[jira] [Reopened] (HBASE-16801) The Append/Increment may return the data from future
[ https://issues.apache.org/jira/browse/HBASE-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai reopened HBASE-16801: --- address [~stack]'s comments. > The Append/Increment may return the data from future > > > Key: HBASE-16801 > URL: https://issues.apache.org/jira/browse/HBASE-16801 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16801.v0.patch > > > OperationContext maintains the mvcc as a static member, so any Append’s and > Increment’s read point will be changed by others. That is, a retrying > Append/Increment may “see” the future data. > This is a master only issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15561) See how G1GC works with MSLAB and chunk pool
[ https://issues.apache.org/jira/browse/HBASE-15561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan resolved HBASE-15561. Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 As per the reports and discussion we have concluded that MSLAB and chunk pool are beneficial. So will close this JIRA linking to HBASE-15513. > See how G1GC works with MSLAB and chunk pool > > > Key: HBASE-15561 > URL: https://issues.apache.org/jira/browse/HBASE-15561 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > Based on the discussion in https://issues.apache.org/jira/browse/HBASE-14613, > we need to identify specifically how MSLAB and G1GC work. This sub-task is > mainly to identify how things work with G1GC and note down the observations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16813) Procedure v2 - Move ProcedureEvent to hbase-procedure module
Matteo Bertozzi created HBASE-16813: --- Summary: Procedure v2 - Move ProcedureEvent to hbase-procedure module Key: HBASE-16813 URL: https://issues.apache.org/jira/browse/HBASE-16813 Project: HBase Issue Type: Sub-task Components: proc-v2 Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 2.0.0 ProcedureEvent was added in MasterProcedureScheduler, but it is generic enough to move to hbase-procedure module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-16663.
Resolution: Fixed
Fix Version/s: 0.98.24
1.2.5
1.3.0
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24
>
> Attachments: HBASE-16663-0.98-V4.patch, HBASE-16663-0.98.patch,
> HBASE-16663-V2.patch, HBASE-16663-V3.patch, HBASE-16663-V4.patch,
> HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queu
[jira] [Created] (HBASE-16812) Cleanup deprecated compact() function
Appy created HBASE-16812: Summary: Cleanup deprecated compact() function Key: HBASE-16812 URL: https://issues.apache.org/jira/browse/HBASE-16812 Project: HBase Issue Type: Task Reporter: Appy Assignee: Appy Priority: Minor compact(CompactionContext compaction, CompactionThroughputController throughputController) is [deprecated in 1.2.0 release|https://github.com/apache/hbase/blob/rel/1.2.0/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java#L222]. Store.java is also marked limited private. Context: I was cleaning up zk table lock which is also used in that method's [override|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L460] in HMobStore. This method isn't being called from anywhere except CompactionTool (which creates HStore object, not HMobStore object). [~jingcheng...@intel.com] Can you PTAL and help me understand what's going on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
@stack, it compiled for me. Also tried few commands, and have to say, it's well designed from user commands perspective. On Tue, Oct 11, 2016 at 5:08 PM, Stack wrote: > On Tue, Oct 11, 2016 at 1:50 PM, Vladimir Rodionov > > wrote: > > > Michael, > > > > Its in HBASE-7912 > > > > This is tip of git log: > > > > commit a072f6f49a26a7259ff2aaef6cb56d85eb592482 > > Author: Frank Welsch > > Date: Fri Sep 23 18:00:42 2016 -0400 > > > > HBASE-16574 Book updates for backup and restore > > > > commit b14e2ab1c24e65ff88dd4c579acf83cb4ed0605e > > Author: tedyu > > Date: Wed Oct 5 16:29:40 2016 -0700 > > > > HBASE-16727 Backup refactoring: remove MR dependencies from HMaster > > (Vladimir Rodionov) > > > > > Thanks. I have that. I tried it and it doesn't compile for me. Does it > compile for you? > Thanks, > M > > > > > -Vlad > > > > On Tue, Oct 11, 2016 at 1:44 PM, Stack wrote: > > > > > Which branch do I check out to try it? HBASE-7912 is not it. I don't > see > > an > > > HBASE-16727... > > > Thanks, > > > M > > > > > > > > > On Tue, Oct 11, 2016 at 11:41 AM, Vladimir Rodionov < > > > vladrodio...@gmail.com> > > > wrote: > > > > > > > The last patch is on review board: > > > > https://reviews.apache.org/r/52748 > > > > > > > > On Mon, Oct 10, 2016 at 3:52 PM, Vladimir Rodionov < > > > vladrodio...@gmail.com > > > > > > > > > wrote: > > > > > > > > > >> How hard to put in an hbase-backup module? hbase-server is fat > > > enough > > > > > >> already. Could be done as a follow-up. > > > > > > > > > > https://issues.apache.org/jira/browse/HBASE-16727? > > > > > focusedCommentId=15531237&page=com.atlassian.jira. > > > > > plugin.system.issuetabpanels:comment-tabpanel#comment-15531237 > > > > > > > > > > Can we do merge first? Then we can discuss separate module. > > > > > > > > > > -Vlad > > > > > > > > > > On Mon, Oct 10, 2016 at 3:44 PM, Ted Yu > wrote: > > > > > > > > > >> Looks like the first quote was cut off. > > > > >> The original sentence was: > > > > >> > > > > >> bq. no mapreduce job launched from master or region server. > > > > >> > > > > >> mapreduce job is launched from the node where command line tool is > > > run. > > > > >> > > > > >> On Mon, Oct 10, 2016 at 3:38 PM, Stack wrote: > > > > >> > > > > >> > bq. launched from master or region server. > > > > >> > > > > > >> > What does this mean please? Has to be run from Master or > > > RegionServer? > > > > >> Can > > > > >> > it be run from another node altogether? > > > > >> > > > > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov < > > > > >> vladrodio...@gmail.com > > > > >> > > > > > > >> > wrote: > > > > >> > > > > > >> > > >> mapreduce dependency has been moved to client side - no > > > mapreduce > > > > >> job > > > > >> > > > > > > >> > > 1. We have no code in the client module anymore, due to > > dependency > > > > on > > > > >> > > internal server API (HFile and WAL access). > > > > >> > > 2. Backup/ restore are client - driven operations, but all the > > > code > > > > >> > resides > > > > >> > > in the server module > > > > >> > > > > > > >> > > > > > >> > How hard to put in an hbase-backup module? hbase-server is fat > > > enough > > > > >> > already. Could be done as a follow-up. > > > > >> > > > > > >> > Thanks, > > > > >> > St.Ack > > > > >> > > > > > >> > > > > > >> > > > > > >> > > 3. No MR in Master, no procedure - driven execution. > > > > >> > > 4. Old good MR from command-line. > > > > >> > > 5. Security was simplified and now only super-user is allowed > to > > > run > > > > >> > > backup/restores. > > > > >> > > 6. HBase Backup API was gone due to 1. Now only command-line > > > access > > > > to > > > > >> > > backup tools. > > > > >> > > > > > > >> > > These consequences of refactoring has been discussed in > > > HBASE-16727. > > > > >> > > > > > > >> > > -Vlad > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu > > > > wrote: > > > > >> > > > > > > >> > > > Reviving this thread. > > > > >> > > > > > > > >> > > > The following has taken place: > > > > >> > > > > > > > >> > > > mapreduce dependency has been moved to client side - no > > > mapreduce > > > > >> job > > > > >> > > > launched from master or region server. > > > > >> > > > document patch (HBASE-16574) has been integrated. > > > > >> > > > Updated mega patch has been attached to HBASE-14123: this > > covers > > > > the > > > > >> > > > refactor in #1 above and the protobuf 3 merge. > > > > >> > > > > > > > >> > > > If community has more feedback on the merge proposal, I > would > > > love > > > > >> to > > > > >> > > hear > > > > >> > > > it. > > > > >> > > > > > > > >> > > > Thanks > > > > >> > > > > > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey < > > > > bus...@cloudera.com> > > > > >> > > wrote: > > > > >> > > > > > > > >> > > > > I'd like to see the docs proposed on HBASE-16574 > integrated > > > into > > > > >> our > > > > >> > > > > project's documentation
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
On Tue, Oct 11, 2016 at 1:50 PM, Vladimir Rodionov wrote: > Michael, > > Its in HBASE-7912 > > This is tip of git log: > > commit a072f6f49a26a7259ff2aaef6cb56d85eb592482 > Author: Frank Welsch > Date: Fri Sep 23 18:00:42 2016 -0400 > > HBASE-16574 Book updates for backup and restore > > commit b14e2ab1c24e65ff88dd4c579acf83cb4ed0605e > Author: tedyu > Date: Wed Oct 5 16:29:40 2016 -0700 > > HBASE-16727 Backup refactoring: remove MR dependencies from HMaster > (Vladimir Rodionov) > > Thanks. I have that. I tried it and it doesn't compile for me. Does it compile for you? Thanks, M > -Vlad > > On Tue, Oct 11, 2016 at 1:44 PM, Stack wrote: > > > Which branch do I check out to try it? HBASE-7912 is not it. I don't see > an > > HBASE-16727... > > Thanks, > > M > > > > > > On Tue, Oct 11, 2016 at 11:41 AM, Vladimir Rodionov < > > vladrodio...@gmail.com> > > wrote: > > > > > The last patch is on review board: > > > https://reviews.apache.org/r/52748 > > > > > > On Mon, Oct 10, 2016 at 3:52 PM, Vladimir Rodionov < > > vladrodio...@gmail.com > > > > > > > wrote: > > > > > > > >> How hard to put in an hbase-backup module? hbase-server is fat > > enough > > > > >> already. Could be done as a follow-up. > > > > > > > > https://issues.apache.org/jira/browse/HBASE-16727? > > > > focusedCommentId=15531237&page=com.atlassian.jira. > > > > plugin.system.issuetabpanels:comment-tabpanel#comment-15531237 > > > > > > > > Can we do merge first? Then we can discuss separate module. > > > > > > > > -Vlad > > > > > > > > On Mon, Oct 10, 2016 at 3:44 PM, Ted Yu wrote: > > > > > > > >> Looks like the first quote was cut off. > > > >> The original sentence was: > > > >> > > > >> bq. no mapreduce job launched from master or region server. > > > >> > > > >> mapreduce job is launched from the node where command line tool is > > run. > > > >> > > > >> On Mon, Oct 10, 2016 at 3:38 PM, Stack wrote: > > > >> > > > >> > bq. launched from master or region server. > > > >> > > > > >> > What does this mean please? Has to be run from Master or > > RegionServer? > > > >> Can > > > >> > it be run from another node altogether? > > > >> > > > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov < > > > >> vladrodio...@gmail.com > > > >> > > > > > >> > wrote: > > > >> > > > > >> > > >> mapreduce dependency has been moved to client side - no > > mapreduce > > > >> job > > > >> > > > > > >> > > 1. We have no code in the client module anymore, due to > dependency > > > on > > > >> > > internal server API (HFile and WAL access). > > > >> > > 2. Backup/ restore are client - driven operations, but all the > > code > > > >> > resides > > > >> > > in the server module > > > >> > > > > > >> > > > > >> > How hard to put in an hbase-backup module? hbase-server is fat > > enough > > > >> > already. Could be done as a follow-up. > > > >> > > > > >> > Thanks, > > > >> > St.Ack > > > >> > > > > >> > > > > >> > > > > >> > > 3. No MR in Master, no procedure - driven execution. > > > >> > > 4. Old good MR from command-line. > > > >> > > 5. Security was simplified and now only super-user is allowed to > > run > > > >> > > backup/restores. > > > >> > > 6. HBase Backup API was gone due to 1. Now only command-line > > access > > > to > > > >> > > backup tools. > > > >> > > > > > >> > > These consequences of refactoring has been discussed in > > HBASE-16727. > > > >> > > > > > >> > > -Vlad > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu > > > wrote: > > > >> > > > > > >> > > > Reviving this thread. > > > >> > > > > > > >> > > > The following has taken place: > > > >> > > > > > > >> > > > mapreduce dependency has been moved to client side - no > > mapreduce > > > >> job > > > >> > > > launched from master or region server. > > > >> > > > document patch (HBASE-16574) has been integrated. > > > >> > > > Updated mega patch has been attached to HBASE-14123: this > covers > > > the > > > >> > > > refactor in #1 above and the protobuf 3 merge. > > > >> > > > > > > >> > > > If community has more feedback on the merge proposal, I would > > love > > > >> to > > > >> > > hear > > > >> > > > it. > > > >> > > > > > > >> > > > Thanks > > > >> > > > > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey < > > > bus...@cloudera.com> > > > >> > > wrote: > > > >> > > > > > > >> > > > > I'd like to see the docs proposed on HBASE-16574 integrated > > into > > > >> our > > > >> > > > > project's documentation prior to merge. > > > >> > > > > > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu < > yuzhih...@gmail.com> > > > >> wrote: > > > >> > > > > > This feature can be marked experimental due to some > > > limitations > > > >> > such > > > >> > > as > > > >> > > > > > security. > > > >> > > > > > > > > >> > > > > > Your previous round of comments have been addressed. > > > >> > > > > > Command line tool has gone through: > > > >> > > > > > > > > >> > > > > > HBASE-16620 Fix backup command-line too
[jira] [Resolved] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling resolved HBASE-16146. --- Resolution: Fixed Assignee: Gary Helmling Hadoop Flags: Reviewed Fix Version/s: 1.4.0 1.3.0 2.0.0 Committed to branch-1.3, branch-1, and master. Counter is no longer used in master, but still present as a deprecated class, so included for consistency. Thanks, [~stack], [~mantonov], and [~enis] for reviews. > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Gary Helmling > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16146.001.patch, HBASE-16146.branch-1.001.patch, > HBASE-16146.branch-1.3.001.patch, counters.patch, less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Or you can apply 14123-master.v29.full.txt (HBASE-14123) to current master. -Vlad On Tue, Oct 11, 2016 at 1:50 PM, Vladimir Rodionov wrote: > Michael, > > Its in HBASE-7912 > > This is tip of git log: > > commit a072f6f49a26a7259ff2aaef6cb56d85eb592482 > Author: Frank Welsch > Date: Fri Sep 23 18:00:42 2016 -0400 > > HBASE-16574 Book updates for backup and restore > > commit b14e2ab1c24e65ff88dd4c579acf83cb4ed0605e > Author: tedyu > Date: Wed Oct 5 16:29:40 2016 -0700 > > HBASE-16727 Backup refactoring: remove MR dependencies from HMaster > (Vladimir Rodionov) > > -Vlad > > On Tue, Oct 11, 2016 at 1:44 PM, Stack wrote: > >> Which branch do I check out to try it? HBASE-7912 is not it. I don't see >> an >> HBASE-16727... >> Thanks, >> M >> >> >> On Tue, Oct 11, 2016 at 11:41 AM, Vladimir Rodionov < >> vladrodio...@gmail.com> >> wrote: >> >> > The last patch is on review board: >> > https://reviews.apache.org/r/52748 >> > >> > On Mon, Oct 10, 2016 at 3:52 PM, Vladimir Rodionov < >> vladrodio...@gmail.com >> > > >> > wrote: >> > >> > > >> How hard to put in an hbase-backup module? hbase-server is fat >> enough >> > > >> already. Could be done as a follow-up. >> > > >> > > https://issues.apache.org/jira/browse/HBASE-16727? >> > > focusedCommentId=15531237&page=com.atlassian.jira. >> > > plugin.system.issuetabpanels:comment-tabpanel#comment-15531237 >> > > >> > > Can we do merge first? Then we can discuss separate module. >> > > >> > > -Vlad >> > > >> > > On Mon, Oct 10, 2016 at 3:44 PM, Ted Yu wrote: >> > > >> > >> Looks like the first quote was cut off. >> > >> The original sentence was: >> > >> >> > >> bq. no mapreduce job launched from master or region server. >> > >> >> > >> mapreduce job is launched from the node where command line tool is >> run. >> > >> >> > >> On Mon, Oct 10, 2016 at 3:38 PM, Stack wrote: >> > >> >> > >> > bq. launched from master or region server. >> > >> > >> > >> > What does this mean please? Has to be run from Master or >> RegionServer? >> > >> Can >> > >> > it be run from another node altogether? >> > >> > >> > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov < >> > >> vladrodio...@gmail.com >> > >> > > >> > >> > wrote: >> > >> > >> > >> > > >> mapreduce dependency has been moved to client side - no >> mapreduce >> > >> job >> > >> > > >> > >> > > 1. We have no code in the client module anymore, due to >> dependency >> > on >> > >> > > internal server API (HFile and WAL access). >> > >> > > 2. Backup/ restore are client - driven operations, but all the >> code >> > >> > resides >> > >> > > in the server module >> > >> > > >> > >> > >> > >> > How hard to put in an hbase-backup module? hbase-server is fat >> enough >> > >> > already. Could be done as a follow-up. >> > >> > >> > >> > Thanks, >> > >> > St.Ack >> > >> > >> > >> > >> > >> > >> > >> > > 3. No MR in Master, no procedure - driven execution. >> > >> > > 4. Old good MR from command-line. >> > >> > > 5. Security was simplified and now only super-user is allowed to >> run >> > >> > > backup/restores. >> > >> > > 6. HBase Backup API was gone due to 1. Now only command-line >> access >> > to >> > >> > > backup tools. >> > >> > > >> > >> > > These consequences of refactoring has been discussed in >> HBASE-16727. >> > >> > > >> > >> > > -Vlad >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu >> > wrote: >> > >> > > >> > >> > > > Reviving this thread. >> > >> > > > >> > >> > > > The following has taken place: >> > >> > > > >> > >> > > > mapreduce dependency has been moved to client side - no >> mapreduce >> > >> job >> > >> > > > launched from master or region server. >> > >> > > > document patch (HBASE-16574) has been integrated. >> > >> > > > Updated mega patch has been attached to HBASE-14123: this >> covers >> > the >> > >> > > > refactor in #1 above and the protobuf 3 merge. >> > >> > > > >> > >> > > > If community has more feedback on the merge proposal, I would >> love >> > >> to >> > >> > > hear >> > >> > > > it. >> > >> > > > >> > >> > > > Thanks >> > >> > > > >> > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey < >> > bus...@cloudera.com> >> > >> > > wrote: >> > >> > > > >> > >> > > > > I'd like to see the docs proposed on HBASE-16574 integrated >> into >> > >> our >> > >> > > > > project's documentation prior to merge. >> > >> > > > > >> > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu > > >> > >> wrote: >> > >> > > > > > This feature can be marked experimental due to some >> > limitations >> > >> > such >> > >> > > as >> > >> > > > > > security. >> > >> > > > > > >> > >> > > > > > Your previous round of comments have been addressed. >> > >> > > > > > Command line tool has gone through: >> > >> > > > > > >> > >> > > > > > HBASE-16620 Fix backup command-line tool usability issues >> > >> > > > > > HBASE-16655 hbase backup describe with incorrect backup id >> > >> results >> > >> > in >> > >> > > > NPE >> > >> > > > > > >> > >> > > > >
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Michael, Its in HBASE-7912 This is tip of git log: commit a072f6f49a26a7259ff2aaef6cb56d85eb592482 Author: Frank Welsch Date: Fri Sep 23 18:00:42 2016 -0400 HBASE-16574 Book updates for backup and restore commit b14e2ab1c24e65ff88dd4c579acf83cb4ed0605e Author: tedyu Date: Wed Oct 5 16:29:40 2016 -0700 HBASE-16727 Backup refactoring: remove MR dependencies from HMaster (Vladimir Rodionov) -Vlad On Tue, Oct 11, 2016 at 1:44 PM, Stack wrote: > Which branch do I check out to try it? HBASE-7912 is not it. I don't see an > HBASE-16727... > Thanks, > M > > > On Tue, Oct 11, 2016 at 11:41 AM, Vladimir Rodionov < > vladrodio...@gmail.com> > wrote: > > > The last patch is on review board: > > https://reviews.apache.org/r/52748 > > > > On Mon, Oct 10, 2016 at 3:52 PM, Vladimir Rodionov < > vladrodio...@gmail.com > > > > > wrote: > > > > > >> How hard to put in an hbase-backup module? hbase-server is fat > enough > > > >> already. Could be done as a follow-up. > > > > > > https://issues.apache.org/jira/browse/HBASE-16727? > > > focusedCommentId=15531237&page=com.atlassian.jira. > > > plugin.system.issuetabpanels:comment-tabpanel#comment-15531237 > > > > > > Can we do merge first? Then we can discuss separate module. > > > > > > -Vlad > > > > > > On Mon, Oct 10, 2016 at 3:44 PM, Ted Yu wrote: > > > > > >> Looks like the first quote was cut off. > > >> The original sentence was: > > >> > > >> bq. no mapreduce job launched from master or region server. > > >> > > >> mapreduce job is launched from the node where command line tool is > run. > > >> > > >> On Mon, Oct 10, 2016 at 3:38 PM, Stack wrote: > > >> > > >> > bq. launched from master or region server. > > >> > > > >> > What does this mean please? Has to be run from Master or > RegionServer? > > >> Can > > >> > it be run from another node altogether? > > >> > > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov < > > >> vladrodio...@gmail.com > > >> > > > > >> > wrote: > > >> > > > >> > > >> mapreduce dependency has been moved to client side - no > mapreduce > > >> job > > >> > > > > >> > > 1. We have no code in the client module anymore, due to dependency > > on > > >> > > internal server API (HFile and WAL access). > > >> > > 2. Backup/ restore are client - driven operations, but all the > code > > >> > resides > > >> > > in the server module > > >> > > > > >> > > > >> > How hard to put in an hbase-backup module? hbase-server is fat > enough > > >> > already. Could be done as a follow-up. > > >> > > > >> > Thanks, > > >> > St.Ack > > >> > > > >> > > > >> > > > >> > > 3. No MR in Master, no procedure - driven execution. > > >> > > 4. Old good MR from command-line. > > >> > > 5. Security was simplified and now only super-user is allowed to > run > > >> > > backup/restores. > > >> > > 6. HBase Backup API was gone due to 1. Now only command-line > access > > to > > >> > > backup tools. > > >> > > > > >> > > These consequences of refactoring has been discussed in > HBASE-16727. > > >> > > > > >> > > -Vlad > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu > > wrote: > > >> > > > > >> > > > Reviving this thread. > > >> > > > > > >> > > > The following has taken place: > > >> > > > > > >> > > > mapreduce dependency has been moved to client side - no > mapreduce > > >> job > > >> > > > launched from master or region server. > > >> > > > document patch (HBASE-16574) has been integrated. > > >> > > > Updated mega patch has been attached to HBASE-14123: this covers > > the > > >> > > > refactor in #1 above and the protobuf 3 merge. > > >> > > > > > >> > > > If community has more feedback on the merge proposal, I would > love > > >> to > > >> > > hear > > >> > > > it. > > >> > > > > > >> > > > Thanks > > >> > > > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey < > > bus...@cloudera.com> > > >> > > wrote: > > >> > > > > > >> > > > > I'd like to see the docs proposed on HBASE-16574 integrated > into > > >> our > > >> > > > > project's documentation prior to merge. > > >> > > > > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu > > >> wrote: > > >> > > > > > This feature can be marked experimental due to some > > limitations > > >> > such > > >> > > as > > >> > > > > > security. > > >> > > > > > > > >> > > > > > Your previous round of comments have been addressed. > > >> > > > > > Command line tool has gone through: > > >> > > > > > > > >> > > > > > HBASE-16620 Fix backup command-line tool usability issues > > >> > > > > > HBASE-16655 hbase backup describe with incorrect backup id > > >> results > > >> > in > > >> > > > NPE > > >> > > > > > > > >> > > > > > The updated doc has been attached to HBASE-16574. > > >> > > > > > > > >> > > > > > Cheers > > >> > > > > > > > >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack > > >> wrote: > > >> > > > > > > > >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu < > yuzhih...@gmail.com > > > > > >> > > wrote: > > >> > > > > >> > > >> > > > >
[jira] [Created] (HBASE-16811) Remove mob sweep job
Appy created HBASE-16811: Summary: Remove mob sweep job Key: HBASE-16811 URL: https://issues.apache.org/jira/browse/HBASE-16811 Project: HBase Issue Type: Task Reporter: Appy Assignee: Appy Priority: Minor Discussed here: http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3CCAAjhxro%3Dt62K44dV2wUtq1hqYLogZ45M3oeNOFZPcnwcSY4_DQ%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
Which branch do I check out to try it? HBASE-7912 is not it. I don't see an HBASE-16727... Thanks, M On Tue, Oct 11, 2016 at 11:41 AM, Vladimir Rodionov wrote: > The last patch is on review board: > https://reviews.apache.org/r/52748 > > On Mon, Oct 10, 2016 at 3:52 PM, Vladimir Rodionov > > wrote: > > > >> How hard to put in an hbase-backup module? hbase-server is fat enough > > >> already. Could be done as a follow-up. > > > > https://issues.apache.org/jira/browse/HBASE-16727? > > focusedCommentId=15531237&page=com.atlassian.jira. > > plugin.system.issuetabpanels:comment-tabpanel#comment-15531237 > > > > Can we do merge first? Then we can discuss separate module. > > > > -Vlad > > > > On Mon, Oct 10, 2016 at 3:44 PM, Ted Yu wrote: > > > >> Looks like the first quote was cut off. > >> The original sentence was: > >> > >> bq. no mapreduce job launched from master or region server. > >> > >> mapreduce job is launched from the node where command line tool is run. > >> > >> On Mon, Oct 10, 2016 at 3:38 PM, Stack wrote: > >> > >> > bq. launched from master or region server. > >> > > >> > What does this mean please? Has to be run from Master or RegionServer? > >> Can > >> > it be run from another node altogether? > >> > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov < > >> vladrodio...@gmail.com > >> > > > >> > wrote: > >> > > >> > > >> mapreduce dependency has been moved to client side - no mapreduce > >> job > >> > > > >> > > 1. We have no code in the client module anymore, due to dependency > on > >> > > internal server API (HFile and WAL access). > >> > > 2. Backup/ restore are client - driven operations, but all the code > >> > resides > >> > > in the server module > >> > > > >> > > >> > How hard to put in an hbase-backup module? hbase-server is fat enough > >> > already. Could be done as a follow-up. > >> > > >> > Thanks, > >> > St.Ack > >> > > >> > > >> > > >> > > 3. No MR in Master, no procedure - driven execution. > >> > > 4. Old good MR from command-line. > >> > > 5. Security was simplified and now only super-user is allowed to run > >> > > backup/restores. > >> > > 6. HBase Backup API was gone due to 1. Now only command-line access > to > >> > > backup tools. > >> > > > >> > > These consequences of refactoring has been discussed in HBASE-16727. > >> > > > >> > > -Vlad > >> > > > >> > > > >> > > > >> > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu > wrote: > >> > > > >> > > > Reviving this thread. > >> > > > > >> > > > The following has taken place: > >> > > > > >> > > > mapreduce dependency has been moved to client side - no mapreduce > >> job > >> > > > launched from master or region server. > >> > > > document patch (HBASE-16574) has been integrated. > >> > > > Updated mega patch has been attached to HBASE-14123: this covers > the > >> > > > refactor in #1 above and the protobuf 3 merge. > >> > > > > >> > > > If community has more feedback on the merge proposal, I would love > >> to > >> > > hear > >> > > > it. > >> > > > > >> > > > Thanks > >> > > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey < > bus...@cloudera.com> > >> > > wrote: > >> > > > > >> > > > > I'd like to see the docs proposed on HBASE-16574 integrated into > >> our > >> > > > > project's documentation prior to merge. > >> > > > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu > >> wrote: > >> > > > > > This feature can be marked experimental due to some > limitations > >> > such > >> > > as > >> > > > > > security. > >> > > > > > > >> > > > > > Your previous round of comments have been addressed. > >> > > > > > Command line tool has gone through: > >> > > > > > > >> > > > > > HBASE-16620 Fix backup command-line tool usability issues > >> > > > > > HBASE-16655 hbase backup describe with incorrect backup id > >> results > >> > in > >> > > > NPE > >> > > > > > > >> > > > > > The updated doc has been attached to HBASE-16574. > >> > > > > > > >> > > > > > Cheers > >> > > > > > > >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack > >> wrote: > >> > > > > > > >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu > > >> > > wrote: > >> > > > > >> > >> > > > > >> > Are there more (review) comments ? > >> > > > > >> > > >> > > > > >> > > >> > > > > >> Are outstanding comments addressed? > >> > > > > >> > >> > > > > >> I don't see answer to my 'is this experimental/will it be > >> marked > >> > > > > >> experimental' question. > >> > > > > >> > >> > > > > >> I ran into some issues trying to use the feature and > suggested > >> > that > >> > > a > >> > > > > >> feature likes this needs polish else it'll just rot, unused. > >> Has > >> > > > polish > >> > > > > >> been applied? All ready for another 'user' test? Suggest that > >> you > >> > > > update > >> > > > > >> here going forward for the benefit of those trying to follow > >> along > >> > > and > >> > > > > who > >> > > > > >> are not watching JIRA change fly-by. > >> > > > > >> > >> > > > > >> It looks like doc got a revision -- I have to chec
[jira] [Created] (HBASE-16810) HBase Balancer throws ArrayIndexOutOfBoundsException when regionservers in /hbase/draining znode and unloaded
Ashu Pachauri created HBASE-16810:
-
Summary: HBase Balancer throws ArrayIndexOutOfBoundsException when
regionservers in /hbase/draining znode and unloaded
Key: HBASE-16810
URL: https://issues.apache.org/jira/browse/HBASE-16810
Project: HBase
Issue Type: Bug
Components: Balancer
Affects Versions: 2.0.0, 1.3.0
Reporter: Ashu Pachauri
1. Add a regionserver znode under /hbase/draining znode.
2. Use RegionMover to unload all regions from the regionserver.
3. Run balancer.
{code}
16/09/21 14:17:33 ERROR ipc.RpcServer: Unexpected throwable object
java.lang.ArrayIndexOutOfBoundsException: 75
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.getLocalityOfRegion(BaseLoadBalancer.java:867)
at
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer$LocalityCostFunction.cost(StochasticLoadBalancer.java:1186)
at
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.computeCost(StochasticLoadBalancer.java:521)
at
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:309)
at
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer.balanceCluster(StochasticLoadBalancer.java:264)
at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1339)
at
org.apache.hadoop.hbase.master.MasterRpcServices.balance(MasterRpcServices.java:442)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58555)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2268)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
The last patch is on review board: https://reviews.apache.org/r/52748 On Mon, Oct 10, 2016 at 3:52 PM, Vladimir Rodionov wrote: > >> How hard to put in an hbase-backup module? hbase-server is fat enough > >> already. Could be done as a follow-up. > > https://issues.apache.org/jira/browse/HBASE-16727? > focusedCommentId=15531237&page=com.atlassian.jira. > plugin.system.issuetabpanels:comment-tabpanel#comment-15531237 > > Can we do merge first? Then we can discuss separate module. > > -Vlad > > On Mon, Oct 10, 2016 at 3:44 PM, Ted Yu wrote: > >> Looks like the first quote was cut off. >> The original sentence was: >> >> bq. no mapreduce job launched from master or region server. >> >> mapreduce job is launched from the node where command line tool is run. >> >> On Mon, Oct 10, 2016 at 3:38 PM, Stack wrote: >> >> > bq. launched from master or region server. >> > >> > What does this mean please? Has to be run from Master or RegionServer? >> Can >> > it be run from another node altogether? >> > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov < >> vladrodio...@gmail.com >> > > >> > wrote: >> > >> > > >> mapreduce dependency has been moved to client side - no mapreduce >> job >> > > >> > > 1. We have no code in the client module anymore, due to dependency on >> > > internal server API (HFile and WAL access). >> > > 2. Backup/ restore are client - driven operations, but all the code >> > resides >> > > in the server module >> > > >> > >> > How hard to put in an hbase-backup module? hbase-server is fat enough >> > already. Could be done as a follow-up. >> > >> > Thanks, >> > St.Ack >> > >> > >> > >> > > 3. No MR in Master, no procedure - driven execution. >> > > 4. Old good MR from command-line. >> > > 5. Security was simplified and now only super-user is allowed to run >> > > backup/restores. >> > > 6. HBase Backup API was gone due to 1. Now only command-line access to >> > > backup tools. >> > > >> > > These consequences of refactoring has been discussed in HBASE-16727. >> > > >> > > -Vlad >> > > >> > > >> > > >> > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu wrote: >> > > >> > > > Reviving this thread. >> > > > >> > > > The following has taken place: >> > > > >> > > > mapreduce dependency has been moved to client side - no mapreduce >> job >> > > > launched from master or region server. >> > > > document patch (HBASE-16574) has been integrated. >> > > > Updated mega patch has been attached to HBASE-14123: this covers the >> > > > refactor in #1 above and the protobuf 3 merge. >> > > > >> > > > If community has more feedback on the merge proposal, I would love >> to >> > > hear >> > > > it. >> > > > >> > > > Thanks >> > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey >> > > wrote: >> > > > >> > > > > I'd like to see the docs proposed on HBASE-16574 integrated into >> our >> > > > > project's documentation prior to merge. >> > > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu >> wrote: >> > > > > > This feature can be marked experimental due to some limitations >> > such >> > > as >> > > > > > security. >> > > > > > >> > > > > > Your previous round of comments have been addressed. >> > > > > > Command line tool has gone through: >> > > > > > >> > > > > > HBASE-16620 Fix backup command-line tool usability issues >> > > > > > HBASE-16655 hbase backup describe with incorrect backup id >> results >> > in >> > > > NPE >> > > > > > >> > > > > > The updated doc has been attached to HBASE-16574. >> > > > > > >> > > > > > Cheers >> > > > > > >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack >> wrote: >> > > > > > >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu >> > > wrote: >> > > > > >> >> > > > > >> > Are there more (review) comments ? >> > > > > >> > >> > > > > >> > >> > > > > >> Are outstanding comments addressed? >> > > > > >> >> > > > > >> I don't see answer to my 'is this experimental/will it be >> marked >> > > > > >> experimental' question. >> > > > > >> >> > > > > >> I ran into some issues trying to use the feature and suggested >> > that >> > > a >> > > > > >> feature likes this needs polish else it'll just rot, unused. >> Has >> > > > polish >> > > > > >> been applied? All ready for another 'user' test? Suggest that >> you >> > > > update >> > > > > >> here going forward for the benefit of those trying to follow >> along >> > > and >> > > > > who >> > > > > >> are not watching JIRA change fly-by. >> > > > > >> >> > > > > >> It looks like doc got a revision -- I have to check -- to take >> on >> > > > > >> suggestion made above but again, suggest, that this thread gets >> > > > updated. >> > > > > >> >> > > > > >> Thanks, >> > > > > >> St.Ack >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> > Thanks >> > > > > >> > >> > > > > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das < >> > > d...@hortonworks.com >> > > > > >> > > > > >> > wrote: >> > > > > >> > >> > > > > >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and >> > others >> > > > for >> >
[jira] [Reopened] (HBASE-16562) ITBLL should fail to start if misconfigured
[
https://issues.apache.org/jira/browse/HBASE-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reopened HBASE-16562:
Testing 1.2.4RC0 with
{noformat}
hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList Loop 1 10 \
1 /tmp/itbll 10 -m slowDeterministic
{noformat}
it seems Integer (note - the object, not the primitive) parameters to
Generator#run can be null
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator.run(IntegrationTestBigLinkedList.java:633)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.runGenerator(IntegrationTestBigLinkedList.java:1260)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.run(IntegrationTestBigLinkedList.java:1310)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.runTestFromCommandLine(IntegrationTestBigLinkedList.java:1662)
at
org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:123)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:106)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.main(IntegrationTestBigLinkedList.java:1702)
{noformat}
> ITBLL should fail to start if misconfigured
> ---
>
> Key: HBASE-16562
> URL: https://issues.apache.org/jira/browse/HBASE-16562
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Andrew Purtell
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.0.4, 1.4.0, 1.3.1, 1.1.7, 0.98.23, 1.2.4
>
> Attachments: HBASE-16562-branch-1.2.patch,
> HBASE-16562-branch-1.2.v1.patch, HBASE-16562.patch, HBASE-16562.v1.patch,
> HBASE-16562.v1.patch-addendum
>
>
> The number of nodes in ITBLL must a multiple of width*wrap (defaults to 25M,
> but can be configured by adding two more args to the test invocation) or else
> verification will fail. This can be very expensive in terms of time or hourly
> billing for on demand test resources. Check the sanity of test parameters
> before launching any MR jobs and fail fast if invariants aren't met with an
> indication what parameter(s) need fixing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: HBase wrong region exception
You maybe hitting this bug: https://issues.apache.org/jira/browse/HBASE-13471 There is no 1.0.11 version. Did you mean 1.0.1? 1.0 is EOL'ed some time ago. I suggest to update both the client and the server to 1.1.x or 1.2.x as soon as possible. Enis On Tue, Oct 11, 2016 at 6:58 AM, Ted Yu wrote: > Suggest keeping client version the same as the version on server. > > Can you check log of the region server where hbase:meta was hosted around > the time this happened ? > There may be some clue. > > Thanks > > On Tue, Oct 11, 2016 at 2:21 AM, Shoubo Li wrote: > > > Hi， I encountered a question bothering me for several days. > > > > my environment: > > > >1. HBase Server: 1.0.11 > >2. HBase Client: 1.0.5-SNAPSHOT > > > > The region server always print warn log as follow: > > > > 2016-10-11 16:52:33,489 WARN > > [RW.default.writeRpcServer.handler=46,queue=4,port=16020] > > regionserver.HRegion: Failed getting lock in batch put, > > row=0799\x00\x00\x01W\x90\x90\xE7&\x0A\x02m2=\x1E\x00\x00\x01 > > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested > > row out of range for row lock on HRegion > > hzop:vtrace,7280,1466058715841.f8db60758f6905bc0d81c23d03aba360., > > startKey='7280', getEndKey()='7290', > > row='0799\x00\x00\x01W\x90\x90\xE7&\x0A\x02m2=\x1E\x00\x00\x01' > > at org.apache.hadoop.hbase.regionserver.HRegion.checkRow( > > HRegion.java:3853) > > at org.apache.hadoop.hbase.regionserver.HRegion. > > getRowLockInternal(HRegion.java:3884) > > at org.apache.hadoop.hbase.regionserver.HRegion. > > doMiniBatchMutation(HRegion.java:2614) > > at org.apache.hadoop.hbase.regionserver.HRegion. > > batchMutate(HRegion.java:2478) > > at org.apache.hadoop.hbase.regionserver.HRegion. > > batchMutate(HRegion.java:2432) > > at org.apache.hadoop.hbase.regionserver.HRegion. > > batchMutate(HRegion.java:2436) > > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > > doBatchOp(RSRpcServices.java:642) > > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > > doNonAtomicRegionMutation(RSRpcServices.java:606) > > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > > multi(RSRpcServices.java:1836) > > ^Cat org.apache.hadoop.hbase.protobuf.generated. > > ClientProtos$ClientService$2.callBlockingMethod > > ClientProtos.java:31451) > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) > > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > > RpcExecutor.java:130) > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor. > java:107) > > at java.lang.Thread.run(Thread.java:745) > > > > There are two questions: > > > >1. why WrongRegionException happen? I known old hbase client version > has > >bugs, but latter version fixed it. > >2. > > > >why region server always print warn log? I also has read the source > >code, I found every time RpcExcecutor will call > > > >consumerLoop(callQueues.get(index)); > > > >so I stop the hbase client job, so that no data would be written to > this > >table. however, the warn log always looped print. I wonder that the > > wrong > >row data is still in the queue, so every loop will fetch from the > queue > > and > >handle it. > > > > any idea? > > >
Fixed: HBase Generate Website
Build status: Fixed
If successful, the website and docs have been generated. To update the live
site, follow the instructions below. If failed, skip to the bottom of this
email.
Use the following commands to download the patch and apply it to a clean branch
based on origin/asf-site. If you prefer to keep the hbase-site repo around
permanently, you can skip the clone step.
git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
cd hbase-site
wget -O-
https://builds.apache.org/job/hbase_generate_website/375/artifact/website.patch.zip
| funzip > 4d091166951a0b53e5fdd9dc694b4e5a12dc527c.patch
git fetch
git checkout -b asf-site-4d091166951a0b53e5fdd9dc694b4e5a12dc527c
origin/asf-site
git am --whitespace=fix 4d091166951a0b53e5fdd9dc694b4e5a12dc527c.patch
At this point, you can preview the changes by opening index.html or any of the
other HTML pages in your local
asf-site-4d091166951a0b53e5fdd9dc694b4e5a12dc527c branch.
There are lots of spurious changes, such as timestamps and CSS styles in
tables, so a generic git diff is not very useful. To see a list of files that
have been added, deleted, renamed, changed type, or are otherwise interesting,
use the following command:
git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
To see only files that had 100 or more lines changed:
git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
When you are satisfied, publish your changes to origin/asf-site using these
commands:
git commit --allow-empty -m "Empty commit" # to work around a current ASF
INFRA bug
git push origin asf-site-4d091166951a0b53e5fdd9dc694b4e5a12dc527c:asf-site
git checkout asf-site
git branch -D asf-site-4d091166951a0b53e5fdd9dc694b4e5a12dc527c
Changes take a couple of minutes to be propagated. You can verify whether they
have been propagated by looking at the Last Published date at the bottom of
http://hbase.apache.org/. It should match the date in the index.html on the
asf-site branch in Git.
As a courtesy- reply-all to this email to let other committers know you pushed
the site.
If failed, see https://builds.apache.org/job/hbase_generate_website/375/console
Re: Failure: HBase Generate Website
Removed.
On Tue, Oct 11, 2016 at 7:31 AM, Dima Spivak wrote:
> Why. Ted, please remove this and debug in a separate Jenkins job that won't
> block site generation.
>
> On Tuesday, October 11, 2016, Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
> > Build status: Failure
> >
> > If successful, the website and docs have been generated. To update the
> > live site, follow the instructions below. If failed, skip to the bottom
> of
> > this email.
> >
> > Use the following commands to download the patch and apply it to a clean
> > branch based on origin/asf-site. If you prefer to keep the hbase-site
> repo
> > around permanently, you can skip the clone step.
> >
> > git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
> >
> > cd hbase-site
> > wget -O- https://builds.apache.org/job/hbase_generate_website/374/
> > artifact/website.patch.zip | funzip > ${GIT_SHA}.patch
> > git fetch
> > git checkout -b asf-site-${GIT_SHA} origin/asf-site
> > git am --whitespace=fix $GIT_SHA.patch
> >
> > At this point, you can preview the changes by opening index.html or any
> of
> > the other HTML pages in your local asf-site-${GIT_SHA} branch.
> >
> > There are lots of spurious changes, such as timestamps and CSS styles in
> > tables, so a generic git diff is not very useful. To see a list of files
> > that have been added, deleted, renamed, changed type, or are otherwise
> > interesting, use the following command:
> >
> > git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
> >
> > To see only files that had 100 or more lines changed:
> >
> > git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
> >
> > When you are satisfied, publish your changes to origin/asf-site using
> > these commands:
> >
> > git commit --allow-empty -m "Empty commit" # to work around a current
> > ASF INFRA bug
> > git push origin asf-site-${GIT_SHA}:asf-site
> > git checkout asf-site
> > git branch -D asf-site-${GIT_SHA}
> >
> > Changes take a couple of minutes to be propagated. You can verify whether
> > they have been propagated by looking at the Last Published date at the
> > bottom of http://hbase.apache.org/. It should match the date in the
> > index.html on the asf-site branch in Git.
> >
> > As a courtesy- reply-all to this email to let other committers know you
> > pushed the site.
> >
> >
> >
> > If failed, see https://builds.apache.org/job/hbase_generate_website/374/
> > console
>
>
>
> --
> -Dima
>
Re: Failure: HBase Generate Website
Why. Ted, please remove this and debug in a separate Jenkins job that won't
block site generation.
On Tuesday, October 11, 2016, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:
> Build status: Failure
>
> If successful, the website and docs have been generated. To update the
> live site, follow the instructions below. If failed, skip to the bottom of
> this email.
>
> Use the following commands to download the patch and apply it to a clean
> branch based on origin/asf-site. If you prefer to keep the hbase-site repo
> around permanently, you can skip the clone step.
>
> git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
>
> cd hbase-site
> wget -O- https://builds.apache.org/job/hbase_generate_website/374/
> artifact/website.patch.zip | funzip > ${GIT_SHA}.patch
> git fetch
> git checkout -b asf-site-${GIT_SHA} origin/asf-site
> git am --whitespace=fix $GIT_SHA.patch
>
> At this point, you can preview the changes by opening index.html or any of
> the other HTML pages in your local asf-site-${GIT_SHA} branch.
>
> There are lots of spurious changes, such as timestamps and CSS styles in
> tables, so a generic git diff is not very useful. To see a list of files
> that have been added, deleted, renamed, changed type, or are otherwise
> interesting, use the following command:
>
> git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
>
> To see only files that had 100 or more lines changed:
>
> git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
>
> When you are satisfied, publish your changes to origin/asf-site using
> these commands:
>
> git commit --allow-empty -m "Empty commit" # to work around a current
> ASF INFRA bug
> git push origin asf-site-${GIT_SHA}:asf-site
> git checkout asf-site
> git branch -D asf-site-${GIT_SHA}
>
> Changes take a couple of minutes to be propagated. You can verify whether
> they have been propagated by looking at the Last Published date at the
> bottom of http://hbase.apache.org/. It should match the date in the
> index.html on the asf-site branch in Git.
>
> As a courtesy- reply-all to this email to let other committers know you
> pushed the site.
>
>
>
> If failed, see https://builds.apache.org/job/hbase_generate_website/374/
> console
--
-Dima
Failure: HBase Generate Website
Build status: Failure
If successful, the website and docs have been generated. To update the live
site, follow the instructions below. If failed, skip to the bottom of this
email.
Use the following commands to download the patch and apply it to a clean branch
based on origin/asf-site. If you prefer to keep the hbase-site repo around
permanently, you can skip the clone step.
git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
cd hbase-site
wget -O-
https://builds.apache.org/job/hbase_generate_website/374/artifact/website.patch.zip
| funzip > ${GIT_SHA}.patch
git fetch
git checkout -b asf-site-${GIT_SHA} origin/asf-site
git am --whitespace=fix $GIT_SHA.patch
At this point, you can preview the changes by opening index.html or any of the
other HTML pages in your local asf-site-${GIT_SHA} branch.
There are lots of spurious changes, such as timestamps and CSS styles in
tables, so a generic git diff is not very useful. To see a list of files that
have been added, deleted, renamed, changed type, or are otherwise interesting,
use the following command:
git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
To see only files that had 100 or more lines changed:
git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
When you are satisfied, publish your changes to origin/asf-site using these
commands:
git commit --allow-empty -m "Empty commit" # to work around a current ASF
INFRA bug
git push origin asf-site-${GIT_SHA}:asf-site
git checkout asf-site
git branch -D asf-site-${GIT_SHA}
Changes take a couple of minutes to be propagated. You can verify whether they
have been propagated by looking at the Last Published date at the bottom of
http://hbase.apache.org/. It should match the date in the index.html on the
asf-site branch in Git.
As a courtesy- reply-all to this email to let other committers know you pushed
the site.
If failed, see https://builds.apache.org/job/hbase_generate_website/374/console
Re: HBase wrong region exception
Suggest keeping client version the same as the version on server. Can you check log of the region server where hbase:meta was hosted around the time this happened ? There may be some clue. Thanks On Tue, Oct 11, 2016 at 2:21 AM, Shoubo Li wrote: > Hi， I encountered a question bothering me for several days. > > my environment: > >1. HBase Server: 1.0.11 >2. HBase Client: 1.0.5-SNAPSHOT > > The region server always print warn log as follow: > > 2016-10-11 16:52:33,489 WARN > [RW.default.writeRpcServer.handler=46,queue=4,port=16020] > regionserver.HRegion: Failed getting lock in batch put, > row=0799\x00\x00\x01W\x90\x90\xE7&\x0A\x02m2=\x1E\x00\x00\x01 > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested > row out of range for row lock on HRegion > hzop:vtrace,7280,1466058715841.f8db60758f6905bc0d81c23d03aba360., > startKey='7280', getEndKey()='7290', > row='0799\x00\x00\x01W\x90\x90\xE7&\x0A\x02m2=\x1E\x00\x00\x01' > at org.apache.hadoop.hbase.regionserver.HRegion.checkRow( > HRegion.java:3853) > at org.apache.hadoop.hbase.regionserver.HRegion. > getRowLockInternal(HRegion.java:3884) > at org.apache.hadoop.hbase.regionserver.HRegion. > doMiniBatchMutation(HRegion.java:2614) > at org.apache.hadoop.hbase.regionserver.HRegion. > batchMutate(HRegion.java:2478) > at org.apache.hadoop.hbase.regionserver.HRegion. > batchMutate(HRegion.java:2432) > at org.apache.hadoop.hbase.regionserver.HRegion. > batchMutate(HRegion.java:2436) > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > doBatchOp(RSRpcServices.java:642) > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > doNonAtomicRegionMutation(RSRpcServices.java:606) > at org.apache.hadoop.hbase.regionserver.RSRpcServices. > multi(RSRpcServices.java:1836) > ^Cat org.apache.hadoop.hbase.protobuf.generated. > ClientProtos$ClientService$2.callBlockingMethod > ClientProtos.java:31451) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > RpcExecutor.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > at java.lang.Thread.run(Thread.java:745) > > There are two questions: > >1. why WrongRegionException happen? I known old hbase client version has >bugs, but latter version fixed it. >2. > >why region server always print warn log? I also has read the source >code, I found every time RpcExcecutor will call > >consumerLoop(callQueues.get(index)); > >so I stop the hbase client job, so that no data would be written to this >table. however, the warn log always looped print. I wonder that the > wrong >row data is still in the queue, so every loop will fetch from the queue > and >handle it. > > any idea? >
HBase wrong region exception
Hi， I encountered a question bothering me for several days. my environment: 1. HBase Server: 1.0.11 2. HBase Client: 1.0.5-SNAPSHOT The region server always print warn log as follow: 2016-10-11 16:52:33,489 WARN [RW.default.writeRpcServer.handler=46,queue=4,port=16020] regionserver.HRegion: Failed getting lock in batch put, row=0799\x00\x00\x01W\x90\x90\xE7&\x0A\x02m2=\x1E\x00\x00\x01 org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of range for row lock on HRegion hzop:vtrace,7280,1466058715841.f8db60758f6905bc0d81c23d03aba360., startKey='7280', getEndKey()='7290', row='0799\x00\x00\x01W\x90\x90\xE7&\x0A\x02m2=\x1E\x00\x00\x01' at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:3853) at org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:3884) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2614) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2478) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2432) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2436) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:642) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:606) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1836) ^Cat org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod ClientProtos.java:31451) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) There are two questions: 1. why WrongRegionException happen? I known old hbase client version has bugs, but latter version fixed it. 2. why region server always print warn log? I also has read the source code, I found every time RpcExcecutor will call consumerLoop(callQueues.get(index)); so I stop the hbase client job, so that no data would be written to this table. however, the warn log always looped print. I wonder that the wrong row data is still in the queue, so every loop will fetch from the queue and handle it. any idea?
[jira] [Created] (HBASE-16809) Save one cell length calculation in HeapMemStoreLAB#copyCellInto
binlijin created HBASE-16809: Summary: Save one cell length calculation in HeapMemStoreLAB#copyCellInto Key: HBASE-16809 URL: https://issues.apache.org/jira/browse/HBASE-16809 Project: HBase Issue Type: Improvement Reporter: binlijin Priority: Minor HeapMemStoreLAB#copyCellInto have already calculate the cell's length, we can pass it to KeyValueUtil#copyCellTo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
