[jira] [Assigned] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain reassigned HBASE-22701:
--
Assignee: Nihal Jain (was: Josh Elser)
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 2.3.0, 2.2.1, 2.1.6
>
> Attachments: HBASE-22701.001.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.(ProtobufUtil.java:260)
> ... 16 more
> {noformat}
> Every subsequent call will result in a NoClassDefFoundError, because we
> already tried to load DynamicClassLoader.class once and failed.
> {noformat}
> 2019-07-14 06:25:34,380 ERROR
> [RpcServer.Metadata.Fifo.handler=2,queue=2,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
>
[jira] [Assigned] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain reassigned HBASE-22701:
--
Assignee: Josh Elser (was: Nihal Jain)
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.2.1, 2.1.6
>
> Attachments: HBASE-22701.001.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.(ProtobufUtil.java:260)
> ... 16 more
> {noformat}
> Every subsequent call will result in a NoClassDefFoundError, because we
> already tried to load DynamicClassLoader.class once and failed.
> {noformat}
> 2019-07-14 06:25:34,380 ERROR
> [RpcServer.Metadata.Fifo.handler=2,queue=2,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
>
[jira] [Commented] (HBASE-22537) Split happened Replica region can not be deleted after deleting table successfully and restarting RegionServer
[ https://issues.apache.org/jira/browse/HBASE-22537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886698#comment-16886698 ] Y. SREENIVASULU REDDY commented on HBASE-22537: --- Attache the patch for the *Master*. And *branch-2.1* test failures are not belongs to the respective patch. [~wchevreuil], can you please review it. > Split happened Replica region can not be deleted after deleting table > successfully and restarting RegionServer > -- > > Key: HBASE-22537 > URL: https://issues.apache.org/jira/browse/HBASE-22537 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.1.1 >Reporter: Y. SREENIVASULU REDDY >Assignee: Y. SREENIVASULU REDDY >Priority: Minor > Fix For: 2.1.6 > > Attachments: HBASE-22537.branch-2.1.002.patch, > HBASE-22537.branch-2.1.003.patch, HBASE-22537.branch-2.1.004.patch, > HBASE-22537.branch-2.1.005.patch, HBASE-22537.branch-2.1.patch, > HBASE-22537.branch-2.patch, HBASE-22537.patch > > > [Test step] > 1.create a table (set RegionReplication=2). > 2.insert data to the table utill region be splitted. > 3.Disable and Drop the table. > 4.Parent replica region holding Regionserver, Kill forcefully > 5.HBase WebUI will show that the replica regions will be in RIT. > [Expect Output] > Parent replica region should be deleted. > [Actual Output] > Parent replica region still exists. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22537) Split happened Replica region can not be deleted after deleting table successfully and restarting RegionServer
[ https://issues.apache.org/jira/browse/HBASE-22537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Y. SREENIVASULU REDDY updated HBASE-22537: -- Attachment: HBASE-22537.patch > Split happened Replica region can not be deleted after deleting table > successfully and restarting RegionServer > -- > > Key: HBASE-22537 > URL: https://issues.apache.org/jira/browse/HBASE-22537 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.1.1 >Reporter: Y. SREENIVASULU REDDY >Assignee: Y. SREENIVASULU REDDY >Priority: Minor > Fix For: 2.1.6 > > Attachments: HBASE-22537.branch-2.1.002.patch, > HBASE-22537.branch-2.1.003.patch, HBASE-22537.branch-2.1.004.patch, > HBASE-22537.branch-2.1.005.patch, HBASE-22537.branch-2.1.patch, > HBASE-22537.branch-2.patch, HBASE-22537.patch > > > [Test step] > 1.create a table (set RegionReplication=2). > 2.insert data to the table utill region be splitted. > 3.Disable and Drop the table. > 4.Parent replica region holding Regionserver, Kill forcefully > 5.HBase WebUI will show that the replica regions will be in RIT. > [Expect Output] > Parent replica region should be deleted. > [Actual Output] > Parent replica region still exists. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22527) [hbck2] Add a master web ui to show the problematic regions
[ https://issues.apache.org/jira/browse/HBASE-22527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-22527: --- Attachment: HBASE-22527.master.004.patch > [hbck2] Add a master web ui to show the problematic regions > --- > > Key: HBASE-22527 > URL: https://issues.apache.org/jira/browse/HBASE-22527 > Project: HBase > Issue Type: Improvement > Components: hbase-operator-tools, hbck2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: 2019-07-16 08-50-07屏幕截图.png, > HBASE-22527.master.001.patch, HBASE-22527.master.002.patch, > HBASE-22527.master.003.patch, HBASE-22527.master.004.patch > > > On our cluster which based 2.2.0, we found one problem: there are some opened > regions which had wrong regionserver in meta. The regionserver is not exist. > We used hbck2 to fix them by the following steps. > # disable table > # bypass the stucked close region procedure (as the target regionserver is > not exist) and disable table procedure. > # setRegionState to CLOSED. > # setTableState to DISABLED. > # enable table > We found this problem by scan the hbase:meta. I thought we should add this > feature to hbck2. The we can use hbck2 to find this problem. Thanks. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22643) Delete region without archiving only if regiondir is present
[
https://issues.apache.org/jira/browse/HBASE-22643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886686#comment-16886686
]
Hudson commented on HBASE-22643:
Results for branch branch-2.2
[build #438 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/438/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/438//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/438//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/438//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Delete region without archiving only if regiondir is present
>
>
> Key: HBASE-22643
> URL: https://issues.apache.org/jira/browse/HBASE-22643
> Project: HBase
> Issue Type: Improvement
> Components: HFile
>Affects Versions: 3.0.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.1, 2.1.7
>
> Attachments: HBASE-22643-master-v1.patch,
> HBASE-22643-master-v2.patch, HBASE-22643-master-v3.patch
>
>
> Put a condition to delete region without archiving only if regionDir is
> present
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[GitHub] [hbase-operator-tools] saintstack commented on a change in pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
saintstack commented on a change in pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService URL: https://github.com/apache/hbase-operator-tools/pull/11#discussion_r304217551 ## File path: hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java ## @@ -88,6 +91,7 @@ private static final String FILESYSTEM = "filesystem"; private static final String VERSION = "version"; private static final String SET_REGION_STATE = "setRegionState"; + private static final String SCHEDULE_SCP = "serverCrashProcedures"; Review comment: Hmm. I'd rather call it 'crash'. The other procedure commands are short. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] saintstack commented on a change in pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
saintstack commented on a change in pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService URL: https://github.com/apache/hbase-operator-tools/pull/11#discussion_r304214827 ## File path: hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java ## @@ -88,6 +91,7 @@ private static final String FILESYSTEM = "filesystem"; private static final String VERSION = "version"; private static final String SET_REGION_STATE = "setRegionState"; + private static final String SCHEDULE_SCP = "serverCrashProcedures"; Review comment: Would rather not use acronym. It is explained in help but seems cryptic as command. scheduleServerCrashProcedures is too long. What you reckon? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22658) region_mover.rb should choose same rsgroup servers as target servers
[
https://issues.apache.org/jira/browse/HBASE-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886670#comment-16886670
]
HBase QA commented on HBASE-22658:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
7s{color} | {color:red} The patch generated 37 new + 354 unchanged - 2 fixed =
391 total (was 356) {color} |
| {color:green}+1{color} | {color:green} ruby-lint {color} | {color:green} 0m
1s{color} | {color:green} There were no new ruby-lint issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 3m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/639/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22658 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974907/HBASE-22658.branch-1.patch
|
| Optional Tests | dupname asflicense rubocop ruby_lint |
| uname | Linux c46e0e9e291f 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-1 / 1340706 |
| maven | version: Apache Maven 3.0.5 |
| rubocop | v0.72.0 |
| rubocop |
https://builds.apache.org/job/PreCommit-HBASE-Build/639/artifact/patchprocess/diff-patch-rubocop.txt
|
| ruby-lint | v2.3.1 |
| Max. process+thread count | 41 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/639/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
> region_mover.rb should choose same rsgroup servers as target servers
> --
>
> Key: HBASE-22658
> URL: https://issues.apache.org/jira/browse/HBASE-22658
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup, shell
>Affects Versions: 1.4.10
>Reporter: liang.feng
>Priority: Major
> Labels: gracefulshutdown, region_mover, rsgroup
> Attachments: HBASE-22658.branch-1.002.patch,
> HBASE-22658.branch-1.patch
>
>
> There are many retries when i am using graceful_stop.sh to shutdown region
> server after using regroup, because the target server in a different rsgroup.
> This makes it slow to graceful shutdown a regionserver. So i think that
> region_mover.rb should only choose same rsgroup servers as target servers.
> Region mover is implemented by jruby in hbase1.x and is implemented by java
> in hbase2.x . I tried to modify the RegionMover.java class to use the same
> logic in hbase2.x, but mvn package failed due to hbase-server module and
> hbase-rsgroup moudle needed to depend on each other, then maven throw a "The
> projects in the reactor contain a cyclic reference". I couldn't solve it.So I
> just uploaded patch for hbase1.x .
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[GitHub] [hbase-operator-tools] saintstack commented on a change in pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
saintstack commented on a change in pull request #11: HBASE-21322 Add a
scheduleServerCrashProcedure() API to HbckService
URL:
https://github.com/apache/hbase-operator-tools/pull/11#discussion_r304214317
##
File path: hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java
##
@@ -254,6 +258,24 @@ int setRegionState(String region, RegionState.State
newState)
}
}
+ List scheduleSCP(String[] args) throws IOException {
+List serverNames = new ArrayList<>();
+for (String serverName: args) {
+ serverNames.add(parseServerName(serverName));
+}
+try (ClusterConnection c =
(ClusterConnection)ConnectionFactory.createConnection(getConf())) {
+ try (Hbck hbck = c.getHbck()) {
+return hbck.scheduleServerCrashProcedure(serverNames);
Review comment:
Thought about that and figured it could get messy having a complicated
matrix of what version support what methods. An old idea is that we'd ask the
server what 'features' it supports and then we'd fail if wanted-api was
missing. This would be a bit tough in this case since we can't change the
server-side after the fact. Also sometimes we want to schedule procedures w/o
checking server version/features -- i.e. during master startup, a check for
version comes back with a PleaseHoldException.
That said, let me make a start. Can do it here w/ this feature.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (HBASE-22658) region_mover.rb should choose same rsgroup servers as target servers
[ https://issues.apache.org/jira/browse/HBASE-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang.feng updated HBASE-22658: --- Attachment: HBASE-22658.branch-1.patch > region_mover.rb should choose same rsgroup servers as target servers > -- > > Key: HBASE-22658 > URL: https://issues.apache.org/jira/browse/HBASE-22658 > Project: HBase > Issue Type: Improvement > Components: rsgroup, shell >Affects Versions: 1.4.10 >Reporter: liang.feng >Priority: Major > Labels: gracefulshutdown, region_mover, rsgroup > Attachments: HBASE-22658.branch-1.002.patch, > HBASE-22658.branch-1.patch > > > There are many retries when i am using graceful_stop.sh to shutdown region > server after using regroup, because the target server in a different rsgroup. > This makes it slow to graceful shutdown a regionserver. So i think that > region_mover.rb should only choose same rsgroup servers as target servers. > Region mover is implemented by jruby in hbase1.x and is implemented by java > in hbase2.x . I tried to modify the RegionMover.java class to use the same > logic in hbase2.x, but mvn package failed due to hbase-server module and > hbase-rsgroup moudle needed to depend on each other, then maven throw a "The > projects in the reactor contain a cyclic reference". I couldn't solve it.So I > just uploaded patch for hbase1.x . > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22658) region_mover.rb should choose same rsgroup servers as target servers
[ https://issues.apache.org/jira/browse/HBASE-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang.feng updated HBASE-22658: --- Attachment: (was: HBASE-22658.branch-1.patch) > region_mover.rb should choose same rsgroup servers as target servers > -- > > Key: HBASE-22658 > URL: https://issues.apache.org/jira/browse/HBASE-22658 > Project: HBase > Issue Type: Improvement > Components: rsgroup, shell >Affects Versions: 1.4.10 >Reporter: liang.feng >Priority: Major > Labels: gracefulshutdown, region_mover, rsgroup > Attachments: HBASE-22658.branch-1.002.patch > > > There are many retries when i am using graceful_stop.sh to shutdown region > server after using regroup, because the target server in a different rsgroup. > This makes it slow to graceful shutdown a regionserver. So i think that > region_mover.rb should only choose same rsgroup servers as target servers. > Region mover is implemented by jruby in hbase1.x and is implemented by java > in hbase2.x . I tried to modify the RegionMover.java class to use the same > logic in hbase2.x, but mvn package failed due to hbase-server module and > hbase-rsgroup moudle needed to depend on each other, then maven throw a "The > projects in the reactor contain a cyclic reference". I couldn't solve it.So I > just uploaded patch for hbase1.x . > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22658) region_mover.rb should choose same rsgroup servers as target servers
[
https://issues.apache.org/jira/browse/HBASE-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886664#comment-16886664
]
HBase QA commented on HBASE-22658:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
7s{color} | {color:red} The patch generated 37 new + 354 unchanged - 2 fixed =
391 total (was 356) {color} |
| {color:green}+1{color} | {color:green} ruby-lint {color} | {color:green} 0m
0s{color} | {color:green} There were no new ruby-lint issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 4m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/638/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22658 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974906/HBASE-22658.branch-1.patch
|
| Optional Tests | dupname asflicense rubocop ruby_lint |
| uname | Linux b36d8f82e74d 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-1 / 1340706 |
| maven | version: Apache Maven 3.0.5 |
| rubocop | v0.72.0 |
| rubocop |
https://builds.apache.org/job/PreCommit-HBASE-Build/638/artifact/patchprocess/diff-patch-rubocop.txt
|
| ruby-lint | v2.3.1 |
| Max. process+thread count | 45 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/638/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
> region_mover.rb should choose same rsgroup servers as target servers
> --
>
> Key: HBASE-22658
> URL: https://issues.apache.org/jira/browse/HBASE-22658
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup, shell
>Affects Versions: 1.4.10
>Reporter: liang.feng
>Priority: Major
> Labels: gracefulshutdown, region_mover, rsgroup
> Attachments: HBASE-22658.branch-1.002.patch,
> HBASE-22658.branch-1.patch
>
>
> There are many retries when i am using graceful_stop.sh to shutdown region
> server after using regroup, because the target server in a different rsgroup.
> This makes it slow to graceful shutdown a regionserver. So i think that
> region_mover.rb should only choose same rsgroup servers as target servers.
> Region mover is implemented by jruby in hbase1.x and is implemented by java
> in hbase2.x . I tried to modify the RegionMover.java class to use the same
> logic in hbase2.x, but mvn package failed due to hbase-server module and
> hbase-rsgroup moudle needed to depend on each other, then maven throw a "The
> projects in the reactor contain a cyclic reference". I couldn't solve it.So I
> just uploaded patch for hbase1.x .
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[GitHub] [hbase] Reidddddd commented on issue #386: HBASE-22610 [BucketCache] Rename hbase.offheapcache.minblocksize.
Reidd commented on issue #386: HBASE-22610 [BucketCache] Rename hbase.offheapcache.minblocksize. URL: https://github.com/apache/hbase/pull/386#issuecomment-512093234 Please refer to [HBASE-22598](https://issues.apache.org/jira/browse/HBASE-22598), it is a typical example. Codes are better than words. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] saintstack commented on a change in pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
saintstack commented on a change in pull request #11: HBASE-21322 Add a
scheduleServerCrashProcedure() API to HbckService
URL:
https://github.com/apache/hbase-operator-tools/pull/11#discussion_r304211667
##
File path: hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java
##
@@ -339,15 +360,22 @@ private static String getCommandUsage() {
writer.println(" $ HBCK2 setTableState users ENABLED");
writer.println(" Returns whatever the previous table state was.");
writer.println();
+writer.println(" " + SCHEDULE_SCP + " ...");
+writer.println(" Schedule ServerCrashProcedure(SCP) for list of
RegionServers. Format");
Review comment:
It actual can do a list of servers it submits them all in the one lump
(see test).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (HBASE-22658) region_mover.rb should choose same rsgroup servers as target servers
[ https://issues.apache.org/jira/browse/HBASE-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang.feng updated HBASE-22658: --- Attachment: HBASE-22658.branch-1.patch > region_mover.rb should choose same rsgroup servers as target servers > -- > > Key: HBASE-22658 > URL: https://issues.apache.org/jira/browse/HBASE-22658 > Project: HBase > Issue Type: Improvement > Components: rsgroup, shell >Affects Versions: 1.4.10 >Reporter: liang.feng >Priority: Major > Labels: gracefulshutdown, region_mover, rsgroup > Attachments: HBASE-22658.branch-1.002.patch, > HBASE-22658.branch-1.patch > > > There are many retries when i am using graceful_stop.sh to shutdown region > server after using regroup, because the target server in a different rsgroup. > This makes it slow to graceful shutdown a regionserver. So i think that > region_mover.rb should only choose same rsgroup servers as target servers. > Region mover is implemented by jruby in hbase1.x and is implemented by java > in hbase2.x . I tried to modify the RegionMover.java class to use the same > logic in hbase2.x, but mvn package failed due to hbase-server module and > hbase-rsgroup moudle needed to depend on each other, then maven throw a "The > projects in the reactor contain a cyclic reference". I couldn't solve it.So I > just uploaded patch for hbase1.x . > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22658) region_mover.rb should choose same rsgroup servers as target servers
[ https://issues.apache.org/jira/browse/HBASE-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang.feng updated HBASE-22658: --- Attachment: (was: HBASE-22658.branch-1.patch) > region_mover.rb should choose same rsgroup servers as target servers > -- > > Key: HBASE-22658 > URL: https://issues.apache.org/jira/browse/HBASE-22658 > Project: HBase > Issue Type: Improvement > Components: rsgroup, shell >Affects Versions: 1.4.10 >Reporter: liang.feng >Priority: Major > Labels: gracefulshutdown, region_mover, rsgroup > Attachments: HBASE-22658.branch-1.002.patch, > HBASE-22658.branch-1.patch > > > There are many retries when i am using graceful_stop.sh to shutdown region > server after using regroup, because the target server in a different rsgroup. > This makes it slow to graceful shutdown a regionserver. So i think that > region_mover.rb should only choose same rsgroup servers as target servers. > Region mover is implemented by jruby in hbase1.x and is implemented by java > in hbase2.x . I tried to modify the RegionMover.java class to use the same > logic in hbase2.x, but mvn package failed due to hbase-server module and > hbase-rsgroup moudle needed to depend on each other, then maven throw a "The > projects in the reactor contain a cyclic reference". I couldn't solve it.So I > just uploaded patch for hbase1.x . > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HBASE-22132) Backport HBASE-22115 intent to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani reassigned HBASE-22132: Assignee: Viraj Jasani > Backport HBASE-22115 intent to branch-1 > --- > > Key: HBASE-22132 > URL: https://issues.apache.org/jira/browse/HBASE-22132 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Viraj Jasani >Priority: Major > Fix For: 1.5.0 > > > Check the exposure of branch-1 code to the problems described on HBASE-22115 > and apply the fix approach there. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [hbase] virajjasani commented on a change in pull request #371: HBASE-22648 : Introducing Snapshot TTL
virajjasani commented on a change in pull request #371: HBASE-22648 :
Introducing Snapshot TTL
URL: https://github.com/apache/hbase/pull/371#discussion_r304207256
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/SnapshotDescription.java
##
@@ -63,46 +64,52 @@ public SnapshotDescription(String name, String table,
SnapshotType type) {
}
public SnapshotDescription(String name, TableName table, SnapshotType type) {
-this(name, table, type, null);
+this(name, table, type, null, -1, -1, -1);
}
/**
- * @deprecated since 2.0.0 and will be removed in 3.0.0. Use the version
with the TableName
- * instance instead.
* @see #SnapshotDescription(String, TableName, SnapshotType, String)
* @see https://issues.apache.org/jira/browse/HBASE-16892;>HBASE-16892
+ * @deprecated since 2.0.0 and will be removed in 3.0.0. Use the version
with the TableName
+ * instance instead.
*/
@Deprecated
public SnapshotDescription(String name, String table, SnapshotType type,
String owner) {
this(name, TableName.valueOf(table), type, owner);
}
public SnapshotDescription(String name, TableName table, SnapshotType type,
String owner) {
-this(name, table, type, owner, -1, -1);
+this(name, table, type, owner, -1, -1, -1);
}
/**
+ * @see #SnapshotDescription(String, TableName, SnapshotType, String, long,
long, int)
+ * @see https://issues.apache.org/jira/browse/HBASE-16892;>HBASE-16892
* @deprecated since 2.0.0 and will be removed in 3.0.0. Use the version
with the TableName
* instance instead.
- * @see #SnapshotDescription(String, TableName, SnapshotType, String, long,
int)
- * @see https://issues.apache.org/jira/browse/HBASE-16892;>HBASE-16892
*/
@Deprecated
public SnapshotDescription(String name, String table, SnapshotType type,
String owner,
long creationTime, int version) {
-this(name, TableName.valueOf(table), type, owner, creationTime, version);
+this(name, TableName.valueOf(table), type, owner, creationTime, -1,
version);
}
public SnapshotDescription(String name, TableName table, SnapshotType type,
String owner,
- long creationTime, int version) {
+ long creationTime, long ttl, int version) {
Review comment:
Totally Agree with having minor attributes Map. Also, given that you are
fine with having getters and setters of ttl present for SnapshotDescription and
protos have well usable builder pattern, I believe it's fine to keep adding
each minor attribute explicitly to protos: Snapshot.proto and HBase.proto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] sunhelly commented on a change in pull request #387: HBASE-22642 Make move operations of RSGroup idempotent
sunhelly commented on a change in pull request #387: HBASE-22642 Make move
operations of RSGroup idempotent
URL: https://github.com/apache/hbase/pull/387#discussion_r304208118
##
File path:
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
##
@@ -652,24 +614,156 @@ public boolean evaluate() throws Exception {
RSGroupInfo newGroup) throws IOException{
// get target server to move, which should has more than one regions
// randomly set a region state to SPLITTING to make move fail
-Map> assignMap =
getTableServerRegionMap().get(tableName);
-String rregion = null;
-ServerName toMoveServer = null;
+return randomlySetRegionState(newGroup,
RegionState.State.SPLITTING,tableName);
+ }
+
+ private Pair randomlySetRegionState(RSGroupInfo
groupInfo,
+ RegionState.State state, TableName... tableNames) throws IOException {
+Preconditions.checkArgument(tableNames.length == 1 || tableNames.length ==
2,
+"only support one or two tables");
+Map> assignMap =
getTableServerRegionMap().get(tableNames[0]);
+if(tableNames.length == 2) {
+ Map> assignMap2 =
getTableServerRegionMap().get(tableNames[1]);
+ assignMap2.forEach((k ,v) -> {
+if(!assignMap.containsKey(k)) {
+ assignMap.remove(k);
+}
+ });
+}
+String toCorrectRegionName = null;
+ServerName srcServer = null;
for (ServerName server : assignMap.keySet()) {
- rregion = assignMap.get(server).size() > 1 &&
- !newGroup.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
- if (rregion != null) {
-toMoveServer = server;
+ toCorrectRegionName = assignMap.get(server).size() >= 1 &&
+ !groupInfo.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
+ if (toCorrectRegionName != null) {
+srcServer = server;
break;
}
}
-assert toMoveServer != null;
-RegionInfo ri =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().
-getRegionInfo(Bytes.toBytesBinary(rregion));
+assert srcServer != null;
+RegionInfo toCorrectRegionInfo =
TEST_UTIL.getMiniHBaseCluster().getMaster()
+
.getAssignmentManager().getRegionInfo(Bytes.toBytesBinary(toCorrectRegionName));
RegionStateNode rsn =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().getRegionStates()
-.getRegionStateNode(ri);
-rsn.setState(RegionState.State.SPLITTING);
-return new Pair<>(toMoveServer, rsn);
+.getRegionStateNode(toCorrectRegionInfo);
+rsn.setState(state);
+return new Pair<>(srcServer, rsn);
+ }
+
+ @Test
+ public void testFailedMoveTablesAndRepair() throws Exception{
+final RSGroupInfo newGroup = addGroup(getGroupName(name.getMethodName()),
1);
+Iterator iterator = newGroup.getServers().iterator();
+Address newGroupServer1 = (Address) iterator.next();
+
+// create table
+// randomly set a region state to SPLITTING to make move abort
+Pair gotPair =
createTableWithRegionSplitting(newGroup,
+new Random().nextInt(8) + 4);
+RegionStateNode rsn = gotPair.getSecond();
+
+// move table to newGroup and check regions
+try {
+ rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+ fail("should get IOException when retry exhausted but there still exists
failed moved "
+ + "regions");
+}catch (Exception e){
+ assertTrue(e.getMessage().contains(
+ gotPair.getSecond().getRegionInfo().getRegionNameAsString()));
+}
+for(RegionInfo regionInfo :
master.getAssignmentManager().getAssignedRegions()){
+ if (regionInfo.getTable().equals(tableName) &&
regionInfo.equals(rsn.getRegionInfo())) {
+assertNotEquals(master.getAssignmentManager().getRegionStates()
+.getRegionServerOfRegion(regionInfo).getAddress(),
newGroupServer1);
+ }
+}
+
+// retry move table to newGroup adn check if all regions are corrected
+rsn.setState(RegionState.State.OPEN);
+rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+for(RegionInfo regionInfo :
master.getAssignmentManager().getAssignedRegions()){
+ if (regionInfo.getTable().equals(tableName)) {
+assertEquals(master.getAssignmentManager().getRegionStates()
+.getRegionServerOfRegion(regionInfo).getAddress(),
newGroupServer1);
+ }
+}
+ }
+
+ @Test
+ public void testFailedMoveServersAndRepair() throws Exception{
Review comment:
Maybe the name needs to be changed? I didn't think a proper name...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[GitHub] [hbase] sunhelly commented on a change in pull request #387: HBASE-22642 Make move operations of RSGroup idempotent
sunhelly commented on a change in pull request #387: HBASE-22642 Make move
operations of RSGroup idempotent
URL: https://github.com/apache/hbase/pull/387#discussion_r304208133
##
File path:
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
##
@@ -652,24 +614,156 @@ public boolean evaluate() throws Exception {
RSGroupInfo newGroup) throws IOException{
// get target server to move, which should has more than one regions
// randomly set a region state to SPLITTING to make move fail
-Map> assignMap =
getTableServerRegionMap().get(tableName);
-String rregion = null;
-ServerName toMoveServer = null;
+return randomlySetRegionState(newGroup,
RegionState.State.SPLITTING,tableName);
+ }
+
+ private Pair randomlySetRegionState(RSGroupInfo
groupInfo,
+ RegionState.State state, TableName... tableNames) throws IOException {
+Preconditions.checkArgument(tableNames.length == 1 || tableNames.length ==
2,
+"only support one or two tables");
+Map> assignMap =
getTableServerRegionMap().get(tableNames[0]);
+if(tableNames.length == 2) {
+ Map> assignMap2 =
getTableServerRegionMap().get(tableNames[1]);
+ assignMap2.forEach((k ,v) -> {
+if(!assignMap.containsKey(k)) {
+ assignMap.remove(k);
+}
+ });
+}
+String toCorrectRegionName = null;
+ServerName srcServer = null;
for (ServerName server : assignMap.keySet()) {
- rregion = assignMap.get(server).size() > 1 &&
- !newGroup.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
- if (rregion != null) {
-toMoveServer = server;
+ toCorrectRegionName = assignMap.get(server).size() >= 1 &&
+ !groupInfo.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
+ if (toCorrectRegionName != null) {
+srcServer = server;
break;
}
}
-assert toMoveServer != null;
-RegionInfo ri =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().
-getRegionInfo(Bytes.toBytesBinary(rregion));
+assert srcServer != null;
+RegionInfo toCorrectRegionInfo =
TEST_UTIL.getMiniHBaseCluster().getMaster()
+
.getAssignmentManager().getRegionInfo(Bytes.toBytesBinary(toCorrectRegionName));
RegionStateNode rsn =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().getRegionStates()
-.getRegionStateNode(ri);
-rsn.setState(RegionState.State.SPLITTING);
-return new Pair<>(toMoveServer, rsn);
+.getRegionStateNode(toCorrectRegionInfo);
+rsn.setState(state);
+return new Pair<>(srcServer, rsn);
+ }
+
+ @Test
+ public void testFailedMoveTablesAndRepair() throws Exception{
Review comment:
I removed checking of equality of source group and target group in the
methods moveTables(), moveServers(), and moveServersAndTables() to make them
idempotent. These three cases are testing the three methods. In each case, I
call the method twice. The first time, the call fails because I make a region
in SPLITTING state, and it will not be moved. And I check if all the regions
except the changed region are on target group servers. The second time, I
recover the region state to OPEN to check if all regions on target group
servers after calling the movement method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] sunhelly commented on a change in pull request #387: HBASE-22642 Make move operations of RSGroup idempotent
sunhelly commented on a change in pull request #387: HBASE-22642 Make move
operations of RSGroup idempotent
URL: https://github.com/apache/hbase/pull/387#discussion_r304206381
##
File path:
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
##
@@ -652,24 +614,156 @@ public boolean evaluate() throws Exception {
RSGroupInfo newGroup) throws IOException{
// get target server to move, which should has more than one regions
// randomly set a region state to SPLITTING to make move fail
-Map> assignMap =
getTableServerRegionMap().get(tableName);
-String rregion = null;
-ServerName toMoveServer = null;
+return randomlySetRegionState(newGroup,
RegionState.State.SPLITTING,tableName);
+ }
+
+ private Pair randomlySetRegionState(RSGroupInfo
groupInfo,
+ RegionState.State state, TableName... tableNames) throws IOException {
+Preconditions.checkArgument(tableNames.length == 1 || tableNames.length ==
2,
+"only support one or two tables");
+Map> assignMap =
getTableServerRegionMap().get(tableNames[0]);
+if(tableNames.length == 2) {
+ Map> assignMap2 =
getTableServerRegionMap().get(tableNames[1]);
+ assignMap2.forEach((k ,v) -> {
+if(!assignMap.containsKey(k)) {
+ assignMap.remove(k);
+}
+ });
+}
+String toCorrectRegionName = null;
+ServerName srcServer = null;
for (ServerName server : assignMap.keySet()) {
- rregion = assignMap.get(server).size() > 1 &&
- !newGroup.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
- if (rregion != null) {
-toMoveServer = server;
+ toCorrectRegionName = assignMap.get(server).size() >= 1 &&
+ !groupInfo.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
+ if (toCorrectRegionName != null) {
+srcServer = server;
break;
}
}
-assert toMoveServer != null;
-RegionInfo ri =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().
-getRegionInfo(Bytes.toBytesBinary(rregion));
+assert srcServer != null;
+RegionInfo toCorrectRegionInfo =
TEST_UTIL.getMiniHBaseCluster().getMaster()
+
.getAssignmentManager().getRegionInfo(Bytes.toBytesBinary(toCorrectRegionName));
RegionStateNode rsn =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().getRegionStates()
-.getRegionStateNode(ri);
-rsn.setState(RegionState.State.SPLITTING);
-return new Pair<>(toMoveServer, rsn);
+.getRegionStateNode(toCorrectRegionInfo);
+rsn.setState(state);
+return new Pair<>(srcServer, rsn);
+ }
+
+ @Test
+ public void testFailedMoveTablesAndRepair() throws Exception{
+final RSGroupInfo newGroup = addGroup(getGroupName(name.getMethodName()),
1);
+Iterator iterator = newGroup.getServers().iterator();
+Address newGroupServer1 = (Address) iterator.next();
+
+// create table
+// randomly set a region state to SPLITTING to make move abort
+Pair gotPair =
createTableWithRegionSplitting(newGroup,
+new Random().nextInt(8) + 4);
+RegionStateNode rsn = gotPair.getSecond();
+
+// move table to newGroup and check regions
+try {
+ rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+ fail("should get IOException when retry exhausted but there still exists
failed moved "
+ + "regions");
+}catch (Exception e){
+ assertTrue(e.getMessage().contains(
+ gotPair.getSecond().getRegionInfo().getRegionNameAsString()));
+}
+for(RegionInfo regionInfo :
master.getAssignmentManager().getAssignedRegions()){
+ if (regionInfo.getTable().equals(tableName) &&
regionInfo.equals(rsn.getRegionInfo())) {
+assertNotEquals(master.getAssignmentManager().getRegionStates()
+.getRegionServerOfRegion(regionInfo).getAddress(),
newGroupServer1);
+ }
+}
+
+// retry move table to newGroup adn check if all regions are corrected
+rsn.setState(RegionState.State.OPEN);
+rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+for(RegionInfo regionInfo :
master.getAssignmentManager().getAssignedRegions()){
+ if (regionInfo.getTable().equals(tableName)) {
+assertEquals(master.getAssignmentManager().getRegionStates()
+.getRegionServerOfRegion(regionInfo).getAddress(),
newGroupServer1);
+ }
+}
+ }
+
+ @Test
+ public void testFailedMoveServersAndRepair() throws Exception{
Review comment:
I named it `repair` to express moving once more can make all the regions on
correct RSs. The first time we calling moveServers fails because one region is
in SPLITTING state, and retring to move it also fails. The seconds time we
calling moveServers the region is moved successfully because its state is
changed to OPEN before movements.
[jira] [Commented] (HBASE-22658) region_mover.rb should choose same rsgroup servers as target servers
[ https://issues.apache.org/jira/browse/HBASE-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886651#comment-16886651 ] Reid Chan commented on HBASE-22658: --- ping [~apurtell] sir, would you mind sparing some cycles to take a look? > region_mover.rb should choose same rsgroup servers as target servers > -- > > Key: HBASE-22658 > URL: https://issues.apache.org/jira/browse/HBASE-22658 > Project: HBase > Issue Type: Improvement > Components: rsgroup, shell >Affects Versions: 1.4.10 >Reporter: liang.feng >Priority: Major > Labels: gracefulshutdown, region_mover, rsgroup > Attachments: HBASE-22658.branch-1.002.patch, > HBASE-22658.branch-1.patch > > > There are many retries when i am using graceful_stop.sh to shutdown region > server after using regroup, because the target server in a different rsgroup. > This makes it slow to graceful shutdown a regionserver. So i think that > region_mover.rb should only choose same rsgroup servers as target servers. > Region mover is implemented by jruby in hbase1.x and is implemented by java > in hbase2.x . I tried to modify the RegionMover.java class to use the same > logic in hbase2.x, but mvn package failed due to hbase-server module and > hbase-rsgroup moudle needed to depend on each other, then maven throw a "The > projects in the reactor contain a cyclic reference". I couldn't solve it.So I > just uploaded patch for hbase1.x . > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22658) region_mover.rb should choose same rsgroup servers as target servers
[
https://issues.apache.org/jira/browse/HBASE-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886648#comment-16886648
]
Reid Chan commented on HBASE-22658:
---
After you checkout branch-1, it's better to {{git pull origin branch-1}} again,
to make sure your branch are up-to-date.
> region_mover.rb should choose same rsgroup servers as target servers
> --
>
> Key: HBASE-22658
> URL: https://issues.apache.org/jira/browse/HBASE-22658
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup, shell
>Affects Versions: 1.4.10
>Reporter: liang.feng
>Priority: Major
> Labels: gracefulshutdown, region_mover, rsgroup
> Attachments: HBASE-22658.branch-1.002.patch,
> HBASE-22658.branch-1.patch
>
>
> There are many retries when i am using graceful_stop.sh to shutdown region
> server after using regroup, because the target server in a different rsgroup.
> This makes it slow to graceful shutdown a regionserver. So i think that
> region_mover.rb should only choose same rsgroup servers as target servers.
> Region mover is implemented by jruby in hbase1.x and is implemented by java
> in hbase2.x . I tried to modify the RegionMover.java class to use the same
> logic in hbase2.x, but mvn package failed due to hbase-server module and
> hbase-rsgroup moudle needed to depend on each other, then maven throw a "The
> projects in the reactor contain a cyclic reference". I couldn't solve it.So I
> just uploaded patch for hbase1.x .
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[GitHub] [hbase] sunhelly commented on a change in pull request #387: HBASE-22642 Make move operations of RSGroup idempotent
sunhelly commented on a change in pull request #387: HBASE-22642 Make move
operations of RSGroup idempotent
URL: https://github.com/apache/hbase/pull/387#discussion_r304205485
##
File path:
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
##
@@ -652,24 +614,156 @@ public boolean evaluate() throws Exception {
RSGroupInfo newGroup) throws IOException{
// get target server to move, which should has more than one regions
// randomly set a region state to SPLITTING to make move fail
-Map> assignMap =
getTableServerRegionMap().get(tableName);
-String rregion = null;
-ServerName toMoveServer = null;
+return randomlySetRegionState(newGroup,
RegionState.State.SPLITTING,tableName);
+ }
+
+ private Pair randomlySetRegionState(RSGroupInfo
groupInfo,
+ RegionState.State state, TableName... tableNames) throws IOException {
+Preconditions.checkArgument(tableNames.length == 1 || tableNames.length ==
2,
+"only support one or two tables");
+Map> assignMap =
getTableServerRegionMap().get(tableNames[0]);
+if(tableNames.length == 2) {
+ Map> assignMap2 =
getTableServerRegionMap().get(tableNames[1]);
+ assignMap2.forEach((k ,v) -> {
+if(!assignMap.containsKey(k)) {
+ assignMap.remove(k);
+}
+ });
+}
+String toCorrectRegionName = null;
+ServerName srcServer = null;
for (ServerName server : assignMap.keySet()) {
- rregion = assignMap.get(server).size() > 1 &&
- !newGroup.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
- if (rregion != null) {
-toMoveServer = server;
+ toCorrectRegionName = assignMap.get(server).size() >= 1 &&
+ !groupInfo.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
+ if (toCorrectRegionName != null) {
+srcServer = server;
break;
}
}
-assert toMoveServer != null;
-RegionInfo ri =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().
-getRegionInfo(Bytes.toBytesBinary(rregion));
+assert srcServer != null;
+RegionInfo toCorrectRegionInfo =
TEST_UTIL.getMiniHBaseCluster().getMaster()
+
.getAssignmentManager().getRegionInfo(Bytes.toBytesBinary(toCorrectRegionName));
RegionStateNode rsn =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().getRegionStates()
-.getRegionStateNode(ri);
-rsn.setState(RegionState.State.SPLITTING);
-return new Pair<>(toMoveServer, rsn);
+.getRegionStateNode(toCorrectRegionInfo);
+rsn.setState(state);
+return new Pair<>(srcServer, rsn);
+ }
+
+ @Test
+ public void testFailedMoveTablesAndRepair() throws Exception{
+final RSGroupInfo newGroup = addGroup(getGroupName(name.getMethodName()),
1);
+Iterator iterator = newGroup.getServers().iterator();
+Address newGroupServer1 = (Address) iterator.next();
+
+// create table
+// randomly set a region state to SPLITTING to make move abort
+Pair gotPair =
createTableWithRegionSplitting(newGroup,
+new Random().nextInt(8) + 4);
+RegionStateNode rsn = gotPair.getSecond();
+
+// move table to newGroup and check regions
+try {
+ rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+ fail("should get IOException when retry exhausted but there still exists
failed moved "
+ + "regions");
+}catch (Exception e){
+ assertTrue(e.getMessage().contains(
+ gotPair.getSecond().getRegionInfo().getRegionNameAsString()));
+}
+for(RegionInfo regionInfo :
master.getAssignmentManager().getAssignedRegions()){
+ if (regionInfo.getTable().equals(tableName) &&
regionInfo.equals(rsn.getRegionInfo())) {
+assertNotEquals(master.getAssignmentManager().getRegionStates()
+.getRegionServerOfRegion(regionInfo).getAddress(),
newGroupServer1);
+ }
+}
+
+// retry move table to newGroup adn check if all regions are corrected
+rsn.setState(RegionState.State.OPEN);
+rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+for(RegionInfo regionInfo :
master.getAssignmentManager().getAssignedRegions()){
+ if (regionInfo.getTable().equals(tableName)) {
+assertEquals(master.getAssignmentManager().getRegionStates()
+.getRegionServerOfRegion(regionInfo).getAddress(),
newGroupServer1);
+ }
+}
+ }
+
+ @Test
+ public void testFailedMoveServersAndRepair() throws Exception{
+final RSGroupInfo newGroup = addGroup(getGroupName(name.getMethodName()),
1);
+
+// create table
+// randomly set a region state to SPLITTING to make move abort
+Pair gotPair =
createTableWithRegionSplitting(newGroup,
+new Random().nextInt(8) + 4);
+RegionStateNode rsn = gotPair.getSecond();
+ServerName srcServer = rsn.getRegionLocation();
+
+//
[GitHub] [hbase] sunhelly commented on a change in pull request #387: HBASE-22642 Make move operations of RSGroup idempotent
sunhelly commented on a change in pull request #387: HBASE-22642 Make move
operations of RSGroup idempotent
URL: https://github.com/apache/hbase/pull/387#discussion_r304204486
##
File path:
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
##
@@ -177,10 +177,7 @@ private void checkServersAndTables(Set servers,
Set tables,
+ " does not exist.");
}
RSGroupInfo srcGrp = new RSGroupInfo(tmpSrcGrp);
-if (srcGrp.getName().equals(targetGroupName)) {
Review comment:
Hi, @xcangCRM , for` the javadoc "@throws IOException if nulls or if servers
and tables not belong to the same group"`, I think it means the check of the
source group of moved servers or tables... If there exists two or more source
groups of the the servers and tables, it will throw exception. The check is on
line 184 to 200.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-22658) region_mover.rb should choose same rsgroup servers as target servers
[ https://issues.apache.org/jira/browse/HBASE-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886644#comment-16886644 ] liang.feng commented on HBASE-22658: ping [~reidchan] How did you get through the robot inspection? I operated as follows ： 1、git clone [https://github.com/apache/hbase.git] 2、git checkout -b branch-1 origin/branch-1 3、modify code 4、git commit -am"HBASE-22658 region_mover.rb should choose same rsgroup servers as target servers" 5、git format-patch HEAD^ 6、upload the patch file What went wrong? > region_mover.rb should choose same rsgroup servers as target servers > -- > > Key: HBASE-22658 > URL: https://issues.apache.org/jira/browse/HBASE-22658 > Project: HBase > Issue Type: Improvement > Components: rsgroup, shell >Affects Versions: 1.4.10 >Reporter: liang.feng >Priority: Major > Labels: gracefulshutdown, region_mover, rsgroup > Attachments: HBASE-22658.branch-1.002.patch, > HBASE-22658.branch-1.patch > > > There are many retries when i am using graceful_stop.sh to shutdown region > server after using regroup, because the target server in a different rsgroup. > This makes it slow to graceful shutdown a regionserver. So i think that > region_mover.rb should only choose same rsgroup servers as target servers. > Region mover is implemented by jruby in hbase1.x and is implemented by java > in hbase2.x . I tried to modify the RegionMover.java class to use the same > logic in hbase2.x, but mvn package failed due to hbase-server module and > hbase-rsgroup moudle needed to depend on each other, then maven throw a "The > projects in the reactor contain a cyclic reference". I couldn't solve it.So I > just uploaded patch for hbase1.x . > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-7191) HBCK - Add offline create/fix hbase.version and hbase.id
[
https://issues.apache.org/jira/browse/HBASE-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886642#comment-16886642
]
HBase QA commented on HBASE-7191:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
42s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} hbase-server: The patch generated 0 new + 84
unchanged - 1 fixed = 84 total (was 85) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
24s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 22s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}245m
11s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}295m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/637/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-7191 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974875/HBASE-7191.branch-2.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux fddf3fdc75db 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / 76c080b339 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.11 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/637/testReport/ |
| Max. process+thread count | 4536 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/637/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
[GitHub] [hbase] Reidddddd commented on issue #371: HBASE-22648 : Introducing Snapshot TTL
Reidd commented on issue #371: HBASE-22648 : Introducing Snapshot TTL URL: https://github.com/apache/hbase/pull/371#issuecomment-512081604 LGTM, and I think Andrew's comments are good and should be followed. And please make sure the QA failed tests warning in JIRA are unrelated, because i can still see a snapshot related failed test, `TestSnapshotTemporaryDirectory` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] sunhelly commented on a change in pull request #375: HBASE-21426 TestEncryptionKeyRotation.testCFKeyRotation is flaky
sunhelly commented on a change in pull request #375: HBASE-21426
TestEncryptionKeyRotation.testCFKeyRotation is flaky
URL: https://github.com/apache/hbase/pull/375#discussion_r304202057
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionKeyRotation.java
##
@@ -91,6 +92,7 @@ public static void setUp() throws Exception {
conf.setInt("hfile.format.version", 3);
conf.set(HConstants.CRYPTO_KEYPROVIDER_CONF_KEY,
KeyProviderForTesting.class.getName());
conf.set(HConstants.CRYPTO_MASTERKEY_NAME_CONF_KEY, "hbase");
+conf.setInt("hbase.hfile.compaction.discharger.interval", 10 * 60 * 1000);
Review comment:
I checked the test class config, it's a MediumTests.class, which will
complete in 50 seconds. So the test method testCFKeyRotation() must complete in
50 secs, less than the cleaner interval(2 mins). I met this problem because
when I ran 100 times of the method, I used parameterized running. All 100 cases
used the same minicluster and when ran after 2 mins the case may met with
checking compacted files fail.
@infraio I will convert the change of the conf.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (HBASE-22700) Incorrect timeout in recommended ZooKeeper configuration
[
https://issues.apache.org/jira/browse/HBASE-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maoling updated HBASE-22700:
Description:
The [Recommended
configuration|https://hbase.apache.org/book.html#recommended_configurations.zk]
section for ZooKeeper states that the default zookeeper.session.timeout is 3
minutes, however, the [default
configuration|https://github.com/apache/hbase/blob/master/hbase-common/src/main/resources/hbase-default.xml#L372-L373]
is 90 seconds(
{code:java}
/** Default value for ZooKeeper session timeout */
public static final int DEFAULT_ZK_SESSION_TIMEOUT = 90 * 1000;
).
{code}
This section in the documentation should be modified to reflect the default
configuration.
was:
The [Recommended
configuration|https://hbase.apache.org/book.html#recommended_configurations.zk]
section for ZooKeeper states that the default zookeeper.session.timeout is 3
minutes, however, the [default
configuration|https://github.com/apache/hbase/blob/master/hbase-common/src/main/resources/hbase-default.xml#L372-L373]
is 90 seconds.
This section in the documentation should be modified to reflect the default
configuration.
> Incorrect timeout in recommended ZooKeeper configuration
>
>
> Key: HBASE-22700
> URL: https://issues.apache.org/jira/browse/HBASE-22700
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Peter Somogyi
>Assignee: maoling
>Priority: Minor
> Labels: beginner
>
> The [Recommended
> configuration|https://hbase.apache.org/book.html#recommended_configurations.zk]
> section for ZooKeeper states that the default zookeeper.session.timeout is 3
> minutes, however, the [default
> configuration|https://github.com/apache/hbase/blob/master/hbase-common/src/main/resources/hbase-default.xml#L372-L373]
> is 90 seconds(
>
> {code:java}
> /** Default value for ZooKeeper session timeout */
> public static final int DEFAULT_ZK_SESSION_TIMEOUT = 90 * 1000;
> ).
> {code}
>
> This section in the documentation should be modified to reflect the default
> configuration.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (HBASE-22672) [hbck2] HBCK2 tool support schedule ServerCrashProcedure
[ https://issues.apache.org/jira/browse/HBASE-22672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-22672. Resolution: Duplicate Resolved as duplicate. > [hbck2] HBCK2 tool support schedule ServerCrashProcedure > > > Key: HBASE-22672 > URL: https://issues.apache.org/jira/browse/HBASE-22672 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [hbase-operator-tools] infraio commented on a change in pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
infraio commented on a change in pull request #11: HBASE-21322 Add a
scheduleServerCrashProcedure() API to HbckService
URL:
https://github.com/apache/hbase-operator-tools/pull/11#discussion_r304192229
##
File path: hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java
##
@@ -339,15 +360,22 @@ private static String getCommandUsage() {
writer.println(" $ HBCK2 setTableState users ENABLED");
writer.println(" Returns whatever the previous table state was.");
writer.println();
+writer.println(" " + SCHEDULE_SCP + " ...");
+writer.println(" Schedule ServerCrashProcedure(SCP) for list of
RegionServers. Format");
Review comment:
Schedule ServerCrashProcedure(SCP) for one regionserver?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-operator-tools] infraio commented on a change in pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
infraio commented on a change in pull request #11: HBASE-21322 Add a
scheduleServerCrashProcedure() API to HbckService
URL:
https://github.com/apache/hbase-operator-tools/pull/11#discussion_r304191908
##
File path: hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java
##
@@ -254,6 +258,24 @@ int setRegionState(String region, RegionState.State
newState)
}
}
+ List scheduleSCP(String[] args) throws IOException {
+List serverNames = new ArrayList<>();
+for (String serverName: args) {
+ serverNames.add(parseServerName(serverName));
+}
+try (ClusterConnection c =
(ClusterConnection)ConnectionFactory.createConnection(getConf())) {
+ try (Hbck hbck = c.getHbck()) {
+return hbck.scheduleServerCrashProcedure(serverNames);
Review comment:
What will happened if user use this for 2.1.1? Should we add a version check
here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-operator-tools] infraio commented on a change in pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
infraio commented on a change in pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService URL: https://github.com/apache/hbase-operator-tools/pull/11#discussion_r304191771 ## File path: hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java ## @@ -88,6 +91,7 @@ private static final String FILESYSTEM = "filesystem"; private static final String VERSION = "version"; private static final String SET_REGION_STATE = "setRegionState"; + private static final String SCHEDULE_SCP = "serverCrashProcedures"; Review comment: How about using scheduleSCP as the cmd? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] infraio commented on a change in pull request #375: HBASE-21426 TestEncryptionKeyRotation.testCFKeyRotation is flaky
infraio commented on a change in pull request #375: HBASE-21426
TestEncryptionKeyRotation.testCFKeyRotation is flaky
URL: https://github.com/apache/hbase/pull/375#discussion_r304189237
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionKeyRotation.java
##
@@ -91,6 +92,7 @@ public static void setUp() throws Exception {
conf.setInt("hfile.format.version", 3);
conf.set(HConstants.CRYPTO_KEYPROVIDER_CONF_KEY,
KeyProviderForTesting.class.getName());
conf.set(HConstants.CRYPTO_MASTERKEY_NAME_CONF_KEY, "hbase");
+conf.setInt("hbase.hfile.compaction.discharger.interval", 10 * 60 * 1000);
Review comment:
How about a bigger value? The test will timeout if it can't finished after
780 seconds. So may be use 30 * 60 * 1000 is better which is bigger than 2 *
780?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (HBASE-22700) Incorrect timeout in recommended ZooKeeper configuration
[ https://issues.apache.org/jira/browse/HBASE-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maoling reassigned HBASE-22700: --- Assignee: maoling > Incorrect timeout in recommended ZooKeeper configuration > > > Key: HBASE-22700 > URL: https://issues.apache.org/jira/browse/HBASE-22700 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Peter Somogyi >Assignee: maoling >Priority: Minor > Labels: beginner > > The [Recommended > configuration|https://hbase.apache.org/book.html#recommended_configurations.zk] > section for ZooKeeper states that the default zookeeper.session.timeout is 3 > minutes, however, the [default > configuration|https://github.com/apache/hbase/blob/master/hbase-common/src/main/resources/hbase-default.xml#L372-L373] > is 90 seconds. > This section in the documentation should be modified to reflect the default > configuration. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (HBASE-22639) Unexpected split when a big table has only one region on a regionServer
[ https://issues.apache.org/jira/browse/HBASE-22639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886604#comment-16886604 ] Xu Cang edited comment on HBASE-22639 at 7/17/19 1:30 AM: -- I don't quite catch what you meant here "don't split the region when table is big enough even it is the only one on a node." Can you please elaborate why you think this is a bad thing? Why is splitting "a big enough' table a bad thing? Thanks! was (Author: xucang): I don't quite catch what you meant here "don't split the region when table is big enough even it is the only one on a node." Can you please elaborate why you think this is a bad thing? Thanks! > Unexpected split when a big table has only one region on a regionServer > > > Key: HBASE-22639 > URL: https://issues.apache.org/jira/browse/HBASE-22639 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Zheng Wang >Priority: Minor > > I am using the default policy named SteppingSplitPolicy. > If restart some nodes,it may occur,because this policy didnot judge if the > table is big enough actually. > It brings some unexpected small regions. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22639) Unexpected split when a big table has only one region on a regionServer
[ https://issues.apache.org/jira/browse/HBASE-22639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886604#comment-16886604 ] Xu Cang commented on HBASE-22639: - I don't quite catch what you meant here "don't split the region when table is big enough even it is the only one on a node." Can you please elaborate why you think this is a bad thing? Thanks! > Unexpected split when a big table has only one region on a regionServer > > > Key: HBASE-22639 > URL: https://issues.apache.org/jira/browse/HBASE-22639 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Zheng Wang >Priority: Minor > > I am using the default policy named SteppingSplitPolicy. > If restart some nodes,it may occur,because this policy didnot judge if the > table is big enough actually. > It brings some unexpected small regions. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22639) Unexpected split when a big table has only one region on a regionServer
[ https://issues.apache.org/jira/browse/HBASE-22639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886602#comment-16886602 ] Toshihiro Suzuki commented on HBASE-22639: -- Is this similar to HBASE-12451? > Unexpected split when a big table has only one region on a regionServer > > > Key: HBASE-22639 > URL: https://issues.apache.org/jira/browse/HBASE-22639 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Zheng Wang >Priority: Minor > > I am using the default policy named SteppingSplitPolicy. > If restart some nodes,it may occur,because this policy didnot judge if the > table is big enough actually. > It brings some unexpected small regions. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22697) when RegionServerStoppedException is received, the client should clear meta cache
[ https://issues.apache.org/jira/browse/HBASE-22697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886601#comment-16886601 ] Xu Cang commented on HBASE-22697: - Is HBASE-4412 's idea similar to what you proposed? > when RegionServerStoppedException is received, the client should clear meta > cache > -- > > Key: HBASE-22697 > URL: https://issues.apache.org/jira/browse/HBASE-22697 > Project: HBase > Issue Type: Improvement >Reporter: Junhong Xu >Assignee: Junhong Xu >Priority: Major > > but now it will retry utill exhausted -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22698) [hbase-connectors] Add license header to README.md
[
https://issues.apache.org/jira/browse/HBASE-22698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886600#comment-16886600
]
Xu Cang commented on HBASE-22698:
-
+1
Since it's merged, closing this JIRA? :)
> [hbase-connectors] Add license header to README.md
> --
>
> Key: HBASE-22698
> URL: https://issues.apache.org/jira/browse/HBASE-22698
> Project: HBase
> Issue Type: Task
> Components: hbase-connectors
>Affects Versions: connector-1.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
>
> Build https://builds.apache.org/job/PreCommit-HBASE-CONNECTORS-Build/55/
> failed because {{spark/hbase-spark/README.md}} does not have license header.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header:
> !? /testptch/hbase-connectors/spark/hbase-spark/README.md
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22699) refactor isMetaClearingException
[ https://issues.apache.org/jira/browse/HBASE-22699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886598#comment-16886598 ] Xu Cang commented on HBASE-22699: - Agreed, just by reading this method, it's kind of cryptic. Will review your patch when it's ready. Thanks [~Joseph295] > refactor isMetaClearingException > > > Key: HBASE-22699 > URL: https://issues.apache.org/jira/browse/HBASE-22699 > Project: HBase > Issue Type: Improvement >Reporter: Junhong Xu >Assignee: Junhong Xu >Priority: Minor > > It is not so readable -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22643) Delete region without archiving only if regiondir is present
[ https://issues.apache.org/jira/browse/HBASE-22643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-22643: --- Fix Version/s: 2.1.7 2.2.1 2.3.0 > Delete region without archiving only if regiondir is present > > > Key: HBASE-22643 > URL: https://issues.apache.org/jira/browse/HBASE-22643 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 3.0.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.1, 2.1.7 > > Attachments: HBASE-22643-master-v1.patch, > HBASE-22643-master-v2.patch, HBASE-22643-master-v3.patch > > > Put a condition to delete region without archiving only if regionDir is > present -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [hbase] xcangCRM commented on a change in pull request #387: HBASE-22642 Make move operations of RSGroup idempotent
xcangCRM commented on a change in pull request #387: HBASE-22642 Make move
operations of RSGroup idempotent
URL: https://github.com/apache/hbase/pull/387#discussion_r304181915
##
File path:
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminServer.java
##
@@ -177,10 +177,7 @@ private void checkServersAndTables(Set servers,
Set tables,
+ " does not exist.");
}
RSGroupInfo srcGrp = new RSGroupInfo(tmpSrcGrp);
-if (srcGrp.getName().equals(targetGroupName)) {
Review comment:
on line 167, need to change the javadoc "@throws IOException if nulls or if
servers and tables not belong to the same group"
this is not accurate anymore :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] xcangCRM commented on a change in pull request #387: HBASE-22642 Make move operations of RSGroup idempotent
xcangCRM commented on a change in pull request #387: HBASE-22642 Make move
operations of RSGroup idempotent
URL: https://github.com/apache/hbase/pull/387#discussion_r304183001
##
File path:
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
##
@@ -652,24 +614,156 @@ public boolean evaluate() throws Exception {
RSGroupInfo newGroup) throws IOException{
// get target server to move, which should has more than one regions
// randomly set a region state to SPLITTING to make move fail
-Map> assignMap =
getTableServerRegionMap().get(tableName);
-String rregion = null;
-ServerName toMoveServer = null;
+return randomlySetRegionState(newGroup,
RegionState.State.SPLITTING,tableName);
+ }
+
+ private Pair randomlySetRegionState(RSGroupInfo
groupInfo,
+ RegionState.State state, TableName... tableNames) throws IOException {
+Preconditions.checkArgument(tableNames.length == 1 || tableNames.length ==
2,
+"only support one or two tables");
+Map> assignMap =
getTableServerRegionMap().get(tableNames[0]);
+if(tableNames.length == 2) {
+ Map> assignMap2 =
getTableServerRegionMap().get(tableNames[1]);
+ assignMap2.forEach((k ,v) -> {
+if(!assignMap.containsKey(k)) {
+ assignMap.remove(k);
+}
+ });
+}
+String toCorrectRegionName = null;
+ServerName srcServer = null;
for (ServerName server : assignMap.keySet()) {
- rregion = assignMap.get(server).size() > 1 &&
- !newGroup.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
- if (rregion != null) {
-toMoveServer = server;
+ toCorrectRegionName = assignMap.get(server).size() >= 1 &&
+ !groupInfo.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
+ if (toCorrectRegionName != null) {
+srcServer = server;
break;
}
}
-assert toMoveServer != null;
-RegionInfo ri =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().
-getRegionInfo(Bytes.toBytesBinary(rregion));
+assert srcServer != null;
+RegionInfo toCorrectRegionInfo =
TEST_UTIL.getMiniHBaseCluster().getMaster()
+
.getAssignmentManager().getRegionInfo(Bytes.toBytesBinary(toCorrectRegionName));
RegionStateNode rsn =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().getRegionStates()
-.getRegionStateNode(ri);
-rsn.setState(RegionState.State.SPLITTING);
-return new Pair<>(toMoveServer, rsn);
+.getRegionStateNode(toCorrectRegionInfo);
+rsn.setState(state);
+return new Pair<>(srcServer, rsn);
+ }
+
+ @Test
+ public void testFailedMoveTablesAndRepair() throws Exception{
+final RSGroupInfo newGroup = addGroup(getGroupName(name.getMethodName()),
1);
+Iterator iterator = newGroup.getServers().iterator();
+Address newGroupServer1 = (Address) iterator.next();
+
+// create table
+// randomly set a region state to SPLITTING to make move abort
+Pair gotPair =
createTableWithRegionSplitting(newGroup,
+new Random().nextInt(8) + 4);
+RegionStateNode rsn = gotPair.getSecond();
+
+// move table to newGroup and check regions
+try {
+ rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+ fail("should get IOException when retry exhausted but there still exists
failed moved "
+ + "regions");
+}catch (Exception e){
+ assertTrue(e.getMessage().contains(
+ gotPair.getSecond().getRegionInfo().getRegionNameAsString()));
+}
+for(RegionInfo regionInfo :
master.getAssignmentManager().getAssignedRegions()){
+ if (regionInfo.getTable().equals(tableName) &&
regionInfo.equals(rsn.getRegionInfo())) {
+assertNotEquals(master.getAssignmentManager().getRegionStates()
+.getRegionServerOfRegion(regionInfo).getAddress(),
newGroupServer1);
+ }
+}
+
+// retry move table to newGroup adn check if all regions are corrected
+rsn.setState(RegionState.State.OPEN);
+rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+for(RegionInfo regionInfo :
master.getAssignmentManager().getAssignedRegions()){
+ if (regionInfo.getTable().equals(tableName)) {
+assertEquals(master.getAssignmentManager().getRegionStates()
+.getRegionServerOfRegion(regionInfo).getAddress(),
newGroupServer1);
+ }
+}
+ }
+
+ @Test
+ public void testFailedMoveServersAndRepair() throws Exception{
Review comment:
what does 'repair' mean here? I must missed something. thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[GitHub] [hbase] xcangCRM commented on a change in pull request #387: HBASE-22642 Make move operations of RSGroup idempotent
xcangCRM commented on a change in pull request #387: HBASE-22642 Make move
operations of RSGroup idempotent
URL: https://github.com/apache/hbase/pull/387#discussion_r304183039
##
File path:
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
##
@@ -652,24 +614,156 @@ public boolean evaluate() throws Exception {
RSGroupInfo newGroup) throws IOException{
// get target server to move, which should has more than one regions
// randomly set a region state to SPLITTING to make move fail
-Map> assignMap =
getTableServerRegionMap().get(tableName);
-String rregion = null;
-ServerName toMoveServer = null;
+return randomlySetRegionState(newGroup,
RegionState.State.SPLITTING,tableName);
+ }
+
+ private Pair randomlySetRegionState(RSGroupInfo
groupInfo,
+ RegionState.State state, TableName... tableNames) throws IOException {
+Preconditions.checkArgument(tableNames.length == 1 || tableNames.length ==
2,
+"only support one or two tables");
+Map> assignMap =
getTableServerRegionMap().get(tableNames[0]);
+if(tableNames.length == 2) {
+ Map> assignMap2 =
getTableServerRegionMap().get(tableNames[1]);
+ assignMap2.forEach((k ,v) -> {
+if(!assignMap.containsKey(k)) {
+ assignMap.remove(k);
+}
+ });
+}
+String toCorrectRegionName = null;
+ServerName srcServer = null;
for (ServerName server : assignMap.keySet()) {
- rregion = assignMap.get(server).size() > 1 &&
- !newGroup.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
- if (rregion != null) {
-toMoveServer = server;
+ toCorrectRegionName = assignMap.get(server).size() >= 1 &&
+ !groupInfo.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
+ if (toCorrectRegionName != null) {
+srcServer = server;
break;
}
}
-assert toMoveServer != null;
-RegionInfo ri =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().
-getRegionInfo(Bytes.toBytesBinary(rregion));
+assert srcServer != null;
+RegionInfo toCorrectRegionInfo =
TEST_UTIL.getMiniHBaseCluster().getMaster()
+
.getAssignmentManager().getRegionInfo(Bytes.toBytesBinary(toCorrectRegionName));
RegionStateNode rsn =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().getRegionStates()
-.getRegionStateNode(ri);
-rsn.setState(RegionState.State.SPLITTING);
-return new Pair<>(toMoveServer, rsn);
+.getRegionStateNode(toCorrectRegionInfo);
+rsn.setState(state);
+return new Pair<>(srcServer, rsn);
+ }
+
+ @Test
+ public void testFailedMoveTablesAndRepair() throws Exception{
+final RSGroupInfo newGroup = addGroup(getGroupName(name.getMethodName()),
1);
+Iterator iterator = newGroup.getServers().iterator();
+Address newGroupServer1 = (Address) iterator.next();
+
+// create table
+// randomly set a region state to SPLITTING to make move abort
+Pair gotPair =
createTableWithRegionSplitting(newGroup,
+new Random().nextInt(8) + 4);
+RegionStateNode rsn = gotPair.getSecond();
+
+// move table to newGroup and check regions
+try {
+ rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+ fail("should get IOException when retry exhausted but there still exists
failed moved "
+ + "regions");
+}catch (Exception e){
+ assertTrue(e.getMessage().contains(
+ gotPair.getSecond().getRegionInfo().getRegionNameAsString()));
+}
+for(RegionInfo regionInfo :
master.getAssignmentManager().getAssignedRegions()){
+ if (regionInfo.getTable().equals(tableName) &&
regionInfo.equals(rsn.getRegionInfo())) {
+assertNotEquals(master.getAssignmentManager().getRegionStates()
+.getRegionServerOfRegion(regionInfo).getAddress(),
newGroupServer1);
+ }
+}
+
+// retry move table to newGroup adn check if all regions are corrected
+rsn.setState(RegionState.State.OPEN);
+rsGroupAdmin.moveTables(Sets.newHashSet(tableName), newGroup.getName());
+for(RegionInfo regionInfo :
master.getAssignmentManager().getAssignedRegions()){
+ if (regionInfo.getTable().equals(tableName)) {
+assertEquals(master.getAssignmentManager().getRegionStates()
+.getRegionServerOfRegion(regionInfo).getAddress(),
newGroupServer1);
+ }
+}
+ }
+
+ @Test
+ public void testFailedMoveServersAndRepair() throws Exception{
+final RSGroupInfo newGroup = addGroup(getGroupName(name.getMethodName()),
1);
+
+// create table
+// randomly set a region state to SPLITTING to make move abort
+Pair gotPair =
createTableWithRegionSplitting(newGroup,
+new Random().nextInt(8) + 4);
+RegionStateNode rsn = gotPair.getSecond();
+ServerName srcServer = rsn.getRegionLocation();
+
+//
[GitHub] [hbase] xcangCRM commented on a change in pull request #387: HBASE-22642 Make move operations of RSGroup idempotent
xcangCRM commented on a change in pull request #387: HBASE-22642 Make move
operations of RSGroup idempotent
URL: https://github.com/apache/hbase/pull/387#discussion_r304183153
##
File path:
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
##
@@ -652,24 +614,156 @@ public boolean evaluate() throws Exception {
RSGroupInfo newGroup) throws IOException{
// get target server to move, which should has more than one regions
// randomly set a region state to SPLITTING to make move fail
-Map> assignMap =
getTableServerRegionMap().get(tableName);
-String rregion = null;
-ServerName toMoveServer = null;
+return randomlySetRegionState(newGroup,
RegionState.State.SPLITTING,tableName);
+ }
+
+ private Pair randomlySetRegionState(RSGroupInfo
groupInfo,
+ RegionState.State state, TableName... tableNames) throws IOException {
+Preconditions.checkArgument(tableNames.length == 1 || tableNames.length ==
2,
+"only support one or two tables");
+Map> assignMap =
getTableServerRegionMap().get(tableNames[0]);
+if(tableNames.length == 2) {
+ Map> assignMap2 =
getTableServerRegionMap().get(tableNames[1]);
+ assignMap2.forEach((k ,v) -> {
+if(!assignMap.containsKey(k)) {
+ assignMap.remove(k);
+}
+ });
+}
+String toCorrectRegionName = null;
+ServerName srcServer = null;
for (ServerName server : assignMap.keySet()) {
- rregion = assignMap.get(server).size() > 1 &&
- !newGroup.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
- if (rregion != null) {
-toMoveServer = server;
+ toCorrectRegionName = assignMap.get(server).size() >= 1 &&
+ !groupInfo.containsServer(server.getAddress()) ?
assignMap.get(server).get(0) : null;
+ if (toCorrectRegionName != null) {
+srcServer = server;
break;
}
}
-assert toMoveServer != null;
-RegionInfo ri =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().
-getRegionInfo(Bytes.toBytesBinary(rregion));
+assert srcServer != null;
+RegionInfo toCorrectRegionInfo =
TEST_UTIL.getMiniHBaseCluster().getMaster()
+
.getAssignmentManager().getRegionInfo(Bytes.toBytesBinary(toCorrectRegionName));
RegionStateNode rsn =
TEST_UTIL.getMiniHBaseCluster().getMaster().getAssignmentManager().getRegionStates()
-.getRegionStateNode(ri);
-rsn.setState(RegionState.State.SPLITTING);
-return new Pair<>(toMoveServer, rsn);
+.getRegionStateNode(toCorrectRegionInfo);
+rsn.setState(state);
+return new Pair<>(srcServer, rsn);
+ }
+
+ @Test
+ public void testFailedMoveTablesAndRepair() throws Exception{
Review comment:
I see you added 3 tests here, can you put some comments about these?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase-operator-tools] asf-ci commented on issue #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
asf-ci commented on issue #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService URL: https://github.com/apache/hbase-operator-tools/pull/11#issuecomment-512049512 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/40/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22132) Backport HBASE-22115 intent to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886569#comment-16886569 ] Andrew Purtell commented on HBASE-22132: Sounds good to me [~vjasani] Also in general would be good to make a pass over where we are opening TraceScopes and ensure that we are keeping around a local reference and closing it, either manually or via try-with-resources. > Backport HBASE-22115 intent to branch-1 > --- > > Key: HBASE-22132 > URL: https://issues.apache.org/jira/browse/HBASE-22132 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > > Check the exposure of branch-1 code to the problems described on HBASE-22115 > and apply the fix approach there. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [hbase-operator-tools] saintstack commented on issue #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
saintstack commented on issue #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService URL: https://github.com/apache/hbase-operator-tools/pull/11#issuecomment-512048150 All about import order of imports... Punting on doing any more here. Was going to merge... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] asf-ci commented on issue #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
asf-ci commented on issue #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService URL: https://github.com/apache/hbase-operator-tools/pull/11#issuecomment-512047141 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/39/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] asf-ci commented on issue #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
asf-ci commented on issue #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService URL: https://github.com/apache/hbase-operator-tools/pull/11#issuecomment-512044226 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/38/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-21393) Add an API ScheduleSCP() to HBCK2
[ https://issues.apache.org/jira/browse/HBASE-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886556#comment-16886556 ] stack commented on HBASE-21393: --- Added PR to address [~wchevreuil] remarks: https://github.com/apache/hbase-operator-tools/pull/11 > Add an API ScheduleSCP() to HBCK2 > -- > > Key: HBASE-21393 > URL: https://issues.apache.org/jira/browse/HBASE-21393 > Project: HBase > Issue Type: Sub-task > Components: hbase-operator-tools, hbck2 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: 0001-HBASE-21393-Add-an-API-ScheduleSCP-to-HBCK2.patch, > 0002-HBASE-21393-Add-an-API-ScheduleSCP-to-HBCK2.patch > > > Add the API of ScheduleSCP() to hbase-operator-tools so that operators can > schedule ServerCrashProcedures of specified regionservers. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [hbase-operator-tools] asf-ci commented on issue #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
asf-ci commented on issue #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService URL: https://github.com/apache/hbase-operator-tools/pull/11#issuecomment-512038124 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/37/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] saintstack opened a new pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService
saintstack opened a new pull request #11: HBASE-21322 Add a scheduleServerCrashProcedure() API to HbckService URL: https://github.com/apache/hbase-operator-tools/pull/11 Signed-off-by: Wellington Chevreuil This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] brfrn169 commented on issue #380: HBASE-22690 Deprecate / Remove OfflineMetaRepair in hbase-2+
brfrn169 commented on issue #380: HBASE-22690 Deprecate / Remove OfflineMetaRepair in hbase-2+ URL: https://github.com/apache/hbase/pull/380#issuecomment-512035024 I fixed some checkstyle warnings. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-21462) Add note for CopyTable section explained it does not perform a diff, but a full write from source to target
[
https://issues.apache.org/jira/browse/HBASE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886534#comment-16886534
]
Hudson commented on HBASE-21462:
Results for branch master
[build #1230 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1230/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1230//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1230//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1230//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add note for CopyTable section explained it does not perform a diff, but a
> full write from source to target
> ---
>
> Key: HBASE-21462
> URL: https://issues.apache.org/jira/browse/HBASE-21462
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.0.0, 2.1.5
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-21462-master.002.patch,
> HBASE-21462.master.001.patch
>
>
> Performance is a common issue with CopyTable is when the key/time range for
> the data to be copied is large because it basically scans all rows on the
> specified range, in the source and perform related puts on the source. We
> should add extra note explaining that on the reference guide, to help
> users/admins understand when to choose between the different tools/approaches
> for syncing clusters.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22685) Add to migration doc that meta should be healthy before upgrade
[
https://issues.apache.org/jira/browse/HBASE-22685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886518#comment-16886518
]
HBase QA commented on HBASE-22685:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 11m
50s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 7m
17s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/636/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22685 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974877/HBASE-22685.master.001.patch
|
| Optional Tests | dupname asflicense refguide |
| uname | Linux 95f131a822c2 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 72e58a8fed |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/636/artifact/patchprocess/branch-site/book.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/636/artifact/patchprocess/whitespace-eol.txt
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/636/artifact/patchprocess/patch-site/book.html
|
| Max. process+thread count | 58 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/636/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
> Add to migration doc that meta should be healthy before upgrade
> ---
>
> Key: HBASE-22685
> URL: https://issues.apache.org/jira/browse/HBASE-22685
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: stack
>Assignee: stack
>Priority: Major
> Labels: documentation
> Attachments: HBASE-22685.master.001.patch
>
>
> Add a note to doc that operator ensures healthy meta before upgrade.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886509#comment-16886509
]
HBase QA commented on HBASE-22701:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 37s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
53s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/635/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22701 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974873/HBASE-22701.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9449dd022de4 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 72e58a8fed |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.11 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/635/testReport/ |
| Max. process+thread count | 301 (vs. ulimit of 1) |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/635/console |
| Powered
[jira] [Commented] (HBASE-21393) Add an API ScheduleSCP() to HBCK2
[ https://issues.apache.org/jira/browse/HBASE-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886492#comment-16886492 ] stack commented on HBASE-21393: --- +1 on [~wchevreuil] review. No guava in hbck2. If need it, use shaded thirdparty version let me update patch to address [~wchevreuil] review. > Add an API ScheduleSCP() to HBCK2 > -- > > Key: HBASE-21393 > URL: https://issues.apache.org/jira/browse/HBASE-21393 > Project: HBase > Issue Type: Sub-task > Components: hbase-operator-tools, hbck2 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: 0001-HBASE-21393-Add-an-API-ScheduleSCP-to-HBCK2.patch, > 0002-HBASE-21393-Add-an-API-ScheduleSCP-to-HBCK2.patch > > > Add the API of ScheduleSCP() to hbase-operator-tools so that operators can > schedule ServerCrashProcedures of specified regionservers. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [hbase] apurtell commented on a change in pull request #371: HBASE-22648 : Introducing Snapshot TTL
apurtell commented on a change in pull request #371: HBASE-22648 : Introducing
Snapshot TTL
URL: https://github.com/apache/hbase/pull/371#discussion_r304135678
##
File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
##
@@ -1372,6 +1372,51 @@ default void snapshot(String snapshotName, TableName
tableName, SnapshotType typ
snapshot(new SnapshotDescription(snapshotName, tableName, type));
}
+ /**
+ * Create typed snapshot of the table. Snapshots are considered unique based
on the name of the
+ * snapshot. Snapshots are taken sequentially even when requested
concurrently, across
+ * all tables. Attempts to take a snapshot with the same name (even a
different type or with
+ * different parameters) will fail with a {@link SnapshotCreationException}
indicating the
+ * duplicate naming. Snapshot names follow the same naming constraints as
tables in HBase. See
+ * {@link
org.apache.hadoop.hbase.TableName#isLegalFullyQualifiedTableName(byte[])}.
+ * Snapshot can live with ttl seconds.
+ *
+ * @param snapshotName name to give the snapshot on the filesystem. Must be
unique from all other
+ * snapshots stored on the cluster
+ * @param tableNamename of the table to snapshot
+ * @param type type of snapshot to take
+ * @param ttl time to live in seconds. After expiry, the snapshot
should be deleted
+ * @throws IOException we fail to reach the master
+ * @throws SnapshotCreationException if snapshot creation failed
+ * @throws IllegalArgumentException if the snapshot request is formatted
incorrectly
+ */
+ default void snapshot(String snapshotName, TableName tableName, SnapshotType
type, Long ttl)
Review comment:
TTL may be the first of more than one snapshot attribute we'd want to pass
through this API. We can create a new method (and more backwards compat
methods) every time, or make only one change: instead of parameter 'ttl' of
type Long, introduce a parameter 'attributes' or 'properties' of type
Map or java.util.Properties?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] apurtell commented on a change in pull request #371: HBASE-22648 : Introducing Snapshot TTL
apurtell commented on a change in pull request #371: HBASE-22648 : Introducing
Snapshot TTL
URL: https://github.com/apache/hbase/pull/371#discussion_r304136426
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/SnapshotDescription.java
##
@@ -139,15 +146,27 @@ public long getCreationTime() {
return this.creationTime;
}
+ // get snapshot ttl in sec
+ public long getTtl() {
Review comment:
Getters and setters for ttl is fine even if also using generic attribute map
with getters and setters for elements of same.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] apurtell commented on a change in pull request #371: HBASE-22648 : Introducing Snapshot TTL
apurtell commented on a change in pull request #371: HBASE-22648 : Introducing Snapshot TTL URL: https://github.com/apache/hbase/pull/371#discussion_r304136760 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java ## @@ -1445,6 +1445,15 @@ "hbase.util.default.lossycounting.errorrate"; public static final String NOT_IMPLEMENTED = "Not implemented"; + // Default TTL - FOREVER + public static final long DEFAULT_SNAPSHOT_TTL = 0; + + // User defined Default TTL config key + public static final String DEFAULT_SNAPSHOT_TTL_CONFIG_KEY = "hbase.master.snapshot.ttl.default"; Review comment: This should be "hbase.master.snapshot.ttl" because you are setting an effective value. (Calling the default constant DEFAULT_SNAPSHOT_TTL is fine because that actually is naming a default value. ) Minor point. If you like. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] apurtell commented on a change in pull request #371: HBASE-22648 : Introducing Snapshot TTL
apurtell commented on a change in pull request #371: HBASE-22648 : Introducing
Snapshot TTL
URL: https://github.com/apache/hbase/pull/371#discussion_r304136202
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/SnapshotDescription.java
##
@@ -63,46 +64,52 @@ public SnapshotDescription(String name, String table,
SnapshotType type) {
}
public SnapshotDescription(String name, TableName table, SnapshotType type) {
-this(name, table, type, null);
+this(name, table, type, null, -1, -1, -1);
}
/**
- * @deprecated since 2.0.0 and will be removed in 3.0.0. Use the version
with the TableName
- * instance instead.
* @see #SnapshotDescription(String, TableName, SnapshotType, String)
* @see https://issues.apache.org/jira/browse/HBASE-16892;>HBASE-16892
+ * @deprecated since 2.0.0 and will be removed in 3.0.0. Use the version
with the TableName
+ * instance instead.
*/
@Deprecated
public SnapshotDescription(String name, String table, SnapshotType type,
String owner) {
this(name, TableName.valueOf(table), type, owner);
}
public SnapshotDescription(String name, TableName table, SnapshotType type,
String owner) {
-this(name, table, type, owner, -1, -1);
+this(name, table, type, owner, -1, -1, -1);
}
/**
+ * @see #SnapshotDescription(String, TableName, SnapshotType, String, long,
long, int)
+ * @see https://issues.apache.org/jira/browse/HBASE-16892;>HBASE-16892
* @deprecated since 2.0.0 and will be removed in 3.0.0. Use the version
with the TableName
* instance instead.
- * @see #SnapshotDescription(String, TableName, SnapshotType, String, long,
int)
- * @see https://issues.apache.org/jira/browse/HBASE-16892;>HBASE-16892
*/
@Deprecated
public SnapshotDescription(String name, String table, SnapshotType type,
String owner,
long creationTime, int version) {
-this(name, TableName.valueOf(table), type, owner, creationTime, version);
+this(name, TableName.valueOf(table), type, owner, creationTime, -1,
version);
}
public SnapshotDescription(String name, TableName table, SnapshotType type,
String owner,
- long creationTime, int version) {
+ long creationTime, long ttl, int version) {
Review comment:
Likewise I would say that the current set of method parameters here are fine
but 'ttl' is more of a minor or situational attribute of the snapshot. We
should stop creating more SnapshotDescription constructors and introduce a
generic map for minor attributes. For your consideration.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] apurtell commented on a change in pull request #371: HBASE-22648 : Introducing Snapshot TTL
apurtell commented on a change in pull request #371: HBASE-22648 : Introducing Snapshot TTL URL: https://github.com/apache/hbase/pull/371#discussion_r304137323 ## File path: hbase-common/src/main/resources/hbase-default.xml ## @@ -1864,4 +1864,23 @@ possible configurations would overwhelm and obscure the important. Default is 5 minutes. Make it 30 seconds for tests. See HBASE-19794 for some context. + +hbase.master.cleaner.snapshot.interval +180 + + Snapshot Cleanup chore interval in milliseconds. + The cleanup thread keeps running at this interval + to find all snapshots that are expired based on TTL + and delete them. + + + +hbase.master.snapshot.ttl.default Review comment: Same This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-21262) [hbck2] AMv2 Lock Picker
[ https://issues.apache.org/jira/browse/HBASE-21262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21262: -- Issue Type: Bug (was: Sub-task) Parent: (was: HBASE-21745) > [hbck2] AMv2 Lock Picker > > > Key: HBASE-21262 > URL: https://issues.apache.org/jira/browse/HBASE-21262 > Project: HBase > Issue Type: Bug > Components: hbck2, Operability >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > This issue is about adding a lock picker to the HbckService > Over the w/e I had interesting case where an enable failed -- a subprocedure > ran into an exclusive lock (I think) -- and then the parent enabletabled > tried rollback. The rollback threw CODE-BUG because some subprocedures were > in unrollbackable states so we ended up skipping out of the enable table > procedure. The enable table procedure was marked ROLLBACKED... so it got > GC'd. But the exclusive lock it had on the table stayed in place. > The above has to be fixed but for the future, we need way to kill locks > otherwise only alternative if removing master proc wal files -- which is a > bigger pain restoring good state. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (HBASE-21262) [hbck2] AMv2 Lock Picker
[ https://issues.apache.org/jira/browse/HBASE-21262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-21262. --- Resolution: Later Resolving as later. I'm not going to run into this any time soon. No one has run into it other than I. Lets come back here if we have another instance where it is needed. > [hbck2] AMv2 Lock Picker > > > Key: HBASE-21262 > URL: https://issues.apache.org/jira/browse/HBASE-21262 > Project: HBase > Issue Type: Sub-task > Components: hbck2, Operability >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > This issue is about adding a lock picker to the HbckService > Over the w/e I had interesting case where an enable failed -- a subprocedure > ran into an exclusive lock (I think) -- and then the parent enabletabled > tried rollback. The rollback threw CODE-BUG because some subprocedures were > in unrollbackable states so we ended up skipping out of the enable table > procedure. The enable table procedure was marked ROLLBACKED... so it got > GC'd. But the exclusive lock it had on the table stayed in place. > The above has to be fixed but for the future, we need way to kill locks > otherwise only alternative if removing master proc wal files -- which is a > bigger pain restoring good state. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22685) Add to migration doc that meta should be healthy before upgrade
[ https://issues.apache.org/jira/browse/HBASE-22685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22685: -- Labels: documentation (was: ) Assignee: stack Tags: documentation Status: Patch Available (was: Open) .001 Add note to fix hbck1 complaint before upgrade where operator will hopefully see it. > Add to migration doc that meta should be healthy before upgrade > --- > > Key: HBASE-22685 > URL: https://issues.apache.org/jira/browse/HBASE-22685 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Assignee: stack >Priority: Major > Labels: documentation > Attachments: HBASE-22685.master.001.patch > > > Add a note to doc that operator ensures healthy meta before upgrade. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-22685) Add to migration doc that meta should be healthy before upgrade
[ https://issues.apache.org/jira/browse/HBASE-22685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22685: -- Attachment: HBASE-22685.master.001.patch > Add to migration doc that meta should be healthy before upgrade > --- > > Key: HBASE-22685 > URL: https://issues.apache.org/jira/browse/HBASE-22685 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: stack >Priority: Major > Attachments: HBASE-22685.master.001.patch > > > Add a note to doc that operator ensures healthy meta before upgrade. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-7191) HBCK - Add offline create/fix hbase.version and hbase.id
[ https://issues.apache.org/jira/browse/HBASE-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886474#comment-16886474 ] stack commented on HBASE-7191: -- Small patch that outputs one message if hbase.version is missing and another if version is old. Points to hbck2 rather than hbck1 for fixing missing hbase.version. > HBCK - Add offline create/fix hbase.version and hbase.id > - > > Key: HBASE-7191 > URL: https://issues.apache.org/jira/browse/HBASE-7191 > Project: HBase > Issue Type: Sub-task > Components: hbck >Affects Versions: 0.94.1 >Reporter: Enis Soztutar >Assignee: xufeng >Priority: Major > Attachments: 7191-2.patch, HBASE-7191.branch-2.001.patch, > hbck1-1.4-v1.patch > > > One of our clients run into a problem, in which they have the hbase.root.dir, > and cluster data, but their hbase.id and hbase.version files are corrupted. > HMaster creates those on start, but not if there is already existing data. > We can add smt like --fixIdFile, and ability for HBCK to do some offline > repairs for the version file. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-7191) HBCK - Add offline create/fix hbase.version and hbase.id
[ https://issues.apache.org/jira/browse/HBASE-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7191: - Issue Type: Sub-task (was: Improvement) Parent: HBASE-21745 > HBCK - Add offline create/fix hbase.version and hbase.id > - > > Key: HBASE-7191 > URL: https://issues.apache.org/jira/browse/HBASE-7191 > Project: HBase > Issue Type: Sub-task > Components: hbck >Affects Versions: 0.94.1 >Reporter: Enis Soztutar >Assignee: xufeng >Priority: Major > Attachments: 7191-2.patch, HBASE-7191.branch-2.001.patch, > hbck1-1.4-v1.patch > > > One of our clients run into a problem, in which they have the hbase.root.dir, > and cluster data, but their hbase.id and hbase.version files are corrupted. > HMaster creates those on start, but not if there is already existing data. > We can add smt like --fixIdFile, and ability for HBCK to do some offline > repairs for the version file. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-7191) HBCK - Add offline create/fix hbase.version and hbase.id
[ https://issues.apache.org/jira/browse/HBASE-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7191: - Attachment: HBASE-7191.branch-2.001.patch > HBCK - Add offline create/fix hbase.version and hbase.id > - > > Key: HBASE-7191 > URL: https://issues.apache.org/jira/browse/HBASE-7191 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.94.1 >Reporter: Enis Soztutar >Assignee: xufeng >Priority: Major > Attachments: 7191-2.patch, HBASE-7191.branch-2.001.patch, > hbck1-1.4-v1.patch > > > One of our clients run into a problem, in which they have the hbase.root.dir, > and cluster data, but their hbase.id and hbase.version files are corrupted. > HMaster creates those on start, but not if there is already existing data. > We can add smt like --fixIdFile, and ability for HBCK to do some offline > repairs for the version file. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886466#comment-16886466
]
Josh Elser commented on HBASE-22701:
.001 just catches any exception thrown by initialization of the temp directory.
Of note, DynamicClassLoader may get used by both client and server (e.g.
deserialization Filters on RegionServer, deserializing RemoteExceptions client
side).
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.2.1, 2.1.6
>
> Attachments: HBASE-22701.001.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.(ProtobufUtil.java:260)
> ... 16 more
> {noformat}
> Every subsequent call will result in a NoClassDefFoundError, because we
> already tried to load DynamicClassLoader.class once and failed.
> {noformat}
> 2019-07-14 06:25:34,380 ERROR
> [RpcServer.Metadata.Fifo.handler=2,queue=2,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder
> at
>
[jira] [Updated] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-22701:
---
Status: Patch Available (was: Open)
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.2.1, 2.1.6
>
> Attachments: HBASE-22701.001.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.(ProtobufUtil.java:260)
> ... 16 more
> {noformat}
> Every subsequent call will result in a NoClassDefFoundError, because we
> already tried to load DynamicClassLoader.class once and failed.
> {noformat}
> 2019-07-14 06:25:34,380 ERROR
> [RpcServer.Metadata.Fifo.handler=2,queue=2,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
>
[jira] [Updated] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-22701:
---
Attachment: HBASE-22701.001.patch
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.2.1, 2.1.6
>
> Attachments: HBASE-22701.001.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.(ProtobufUtil.java:260)
> ... 16 more
> {noformat}
> Every subsequent call will result in a NoClassDefFoundError, because we
> already tried to load DynamicClassLoader.class once and failed.
> {noformat}
> 2019-07-14 06:25:34,380 ERROR
> [RpcServer.Metadata.Fifo.handler=2,queue=2,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
>
[jira] [Commented] (HBASE-7191) HBCK - Add offline create/fix hbase.version and hbase.id
[
https://issues.apache.org/jira/browse/HBASE-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886440#comment-16886440
]
stack commented on HBASE-7191:
--
Back looking at this issue. This is what I get if I remove the hbase.version
file:
{code}
org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout
needs to be upgraded. You have version null and I want version 8. Consult
http://hbase.apache.org/book.html for further information about upgrading
HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck
-fixVersionFile'.
at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:446)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:273)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:153)
at
org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:124)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:901)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2120)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:577)
at java.base/java.lang.Thread.run(Thread.java:844)
{code}
After commit of HBASE-22688 this morning, the fix is running the filesystem
command from hbck2 with the -fix option. It restores the hbase.version file.
{code}$
HBASE_CLASSPATH_PREFIX=../hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar
bin/hbase org.apache.hbase.HBCK2 filesystem -fix{code}
It does NOT put back the hbase.id file. Launching the hbase process creates a
new one if missing which is probably not ideal; should probably try and find
old cluster id and put that back rather than create a new one. hbase cluster id
doesn't seem critical so will leave it for now until someone complains.
Meantime let me close out this issue with a patch that fixes the message that
is spewed when hbase.version is missing.
> HBCK - Add offline create/fix hbase.version and hbase.id
> -
>
> Key: HBASE-7191
> URL: https://issues.apache.org/jira/browse/HBASE-7191
> Project: HBase
> Issue Type: Improvement
> Components: hbck
>Affects Versions: 0.94.1
>Reporter: Enis Soztutar
>Assignee: xufeng
>Priority: Major
> Attachments: 7191-2.patch, hbck1-1.4-v1.patch
>
>
> One of our clients run into a problem, in which they have the hbase.root.dir,
> and cluster data, but their hbase.id and hbase.version files are corrupted.
> HMaster creates those on start, but not if there is already existing data.
> We can add smt like --fixIdFile, and ability for HBCK to do some offline
> repairs for the version file.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
Josh Elser created HBASE-22701:
--
Summary: Better handle invalid local directory for
DynamicClassLoader
Key: HBASE-22701
URL: https://issues.apache.org/jira/browse/HBASE-22701
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: 2.3.0, 2.2.1, 2.1.6
If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
which is not writable to it, you will get some weird errors on the scan path. I
just saw this (again?) with Phoenix.
Specifically, the first attempt to reference DynamicClassLoader (via
ProtobufUtil), will result in an ExceptionInInitializationError because the
unchecked exception coming out of DynamicClassLoader's constructor interrupts
the loading of {{DynamicClassLoader.class}}.
{noformat}
2019-07-14 06:25:34,284 ERROR
[RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
coprocessor.MetaDataEndpointImpl: dropTable failed
org.apache.hadoop.hbase.DoNotRetryIOException:
java.lang.ExceptionInInitializerError
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ExceptionInInitializerError
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
at
org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
at
org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
at
org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
... 7 more
Caused by: java.lang.RuntimeException: Failed to create local dir
/hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
at
org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
at
org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.(ProtobufUtil.java:260)
... 16 more
{noformat}
Every subsequent call will result in a NoClassDefFoundError, because we already
tried to load DynamicClassLoader.class once and failed.
{noformat}
2019-07-14 06:25:34,380 ERROR
[RpcServer.Metadata.Fifo.handler=2,queue=2,port=16020]
coprocessor.MetaDataEndpointImpl: dropTable failed
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError:
Could not initialize class
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at
[jira] [Created] (HBASE-22700) Incorrect timeout in recommended ZooKeeper configuration
Peter Somogyi created HBASE-22700: - Summary: Incorrect timeout in recommended ZooKeeper configuration Key: HBASE-22700 URL: https://issues.apache.org/jira/browse/HBASE-22700 Project: HBase Issue Type: Bug Components: documentation Reporter: Peter Somogyi The [Recommended configuration|https://hbase.apache.org/book.html#recommended_configurations.zk] section for ZooKeeper states that the default zookeeper.session.timeout is 3 minutes, however, the [default configuration|https://github.com/apache/hbase/blob/master/hbase-common/src/main/resources/hbase-default.xml#L372-L373] is 90 seconds. This section in the documentation should be modified to reflect the default configuration. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[
https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886408#comment-16886408
]
Andrew Purtell edited comment on HBASE-22623 at 7/16/19 7:43 PM:
-
{quote}The signatures for the Append/Increment coprocs and the Batch mutation
coprocs are different enough that I'm somewhat skeptical we could make a single
hook that would work for both
{quote}
So don't modify existing hooks.
The original proposal on this issue was a new hook at every call site of
WAL#append in HRegion: "Add RegionObserver coprocessor hook for preWALAppend"
was (Author: apurtell):
{quote}The signatures for the Append/Increment coprocs and the Batch mutation
coprocs are different enough that I'm somewhat skeptical we could make a single
hook that would work for both
{quote}
So don't.
The original proposal on this issue was a new hook at every call site of
WAL#append in HRegion: "Add RegionObserver coprocessor hook for preWALAppend"
> Add RegionObserver coprocessor hook for preWALAppend
>
>
> Key: HBASE-22623
> URL: https://issues.apache.org/jira/browse/HBASE-22623
> Project: HBase
> Issue Type: New Feature
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
>
> While many coprocessor hooks expose the WALEdit to implementing coprocs,
> there aren't any that expose the WALKey before it's created and added to the
> WALEntry.
> It's sometimes useful for coprocessors to be able to edit the WALKey, for
> example to add extended attributes using the fields to be added in
> HBASE-22622.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[
https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886410#comment-16886410
]
Andrew Purtell commented on HBASE-22623:
{quote}instrumentation for the two code paths could be done in separate JIRAs,
a new one for Append + Increment and this one for Batch Mutate. Both would need
to be done to get full coverage, and we'd want them in the same release
{quote}
Yes it could be done this way too.
I'm not making my mind up about this or anything ahead of seeing a patch to
evaluate. Let's consider the code.
> Add RegionObserver coprocessor hook for preWALAppend
>
>
> Key: HBASE-22623
> URL: https://issues.apache.org/jira/browse/HBASE-22623
> Project: HBase
> Issue Type: New Feature
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
>
> While many coprocessor hooks expose the WALEdit to implementing coprocs,
> there aren't any that expose the WALKey before it's created and added to the
> WALEntry.
> It's sometimes useful for coprocessors to be able to edit the WALKey, for
> example to add extended attributes using the fields to be added in
> HBASE-22622.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[
https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886408#comment-16886408
]
Andrew Purtell edited comment on HBASE-22623 at 7/16/19 7:41 PM:
-
{quote}The signatures for the Append/Increment coprocs and the Batch mutation
coprocs are different enough that I'm somewhat skeptical we could make a single
hook that would work for both
{quote}
So don't.
The original proposal on this issue was a new hook at every call site of
WAL#append in HRegion: "Add RegionObserver coprocessor hook for preWALAppend"
was (Author: apurtell):
{quote}The signatures for the Append/Increment coprocs and the Batch mutation
coprocs are different enough that I'm somewhat skeptical we could make a single
hook that would work for both
{quote}
So don't.
The proposal is a new hook at every call site of WAL#append in HRegion: "Add
RegionObserver coprocessor hook for preWALAppend"
> Add RegionObserver coprocessor hook for preWALAppend
>
>
> Key: HBASE-22623
> URL: https://issues.apache.org/jira/browse/HBASE-22623
> Project: HBase
> Issue Type: New Feature
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
>
> While many coprocessor hooks expose the WALEdit to implementing coprocs,
> there aren't any that expose the WALKey before it's created and added to the
> WALEntry.
> It's sometimes useful for coprocessors to be able to edit the WALKey, for
> example to add extended attributes using the fields to be added in
> HBASE-22622.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[
https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886408#comment-16886408
]
Andrew Purtell commented on HBASE-22623:
{quote}The signatures for the Append/Increment coprocs and the Batch mutation
coprocs are different enough that I'm somewhat skeptical we could make a single
hook that would work for both
{quote}
So don't.
The proposal is a new hook at every call site of WAL#append: "Add
RegionObserver coprocessor hook for preWALAppend"
> Add RegionObserver coprocessor hook for preWALAppend
>
>
> Key: HBASE-22623
> URL: https://issues.apache.org/jira/browse/HBASE-22623
> Project: HBase
> Issue Type: New Feature
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
>
> While many coprocessor hooks expose the WALEdit to implementing coprocs,
> there aren't any that expose the WALKey before it's created and added to the
> WALEntry.
> It's sometimes useful for coprocessors to be able to edit the WALKey, for
> example to add extended attributes using the fields to be added in
> HBASE-22622.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[
https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886408#comment-16886408
]
Andrew Purtell edited comment on HBASE-22623 at 7/16/19 7:40 PM:
-
{quote}The signatures for the Append/Increment coprocs and the Batch mutation
coprocs are different enough that I'm somewhat skeptical we could make a single
hook that would work for both
{quote}
So don't.
The proposal is a new hook at every call site of WAL#append in HRegion: "Add
RegionObserver coprocessor hook for preWALAppend"
was (Author: apurtell):
{quote}The signatures for the Append/Increment coprocs and the Batch mutation
coprocs are different enough that I'm somewhat skeptical we could make a single
hook that would work for both
{quote}
So don't.
The proposal is a new hook at every call site of WAL#append: "Add
RegionObserver coprocessor hook for preWALAppend"
> Add RegionObserver coprocessor hook for preWALAppend
>
>
> Key: HBASE-22623
> URL: https://issues.apache.org/jira/browse/HBASE-22623
> Project: HBase
> Issue Type: New Feature
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
>
> While many coprocessor hooks expose the WALEdit to implementing coprocs,
> there aren't any that expose the WALKey before it's created and added to the
> WALEntry.
> It's sometimes useful for coprocessors to be able to edit the WALKey, for
> example to add extended attributes using the fields to be added in
> HBASE-22622.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[ https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886378#comment-16886378 ] Geoffrey Jacoby commented on HBASE-22623: - What I meant is that for ease of evaluation and to keep patches at a modest size, instrumentation for the two code paths could be done in separate JIRAs, a new one for Append + Increment and this one for Batch Mutate. Both would need to be done to get full coverage, and we'd want them in the same release. (Phoenix does use Append and Increment in a few cases, such as atomic upserts.) I can also combine them in this JIRA, which it sounds like you'd prefer and would also be fine. The signatures for the Append/Increment coprocs and the Batch mutation coprocs are different enough that I'm somewhat skeptical we could make a single hook that would work for both, and doing the batch mutation instrumentation through the MiniBatchOperationInProgress eliminates the need for new coprocessor hooks on that path, and limiting those to a minimum (such as the seemingly-unavoidable ones on Append and Increment) seems good to me. > Add RegionObserver coprocessor hook for preWALAppend > > > Key: HBASE-22623 > URL: https://issues.apache.org/jira/browse/HBASE-22623 > Project: HBase > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.3.0 > > > While many coprocessor hooks expose the WALEdit to implementing coprocs, > there aren't any that expose the WALKey before it's created and added to the > WALEntry. > It's sometimes useful for coprocessors to be able to edit the WALKey, for > example to add extended attributes using the fields to be added in > HBASE-22622. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (HBASE-22688) [HBCK2] Add filesystem fixup to hbck2
[ https://issues.apache.org/jira/browse/HBASE-22688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-22688. --- Resolution: Fixed Assignee: stack Hadoop Flags: Reviewed Fix Version/s: hbase-operator-tools-1.0.0 Release Note: Adds a 'filesystem' command to HBCK2. Checks for bad references, links, and corrupt hfiles with the option of sidelining if pass --fix option. Merged PR. Thanks for blessing [~busbey] > [HBCK2] Add filesystem fixup to hbck2 > - > > Key: HBASE-22688 > URL: https://issues.apache.org/jira/browse/HBASE-22688 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: hbase-operator-tools-1.0.0 > > Attachments: > 0001-HBASE-22688-HBCK2-Add-filesystem-fixup-to-hbck2.patch > > > Poking around, was going to add a 'filesystem' command to hbck2. It would > take nothing or a list of one or more table names and optionally a path to a > region. If nothing passed, it would review the fs for corrupt hfiles and bad > links reporting the bad as it went. If --fix is passed, it will sideline > corrupt files and mis-links. Would advise reopen/rolling-restart if change > made. For starters would make use of the hbck1 fs tools added by HBASE-22680. > Above would exploit the following facillty from hbck1. > -checkCorruptHFiles > -sidelineCorruptHFiles > -fixVersionFile > -fixReferenceFiles > -fixHFileLinks > This would address the following from parent issue at least: > * Fix hfile link problems... > * Enumerate store files to determine file level corruption... > I noticed that the 'read-only' hbck1 tool HBaseFsck which is present in the > hbase2 codebase actually allows fixes in the filesystem; i.e. you'd run > HBaseFsck and pass say -fixVersionFile. This is mixed in w/ all the other > hbck1 noise but an operator should be able to make progress with what is > there regards hfile corruption, bad links, etc. In other words, could just > doc this facility in the hbck2 tool point folks to the latent hbck1 HBaseFsck > if they need to do fixup of the fs (Would be confusing but it exists). -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [hbase-operator-tools] saintstack merged pull request #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2
saintstack merged pull request #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2 URL: https://github.com/apache/hbase-operator-tools/pull/10 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[
https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886361#comment-16886361
]
Andrew Purtell edited comment on HBASE-22623 at 7/16/19 6:20 PM:
-
{quote}They're mostly unrelated changes though, so it would either work to do
them together in this JIRA or separately.
{quote}
I don't follow that. My understanding of the intent here is to intercept
control of execution just prior to append of WAL entries to the log for
potential modification or decoration. If the solution misses some code paths
doesn't it fall short of the goal?
When I looked at this, I thought a new hook to be called prior to WAL#append at
all appropriate call sites would provide full coverage.
Wouldn't an application processing WALedits produced by Put, Delete, etc. also
need to process ones produced by Increment. Does Phoenix use Increments?
was (Author: apurtell):
{quote}They're mostly unrelated changes though, so it would either work to do
them together in this JIRA or separately.
{quote}
I don't follow that. My understanding of the intent here is to intercept
control of execution just prior to append of WAL entries to the log for
potential modification or decoration. If the solution misses some code paths
doesn't it fall short of the goal?
> Add RegionObserver coprocessor hook for preWALAppend
>
>
> Key: HBASE-22623
> URL: https://issues.apache.org/jira/browse/HBASE-22623
> Project: HBase
> Issue Type: New Feature
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
>
> While many coprocessor hooks expose the WALEdit to implementing coprocs,
> there aren't any that expose the WALKey before it's created and added to the
> WALEntry.
> It's sometimes useful for coprocessors to be able to edit the WALKey, for
> example to add extended attributes using the fields to be added in
> HBASE-22622.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[
https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886361#comment-16886361
]
Andrew Purtell commented on HBASE-22623:
{quote}They're mostly unrelated changes though, so it would either work to do
them together in this JIRA or separately.
{quote}
I don't follow that. My understanding of the intent here is to intercept
control of execution just prior to append of WAL entries to the log for
potential modification or decoration. If the solution misses some code paths
doesn't it fall short of the goal?
> Add RegionObserver coprocessor hook for preWALAppend
>
>
> Key: HBASE-22623
> URL: https://issues.apache.org/jira/browse/HBASE-22623
> Project: HBase
> Issue Type: New Feature
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
>
> While many coprocessor hooks expose the WALEdit to implementing coprocs,
> there aren't any that expose the WALKey before it's created and added to the
> WALEntry.
> It's sometimes useful for coprocessors to be able to edit the WALKey, for
> example to add extended attributes using the fields to be added in
> HBASE-22622.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[GitHub] [hbase-operator-tools] asf-ci commented on issue #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2
asf-ci commented on issue #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2 URL: https://github.com/apache/hbase-operator-tools/pull/10#issuecomment-511923563 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/36/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[ https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886347#comment-16886347 ] Geoffrey Jacoby edited comment on HBASE-22623 at 7/16/19 5:57 PM: -- I think Append and Increment use a different code path (see HRegion.doDelta) and so would have to be instrumented a different way, perhaps with extra coprocessor hooks, one for Increment and one for Append. (The existing Append and Increment pre-hooks don't take a convenient "carrier" object like the batch mutations do, and already use their return values for other purposes.) checkAndXXX seems to eventually go down the same batch mutation path as Puts and Deletes (see HRegion:4287) so the MiniBatchOperationInProgress solution above would work for them. They're mostly unrelated changes though, so it would either work to do them together in this JIRA or separately. was (Author: gjacoby): I think Append and Increment use a different code path (see HRegion.doDelta) and so would have to be instrumented a different way, perhaps with extra coprocessor hooks, one for Increment and one for Append. (The existing Append and Increment pre-hooks don't take a convenient "carrier" object like the batch mutations do, and already use their return values for other purposes.) checkAndXXX seems to eventually go down the same batch mutation path (see HRegion:4287) so the MiniBatchOperationInProgress solution above would work for them. They're mostly unrelated changes though, so it would either work to do them together in this JIRA or separately. > Add RegionObserver coprocessor hook for preWALAppend > > > Key: HBASE-22623 > URL: https://issues.apache.org/jira/browse/HBASE-22623 > Project: HBase > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.3.0 > > > While many coprocessor hooks expose the WALEdit to implementing coprocs, > there aren't any that expose the WALKey before it's created and added to the > WALEntry. > It's sometimes useful for coprocessors to be able to edit the WALKey, for > example to add extended attributes using the fields to be added in > HBASE-22622. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[ https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886347#comment-16886347 ] Geoffrey Jacoby edited comment on HBASE-22623 at 7/16/19 5:55 PM: -- I think Append and Increment use a different code path (see HRegion.doDelta) and so would have to be instrumented a different way, perhaps with extra coprocessor hooks, one for Increment and one for Append. (The existing Append and Increment pre-hooks don't take a convenient "carrier" object like the batch mutations do, and already use their return values for other purposes.) checkAndXXX seems to eventually go down the same batch mutation path (see HRegion:4287) so the MiniBatchOperationInProgress solution above would work for them. They're mostly unrelated changes though, so it would either work to do them together in this JIRA or separately. was (Author: gjacoby): I think Append and Increment use a different code path (see HRegion.doDelta) and so would have to be instrumented a different way, perhaps with extra coprocessor hooks, one for Increment and one for Append. (The existing Append and Increment pre-hooks don't take a convenient "carrier" object like the batch mutations do, and already use their return values for other purposes.) checkAndXXX seems to eventually go down the same batch mutation path (see HRegion:4287) so the MiniBatchOperationInProgress solution above would work for them. They're unrelated changes though, so it > Add RegionObserver coprocessor hook for preWALAppend > > > Key: HBASE-22623 > URL: https://issues.apache.org/jira/browse/HBASE-22623 > Project: HBase > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.3.0 > > > While many coprocessor hooks expose the WALEdit to implementing coprocs, > there aren't any that expose the WALKey before it's created and added to the > WALEntry. > It's sometimes useful for coprocessors to be able to edit the WALKey, for > example to add extended attributes using the fields to be added in > HBASE-22622. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[ https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886347#comment-16886347 ] Geoffrey Jacoby commented on HBASE-22623: - I think Append and Increment use a different code path (see HRegion.doDelta) and so would have to be instrumented a different way, perhaps with extra coprocessor hooks, one for Increment and one for Append. (The existing Append and Increment pre-hooks don't take a convenient "carrier" object like the batch mutations do, and already use their return values for other purposes.) checkAndXXX seems to eventually go down the same batch mutation path (see HRegion:4287) so the MiniBatchOperationInProgress solution above would work for them. They're unrelated changes though, so it > Add RegionObserver coprocessor hook for preWALAppend > > > Key: HBASE-22623 > URL: https://issues.apache.org/jira/browse/HBASE-22623 > Project: HBase > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.3.0 > > > While many coprocessor hooks expose the WALEdit to implementing coprocs, > there aren't any that expose the WALKey before it's created and added to the > WALEntry. > It's sometimes useful for coprocessors to be able to edit the WALKey, for > example to add extended attributes using the fields to be added in > HBASE-22622. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[GitHub] [hbase-operator-tools] asf-ci commented on issue #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2
asf-ci commented on issue #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2 URL: https://github.com/apache/hbase-operator-tools/pull/10#issuecomment-511919538 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/35/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] asf-ci commented on issue #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2
asf-ci commented on issue #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2 URL: https://github.com/apache/hbase-operator-tools/pull/10#issuecomment-511917887 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/34/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-21462) Add note for CopyTable section explained it does not perform a diff, but a full write from source to target
[ https://issues.apache.org/jira/browse/HBASE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-21462: - Affects Version/s: 3.0.0 2.1.5 > Add note for CopyTable section explained it does not perform a diff, but a > full write from source to target > --- > > Key: HBASE-21462 > URL: https://issues.apache.org/jira/browse/HBASE-21462 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0, 2.1.5 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21462-master.002.patch, > HBASE-21462.master.001.patch > > > Performance is a common issue with CopyTable is when the key/time range for > the data to be copied is large because it basically scans all rows on the > specified range, in the source and perform related puts on the source. We > should add extra note explaining that on the reference guide, to help > users/admins understand when to choose between the different tools/approaches > for syncing clusters. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HBASE-21462) Add note for CopyTable section explained it does not perform a diff, but a full write from source to target
[ https://issues.apache.org/jira/browse/HBASE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-21462: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Add note for CopyTable section explained it does not perform a diff, but a > full write from source to target > --- > > Key: HBASE-21462 > URL: https://issues.apache.org/jira/browse/HBASE-21462 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21462-master.002.patch, > HBASE-21462.master.001.patch > > > Performance is a common issue with CopyTable is when the key/time range for > the data to be copied is large because it basically scans all rows on the > specified range, in the source and perform related puts on the source. We > should add extra note explaining that on the reference guide, to help > users/admins understand when to choose between the different tools/approaches > for syncing clusters. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HBASE-22623) Add RegionObserver coprocessor hook for preWALAppend
[
https://issues.apache.org/jira/browse/HBASE-22623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886322#comment-16886322
]
Andrew Purtell commented on HBASE-22623:
{quote} MiniBatchOperationInProgress object that gets created at the start of a
batch Put or Delete operation (doMiniBatchMutate), and which gets passed to
each subsequent coprocessor in the batch mutation pipeline. Both the main
pipeline and the coprocs would be able to read or write WALKey attributes to
the MiniBatchOperationInProgress, and all that has to change in the main
pipeline is an additional optional parameter on doWALAppend (which is private)
to instrument the WALKeyImpls it creates.
{quote}
Does this cover all cases where a WAL entry might be appended to the WAL?
Thinking of append, increment, and checkAndXXX operations.
> Add RegionObserver coprocessor hook for preWALAppend
>
>
> Key: HBASE-22623
> URL: https://issues.apache.org/jira/browse/HBASE-22623
> Project: HBase
> Issue Type: New Feature
>Reporter: Geoffrey Jacoby
>Assignee: Geoffrey Jacoby
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
>
> While many coprocessor hooks expose the WALEdit to implementing coprocs,
> there aren't any that expose the WALKey before it's created and added to the
> WALEntry.
> It's sometimes useful for coprocessors to be able to edit the WALKey, for
> example to add extended attributes using the fields to be added in
> HBASE-22622.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[GitHub] [hbase-operator-tools] asf-ci commented on issue #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2
asf-ci commented on issue #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2 URL: https://github.com/apache/hbase-operator-tools/pull/10#issuecomment-511910152 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/33/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] saintstack opened a new pull request #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2
saintstack opened a new pull request #10: HBASE-22688 [HBCK2] Add filesystem fixup to hbck2 URL: https://github.com/apache/hbase-operator-tools/pull/10 Adds a general filesystem check command to the HBCK2 list. Runs just the offline filesystem checks from the old hbck1 HBaseFsck tool. Checks hfile validity, if references and hfile links are wholesome, and whether the hbase.version file is present. Pass '--fix' to do fixup. Depends on hbck1 HBaseFsck being present in hbck2 (HBASE-22680 adds it so this could go in after it) Does NOT do regiondir or tabledir fixup or fixing of orphans in hdfs or plugging holes in hdfs. A hbase-hbck2/src/main/java/org/apache/hbase/FileSystemFsck.java Adds a file to handle filesystem fsck checking and fixing. M hbase-hbck2/src/main/java/org/apache/hbase/hbck1/HFileCorruptionChecker.java Formatting fixes and filling out the MOB report so like the hfile report. Signed-off-by: Sean Busbey This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
