[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615568#comment-16615568
]
Reid Chan commented on HBASE-20993:
---
ping sir [~apurtell], have free cycles to take a look?
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
> Fix For: 1.5.0, 1.4.8
>
> Attachments: HBASE-20993.001.patch,
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch,
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch,
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch,
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch,
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.009.patch,
> HBASE-20993.branch-1.2.001.patch, HBASE-20993.branch-1.wip.002.patch,
> HBASE-20993.branch-1.wip.patch, yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:607)
> at
>
[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[
https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615557#comment-16615557
]
Hadoop QA commented on HBASE-21191:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
15s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
33s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
29s{color} | {color:red} hbase-server: The patch generated 4 new + 366
unchanged - 0 fixed = 370 total (was 366) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
8s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 52s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
48s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 52s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}177m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-21191 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939808/HBASE-21191.branch-2.1.005.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 5c02e71bf859 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-21023) Add bypassProcedureToCompletion() API to HbckService
[
https://issues.apache.org/jira/browse/HBASE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615549#comment-16615549
]
Hadoop QA commented on HBASE-21023:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
13s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 18s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
20s{color} | {color:red} hbase-client generated 1 new + 2 unchanged - 0 fixed =
3 total (was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
0s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}136m
43s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
5s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}195m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21023 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939797/hbase-21023.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti
[jira] [Commented] (HBASE-21182) Failed to execute start-hbase.sh
[
https://issues.apache.org/jira/browse/HBASE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615519#comment-16615519
]
Hudson commented on HBASE-21182:
Results for branch branch-2.1
[build #325 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/325/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/325//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/325//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/325//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Failed to execute start-hbase.sh
>
>
> Key: HBASE-21182
> URL: https://issues.apache.org/jira/browse/HBASE-21182
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0, 2.1.1
>Reporter: Subrat Mishra
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21182.master.001.patch,
> HBASE-21182.master.002.patch
>
>
> Built master branch like below:
> {code:java}
> mvn clean install -DskipTests{code}
> Then tried to execute start-hbase.sh failed with NoClassDefFoundError
> {code:java}
> ./bin/start-hbase.sh
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/org/eclipse/jetty/server/Connector
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.org.eclipse.jetty.server.Connector{code}
> Note: It worked after reverting HBASE-21153
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615517#comment-16615517
]
Hadoop QA commented on HBASE-21196:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
20s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
16s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 24s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
16s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}218m
7s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}278m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939771/HBASE-21196.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a0cea86c32d4 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 38a697ad1e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs |
[jira] [Commented] (HBASE-21182) Failed to execute start-hbase.sh
[
https://issues.apache.org/jira/browse/HBASE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615515#comment-16615515
]
Hudson commented on HBASE-21182:
Results for branch branch-2
[build #1249 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1249/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1249//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1249//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1249//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Failed to execute start-hbase.sh
>
>
> Key: HBASE-21182
> URL: https://issues.apache.org/jira/browse/HBASE-21182
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0, 2.1.1
>Reporter: Subrat Mishra
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21182.master.001.patch,
> HBASE-21182.master.002.patch
>
>
> Built master branch like below:
> {code:java}
> mvn clean install -DskipTests{code}
> Then tried to execute start-hbase.sh failed with NoClassDefFoundError
> {code:java}
> ./bin/start-hbase.sh
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/org/eclipse/jetty/server/Connector
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.org.eclipse.jetty.server.Connector{code}
> Note: It worked after reverting HBASE-21153
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615489#comment-16615489 ] stack commented on HBASE-21191: --- .005 addresses feedback from [~Apache9] up in rb. > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch, > HBASE-21191.branch-2.1.002.patch, HBASE-21191.branch-2.1.003.patch, > HBASE-21191.branch-2.1.004.patch, HBASE-21191.branch-2.1.005.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21191: -- Attachment: HBASE-21191.branch-2.1.005.patch > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21191.branch-2.1.001.patch, > HBASE-21191.branch-2.1.002.patch, HBASE-21191.branch-2.1.003.patch, > HBASE-21191.branch-2.1.004.patch, HBASE-21191.branch-2.1.005.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-20786 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21023) Add bypassProcedureToCompletion() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615446#comment-16615446 ] Umesh Agashe commented on HBASE-21023: -- Attached patch 002 with changes as per review comments. > Add bypassProcedureToCompletion() API to HbckService > > > Key: HBASE-21023 > URL: https://issues.apache.org/jira/browse/HBASE-21023 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Affects Versions: 2.0.1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Fix For: 2.2.0 > > Attachments: hbase-21023.master.001.patch, > hbase-21023.master.001.patch, hbase-21023.master.002.patch > > > completeProcedure/s(): some procedures do not support abort at every step. > When these procedures get stuck then they can not be aborted or make further > progress. Corrective action is to bypass these procedures to completion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21023) Add bypassProcedureToCompletion() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-21023: - Attachment: hbase-21023.master.002.patch > Add bypassProcedureToCompletion() API to HbckService > > > Key: HBASE-21023 > URL: https://issues.apache.org/jira/browse/HBASE-21023 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Affects Versions: 2.0.1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Fix For: 2.2.0 > > Attachments: hbase-21023.master.001.patch, > hbase-21023.master.001.patch, hbase-21023.master.002.patch > > > completeProcedure/s(): some procedures do not support abort at every step. > When these procedures get stuck then they can not be aborted or make further > progress. Corrective action is to bypass these procedures to completion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21169) Initiate hbck2 tool in hbase-operator-tools repo
[ https://issues.apache.org/jira/browse/HBASE-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-21169: - Attachment: hbase-21169.master.001.patch > Initiate hbck2 tool in hbase-operator-tools repo > > > Key: HBASE-21169 > URL: https://issues.apache.org/jira/browse/HBASE-21169 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Affects Versions: 2.1.0 >Reporter: Umesh Agashe >Assignee: stack >Priority: Major > Attachments: hbase-21169.master.001.patch > > > Create hbck2 tool in hbase-operator-tools > (https://github.com/apache/hbase-operator-tools.git) repo. This is not > intended to be complete tool but initial changes with usage, ability to > connect to server, logging, and using newly added HbckService etc. Code > changes to address specific use cases can be added later and tool will evolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615332#comment-16615332
]
Josh Elser commented on HBASE-20952:
{quote}Possible the replication itself requires a new Jira 'Revisit Replication
approach' especially if we are talking about different kind of wal providers
{quote}
{quote}And that would require another layer of abstraction for replication
itself. Should it be done under this jira? Honestly speaking I don't think so.
{quote}
When it comes to implementation, I think that's a good point. Playing with an
implementation can be helpful, but (largely to what I think Duo is worried
about) building an API based on such hacking never gives the correct product.
That said, I think we must include all systems in the design doc up-front. I,
at least, learned something new about sync replication which is a good example
of why this process is beneficial.
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615326#comment-16615326
]
Sergey Soldatov commented on HBASE-20952:
-
A few of my personal thoughts that are based on previous experience of
implementing such kind of functionality in HBase (C5, non-stop hbase,
hydrabase). It would be better to perform atomic meaningful changes that
address a particular problem without breaking the existing functionality and
without adding an additional complexity. We need a 'pluggable' WALProvider
that will be agnostic to file system structure. Fine. Let's make it
configurable and remove the dependency on file systems (introducing WALIdentity
or whatever we call it). It doesn't work with the current implementation of
replication? Not good, but we may mention that it in the Limitation section.
All we need is an understanding that it's possible to implement even if it
would require a lot of efforts. Possible the replication itself requires a new
Jira 'Revisit Replication approach' especially if we are talking about
different kind of wal providers. Possible someone would love to have a setup
with replication between on-prem cluster that is using regular WAL and another
in AWS using Bookeeper wal. And that would require another layer of abstraction
for replication itself. Should it be done under this jira? Honestly speaking I
don't think so.
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21164) reportForDuty to spew less log if master is initializing
[
https://issues.apache.org/jira/browse/HBASE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615316#comment-16615316
]
Hadoop QA commented on HBASE-21164:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
44s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
48s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 20s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}143m
37s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}191m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21164 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939754/HBASE-21164.009.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 14dbcc1e9836 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 38a697ad1e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14420/testReport/ |
| Max. process+thread count | 5367 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14420/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> reportForDuty to spew less log if
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615267#comment-16615267
]
Josh Elser commented on HBASE-20952:
{quote}The current implementation has a bunch of dependencies on file
operations such as temporary files, list of corrupted files, etc. From HBase
perspective, it would be much easier to keep it as is and make log splitter an
interface that should take log and create list of recovery logs.
{quote}
{quote}In other hands for the 1st iteration, this may be hidden by the schema
where 3rd party wal may not use the splitter at all and recovery would be
reading a stream of records provided by the WALProvider for a particular region
{quote}
Thanks for your assessment, Sergey. Your take makes sense to me (e.g. let WAL
implementation decide if it needs splitting). This concern and assessment is
something that should be captured in the design doc, [~yuzhih...@gmail.com].
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615265#comment-16615265
]
Sergey Soldatov commented on HBASE-20952:
-
{quote}In the old time we will roll the wal writer, and it is done by RS, so
closing the wal file is not enough, as the RS will try to open a new one and
write to it. That's why we need to rename the wal directory.
{quote}
For the WAL provider that doesn't depend on the HDFS directory structure, there
should be a manager that keeps information about existing logs. Internals are
implementation specific (i.e. for Kafka wal provider it may be a separate topic
or some internal DB. For consensus-based logs like Ratis LogService that might
be a separate state machine), But any new log should be registered there.
Adding a new method to WALProvider like 'disable'/'decommission' that would
tell the manager to reject new logs for particular RS (or even region if we
consider wal per region schema) is not a problem. For the existing wal
providers, that method may rename the wal directory.
{quote}In your words above, it seems to me that we will only have one stream
opened forever for a RS, then how do we drop the old edits after flush? And how
do we setup the wal stream? Only once at the RS start up? And if there are
errors later, we just abort?
{quote}
Not necessary. There is no problem to have wal per region. Actually, in some
cases, it would be preferable. For example Kafka topic per region. Any kind of
recovery would be a simple subscribe/replay the particular topic. No log
splits, less offline time. For a regular case, we are not talking about
streams. It's just a WAL implementation that supports the append operation. For
replication/recovery we should be able to get a stream and read from the
particular ID/offset. Error handling should be hidden by the implementation. A
simple example for quorum based implementation. We have 3 nodes quorum for log
'RS1.1' (RS1, RS2,RS3). RS2, RS3 went down due some reason, so we lost the
majority and this quorum becomes read-only. A new log 'RS1.2' is created with
the quorum (RS1, RS4, RS5) and all writes are going there. But if we speak
about reading stream it would provide a single instance that iterates through
RS1.1 and RS1.2 continuously. The same approach may be applied to the existing
wal files as well.
{quote}And for the FileSystem, we will use multi wal to increase the
performance, and the logic is messed up with WALProvider. Does ratis still need
multi wal to increase the performance? And if not, what's the plan? We need to
refactor the multi wal related code, to not work against the WALProvider but
something with the FileSystem related stuffs directly?
{quote}
That might be done in the further refactoring of multiwal. At the moment the
approach is that we may specify 3rd party wal provider class in WALFactory. So
if it's there, multiwal would not be used at all as it's the provider class. In
other hands, it could be refactored to something like 'wal strategy' and works
with any kind of providers.
{quote}had mentioned offline yesterday that he thinks some gaps still exist
around WAL splitting – do you understand that well enough to suggest what needs
to be addressed in the doc which is not already there?
{quote}
WALSplitter is a separate topic for the discussion. The current implementation
has a bunch of dependencies on file operations such as temporary files, list of
corrupted files, etc. From HBase perspective, it would be much easier to keep
it as is and make log splitter an interface that should take log and create
list of recovery logs. But from the perspective of 3rd party wal developer that
would be a nightmare to handle all possible cases and fit into the split log
chore logic. In other hands for the 1st iteration, this may be hidden by the
schema where 3rd party wal may not use the splitter at all and recovery would
be reading a stream of records provided by the WALProvider for a particular
region
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615255#comment-16615255
]
Nihal Jain commented on HBASE-21196:
Submitted [^HBASE-21196.master.002.patch]. Please review..
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HBASE-21196.master.002.patch,
> HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
> up meta region location in ZK" are present for every put.
> *Analysis:*
> The problem occurs as we call the {{cleanServerCache}} method always clears
> the server cache in case tablename is null and exception is null. See
> [AsyncRequestFutureImpl.java#L918|https://github.com/apache/hbase/blob/5d14c1af65c02f4e87059337c35e4431505de91c/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L918]
> {code:java}
> private void cleanServerCache(ServerName server, Throwable regionException) {
> if (tableName == null &&
> ClientExceptionsUtil.isMetaClearingException(regionException)) {
> // For multi-actions, we don't have a table name, but we want to make
> sure to clear the
> // cache in case there were location-related exceptions. We don't to
> clear the cache
> // for every possible exception
[jira] [Updated] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-21196:
---
Attachment: HBASE-21196.master.002.patch
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HBASE-21196.master.002.patch,
> HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
> up meta region location in ZK" are present for every put.
> *Analysis:*
> The problem occurs as we call the {{cleanServerCache}} method always clears
> the server cache in case tablename is null and exception is null. See
> [AsyncRequestFutureImpl.java#L918|https://github.com/apache/hbase/blob/5d14c1af65c02f4e87059337c35e4431505de91c/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L918]
> {code:java}
> private void cleanServerCache(ServerName server, Throwable regionException) {
> if (tableName == null &&
> ClientExceptionsUtil.isMetaClearingException(regionException)) {
> // For multi-actions, we don't have a table name, but we want to make
> sure to clear the
> // cache in case there were location-related exceptions. We don't to
> clear the cache
> // for every possible exception that comes through, however.
>
[jira] [Commented] (HBASE-21169) Initiate hbck2 tool in hbase-operator-tools repo
[ https://issues.apache.org/jira/browse/HBASE-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615245#comment-16615245 ] Umesh Agashe commented on HBASE-21169: -- "working' is vague in the comment above. To be clear, I have some code changes. I am not working on doc. Thanks for the doc link [~stack]! > Initiate hbck2 tool in hbase-operator-tools repo > > > Key: HBASE-21169 > URL: https://issues.apache.org/jira/browse/HBASE-21169 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Affects Versions: 2.1.0 >Reporter: Umesh Agashe >Assignee: stack >Priority: Major > > Create hbck2 tool in hbase-operator-tools > (https://github.com/apache/hbase-operator-tools.git) repo. This is not > intended to be complete tool but initial changes with usage, ability to > connect to server, logging, and using newly added HbckService etc. Code > changes to address specific use cases can be added later and tool will evolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21185) WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes
[
https://issues.apache.org/jira/browse/HBASE-21185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615214#comment-16615214
]
Hadoop QA commented on HBASE-21185:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
10s{color} | {color:red} hbase-server: The patch generated 3 new + 17 unchanged
- 0 fixed = 20 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
8s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 29s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}132m 7s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}173m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.procedure.TestDisableTableProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21185 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939546/HBASE-21185.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 03a6fc24e8e2 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 38a697ad1e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615209#comment-16615209
]
Nihal Jain commented on HBASE-21196:
{quote} Is seems the new test class can be medium, judging from the duration it
took.
{quote}
Yes, it will take a few seconds. Will update the test class category in the
next patch. Thank you for the review [~yuzhih...@gmail.com].
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
> up meta region location in ZK" are present for every put.
> *Analysis:*
> The problem occurs as we call the {{cleanServerCache}} method always clears
> the server cache in case tablename is null and exception is null. See
> [AsyncRequestFutureImpl.java#L918|https://github.com/apache/hbase/blob/5d14c1af65c02f4e87059337c35e4431505de91c/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L918]
> {code:java}
> private void cleanServerCache(ServerName server, Throwable regionException) {
> if (tableName == null &&
> ClientExceptionsUtil.isMetaClearingException(regionException)) {
> // For multi-actions, we don't have a table name, but we want to make
> sure to clear
[jira] [Commented] (HBASE-21135) Build fails on windows as it fails to parse windows path during license check
[
https://issues.apache.org/jira/browse/HBASE-21135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615205#comment-16615205
]
Nihal Jain commented on HBASE-21135:
{quote}But we should have a DISCUSS thread on dev@ about the future of building
on windows.
{quote}
That would be great. I think we should at least be able to compile hbase
correctly on windows. I use windows for development at my workplace; I need to
compile the code before pushing any changes to my work-repo. Sometimes, I also
need to run unit tests via eclipse. I believe many other developers might be
using windows as their development environment. Although I doubt very few would
be using windows as test environment. Ensuring that hbase can be compiled on
windows would save a lot of trouble windows developers may face initially while
setting up hbase for the first time. In fact today I found we cannot compile on
windows with "{{-Prelease"}} profile. I was thinking about fixing it and
submitting a patch for the same; given that hbase community is eager to support
compilation on windows.
> Build fails on windows as it fails to parse windows path during license check
> -
>
> Key: HBASE-21135
> URL: https://issues.apache.org/jira/browse/HBASE-21135
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0, 2.1.1
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21135.master.001.patch
>
>
> License check via enforce plugin throws following error during build on
> windows:
> {code:java}
> Sourced file: inline evaluation of: ``File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-ar . . . '' Token
> Parsing Error: Lexical error at line 1, column 29. Encountered: "D" (68),
> after : "\"D:\\": {code}
> Complete stacktrace with command
> {code:java}
> mvn clean install -DskipTests -X
> {code}
> is as follows:
> {noformat}
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (check-aggregate-license) @
> hbase-shaded ---
> [DEBUG] Configuring mojo
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce from plugin
> realm
> ClassRealm[plugin>org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1,
> parent: sun.misc.Launcher$AppClassLoader@55f96302]
> [DEBUG] Configuring mojo
> 'org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce' with basic
> configurator -->
> [DEBUG] (s) fail = true
> [DEBUG] (s) failFast = false
> [DEBUG] (f) ignoreCache = false
> [DEBUG] (f) mojoExecution =
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce {execution:
> check-aggregate-license}
> [DEBUG] (s) project = MavenProject:
> org.apache.hbase:hbase-shaded:2.1.1-SNAPSHOT @
> D:\DS\HBase_2\hbase\hbase-shaded\pom.xml
> [DEBUG] (s) condition = File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE");
> // Beanshell does not support try-with-resources,
> // so we must close this scanner manually
> Scanner scanner = new Scanner(license);
> while (scanner.hasNextLine()) {
> if (scanner.nextLine().startsWith("ERROR:")) {
> scanner.close();
> return false;
> }
> }
> scanner.close();
> return true;
> [DEBUG] (s) message = License errors detected, for more detail find ERROR in
>
> D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE
> [DEBUG] (s) rules =
> [org.apache.maven.plugins.enforcer.EvaluateBeanshell@7e307087]
> [DEBUG] (s) session = org.apache.maven.execution.MavenSession@5e1218b4
> [DEBUG] (s) skip = false
> [DEBUG] -- end configuration --
> [DEBUG] Executing rule: org.apache.maven.plugins.enforcer.EvaluateBeanshell
> [DEBUG] Echo condition : File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE");
> // Beanshell does not support try-with-resources,
> // so we must close this scanner manually
> Scanner scanner = new Scanner(license);
> while (scanner.hasNextLine()) {
> if (scanner.nextLine().startsWith("ERROR:")) {
> scanner.close();
> return false;
> }
> }
> scanner.close();
> return true;
> [DEBUG] Echo script : File license = new
>
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615189#comment-16615189
]
Ted Yu commented on HBASE-21196:
You can keep them.
https://builds.apache.org/job/PreCommit-HBASE-Build/14415/testReport/org.apache.hadoop.hbase.client/TestRegionLocationCaching/
Is seems the new test class can be medium, judging from the duration it took.
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
> up meta region location in ZK" are present for every put.
> *Analysis:*
> The problem occurs as we call the {{cleanServerCache}} method always clears
> the server cache in case tablename is null and exception is null. See
> [AsyncRequestFutureImpl.java#L918|https://github.com/apache/hbase/blob/5d14c1af65c02f4e87059337c35e4431505de91c/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L918]
> {code:java}
> private void cleanServerCache(ServerName server, Throwable regionException) {
> if (tableName == null &&
> ClientExceptionsUtil.isMetaClearingException(regionException)) {
> // For multi-actions, we don't have a table name, but we want to make
> sure to clear the
> //
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615180#comment-16615180
]
Nihal Jain commented on HBASE-21196:
bq. Please fix the assertion ('it is0').
Sure will remove the above string from the message as the actual value is
anyways logged in case of failure
bq. Since the other two tests don't fail without fix, I think you may take them
out.
I added the other 2 tests deliberately for greater coverage. I think it's good
to have tests which can ensure we don't break anything like this in future.
Should I keep them?
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
> up meta region location in ZK" are present for every put.
> *Analysis:*
> The problem occurs as we call the {{cleanServerCache}} method always clears
> the server cache in case tablename is null and exception is null. See
> [AsyncRequestFutureImpl.java#L918|https://github.com/apache/hbase/blob/5d14c1af65c02f4e87059337c35e4431505de91c/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L918]
> {code:java}
> private void cleanServerCache(ServerName server, Throwable regionException) {
> if
[jira] [Commented] (HBASE-21169) Initiate hbck2 tool in hbase-operator-tools repo
[ https://issues.apache.org/jira/browse/HBASE-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615158#comment-16615158 ] Umesh Agashe commented on HBASE-21169: -- [~stack], I was working on it and thought I assigned it to myself to indicate the same. Lets talk about it. > Initiate hbck2 tool in hbase-operator-tools repo > > > Key: HBASE-21169 > URL: https://issues.apache.org/jira/browse/HBASE-21169 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Affects Versions: 2.1.0 >Reporter: Umesh Agashe >Assignee: stack >Priority: Major > > Create hbck2 tool in hbase-operator-tools > (https://github.com/apache/hbase-operator-tools.git) repo. This is not > intended to be complete tool but initial changes with usage, ability to > connect to server, logging, and using newly added HbckService etc. Code > changes to address specific use cases can be added later and tool will evolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21035) Meta Table should be able to online even if all procedures are lost
[ https://issues.apache.org/jira/browse/HBASE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615142#comment-16615142 ] stack commented on HBASE-21035: --- [~Apache9] Yes. Github. Am taking the opportunity to do some experiments. Lets play with githhub'ing it. HBASE-21169 is the covering issue. I'll fill in some outline in attached doc for the hbck2 'form'. Would love commentary/push-back. There is little there at the moment. > Meta Table should be able to online even if all procedures are lost > --- > > Key: HBASE-21035 > URL: https://issues.apache.org/jira/browse/HBASE-21035 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-21035.branch-2.0.001.patch, > HBASE-21035.branch-2.1.001.patch > > > After HBASE-20708, we changed the way we init after master starts. It will > only check WAL dirs and compare to Zookeeper RS nodes to decide which server > need to expire. For servers which's dir is ending with 'SPLITTING', we assure > that there will be a SCP for it. > But, if the server with the meta region crashed before master restarts, and > if all the procedure wals are lost (due to bug, or deleted manually, > whatever), the new restarted master will be stuck when initing. Since no one > will bring meta region online. > Although it is an anomaly case, but I think no matter what happens, we need > to online meta region. Otherwise, we are sitting ducks, noting can be done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21169) Initiate hbck2 tool in hbase-operator-tools repo
[ https://issues.apache.org/jira/browse/HBASE-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615128#comment-16615128 ] stack commented on HBASE-21169: --- Link to HBCK2 Design. Commentary please. > Initiate hbck2 tool in hbase-operator-tools repo > > > Key: HBASE-21169 > URL: https://issues.apache.org/jira/browse/HBASE-21169 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Affects Versions: 2.1.0 >Reporter: Umesh Agashe >Assignee: stack >Priority: Major > > Create hbck2 tool in hbase-operator-tools > (https://github.com/apache/hbase-operator-tools.git) repo. This is not > intended to be complete tool but initial changes with usage, ability to > connect to server, logging, and using newly added HbckService etc. Code > changes to address specific use cases can be added later and tool will evolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21169) Initiate hbck2 tool in hbase-operator-tools repo
[ https://issues.apache.org/jira/browse/HBASE-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615125#comment-16615125 ] stack commented on HBASE-21169: --- I took this since I've been at it for the last day or so [~uagashe]. Let me start up a doc for discussion of what the tool looks like. Will attach here. > Initiate hbck2 tool in hbase-operator-tools repo > > > Key: HBASE-21169 > URL: https://issues.apache.org/jira/browse/HBASE-21169 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Affects Versions: 2.1.0 >Reporter: Umesh Agashe >Assignee: stack >Priority: Major > > Create hbck2 tool in hbase-operator-tools > (https://github.com/apache/hbase-operator-tools.git) repo. This is not > intended to be complete tool but initial changes with usage, ability to > connect to server, logging, and using newly added HbckService etc. Code > changes to address specific use cases can be added later and tool will evolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21169) Initiate hbck2 tool in hbase-operator-tools repo
[ https://issues.apache.org/jira/browse/HBASE-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-21169: - Assignee: stack (was: Umesh Agashe) > Initiate hbck2 tool in hbase-operator-tools repo > > > Key: HBASE-21169 > URL: https://issues.apache.org/jira/browse/HBASE-21169 > Project: HBase > Issue Type: Sub-task > Components: hbck2 >Affects Versions: 2.1.0 >Reporter: Umesh Agashe >Assignee: stack >Priority: Major > > Create hbck2 tool in hbase-operator-tools > (https://github.com/apache/hbase-operator-tools.git) repo. This is not > intended to be complete tool but initial changes with usage, ability to > connect to server, logging, and using newly added HbckService etc. Code > changes to address specific use cases can be added later and tool will evolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615075#comment-16615075
]
Reid Chan commented on HBASE-20993:
---
Thanks for the info! It sounds like not the problem caused by patch and i can
commit it.
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
> Fix For: 1.5.0, 1.4.8
>
> Attachments: HBASE-20993.001.patch,
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch,
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch,
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch,
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch,
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.009.patch,
> HBASE-20993.branch-1.2.001.patch, HBASE-20993.branch-1.wip.002.patch,
> HBASE-20993.branch-1.wip.patch, yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:607)
> at
>
[jira] [Updated] (HBASE-21164) reportForDuty to spew less log if master is initializing
[
https://issues.apache.org/jira/browse/HBASE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HBASE-21164:
--
Attachment: HBASE-21164.009.patch
> reportForDuty to spew less log if master is initializing
>
>
> Key: HBASE-21164
> URL: https://issues.apache.org/jira/browse/HBASE-21164
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: stack
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HBASE-21164.005.patch, HBASE-21164.006.patch,
> HBASE-21164.007.patch, HBASE-21164.008.patch, HBASE-21164.009.patch,
> HBASE-21164.branch-2.1.001.patch, HBASE-21164.branch-2.1.002.patch,
> HBASE-21164.branch-2.1.003.patch, HBASE-21164.branch-2.1.004.patch
>
>
> RegionServers do reportForDuty on startup to tell Master they are available.
> If Master is initializing, and especially on a big cluster when it can take a
> while particularly if something is amiss, the log every three seconds is
> annoying and doesn't do anything of use. We should spew less those logs. Here
> is example:
> {code:java}
> 2018-09-06 14:01:39,312 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: reportForDuty to
> master=vc0207.halxg.cloudera.com,22001,1536266763109 with port=22001,
> startcode=1536266763109
> 2018-09-06 14:01:39,312 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: reportForDuty failed;
> sleeping and then retrying.
>
> {code}
> For example, I am looking at a large cluster now that had a backlog of
> procedure WALs. It is taking a couple of hours recreating the procedure-state
> because there are millions of procedures outstanding. Meantime, the Master
> log is just full of the above message – every three seconds...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615046#comment-16615046
]
Sean Busbey commented on HBASE-20993:
-
FYI, I filed YETUS-693 to try to improve the failing of the xml plugin to be
more clear that it isn't necessarily a problem with the file.
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
> Fix For: 1.5.0, 1.4.8
>
> Attachments: HBASE-20993.001.patch,
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch,
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch,
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch,
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch,
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.009.patch,
> HBASE-20993.branch-1.2.001.patch, HBASE-20993.branch-1.wip.002.patch,
> HBASE-20993.branch-1.wip.patch, yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
>
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615020#comment-16615020
]
Sean Busbey commented on HBASE-20993:
-
oh neat. I just hit this error somewhere else, so yes I do! :)
Yetus' xml checker relies on Java's scripting engine to run javascript to parse
the file as a valid XML document. Some JVMs don't have a script engine for
javascript, specifically I think the Azul JDK7 we rely on as default for
branch-1 doesn't.
When I ran into this I used xmllint to manually check the file I had changed.
For this case it looks fine also:
{code}
Busbey-MBA:hbase busbey$ git checkout branch-1
Checking out files: 100% (5165/5165), done.
Switched to branch 'branch-1'
Your branch is up to date with 'origin/branch-1'.
Busbey-MBA:hbase busbey$ smart-apply-patch --committer --project=hbase
HBASE-20993
Processing: HBASE-20993
HBASE-20993 patch is being downloaded at Fri Sep 14 11:05:00 CDT 2018 from
https://issues.apache.org/jira/secure/attachment/12939639/HBASE-20993.branch-1.009.patch
-> Downloaded
Applying the patch:
Fri Sep 14 11:05:01 CDT 2018
cd /Users/busbey/tmp_projects/hbase
git am --signoff --whitespace=fix -p1 /tmp/yetus-15038.23489/patch
Applying: HBASE-20993. [Auth] IPC client fallback to simple auth allowed
doesn't work
Busbey-MBA:hbase busbey$ xmllint --dtdvalid
'http://www.puppycrawl.com/dtds/suppressions_1_0.dtd' --noout
hbase-checkstyle/src/main/resources/hbase/checkstyle-suppressions.xml
Busbey-MBA:hbase busbey$ echo $?
0
{code}
I'm not sure what the general solution is here for branch-1.
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
> Fix For: 1.5.0, 1.4.8
>
> Attachments: HBASE-20993.001.patch,
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch,
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch,
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch,
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch,
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.009.patch,
> HBASE-20993.branch-1.2.001.patch, HBASE-20993.branch-1.wip.002.patch,
> HBASE-20993.branch-1.wip.patch, yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
>
[jira] [Commented] (HBASE-21135) Build fails on windows as it fails to parse windows path during license check
[
https://issues.apache.org/jira/browse/HBASE-21135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615009#comment-16615009
]
Sean Busbey commented on HBASE-21135:
-
No, I do not have cygwin since the ref guide didn't mention anything about it.
I'm +1 on including this to help out short term. But we should have a DISCUSS
thread on dev@ about the future of building on windows. There's too many gaps
right now e.g. nothing in the ref guide about set up and no nightly tests that
building on windows works. If we're not going to make those efforts we should
expressly state that it isn't expected to work.
> Build fails on windows as it fails to parse windows path during license check
> -
>
> Key: HBASE-21135
> URL: https://issues.apache.org/jira/browse/HBASE-21135
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0, 2.1.1
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21135.master.001.patch
>
>
> License check via enforce plugin throws following error during build on
> windows:
> {code:java}
> Sourced file: inline evaluation of: ``File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-ar . . . '' Token
> Parsing Error: Lexical error at line 1, column 29. Encountered: "D" (68),
> after : "\"D:\\": {code}
> Complete stacktrace with command
> {code:java}
> mvn clean install -DskipTests -X
> {code}
> is as follows:
> {noformat}
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (check-aggregate-license) @
> hbase-shaded ---
> [DEBUG] Configuring mojo
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce from plugin
> realm
> ClassRealm[plugin>org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1,
> parent: sun.misc.Launcher$AppClassLoader@55f96302]
> [DEBUG] Configuring mojo
> 'org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce' with basic
> configurator -->
> [DEBUG] (s) fail = true
> [DEBUG] (s) failFast = false
> [DEBUG] (f) ignoreCache = false
> [DEBUG] (f) mojoExecution =
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce {execution:
> check-aggregate-license}
> [DEBUG] (s) project = MavenProject:
> org.apache.hbase:hbase-shaded:2.1.1-SNAPSHOT @
> D:\DS\HBase_2\hbase\hbase-shaded\pom.xml
> [DEBUG] (s) condition = File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE");
> // Beanshell does not support try-with-resources,
> // so we must close this scanner manually
> Scanner scanner = new Scanner(license);
> while (scanner.hasNextLine()) {
> if (scanner.nextLine().startsWith("ERROR:")) {
> scanner.close();
> return false;
> }
> }
> scanner.close();
> return true;
> [DEBUG] (s) message = License errors detected, for more detail find ERROR in
>
> D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE
> [DEBUG] (s) rules =
> [org.apache.maven.plugins.enforcer.EvaluateBeanshell@7e307087]
> [DEBUG] (s) session = org.apache.maven.execution.MavenSession@5e1218b4
> [DEBUG] (s) skip = false
> [DEBUG] -- end configuration --
> [DEBUG] Executing rule: org.apache.maven.plugins.enforcer.EvaluateBeanshell
> [DEBUG] Echo condition : File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE");
> // Beanshell does not support try-with-resources,
> // so we must close this scanner manually
> Scanner scanner = new Scanner(license);
> while (scanner.hasNextLine()) {
> if (scanner.nextLine().startsWith("ERROR:")) {
> scanner.close();
> return false;
> }
> }
> scanner.close();
> return true;
> [DEBUG] Echo script : File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE");
> // Beanshell does not support try-with-resources,
> // so we must close this scanner manually
> Scanner scanner = new Scanner(license);
> while (scanner.hasNextLine()) {
> if (scanner.nextLine().startsWith("ERROR:")) {
> scanner.close();
> return false;
> }
[jira] [Updated] (HBASE-21185) WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes
[ https://issues.apache.org/jira/browse/HBASE-21185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-21185: - Status: Patch Available (was: Open) > WALPrettyPrinter: Additional useful info to be printed by wal printer tool, > for debugability purposes > - > > Key: HBASE-21185 > URL: https://issues.apache.org/jira/browse/HBASE-21185 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Attachments: HBASE-21185.master.001.patch, > HBASE-21185.master.002.patch > > > *WALPrettyPrinter* is very useful for troubleshooting wal issues, such as > faulty replication sinks. An useful information one might want to track is > the size of a single WAL entry edit, as well as size for each edit cell. Am > proposing a patch that adds calculations for these two, as well an option to > seek straight to a given position on the WAL file being analysed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614920#comment-16614920
]
Josh Elser commented on HBASE-20952:
{quote}If so, I was wondering if this doc was going to get a revision? (Lets
add link to this doc up at the top of this issue)
{quote}
It's been receiving updates already. I think GDocs will let you see that? Will
link that sucker as well if not already done.
{quote}Is it that he's done more homework that he has these questions or that
he just has a better understanding of how the system works?
{quote}
Well, he's obviously an expert at the brand new sync replication stuff. I'll
happily admit my ignorance around that. The fencing was me missing the
importance of the directory rename. Good outcome from review.
{quote}Is the 'design doc' we talk of above the place to work through his
concerns or is that somewhere else?
{quote}
My assumption was that we would be able to capture this information in the
design doc, and any others that people see as a gap. [~sergey.soldatov] had
mentioned offline yesterday that he thinks some gaps still exist around WAL
splitting – do you understand that well enough to suggest what needs to be
addressed in the doc which is not already there?
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614916#comment-16614916
]
Josh Elser commented on HBASE-20952:
{quote}Yes this is what I really want to discuss, not something like whether we
should use WALInfo or WALIdentity.
{quote}
We're in agreement... API just gives something tangible to look at when
thinking about requirements of the system. I think it's helpful, that's fine if
you don't :). Having a pissing match about the name of a class at this point is
a waste of time.
{quote}it seems to me that we will only have one stream opened forever for a
RS, then how do we drop the old edits after flush
{quote}
No, the plan would be to mimic the RS "rolling" logs in Ratis as well. This was
discussed over on the Ratis side. Seems to not have made it here – sorry.
{quote}In the old time we will roll the wal writer, and it is done by RS, so
closing the wal file is not enough, as the RS will try to open a new one and
write to it. That's why we need to rename the wal directory
{quote}
Crap. The current vision has a half-fix for this: just like we can do for the
Log used by one RS, we can list all Logs used by this RS and force them closed.
We don't (presently) have a mechanism spec'ed out in the LogService to prevent
a client from creating new Logs. We will have a "centralized" service that
manages the metadata for the Ratis logs in the system, so I think we _could_ do
something equivalent in the LogService (prevent a specific client from creating
a new Log). Need to think about that some more.
{quote}And for the FileSystem, we will use multi wal to increase the
performance, and the logic is messed up with WALProvider.
{quote}
Tell me more, please.
{quote}Does ratis still need multi wal to increase the performance? And if not,
what's the plan? We need to refactor the multi wal related code, to not work
against the WALProvider but something with the FileSystem related stuffs
directly?
{quote}
It's a goal that MultiWAL can work with any other WAL implementation, but not a
"priority 0" kind of concern to me.
{quote} For FileSystem based wals, it is just a directory on a remote cluster,
for example, "hdfs://cluster-name/path". We need to find a way to describe
other log systems.
{quote}
Got it. Yeah, this is another hole right now. [~rajeshbabu] and
[~sergey.soldatov] would probably be interested in this one; they've been
thinking about what logic we'll need to build in LogService (generic) and what
logic we'll have to build in HBase (specific).
Probably the biggest open question around this is how HBase will find the
LogService "metadata" service (e.g. the aforementioned piece that gives DDL
operations around Logs). Should the LogService provide some mechanism to
publish this? Or just leave it as an example to the implementation (e.g. HBase
master handles it)?
Thinking through the basics: an HDFS URI is logically equivalent to what we'd
be using in the LogService: HDFS UI gives us a nameservice and the unique name
of a file. For the LogService, we'd have the Raft quorum for the discovery
service and the name of the log to create. Needs some more thought around a
more elegant abstraction, I think...
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[
https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614914#comment-16614914
]
Hadoop QA commented on HBASE-21102:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
14s{color} | {color:red} hbase-server: The patch generated 1 new + 74 unchanged
- 0 fixed = 75 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
9s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
43s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}123m
4s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21102 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939712/HBASE-21102_addendum.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4d5c3a52de96 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5d14c1af65 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14418/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| hadoopcheck |
[jira] [Commented] (HBASE-21178) [BC break] : Get and Scan operation with a custom converter_class not working
[
https://issues.apache.org/jira/browse/HBASE-21178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614891#comment-16614891
]
Ted Yu commented on HBASE-21178:
That should be fine.
> [BC break] : Get and Scan operation with a custom converter_class not working
> -
>
> Key: HBASE-21178
> URL: https://issues.apache.org/jira/browse/HBASE-21178
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Subrat Mishra
>Assignee: Subrat Mishra
>Priority: Critical
> Attachments: HBASE-21178.master.001.patch
>
>
> Consider a simple scenario:
> {code:java}
> create 'foo', {NAME => 'f1'}
> put 'foo','r1','f1:a',1000
> get 'foo','r1',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}
> scan 'foo',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}{code}
> Both get and scan fails with ERROR
> {code:java}
> ERROR: wrong number of arguments (3 for 1) {code}
> Looks like in table.rb file converter_method expects 3 arguments [(bytes,
> offset, len)] since version 2.0.0, prior to version 2.0.0 it was taking only
> 1 argument [(bytes)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21178) [BC break] : Get and Scan operation with a custom converter_class not working
[
https://issues.apache.org/jira/browse/HBASE-21178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614885#comment-16614885
]
Subrat Mishra commented on HBASE-21178:
---
{quote}Is it possible that one of converter_class / converter_method is null ?
In that case the conversion should be skipped, right ?
{quote}
[~yuzhih...@gmail.com], I think converter_class / converter_method won't be
null in this particular case as null check has already taken care before in
"convert" method. Just added as a defensive check and tried to keep it similar
to
convert_bytes_with_position method.
{quote}Also, is it possible to add a test with conversion to prevent regression
?
{quote}
Is it fine to add the same example of "org.apache.hadoop.hbase.util.Bytes.len"
(mentioned in description) as a test case under table_test.rb?
> [BC break] : Get and Scan operation with a custom converter_class not working
> -
>
> Key: HBASE-21178
> URL: https://issues.apache.org/jira/browse/HBASE-21178
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Subrat Mishra
>Assignee: Subrat Mishra
>Priority: Critical
> Attachments: HBASE-21178.master.001.patch
>
>
> Consider a simple scenario:
> {code:java}
> create 'foo', {NAME => 'f1'}
> put 'foo','r1','f1:a',1000
> get 'foo','r1',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}
> scan 'foo',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}{code}
> Both get and scan fails with ERROR
> {code:java}
> ERROR: wrong number of arguments (3 for 1) {code}
> Looks like in table.rb file converter_method expects 3 arguments [(bytes,
> offset, len)] since version 2.0.0, prior to version 2.0.0 it was taking only
> 1 argument [(bytes)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21182) Failed to execute start-hbase.sh
[
https://issues.apache.org/jira/browse/HBASE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-21182:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Failed to execute start-hbase.sh
>
>
> Key: HBASE-21182
> URL: https://issues.apache.org/jira/browse/HBASE-21182
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0, 2.1.1
>Reporter: Subrat Mishra
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21182.master.001.patch,
> HBASE-21182.master.002.patch
>
>
> Built master branch like below:
> {code:java}
> mvn clean install -DskipTests{code}
> Then tried to execute start-hbase.sh failed with NoClassDefFoundError
> {code:java}
> ./bin/start-hbase.sh
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/org/eclipse/jetty/server/Connector
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.org.eclipse.jetty.server.Connector{code}
> Note: It worked after reverting HBASE-21153
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21182) Failed to execute start-hbase.sh
[
https://issues.apache.org/jira/browse/HBASE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614829#comment-16614829
]
Toshihiro Suzuki commented on HBASE-21182:
--
Pushed to master, branch-2 and branch-2.1.
> Failed to execute start-hbase.sh
>
>
> Key: HBASE-21182
> URL: https://issues.apache.org/jira/browse/HBASE-21182
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0, 2.1.1
>Reporter: Subrat Mishra
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21182.master.001.patch,
> HBASE-21182.master.002.patch
>
>
> Built master branch like below:
> {code:java}
> mvn clean install -DskipTests{code}
> Then tried to execute start-hbase.sh failed with NoClassDefFoundError
> {code:java}
> ./bin/start-hbase.sh
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/org/eclipse/jetty/server/Connector
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.org.eclipse.jetty.server.Connector{code}
> Note: It worked after reverting HBASE-21153
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21182) Failed to execute start-hbase.sh
[
https://issues.apache.org/jira/browse/HBASE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614814#comment-16614814
]
Toshihiro Suzuki commented on HBASE-21182:
--
Thank you for reviewing [~busbey]. Let me commit.
> Failed to execute start-hbase.sh
>
>
> Key: HBASE-21182
> URL: https://issues.apache.org/jira/browse/HBASE-21182
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0, 2.1.1
>Reporter: Subrat Mishra
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21182.master.001.patch,
> HBASE-21182.master.002.patch
>
>
> Built master branch like below:
> {code:java}
> mvn clean install -DskipTests{code}
> Then tried to execute start-hbase.sh failed with NoClassDefFoundError
> {code:java}
> ./bin/start-hbase.sh
> Error: A JNI error has occurred, please check your installation and try again
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/shaded/org/eclipse/jetty/server/Connector
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
> at java.lang.Class.getMethod0(Class.java:3018)
> at java.lang.Class.getMethod(Class.java:1784)
> at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.org.eclipse.jetty.server.Connector{code}
> Note: It worked after reverting HBASE-21153
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[
https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614687#comment-16614687
]
Hadoop QA commented on HBASE-21102:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
45s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
18s{color} | {color:red} hbase-server: The patch generated 1 new + 74 unchanged
- 0 fixed = 75 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
56s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 40s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestBaseLoadBalancer |
| | hadoop.hbase.master.assignment.TestAMAssignWithRandExec |
| | hadoop.hbase.master.assignment.TestAssignmentManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21102 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939683/HBASE-21102_addendum.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 12a4cae49838 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5d14c1af65 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
[jira] [Updated] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-21102: --- Status: Patch Available (was: Open) Solved the test case issue. The services could be null in test cases where cluster is not spawned. [~Apache9] Still I cannot see the failed test case running after the addendum. It is not in the list of tests. Is it because it is flaky test? > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch, > HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch, > HBASE-21102_addendum.patch, HBASE-21102_addendum.patch, > HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-21102: --- Status: Open (was: Patch Available) > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch, > HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch, > HBASE-21102_addendum.patch, HBASE-21102_addendum.patch, > HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-21102: --- Attachment: HBASE-21102_addendum.patch > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch, > HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch, > HBASE-21102_addendum.patch, HBASE-21102_addendum.patch, > HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[
https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614666#comment-16614666
]
Hadoop QA commented on HBASE-21102:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
28s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
18s{color} | {color:red} hbase-server: The patch generated 2 new + 74 unchanged
- 0 fixed = 76 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
23s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}138m 16s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}178m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.assignment.TestAssignmentManager |
| | hadoop.hbase.master.balancer.TestBaseLoadBalancer |
| | hadoop.hbase.master.assignment.TestAMAssignWithRandExec |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21102 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939671/HBASE-21102_addendum.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 44883cbfad62 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5d14c1af65 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
[jira] [Commented] (HBASE-21135) Build fails on windows as it fails to parse windows path during license check
[
https://issues.apache.org/jira/browse/HBASE-21135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614612#comment-16614612
]
Nihal Jain commented on HBASE-21135:
{quote}I can't build master on my windows box before or after this patch. Maven
fails in both cases with an error in hbase-common about finding bash.
{quote}
[~busbey] Do you have [cygwin|https://hbase.apache.org/cygwin.html] installed?
I compile hbase using cygwin.
> Build fails on windows as it fails to parse windows path during license check
> -
>
> Key: HBASE-21135
> URL: https://issues.apache.org/jira/browse/HBASE-21135
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0, 2.1.1
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21135.master.001.patch
>
>
> License check via enforce plugin throws following error during build on
> windows:
> {code:java}
> Sourced file: inline evaluation of: ``File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-ar . . . '' Token
> Parsing Error: Lexical error at line 1, column 29. Encountered: "D" (68),
> after : "\"D:\\": {code}
> Complete stacktrace with command
> {code:java}
> mvn clean install -DskipTests -X
> {code}
> is as follows:
> {noformat}
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (check-aggregate-license) @
> hbase-shaded ---
> [DEBUG] Configuring mojo
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce from plugin
> realm
> ClassRealm[plugin>org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1,
> parent: sun.misc.Launcher$AppClassLoader@55f96302]
> [DEBUG] Configuring mojo
> 'org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce' with basic
> configurator -->
> [DEBUG] (s) fail = true
> [DEBUG] (s) failFast = false
> [DEBUG] (f) ignoreCache = false
> [DEBUG] (f) mojoExecution =
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce {execution:
> check-aggregate-license}
> [DEBUG] (s) project = MavenProject:
> org.apache.hbase:hbase-shaded:2.1.1-SNAPSHOT @
> D:\DS\HBase_2\hbase\hbase-shaded\pom.xml
> [DEBUG] (s) condition = File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE");
> // Beanshell does not support try-with-resources,
> // so we must close this scanner manually
> Scanner scanner = new Scanner(license);
> while (scanner.hasNextLine()) {
> if (scanner.nextLine().startsWith("ERROR:")) {
> scanner.close();
> return false;
> }
> }
> scanner.close();
> return true;
> [DEBUG] (s) message = License errors detected, for more detail find ERROR in
>
> D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE
> [DEBUG] (s) rules =
> [org.apache.maven.plugins.enforcer.EvaluateBeanshell@7e307087]
> [DEBUG] (s) session = org.apache.maven.execution.MavenSession@5e1218b4
> [DEBUG] (s) skip = false
> [DEBUG] -- end configuration --
> [DEBUG] Executing rule: org.apache.maven.plugins.enforcer.EvaluateBeanshell
> [DEBUG] Echo condition : File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE");
> // Beanshell does not support try-with-resources,
> // so we must close this scanner manually
> Scanner scanner = new Scanner(license);
> while (scanner.hasNextLine()) {
> if (scanner.nextLine().startsWith("ERROR:")) {
> scanner.close();
> return false;
> }
> }
> scanner.close();
> return true;
> [DEBUG] Echo script : File license = new
> File("D:\DS\HBase_2\hbase\hbase-shaded\target/maven-shared-archive-resources/META-INF/LICENSE");
> // Beanshell does not support try-with-resources,
> // so we must close this scanner manually
> Scanner scanner = new Scanner(license);
> while (scanner.hasNextLine()) {
> if (scanner.nextLine().startsWith("ERROR:")) {
> scanner.close();
> return false;
> }
> }
> scanner.close();
> return true;
> [DEBUG] Adding failure due to exception
>
[jira] [Commented] (HBASE-15631) Backport Regionserver Groups (HBASE-6721) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614586#comment-16614586 ] Anoop Sam John commented on HBASE-15631: [~apurtell] Any specific/changed configs for the branch-1 feature usage? If so a RL would be the best thing to have. > Backport Regionserver Groups (HBASE-6721) to branch-1 > -- > > Key: HBASE-15631 > URL: https://issues.apache.org/jira/browse/HBASE-15631 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Francis Liu >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.4.0 > > Attachments: HBASE-15631-branch-1-addendum.patch, > HBASE-15631-branch-1.patch, HBASE-15631-branch-1.patch, > HBASE-15631-branch-1.patch, HBASE-15631.branch-1.patch, HBASE-15631.patch > > > Based on dev list discussion backporting region server group should not be an > issue as it does not: 1. destabilize the code. 2. cause backward > incompatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21178) [BC break] : Get and Scan operation with a custom converter_class not working
[
https://issues.apache.org/jira/browse/HBASE-21178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614584#comment-16614584
]
Ted Yu commented on HBASE-21178:
{code}
782 converter_class = 'org.apache.hadoop.hbase.util.Bytes' unless
converter_class
783 converter_method = 'toStringBinary' unless converter_method
784 eval(converter_class).method(converter_method).call(bytes)
{code}
Is it possible that one of converter_class / converter_method is null ?
In that case the conversion should be skipped, right ?
Also, is it possible to add a test with conversion to prevent regression ?
Thanks for the finding.
> [BC break] : Get and Scan operation with a custom converter_class not working
> -
>
> Key: HBASE-21178
> URL: https://issues.apache.org/jira/browse/HBASE-21178
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Subrat Mishra
>Assignee: Subrat Mishra
>Priority: Critical
> Attachments: HBASE-21178.master.001.patch
>
>
> Consider a simple scenario:
> {code:java}
> create 'foo', {NAME => 'f1'}
> put 'foo','r1','f1:a',1000
> get 'foo','r1',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}
> scan 'foo',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}{code}
> Both get and scan fails with ERROR
> {code:java}
> ERROR: wrong number of arguments (3 for 1) {code}
> Looks like in table.rb file converter_method expects 3 arguments [(bytes,
> offset, len)] since version 2.0.0, prior to version 2.0.0 it was taking only
> 1 argument [(bytes)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21198) Exclude dependency on net.minidev:json-smart
Ted Yu created HBASE-21198:
--
Summary: Exclude dependency on net.minidev:json-smart
Key: HBASE-21198
URL: https://issues.apache.org/jira/browse/HBASE-21198
Project: HBase
Issue Type: Task
Reporter: Ted Yu
>From
>https://builds.apache.org/job/PreCommit-HBASE-Build/14414/artifact/patchprocess/patch-javac-3.0.0.txt
> :
{code}
[ERROR] Failed to execute goal on project hbase-common: Could not resolve
dependencies for project org.apache.hbase:hbase-common:jar:3.0.0-SNAPSHOT:
Failed to collect dependencies at org.apache.hadoop:hadoop-common:jar:3.0.0 ->
org.apache.hadoop:hadoop-auth:jar:3.0.0 ->
com.nimbusds:nimbus-jose-jwt:jar:4.41.1 ->
net.minidev:json-smart:jar:2.3-SNAPSHOT: Failed to read artifact descriptor for
net.minidev:json-smart:jar:2.3-SNAPSHOT: Could not transfer artifact
net.minidev:json-smart:pom:2.3-SNAPSHOT from/to dynamodb-local-oregon
(https://s3-us-west-2.amazonaws.com/dynamodb-local/release): Access denied to:
https://s3-us-west-2.amazonaws.com/dynamodb-local/release/net/minidev/json-smart/2.3-SNAPSHOT/json-smart-2.3-SNAPSHOT.pom
, ReasonPhrase:Forbidden. -> [Help 1]
{code}
We should exclude dependency on net.minidev:json-smart
hbase-common/bin/pom.xml has done so.
The other pom.xml should do the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21160) Assertion in TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels is ignored
[
https://issues.apache.org/jira/browse/HBASE-21160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614570#comment-16614570
]
Ted Yu commented on HBASE-21160:
Can you address these ?
{code}
[WARNING]
/testptch/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDefaultVisLabelService.java:[94,9]
[CatchFail] Ignoring exceptions and calling fail() is unnecessary, and makes
test output less useful
[WARNING]
/testptch/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDefaultVisLabelService.java:[183,9]
[CatchFail] Ignoring exceptions and calling fail() is unnecessary, and makes
test output less useful
[WARNING]
/testptch/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDefaultVisLabelService.java:[195,71]
[DefaultCharset] Implicit use of the platform default charset, which can
result in e.g. non-ASCII characters being silently replaced with '?' in many
environments
[WARNING]
/testptch/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDefaultVisLabelService.java:[196,74]
[DefaultCharset] Implicit use of the platform default charset, which can
result in e.g. non-ASCII characters being silently replaced with '?' in many
environments
[WARNING]
/testptch/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDefaultVisLabelService.java:[197,77]
[DefaultCharset] Implicit use of the platform default charset, which can
result in e.g. non-ASCII characters being silently replaced with '?' in many
environments
{code}
> Assertion in
> TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels
> is ignored
> ---
>
> Key: HBASE-21160
> URL: https://issues.apache.org/jira/browse/HBASE-21160
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Trivial
> Attachments: HBASE-21160-1.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14327/artifact/patchprocess/diff-compile-javac-hbase-server.txt
> (HBASE-21138 QA run):
> {code}
> [WARNING]
> /testptch/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDeletes.java:[315,25]
> [AssertionFailureIgnored] This assertion throws an AssertionError if it
> fails, which will be caught by an enclosing try block.
> {code}
> Here is related code:
> {code}
> PrivilegedExceptionAction scanAction = new
> PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> try (Connection connection =
> ConnectionFactory.createConnection(conf);
> ...
> assertEquals(1, next.length);
> } catch (Throwable t) {
> throw new IOException(t);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21160) Assertion in TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels is ignored
[
https://issues.apache.org/jira/browse/HBASE-21160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614566#comment-16614566
]
Hadoop QA commented on HBASE-21160:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 49s{color}
| {color:red} hbase-server generated 8 new + 180 unchanged - 8 fixed = 188
total (was 188) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
58s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}143m
55s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}183m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21160 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939653/HBASE-21160-1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 98e4d6a87a20 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 5d14c1af65 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/14414/artifact/patchprocess/diff-compile-javac-hbase-server.txt
|
| hadoopcheck |
https://builds.apache.org/job/PreCommit-HBASE-Build/14414/artifact/patchprocess/patch-javac-3.0.0.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14414/testReport/ |
| Max. process+thread count | 4961 (vs. ulimit of 1) |
[jira] [Commented] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[
https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614563#comment-16614563
]
Ted Yu commented on HBASE-21102:
Addendum looks good.
{code}
public boolean isReplicaAvailbleForRegion(final RegionInfo info) {
{code}
Missing 'a' in 'Availble'
> ServerCrashProcedure should select target server where no other replicas
> exist for the current region
> -
>
> Key: HBASE-21102
> URL: https://issues.apache.org/jira/browse/HBASE-21102
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 3.0.0, 2.2.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Major
> Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch,
> HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch,
> HBASE-21102_addendum.patch, HBASE-21102_initial.patch
>
>
> Currently when a server with region replica crashes, when the target server
> is created for the replica region assignment there is no guarentee that a
> server is selected where there is no other replica for the current region
> getting assigned. It so happens that currently we do an assignment randomly
> and later the LB comes and identifies these cases and again does MOVE for
> such regions. It will be better if we can identify target servers at least
> minimally ensuring that replicas are not colocated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614559#comment-16614559
]
Hadoop QA commented on HBASE-21196:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
9s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
9s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m
38s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
59s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}122m
22s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939655/HBASE-21196.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2295a3fd1e7c 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5d14c1af65 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
|
[jira] [Commented] (HBASE-21035) Meta Table should be able to online even if all procedures are lost
[ https://issues.apache.org/jira/browse/HBASE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614558#comment-16614558 ] Duo Zhang commented on HBASE-21035: --- We could also help on HBCK2. What is the working flow for it? Github based? > Meta Table should be able to online even if all procedures are lost > --- > > Key: HBASE-21035 > URL: https://issues.apache.org/jira/browse/HBASE-21035 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-21035.branch-2.0.001.patch, > HBASE-21035.branch-2.1.001.patch > > > After HBASE-20708, we changed the way we init after master starts. It will > only check WAL dirs and compare to Zookeeper RS nodes to decide which server > need to expire. For servers which's dir is ending with 'SPLITTING', we assure > that there will be a SCP for it. > But, if the server with the meta region crashed before master restarts, and > if all the procedure wals are lost (due to bug, or deleted manually, > whatever), the new restarted master will be stuck when initing. Since no one > will bring meta region online. > Although it is an anomaly case, but I think no matter what happens, we need > to online meta region. Otherwise, we are sitting ducks, noting can be done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21160) Assertion in TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels is ignored
[
https://issues.apache.org/jira/browse/HBASE-21160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614557#comment-16614557
]
liubangchen edited comment on HBASE-21160 at 9/14/18 9:16 AM:
--
Areadly has run with commd :
{code}
mvn
-Dtest=org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
surefire-report:report test
{code}
And result is:
{code}
Unable to find source-code formatter for language: shell. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yaml[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.421
s - in
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-surefire-plugin:2.21.0:test (secondPartTestsExecution) @
hbase-server ---
[INFO] Tests are skipped.
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 03:59 min
[INFO] Finished at: 2018-09-14T14:13:15+08:00
[INFO]
{code}
was (Author: liubangchen):
Areadly has run with commd :
{code:shell}
mvn
-Dtest=org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
surefire-report:report test
{code}
And result is:
{code:shell}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.421
s - in
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-surefire-plugin:2.21.0:test (secondPartTestsExecution) @
hbase-server ---
[INFO] Tests are skipped.
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 03:59 min
[INFO] Finished at: 2018-09-14T14:13:15+08:00
[INFO]
{code}
> Assertion in
> TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels
> is ignored
> ---
>
> Key: HBASE-21160
> URL: https://issues.apache.org/jira/browse/HBASE-21160
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Trivial
> Attachments: HBASE-21160-1.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14327/artifact/patchprocess/diff-compile-javac-hbase-server.txt
> (HBASE-21138 QA run):
> {code}
> [WARNING]
> /testptch/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDeletes.java:[315,25]
> [AssertionFailureIgnored] This assertion throws an AssertionError if it
> fails, which will be caught by an enclosing try block.
> {code}
> Here is related code:
> {code}
> PrivilegedExceptionAction scanAction = new
> PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> try (Connection connection =
> ConnectionFactory.createConnection(conf);
> ...
> assertEquals(1, next.length);
> } catch (Throwable t) {
> throw new IOException(t);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21160) Assertion in TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels is ignored
[
https://issues.apache.org/jira/browse/HBASE-21160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614557#comment-16614557
]
liubangchen commented on HBASE-21160:
-
Areadly has run with commd :
{code:shell}
mvn
-Dtest=org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
surefire-report:report test
{code}
And result is:
{code:shell}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.421
s - in
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-surefire-plugin:2.21.0:test (secondPartTestsExecution) @
hbase-server ---
[INFO] Tests are skipped.
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 03:59 min
[INFO] Finished at: 2018-09-14T14:13:15+08:00
[INFO]
{code}
> Assertion in
> TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels
> is ignored
> ---
>
> Key: HBASE-21160
> URL: https://issues.apache.org/jira/browse/HBASE-21160
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Trivial
> Attachments: HBASE-21160-1.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14327/artifact/patchprocess/diff-compile-javac-hbase-server.txt
> (HBASE-21138 QA run):
> {code}
> [WARNING]
> /testptch/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDeletes.java:[315,25]
> [AssertionFailureIgnored] This assertion throws an AssertionError if it
> fails, which will be caught by an enclosing try block.
> {code}
> Here is related code:
> {code}
> PrivilegedExceptionAction scanAction = new
> PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> try (Connection connection =
> ConnectionFactory.createConnection(conf);
> ...
> assertEquals(1, next.length);
> } catch (Throwable t) {
> throw new IOException(t);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614555#comment-16614555
]
Ted Yu commented on HBASE-21196:
Ran test without fix:
{code}
testCachingForHTableMultiplexerMultiPut(org.apache.hadoop.hbase.client.TestRegionLocationCaching)
Time elapsed: 0.469 sec <<< FAILURE!
java.lang.AssertionError: Expected non-zero number of cached region locations,
but it is0. Actual: 0
at
org.apache.hadoop.hbase.client.TestRegionLocationCaching.checkRegionLocationIsCached(TestRegionLocationCaching.java:156)
at
org.apache.hadoop.hbase.client.TestRegionLocationCaching.testCachingForHTableMultiplexerMultiPut(TestRegionLocationCaching.java:101)
testCachingForHTableMultiplexerSinglePut(org.apache.hadoop.hbase.client.TestRegionLocationCaching)
Time elapsed: 0.305 sec <<< FAILURE!
java.lang.AssertionError: Expected non-zero number of cached region locations,
but it is0. Actual: 0
at
org.apache.hadoop.hbase.client.TestRegionLocationCaching.checkRegionLocationIsCached(TestRegionLocationCaching.java:156)
at
org.apache.hadoop.hbase.client.TestRegionLocationCaching.testCachingForHTableMultiplexerSinglePut(TestRegionLocationCaching.java:79)
{code}
Please fix the assertion ('it is0').
Since the other two tests don't fail without fix, I think you may take them out.
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
>
[jira] [Commented] (HBASE-21160) Assertion in TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels is ignored
[
https://issues.apache.org/jira/browse/HBASE-21160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614551#comment-16614551
]
Ted Yu commented on HBASE-21160:
Ran the test with patch locally which passed.
> Assertion in
> TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels
> is ignored
> ---
>
> Key: HBASE-21160
> URL: https://issues.apache.org/jira/browse/HBASE-21160
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Trivial
> Attachments: HBASE-21160-1.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14327/artifact/patchprocess/diff-compile-javac-hbase-server.txt
> (HBASE-21138 QA run):
> {code}
> [WARNING]
> /testptch/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDeletes.java:[315,25]
> [AssertionFailureIgnored] This assertion throws an AssertionError if it
> fails, which will be caught by an enclosing try block.
> {code}
> Here is related code:
> {code}
> PrivilegedExceptionAction scanAction = new
> PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> try (Connection connection =
> ConnectionFactory.createConnection(conf);
> ...
> assertEquals(1, next.length);
> } catch (Throwable t) {
> throw new IOException(t);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-21102: --- Status: Patch Available (was: Open) This new addendum is better since we get the regions list from the tail map of the RegionState's regionMap. And if we move to the next region we immediately break the loop. > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch, > HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch, > HBASE-21102_addendum.patch, HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-21102: --- Status: Open (was: Patch Available) > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch, > HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch, > HBASE-21102_addendum.patch, HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-21102: --- Attachment: HBASE-21102_addendum.patch > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch, > HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch, > HBASE-21102_addendum.patch, HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21164) reportForDuty to spew less log if master is initializing
[
https://issues.apache.org/jira/browse/HBASE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614526#comment-16614526
]
Hadoop QA commented on HBASE-21164:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
1s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}140m 48s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}177m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.token.TestZKSecretWatcher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21164 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939652/HBASE-21164.008.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux db8a0f3ef3ed 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5d14c1af65 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| hadoopcheck |
https://builds.apache.org/job/PreCommit-HBASE-Build/14413/artifact/patchprocess/patch-javac-3.0.0.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14413/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14413/testReport/ |
| Max. process+thread count
[jira] [Commented] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614512#comment-16614512 ] ramkrishna.s.vasudevan commented on HBASE-21102: Let me make the method that says whether replica or not - better. The current one in the addendum patch is not that great. > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch, > HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch, > HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21178) Get and Scan operation with a custom converter_class not working
[
https://issues.apache.org/jira/browse/HBASE-21178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-21178:
---
Summary: Get and Scan operation with a custom converter_class not working
(was: Get and Scan operation with converter_class not working)
> Get and Scan operation with a custom converter_class not working
>
>
> Key: HBASE-21178
> URL: https://issues.apache.org/jira/browse/HBASE-21178
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Subrat Mishra
>Assignee: Subrat Mishra
>Priority: Critical
> Attachments: HBASE-21178.master.001.patch
>
>
> Consider a simple scenario:
> {code:java}
> create 'foo', {NAME => 'f1'}
> put 'foo','r1','f1:a',1000
> get 'foo','r1',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}
> scan 'foo',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}{code}
> Both get and scan fails with ERROR
> {code:java}
> ERROR: wrong number of arguments (3 for 1) {code}
> Looks like in table.rb file converter_method expects 3 arguments [(bytes,
> offset, len)] since version 2.0.0, prior to version 2.0.0 it was taking only
> 1 argument [(bytes)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21178) Get and Scan operation with converter_class not working
[
https://issues.apache.org/jira/browse/HBASE-21178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614511#comment-16614511
]
Anoop Sam John commented on HBASE-21178:
As this is a break on the shell exposed API, I made it to be critical. Need to
be committed all 2.x branches. Ping [~stack].
> Get and Scan operation with converter_class not working
> ---
>
> Key: HBASE-21178
> URL: https://issues.apache.org/jira/browse/HBASE-21178
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Subrat Mishra
>Assignee: Subrat Mishra
>Priority: Critical
> Attachments: HBASE-21178.master.001.patch
>
>
> Consider a simple scenario:
> {code:java}
> create 'foo', {NAME => 'f1'}
> put 'foo','r1','f1:a',1000
> get 'foo','r1',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}
> scan 'foo',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}{code}
> Both get and scan fails with ERROR
> {code:java}
> ERROR: wrong number of arguments (3 for 1) {code}
> Looks like in table.rb file converter_method expects 3 arguments [(bytes,
> offset, len)] since version 2.0.0, prior to version 2.0.0 it was taking only
> 1 argument [(bytes)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21178) [BC break] : Get and Scan operation with a custom converter_class not working
[
https://issues.apache.org/jira/browse/HBASE-21178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-21178:
---
Summary: [BC break] : Get and Scan operation with a custom converter_class
not working (was: Get and Scan operation with a custom converter_class not
working)
> [BC break] : Get and Scan operation with a custom converter_class not working
> -
>
> Key: HBASE-21178
> URL: https://issues.apache.org/jira/browse/HBASE-21178
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Subrat Mishra
>Assignee: Subrat Mishra
>Priority: Critical
> Attachments: HBASE-21178.master.001.patch
>
>
> Consider a simple scenario:
> {code:java}
> create 'foo', {NAME => 'f1'}
> put 'foo','r1','f1:a',1000
> get 'foo','r1',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}
> scan 'foo',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}{code}
> Both get and scan fails with ERROR
> {code:java}
> ERROR: wrong number of arguments (3 for 1) {code}
> Looks like in table.rb file converter_method expects 3 arguments [(bytes,
> offset, len)] since version 2.0.0, prior to version 2.0.0 it was taking only
> 1 argument [(bytes)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21178) Get and Scan operation with converter_class not working
[
https://issues.apache.org/jira/browse/HBASE-21178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-21178:
---
Affects Version/s: 2.0.0
Priority: Critical (was: Major)
Hadoop Flags: Reviewed
Component/s: shell
> Get and Scan operation with converter_class not working
> ---
>
> Key: HBASE-21178
> URL: https://issues.apache.org/jira/browse/HBASE-21178
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Subrat Mishra
>Assignee: Subrat Mishra
>Priority: Critical
> Attachments: HBASE-21178.master.001.patch
>
>
> Consider a simple scenario:
> {code:java}
> create 'foo', {NAME => 'f1'}
> put 'foo','r1','f1:a',1000
> get 'foo','r1',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}
> scan 'foo',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}{code}
> Both get and scan fails with ERROR
> {code:java}
> ERROR: wrong number of arguments (3 for 1) {code}
> Looks like in table.rb file converter_method expects 3 arguments [(bytes,
> offset, len)] since version 2.0.0, prior to version 2.0.0 it was taking only
> 1 argument [(bytes)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-21102: --- Status: Patch Available (was: Open) > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch, > HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch, > HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614493#comment-16614493 ] ramkrishna.s.vasudevan commented on HBASE-21102: Fixed the reason for the UT failure. It was because in the LB code I was trying to see if the list of regions had atleast one replica. It was so happening that in my test env (even today I ran it 25 times) every time I was getting more than 5 regions to the retain Assignment API. So one of them turned out to be a replica region always. But in the jenkins build it was so happening that the list of regions had only one region and that one region was a primary region. So when it was the case we were not taking the entire snapshot of assignment so we were doing its assignment to the server where there was another replica. It would be better if we can centrally know some where that for a given region are there any replicas without iterating through the AM's region states. Other way is to get the details from the TableDescriptor of Master but I fear it is not using the cache in the master side and we may end up doing a FS read there. > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch, > HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch, > HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-21102: --- Attachment: HBASE-21102_addendum.patch > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: HBASE-21102_1.patch, HBASE-21102_2.patch, > HBASE-21102_3.patch, HBASE-21102_4.patch, HBASE-21102_addendum.patch, > HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614426#comment-16614426
]
Nihal Jain commented on HBASE-21196:
Attached a new patch which removes whitespace, adds class rule and proper test
category. Other failures are not relevant. Pass locally.
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
> up meta region location in ZK" are present for every put.
> *Analysis:*
> The problem occurs as we call the {{cleanServerCache}} method always clears
> the server cache in case tablename is null and exception is null. See
> [AsyncRequestFutureImpl.java#L918|https://github.com/apache/hbase/blob/5d14c1af65c02f4e87059337c35e4431505de91c/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L918]
> {code:java}
> private void cleanServerCache(ServerName server, Throwable regionException) {
> if (tableName == null &&
> ClientExceptionsUtil.isMetaClearingException(regionException)) {
> // For multi-actions, we don't have a table name, but we want to make
> sure to clear the
> // cache in case there were location-related exceptions. We don't to
> clear the
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614427#comment-16614427
]
Nihal Jain commented on HBASE-21196:
Ping [~elserj], [~apurtell], [~yuzhih...@gmail.com].
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
> up meta region location in ZK" are present for every put.
> *Analysis:*
> The problem occurs as we call the {{cleanServerCache}} method always clears
> the server cache in case tablename is null and exception is null. See
> [AsyncRequestFutureImpl.java#L918|https://github.com/apache/hbase/blob/5d14c1af65c02f4e87059337c35e4431505de91c/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L918]
> {code:java}
> private void cleanServerCache(ServerName server, Throwable regionException) {
> if (tableName == null &&
> ClientExceptionsUtil.isMetaClearingException(regionException)) {
> // For multi-actions, we don't have a table name, but we want to make
> sure to clear the
> // cache in case there were location-related exceptions. We don't to
> clear the cache
> // for every possible exception that comes through, however.
>
[jira] [Updated] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-21196:
---
Attachment: HBASE-21196.master.001.patch
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
> up meta region location in ZK" are present for every put.
> *Analysis:*
> The problem occurs as we call the {{cleanServerCache}} method always clears
> the server cache in case tablename is null and exception is null. See
> [AsyncRequestFutureImpl.java#L918|https://github.com/apache/hbase/blob/5d14c1af65c02f4e87059337c35e4431505de91c/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L918]
> {code:java}
> private void cleanServerCache(ServerName server, Throwable regionException) {
> if (tableName == null &&
> ClientExceptionsUtil.isMetaClearingException(regionException)) {
> // For multi-actions, we don't have a table name, but we want to make
> sure to clear the
> // cache in case there were location-related exceptions. We don't to
> clear the cache
> // for every possible exception that comes through, however.
> asyncProcess.connection.clearCaches(server);
> }
>
[jira] [Updated] (HBASE-21160) Assertion in TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels is ignored
[
https://issues.apache.org/jira/browse/HBASE-21160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen updated HBASE-21160:
Attachment: HBASE-21160-1.patch
Status: Patch Available (was: Open)
> Assertion in
> TestVisibilityLabelsWithDeletes#testDeleteColumnsWithoutAndWithVisibilityLabels
> is ignored
> ---
>
> Key: HBASE-21160
> URL: https://issues.apache.org/jira/browse/HBASE-21160
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Trivial
> Attachments: HBASE-21160-1.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14327/artifact/patchprocess/diff-compile-javac-hbase-server.txt
> (HBASE-21138 QA run):
> {code}
> [WARNING]
> /testptch/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDeletes.java:[315,25]
> [AssertionFailureIgnored] This assertion throws an AssertionError if it
> fails, which will be caught by an enclosing try block.
> {code}
> Here is related code:
> {code}
> PrivilegedExceptionAction scanAction = new
> PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> try (Connection connection =
> ConnectionFactory.createConnection(conf);
> ...
> assertEquals(1, next.length);
> } catch (Throwable t) {
> throw new IOException(t);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614410#comment-16614410
]
Reid Chan commented on HBASE-20993:
---
ping [~busbey], do you know what this error {{"script engine for language js
can not be found"}} indicates?
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
> Fix For: 1.5.0, 1.4.8
>
> Attachments: HBASE-20993.001.patch,
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch,
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch,
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch,
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch,
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.009.patch,
> HBASE-20993.branch-1.2.001.patch, HBASE-20993.branch-1.wip.002.patch,
> HBASE-20993.branch-1.wip.patch, yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:607)
> at
>
