[jira] [Commented] (HBASE-22514) Move rsgroup feature into core of HBase
[
https://issues.apache.org/jira/browse/HBASE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032828#comment-17032828
]
Hudson commented on HBASE-22514:
Results for branch HBASE-22514
[build #266 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/266/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/266//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/266//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/266//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move rsgroup feature into core of HBase
> ---
>
> Key: HBASE-22514
> URL: https://issues.apache.org/jira/browse/HBASE-22514
> Project: HBase
> Issue Type: Umbrella
> Components: Admin, Client, rsgroup
>Reporter: Yechao Chen
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-22514.master.001.patch,
> image-2019-05-31-18-25-38-217.png
>
>
> The class RSGroupAdminClient is not public
> we need to use java api RSGroupAdminClient to manager RSG
> so RSGroupAdminClient should be public
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1142: HBASE-23809 The RSGroup shell test is missing
Apache-HBase commented on issue #1142: HBASE-23809 The RSGroup shell test is missing URL: https://github.com/apache/hbase/pull/1142#issuecomment-583702242 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 18s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 52s | master passed | | +1 :green_heart: | compile | 0m 51s | master passed | | +1 :green_heart: | checkstyle | 0m 26s | master passed | | +1 :green_heart: | shadedjars | 5m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | | +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 48s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 36s | the patch passed | | +1 :green_heart: | compile | 0m 51s | the patch passed | | +1 :green_heart: | javac | 0m 51s | the patch passed | | +1 :green_heart: | checkstyle | 0m 24s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 47s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | | +1 :green_heart: | findbugs | 0m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 5m 21s | hbase-rsgroup in the patch passed. | | +1 :green_heart: | unit | 7m 1s | hbase-shell in the patch passed. | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 66m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1142 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux bfd739dfb6c6 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1142/out/precommit/personality/provided.sh | | git revision | master / d554689cc4 | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/3/testReport/ | | Max. process+thread count | 5665 (vs. ulimit of 1) | | modules | C: hbase-rsgroup hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/3/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23812) [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did not replicate enough rows expected:<10> but was:<7>
[
https://issues.apache.org/jira/browse/HBASE-23812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032809#comment-17032809
]
Hudson commented on HBASE-23812:
Results for branch branch-2
[build #2459 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2459/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2459//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2459//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2459//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did
> not replicate enough rows expected:<10> but was:<7>
> -
>
> Key: HBASE-23812
> URL: https://issues.apache.org/jira/browse/HBASE-23812
> Project: HBase
> Issue Type: Bug
> Components: flakies
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments:
> 0001-HBASE-23812-Flakey-Test-TestReplicator-testReplicato.patch
>
>
> I've seen this fail. Here is the latest:
> {code}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 21.044 s <<< FAILURE! - in
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator
> [ERROR]
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors
> Time elapsed: 7.139 s <<< FAILURE!
> java.lang.AssertionError: We did not replicate enough rows expected:<10> but
> was:<7>
>at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors(TestReplicator.java:158)
> {code}
> In test logs there are a few (in groups of three) of these:
> {code}
> 4247 2020-02-07 13:50:29,070 WARN
> [RS_REFRESH_PEER-regionserver/localhost:0-1.replicationSource,testReplicatorWithErrors.replicationSource.shipperlocalhost%2C49531%2C1581112194639,testReplicatorWithErrors]
> regionserver. ReplicationSourceShipper(223):
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$FailureInjectingReplicationEndpointForTest
> threw unknown exception:
> 4248 java.lang.ClassCastException:
> org.apache.hbase.thirdparty.com.google.protobuf.ServiceException cannot be
> cast to java.io.IOException
> 4249 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.parallelReplicate(HBaseInterClusterReplicationEndpoint.java:361)
> 4250 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:404)
> 4251 at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$ReplicationEndpointForTest.replicate(TestReplicator.java:228)
> 4252 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:188)
> 4253 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:118)
> {code}
> Let me fix this and see if it makes TestReplicator stable again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[
https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032810#comment-17032810
]
Hudson commented on HBASE-23779:
Results for branch branch-2
[build #2459 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2459/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2459//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2459//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2459//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Up the default fork count to make builds complete faster; make count relative
> to CPU count
> --
>
> Key: HBASE-23779
> URL: https://issues.apache.org/jira/browse/HBASE-23779
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: addendum2.patch, test_yetus_934.0.patch
>
>
> Tests take a long time. Our fork count running all tests are conservative --
> 1 (small) for first part and 5 for second part (medium and large). Rather
> than hardcoding we should set the fork count to be relative to machine size.
> Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.
> Looking up at jenkins, it seems like the boxes are 24 cores... at least going
> by my random survey. The load reported on a few seems low though this not
> representative (looking at machine/uptime).
> More parallelism willl probably mean more test failure. Let me take a look
> see.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
Apache-HBase commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#issuecomment-583696957 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 15s | master passed | | +1 :green_heart: | compile | 0m 52s | master passed | | +1 :green_heart: | checkstyle | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 4m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | | +0 :ok: | spotbugs | 1m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 8s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 58s | the patch passed | | +1 :green_heart: | compile | 0m 52s | the patch passed | | +1 :green_heart: | javac | 0m 52s | the patch passed | | +1 :green_heart: | checkstyle | 0m 42s | the patch passed | | +1 :green_heart: | rubocop | 0m 9s | There were no new rubocop issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 54s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | | +1 :green_heart: | findbugs | 1m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 56s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 6m 54s | hbase-shell in the patch passed. | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 59m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1138/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1138 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile rubocop | | uname | Linux 98cb0c885457 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1138/out/precommit/personality/provided.sh | | git revision | master / d554689cc4 | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1138/2/testReport/ | | Max. process+thread count | 3689 (vs. ulimit of 1) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1138/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 rubocop=0.79.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1147: HBASE-23814 Add null checks and logging to misc set of tests
Apache-HBase commented on issue #1147: HBASE-23814 Add null checks and logging to misc set of tests URL: https://github.com/apache/hbase/pull/1147#issuecomment-583695034 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 12s | branch-2 passed | | +1 :green_heart: | compile | 1m 4s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 20s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | branch-2 passed | | +0 :ok: | spotbugs | 3m 55s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 53s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 32s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | checkstyle | 1m 15s | hbase-server: The patch generated 0 new + 5 unchanged - 1 fixed = 5 total (was 6) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 21m 21s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 48s | the patch passed | | +1 :green_heart: | findbugs | 5m 12s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 136m 10s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 205m 24s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.replication.TestReplicationStatus | | | hadoop.hbase.replication.TestReplicationDroppedTables | | | hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaSameHosts | | | hadoop.hbase.security.access.TestAccessController3 | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1147/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1147 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux f9e3363805c0 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1147/out/precommit/personality/provided.sh | | git revision | branch-2 / 29fb52b5d2 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1147/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1147/1/testReport/ | | Max. process+thread count | 6941 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1147/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1142: HBASE-23809 The RSGroup shell test is missing
Apache-HBase commented on issue #1142: HBASE-23809 The RSGroup shell test is missing URL: https://github.com/apache/hbase/pull/1142#issuecomment-583693638 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 46s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | checkstyle | 0m 25s | master passed | | +1 :green_heart: | shadedjars | 5m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | | +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 48s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 36s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | checkstyle | 0m 23s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 38s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | | +1 :green_heart: | findbugs | 0m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 5m 28s | hbase-rsgroup in the patch passed. | | -1 :x: | unit | 7m 3s | hbase-shell in the patch failed. | | +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | | | | 67m 1s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestAdminShell2 | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1142 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 84a18307fc50 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1142/out/precommit/personality/provided.sh | | git revision | master / d554689cc4 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/2/artifact/out/patch-unit-hbase-shell.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/2/testReport/ | | Max. process+thread count | 6102 (vs. ulimit of 1) | | modules | C: hbase-rsgroup hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1146: HBASE-23781 Removed deprecated createTableDescriptor(String) from HBaseTestingUtility
Apache-HBase commented on issue #1146: HBASE-23781 Removed deprecated createTableDescriptor(String) from HBaseTestingUtility URL: https://github.com/apache/hbase/pull/1146#issuecomment-583692290 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 54s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 14 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 9s | master passed | | +1 :green_heart: | compile | 1m 1s | master passed | | +1 :green_heart: | checkstyle | 1m 20s | master passed | | +1 :green_heart: | shadedjars | 5m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | | +0 :ok: | spotbugs | 4m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 50s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 2s | the patch passed | | +1 :green_heart: | compile | 1m 6s | the patch passed | | +1 :green_heart: | javac | 1m 6s | the patch passed | | +1 :green_heart: | checkstyle | 1m 17s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 18m 5s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | | +1 :green_heart: | findbugs | 4m 55s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 131m 28s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | | | | 193m 44s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestAsyncMetaRegionLocator | | | hadoop.hbase.client.TestTableFavoredNodes | | | hadoop.hbase.master.procedure.TestSplitWALProcedure | | | hadoop.hbase.client.TestAvoidCellReferencesIntoShippedBlocks | | | hadoop.hbase.replication.regionserver.TestWALEntryStream | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1146/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1146 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux b0ee019459c2 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1146/out/precommit/personality/provided.sh | | git revision | master / ef380e0a2e | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1146/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1146/1/testReport/ | | Max. process+thread count | 6648 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1146/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[
https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032790#comment-17032790
]
Hudson commented on HBASE-18095:
Results for branch HBASE-18095/client-locate-meta-no-zookeeper
[build #66 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/66/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/66//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/66//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/66//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/66//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Provide an option for clients to find the server hosting META that does not
> involve the ZooKeeper client
>
>
> Key: HBASE-18095
> URL: https://issues.apache.org/jira/browse/HBASE-18095
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Reporter: Andrew Kyle Purtell
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch
>
>
> Clients are required to connect to ZooKeeper to find the location of the
> regionserver hosting the meta table region. Site configuration provides the
> client a list of ZK quorum peers and the client uses an embedded ZK client to
> query meta location. Timeouts and retry behavior of this embedded ZK client
> are managed orthogonally to HBase layer settings and in some cases the ZK
> cannot manage what in theory the HBase client can, i.e. fail fast upon outage
> or network partition.
> We should consider new configuration settings that provide a list of
> well-known master and backup master locations, and with this information the
> client can contact any of the master processes directly. Any master in either
> active or passive state will track meta location and respond to requests for
> it with its cached last known location. If this location is stale, the client
> can ask again with a flag set that requests the master refresh its location
> cache and return the up-to-date location. Every client interaction with the
> cluster thus uses only HBase RPC as transport, with appropriate settings
> applied to the connection. The configuration toggle that enables this
> alternative meta location lookup should be false by default.
> This removes the requirement that HBase clients embed the ZK client and
> contact the ZK service directly at the beginning of the connection lifecycle.
> This has several benefits. ZK service need not be exposed to clients, and
> their potential abuse, yet no benefit ZK provides the HBase server cluster is
> compromised. Normalizing HBase client and ZK client timeout settings and
> retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer
> necessary.
> And, from [~ghelmling]: There is an additional complication here for
> token-based authentication. When a delegation token is used for SASL
> authentication, the client uses the cluster ID obtained from Zookeeper to
> select the token identifier to use. So there would also need to be some
> Zookeeper-less, unauthenticated way to obtain the cluster ID as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23812) [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did not replicate enough rows expected:<10> but was:<7>
[
https://issues.apache.org/jira/browse/HBASE-23812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032789#comment-17032789
]
Michael Stack commented on HBASE-23812:
---
bq. You want to see it pass after repeated builds?
Yeah. Its a rare case. I want it to clear flakies
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html
or just stop showing up in my local runs. Day or two? Want something more
clean than this?
> [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did
> not replicate enough rows expected:<10> but was:<7>
> -
>
> Key: HBASE-23812
> URL: https://issues.apache.org/jira/browse/HBASE-23812
> Project: HBase
> Issue Type: Bug
> Components: flakies
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments:
> 0001-HBASE-23812-Flakey-Test-TestReplicator-testReplicato.patch
>
>
> I've seen this fail. Here is the latest:
> {code}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 21.044 s <<< FAILURE! - in
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator
> [ERROR]
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors
> Time elapsed: 7.139 s <<< FAILURE!
> java.lang.AssertionError: We did not replicate enough rows expected:<10> but
> was:<7>
>at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors(TestReplicator.java:158)
> {code}
> In test logs there are a few (in groups of three) of these:
> {code}
> 4247 2020-02-07 13:50:29,070 WARN
> [RS_REFRESH_PEER-regionserver/localhost:0-1.replicationSource,testReplicatorWithErrors.replicationSource.shipperlocalhost%2C49531%2C1581112194639,testReplicatorWithErrors]
> regionserver. ReplicationSourceShipper(223):
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$FailureInjectingReplicationEndpointForTest
> threw unknown exception:
> 4248 java.lang.ClassCastException:
> org.apache.hbase.thirdparty.com.google.protobuf.ServiceException cannot be
> cast to java.io.IOException
> 4249 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.parallelReplicate(HBaseInterClusterReplicationEndpoint.java:361)
> 4250 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:404)
> 4251 at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$ReplicationEndpointForTest.replicate(TestReplicator.java:228)
> 4252 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:188)
> 4253 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:118)
> {code}
> Let me fix this and see if it makes TestReplicator stable again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] jatsakthi commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
jatsakthi commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#issuecomment-583690878 Thanks for your reviews @esteban I have updated the PR. Now the old syntax is retained which supports the addition of more than 2 regions as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] jatsakthi commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
jatsakthi commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#discussion_r376678590 ## File path: hbase-shell/src/main/ruby/shell/commands/merge_region.rb ## @@ -33,16 +33,18 @@ def help Examples: - hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME' - hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME', true + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME'] + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ...] + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ...], true - hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME' - hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', true + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME'] + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ...] + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ...], true Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] jatsakthi commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
jatsakthi commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#discussion_r376678595 ## File path: hbase-shell/src/main/ruby/hbase/admin.rb ## @@ -535,14 +535,20 @@ def move(encoded_region_name, server = nil) end #-- -# Merge two regions -def merge_region(region_a_name, region_b_name, force) - @admin.mergeRegionsAsync( -region_a_name.to_java_bytes, -region_b_name.to_java_bytes, -java.lang.Boolean.valueOf(force) +# Merge multiple regions +def merge_region(regions = [], force) Review comment: Sure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on issue #1142: HBASE-23809 The RSGroup shell test is missing
Apache9 commented on issue #1142: HBASE-23809 The RSGroup shell test is missing URL: https://github.com/apache/hbase/pull/1142#issuecomment-583682820 Seems easy to fix, just add more region servers when start mini cluster so we have enough region servers to move between rs groups. Let's see if this works. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on issue #1094: HBASE-23744 - FastPathBalancedQueueRpcExecutor should enforce queue l…
ndimiduk commented on issue #1094: HBASE-23744 - FastPathBalancedQueueRpcExecutor should enforce queue l… URL: https://github.com/apache/hbase/pull/1094#issuecomment-583681720 I have the same question as @xcangCRM . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23747) Support HBase getconf in hbase shell similar to hdfs
[
https://issues.apache.org/jira/browse/HBASE-23747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032778#comment-17032778
]
Nick Dimiduk commented on HBASE-23747:
--
In the shell, so that's it's available from a ruby/shell script? It should
already be there. If you want to inspect on the command line, we already have
{{org.apache.hadoop.hbase.util.HBaseConfTool}}. You want to add an alias to the
{{bin/hbase}} command?
> Support HBase getconf in hbase shell similar to hdfs
>
>
> Key: HBASE-23747
> URL: https://issues.apache.org/jira/browse/HBASE-23747
> Project: HBase
> Issue Type: Improvement
>Reporter: Swaroopa Kadam
>Assignee: Swaroopa Kadam
>Priority: Minor
>
> Similar to HDFS getconf confkey -[key], HBase should provide a way to read
> configurations from HBase shell
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1147: HBASE-23814 Add null checks and logging to misc set of tests
ndimiduk commented on a change in pull request #1147: HBASE-23814 Add null checks and logging to misc set of tests URL: https://github.com/apache/hbase/pull/1147#discussion_r376672487 ## File path: hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableScanException.java ## @@ -74,6 +74,8 @@ private static volatile boolean DO_NOT_RETRY; + private static final int ROW_COUNT = 10; Review comment: > TestAsyncTableScanException use a define for row counts; count 100 instead of 1000 and see if helps You're off by an order of magnitude. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1147: HBASE-23814 Add null checks and logging to misc set of tests
ndimiduk commented on a change in pull request #1147: HBASE-23814 Add null checks and logging to misc set of tests URL: https://github.com/apache/hbase/pull/1147#discussion_r376672782 ## File path: hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableScanException.java ## @@ -74,6 +74,8 @@ private static volatile boolean DO_NOT_RETRY; + private static final int ROW_COUNT = 10; Review comment: Oh, you meant for the earlier test. Same question as @z-york asks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1147: HBASE-23814 Add null checks and logging to misc set of tests
ndimiduk commented on a change in pull request #1147: HBASE-23814 Add null
checks and logging to misc set of tests
URL: https://github.com/apache/hbase/pull/1147#discussion_r376672667
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java
##
@@ -267,9 +267,10 @@ public void testZooKeeperAbort() throws Exception {
public void testZooKeeperNormal() throws Exception {
ReplicationLogCleaner cleaner = new ReplicationLogCleaner();
+long modTime = System.currentTimeMillis() - 1000;
Review comment:
> TestLogsCleaner use single mod time. Make it for sure less than now in
case test runs all in the same millisecond (would cause test fail)
Mind adding this note as a comment here in the test as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-23812) [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did not replicate enough rows expected:<10> but was:<7>
[
https://issues.apache.org/jira/browse/HBASE-23812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032777#comment-17032777
]
Nick Dimiduk commented on HBASE-23812:
--
bq. Let me push what I have and leave this open to see if it helps.
Pre-commit here or on a PR isn't good enough to track the problem? You want to
see it pass after repeated builds?
> [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did
> not replicate enough rows expected:<10> but was:<7>
> -
>
> Key: HBASE-23812
> URL: https://issues.apache.org/jira/browse/HBASE-23812
> Project: HBase
> Issue Type: Bug
> Components: flakies
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments:
> 0001-HBASE-23812-Flakey-Test-TestReplicator-testReplicato.patch
>
>
> I've seen this fail. Here is the latest:
> {code}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 21.044 s <<< FAILURE! - in
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator
> [ERROR]
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors
> Time elapsed: 7.139 s <<< FAILURE!
> java.lang.AssertionError: We did not replicate enough rows expected:<10> but
> was:<7>
>at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors(TestReplicator.java:158)
> {code}
> In test logs there are a few (in groups of three) of these:
> {code}
> 4247 2020-02-07 13:50:29,070 WARN
> [RS_REFRESH_PEER-regionserver/localhost:0-1.replicationSource,testReplicatorWithErrors.replicationSource.shipperlocalhost%2C49531%2C1581112194639,testReplicatorWithErrors]
> regionserver. ReplicationSourceShipper(223):
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$FailureInjectingReplicationEndpointForTest
> threw unknown exception:
> 4248 java.lang.ClassCastException:
> org.apache.hbase.thirdparty.com.google.protobuf.ServiceException cannot be
> cast to java.io.IOException
> 4249 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.parallelReplicate(HBaseInterClusterReplicationEndpoint.java:361)
> 4250 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:404)
> 4251 at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$ReplicationEndpointForTest.replicate(TestReplicator.java:228)
> 4252 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:188)
> 4253 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:118)
> {code}
> Let me fix this and see if it makes TestReplicator stable again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1116: HBASE-23783: Address tests writing and reading SSL/Security files in …
ndimiduk commented on a change in pull request #1116: HBASE-23783: Address
tests writing and reading SSL/Security files in …
URL: https://github.com/apache/hbase/pull/1116#discussion_r376671321
##
File path: pom.xml
##
@@ -729,6 +729,8 @@
${surefire.skipFirstPart}
${surefire.firstPartForkCount}
false
+${surefire.reportsDirectory}
+${surefire.tempDir}
Review comment:
According to the docs, it can already be overridden via the `tempDir`
property. that not working for you?
http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#tempDir
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-23765) unable to communicate with localhost when setting up Kerberos-related tests
[
https://issues.apache.org/jira/browse/HBASE-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032775#comment-17032775
]
Nick Dimiduk commented on HBASE-23765:
--
*sigh* the attached is from JDK8. And the problem has passed again -- test
passes on both JDK8 and JDK11. This is how I've been running.
{noformat}
$ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 mvn clean install -DskipTests
-Dhadoop.profile=3.0 && JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 mvn -pl
hbase-rest clean test -Dtest=TestSecureRESTServer -Dhadoop.profile=3.0
$ JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 mvn clean install -DskipTests
-Dhadoop.profile=3.0 && JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 mvn -pl
hbase-rest clean test -Dtest=TestSecureRESTServer -Dhadoop.profile=3.0
{noformat}
> unable to communicate with localhost when setting up Kerberos-related tests
> ---
>
> Key: HBASE-23765
> URL: https://issues.apache.org/jira/browse/HBASE-23765
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Josh Elser
>Priority: Minor
> Attachments:
> TEST-org.apache.hadoop.hbase.rest.TestSecureRESTServer.xml
>
>
> [~ndimiduk] gave an ask over on HBASE-23760 to change some of the
> Hadoop-level configuration properties around secure cluster setup from
> localhost to 127.0.0.1. He and [~bharathv] have been chasing some issues with
> ZooKeeper and not having a resolution of localhost to 127.0.0.1.
> Before I start making a change, how sure are we that this is an issue?
> Assuming that it's the nightlies that we see these on, how about we make a
> change to increase the krb5 and spnego debugging to see if we aren't
> resolving names properly?
> There might be a debug property for DNS lookups in Java too maybe?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23765) unable to communicate with localhost when setting up Kerberos-related tests
[ https://issues.apache.org/jira/browse/HBASE-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23765: - Summary: unable to communicate with localhost when setting up Kerberos-related tests (was: [JDK11] unable to communicate with localhost when setting up Kerberos-related tests) > unable to communicate with localhost when setting up Kerberos-related tests > --- > > Key: HBASE-23765 > URL: https://issues.apache.org/jira/browse/HBASE-23765 > Project: HBase > Issue Type: Task > Components: test >Reporter: Josh Elser >Priority: Minor > Attachments: > TEST-org.apache.hadoop.hbase.rest.TestSecureRESTServer.xml > > > [~ndimiduk] gave an ask over on HBASE-23760 to change some of the > Hadoop-level configuration properties around secure cluster setup from > localhost to 127.0.0.1. He and [~bharathv] have been chasing some issues with > ZooKeeper and not having a resolution of localhost to 127.0.0.1. > Before I start making a change, how sure are we that this is an issue? > Assuming that it's the nightlies that we see these on, how about we make a > change to increase the krb5 and spnego debugging to see if we aren't > resolving names properly? > There might be a debug property for DNS lookups in Java too maybe? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] z-york commented on issue #1145: HBASE-23813 Get surefire version from parent pom rather than define i…
z-york commented on issue #1145: HBASE-23813 Get surefire version from parent pom rather than define i… URL: https://github.com/apache/hbase/pull/1145#issuecomment-583675687 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #934: HBASE-23568: Improve Threading of Replication
bharathv commented on a change in pull request #934: HBASE-23568: Improve
Threading of Replication
URL: https://github.com/apache/hbase/pull/934#discussion_r37733
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReader.java
##
@@ -409,17 +413,7 @@ private boolean acquireBufferQuota(long size) {
return totalBufferUsed.addAndGet(size) >= totalBufferQuota;
}
- /**
- * @return whether the reader thread is running
- */
- public boolean isReaderRunning() {
-return isReaderRunning && !isInterrupted();
Review comment:
Shouldn't it also be looking for an interrupt? (like in a thread pool
shutdown).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #934: HBASE-23568: Improve Threading of Replication
bharathv commented on a change in pull request #934: HBASE-23568: Improve
Threading of Replication
URL: https://github.com/apache/hbase/pull/934#discussion_r376665954
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -690,13 +705,8 @@ public ReplicationQueueInfo getReplicationQueueInfo() {
return replicationQueueInfo;
}
- public boolean isWorkerRunning(){
-for(ReplicationSourceShipper worker : this.workerThreads.values()){
- if(worker.isActive()){
-return worker.isActive();
- }
-}
-return false;
+ public boolean isWorkerRunning() {
+return !this.workerThreads.isEmpty();
Review comment:
isn't this a change of semantics? (there could be window where it isn't
active but still in the map..). Doesn't probably matter that much but just
wondering..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #934: HBASE-23568: Improve Threading of Replication
bharathv commented on a change in pull request #934: HBASE-23568: Improve
Threading of Replication
URL: https://github.com/apache/hbase/pull/934#discussion_r376665200
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -585,50 +626,24 @@ public void terminate(String reason, Exception cause,
boolean clearMetrics, bool
initThread.interrupt();
Threads.shutdown(initThread, this.sleepForRetries);
}
-Collection workers = workerThreads.values();
-for (ReplicationSourceShipper worker : workers) {
- worker.stopWorker();
- if(worker.entryReader != null) {
-worker.entryReader.setReaderRunning(false);
- }
-}
-for (ReplicationSourceShipper worker : workers) {
- if (worker.isAlive() || worker.entryReader.isAlive()) {
-try {
- // Wait worker to stop
- Thread.sleep(this.sleepForRetries);
-} catch (InterruptedException e) {
- LOG.info("{} Interrupted while waiting {} to stop", logPeerId(),
worker.getName());
- Thread.currentThread().interrupt();
-}
-// If worker still is alive after waiting, interrupt it
-if (worker.isAlive()) {
- worker.interrupt();
-}
-// If entry reader is alive after waiting, interrupt it
-if (worker.entryReader.isAlive()) {
- worker.entryReader.interrupt();
-}
- }
+this.workerThreads.values().forEach(worker -> worker.stopWorker());
+
+final boolean shutdownSuccess =
MoreExecutors.shutdownAndAwaitTermination(this.executorService,
+ workerThreads.values().size() * this.sleepForRetries, TimeUnit.SECONDS);
+if (!shutdownSuccess) {
+ LOG.info("{} Unable to shutdown thread pool completely", logPeerId());
Review comment:
nit: Shouldn't this be a warn?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #934: HBASE-23568: Improve Threading of Replication
bharathv commented on a change in pull request #934: HBASE-23568: Improve
Threading of Replication
URL: https://github.com/apache/hbase/pull/934#discussion_r376664834
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -304,31 +318,56 @@ private void initializeWALEntryFilter(UUID
peerClusterId) {
this.walEntryFilter = new ChainWALEntryFilter(filters);
}
- private void tryStartNewShipper(String walGroupId,
PriorityBlockingQueue queue) {
-workerThreads.compute(walGroupId, (key, value) -> {
- if (value != null) {
-if (LOG.isDebugEnabled()) {
- LOG.debug(
- "{} Someone has beat us to start a worker thread for wal group
{}",
- logPeerId(), key);
-}
-return value;
- } else {
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} Starting up worker for wal group {}", logPeerId(),
key);
+ /**
+ * Synchronized method so that only one item is inserted at a time. Should
be the only place that
Review comment:
Oh man, this is super subtle. While I agree that there won't be considerable
perf drop (since most methods are just spinning up threads and not doing heavy
weight stuff), I still didn't understand how it can cause a deadlock.
> However, in the original code, the main thread was running within the lock
of the Map, so trying to then remove an entry from the Map that is already had
a lock on caused a deadlock.
I agree that this was in the original implementation, it was inside a
"compute()", so it grabbed the lock and it would deadlock. But with your
change, you are not inside a lock scope right? (or did I miss something). IIUC,
you are saying "workerThreads.remove(walGroupId);" deadlocks with something.
Whats that something?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #934: HBASE-23568: Improve Threading of Replication
bharathv commented on a change in pull request #934: HBASE-23568: Improve Threading of Replication URL: https://github.com/apache/hbase/pull/934#discussion_r376661463 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java ## @@ -126,10 +134,16 @@ private long defaultBandwidth; private long currentBandwidth; private WALFileLengthProvider walFileLengthProvider; + @VisibleForTesting - protected final ConcurrentHashMap workerThreads = + protected final Map workerThreads = new ConcurrentHashMap<>(); + private final ListeningExecutorService executorService = Review comment: This could use some comments (what is in this executor service?).. While you are here, mind adding some javadocs for the workerThreads too? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #934: HBASE-23568: Improve Threading of Replication
bharathv commented on a change in pull request #934: HBASE-23568: Improve
Threading of Replication
URL: https://github.com/apache/hbase/pull/934#discussion_r376661329
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -126,10 +134,16 @@
private long defaultBandwidth;
private long currentBandwidth;
private WALFileLengthProvider walFileLengthProvider;
+
@VisibleForTesting
- protected final ConcurrentHashMap
workerThreads =
+ protected final Map workerThreads =
new ConcurrentHashMap<>();
+ private final ListeningExecutorService executorService =
+ MoreExecutors.listeningDecorator(Executors.newCachedThreadPool(
+new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("replicationSource-%d")
Review comment:
I think there is some loss of debugging information here. Earlier the worker
and reader threads were annotated with walGroupId and stuff, any chance to get
that back? I think that is some useful information in jstacks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (HBASE-23765) [JDK11] unable to communicate with localhost when setting up Kerberos-related tests
[
https://issues.apache.org/jira/browse/HBASE-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23765:
-
Attachment: TEST-org.apache.hadoop.hbase.rest.TestSecureRESTServer.xml
Here's a run from master ({{d554689cc4}}) with JDK11 and Hadoop3 profile. The
Hadoop profile gets me a bit farther. See the region server unable to
communicate with the master, both on localhost. I've added
{{-Dsun.security.krb5.debug=true}} as well.
> [JDK11] unable to communicate with localhost when setting up Kerberos-related
> tests
> ---
>
> Key: HBASE-23765
> URL: https://issues.apache.org/jira/browse/HBASE-23765
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Josh Elser
>Priority: Minor
> Attachments:
> TEST-org.apache.hadoop.hbase.rest.TestSecureRESTServer.xml
>
>
> [~ndimiduk] gave an ask over on HBASE-23760 to change some of the
> Hadoop-level configuration properties around secure cluster setup from
> localhost to 127.0.0.1. He and [~bharathv] have been chasing some issues with
> ZooKeeper and not having a resolution of localhost to 127.0.0.1.
> Before I start making a change, how sure are we that this is an issue?
> Assuming that it's the nightlies that we see these on, how about we make a
> change to increase the krb5 and spnego debugging to see if we aren't
> resolving names properly?
> There might be a debug property for DNS lookups in Java too maybe?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23765) [JDK11] unable to communicate with localhost when setting up Kerberos-related tests
[ https://issues.apache.org/jira/browse/HBASE-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23765: - Summary: [JDK11] unable to communicate with localhost when setting up Kerberos-related tests (was: [JDK11] Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests) > [JDK11] unable to communicate with localhost when setting up Kerberos-related > tests > --- > > Key: HBASE-23765 > URL: https://issues.apache.org/jira/browse/HBASE-23765 > Project: HBase > Issue Type: Task > Components: test >Reporter: Josh Elser >Priority: Minor > > [~ndimiduk] gave an ask over on HBASE-23760 to change some of the > Hadoop-level configuration properties around secure cluster setup from > localhost to 127.0.0.1. He and [~bharathv] have been chasing some issues with > ZooKeeper and not having a resolution of localhost to 127.0.0.1. > Before I start making a change, how sure are we that this is an issue? > Assuming that it's the nightlies that we see these on, how about we make a > change to increase the krb5 and spnego debugging to see if we aren't > resolving names properly? > There might be a debug property for DNS lookups in Java too maybe? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23765) [JDK11] Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests
[ https://issues.apache.org/jira/browse/HBASE-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23765: - Summary: [JDK11] Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests (was: Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests) > [JDK11] Use 127.0.0.1 instead of localhost when setting up Kerberos-related > tests > - > > Key: HBASE-23765 > URL: https://issues.apache.org/jira/browse/HBASE-23765 > Project: HBase > Issue Type: Task > Components: test >Reporter: Josh Elser >Priority: Minor > > [~ndimiduk] gave an ask over on HBASE-23760 to change some of the > Hadoop-level configuration properties around secure cluster setup from > localhost to 127.0.0.1. He and [~bharathv] have been chasing some issues with > ZooKeeper and not having a resolution of localhost to 127.0.0.1. > Before I start making a change, how sure are we that this is an issue? > Assuming that it's the nightlies that we see these on, how about we make a > change to increase the krb5 and spnego debugging to see if we aren't > resolving names properly? > There might be a debug property for DNS lookups in Java too maybe? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23779: -- Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Pushed the patch to branch-2 and master. Then spent time playing w/ the changes made to hbase-personality seeing if I could speed jenkins even more by adding the -T flag on each jenkins invocation. Took a few ADDENDUMs which show in the master log. In the end, the -T seems to destabilize jenkins build so the last ADDENDUM reverts changes made to hbase-personality in both master and branch-2.3. All the rest of the patch making forkcount a factor of cpu count remains. Resolving. > Up the default fork count to make builds complete faster; make count relative > to CPU count > -- > > Key: HBASE-23779 > URL: https://issues.apache.org/jira/browse/HBASE-23779 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: addendum2.patch, test_yetus_934.0.patch > > > Tests take a long time. Our fork count running all tests are conservative -- > 1 (small) for first part and 5 for second part (medium and large). Rather > than hardcoding we should set the fork count to be relative to machine size. > Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box. > Looking up at jenkins, it seems like the boxes are 24 cores... at least going > by my random survey. The load reported on a few seems low though this not > representative (looking at machine/uptime). > More parallelism willl probably mean more test failure. Let me take a look > see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1145: HBASE-23813 Get surefire version from parent pom rather than define i…
Apache-HBase commented on issue #1145: HBASE-23813 Get surefire version from parent pom rather than define i… URL: https://github.com/apache/hbase/pull/1145#issuecomment-583665389 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 48s | branch-2 passed | | +1 :green_heart: | compile | 0m 40s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 29s | the patch passed | | +1 :green_heart: | compile | 0m 39s | the patch passed | | +1 :green_heart: | javac | 0m 39s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedjars | 4m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 17s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 45s | hbase-client-project in the patch passed. | | +1 :green_heart: | unit | 0m 46s | hbase-shaded-client-project in the patch passed. | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 50m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1145/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1145 | | Optional Tests | dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile | | uname | Linux 0891dce3324f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1145/out/precommit/personality/provided.sh | | git revision | branch-2 / 29fb52b5d2 | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1145/1/testReport/ | | Max. process+thread count | 432 (vs. ulimit of 1) | | modules | C: hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project U: hbase-archetypes | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1145/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] z-york commented on a change in pull request #1147: HBASE-23814 Add null checks and logging to misc set of tests
z-york commented on a change in pull request #1147: HBASE-23814 Add null checks and logging to misc set of tests URL: https://github.com/apache/hbase/pull/1147#discussion_r376659358 ## File path: hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableRegionReplicasScan.java ## @@ -39,7 +39,7 @@ public static final HBaseClassTestRule CLASS_RULE = HBaseClassTestRule.forClass(TestAsyncTableRegionReplicasScan.class); - private static int ROW_COUNT = 1000; + private static int ROW_COUNT = 100; Review comment: What's the rationale for reducing row counts for the Async tests? Decrease runtime? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032751#comment-17032751 ] Michael Stack commented on HBASE-23779: --- Ok. An the next run, https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2/2458/artifact/, hadoop2 test got cut off as in previous run. hadoop3 test made it to the end but a bunch of them timedout w/ process exit complaint. Reverting the changes in dev-support/hbase-personality.sh from my patch where I try to add -T0.5C to all mvn invocations. Will leave it as default single-threaded for now. > Up the default fork count to make builds complete faster; make count relative > to CPU count > -- > > Key: HBASE-23779 > URL: https://issues.apache.org/jira/browse/HBASE-23779 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: addendum2.patch, test_yetus_934.0.patch > > > Tests take a long time. Our fork count running all tests are conservative -- > 1 (small) for first part and 5 for second part (medium and large). Rather > than hardcoding we should set the fork count to be relative to machine size. > Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box. > Looking up at jenkins, it seems like the boxes are 24 cores... at least going > by my random survey. The load reported on a few seems low though this not > representative (looking at machine/uptime). > More parallelism willl probably mean more test failure. Let me take a look > see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23811) [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032743#comment-17032743 ] Michael Stack commented on HBASE-23811: --- The gson dependency is problematic. HBase uses shaded gson in thirdparty. kotlin? Sure you didn't forget the scala include (smile). For a lib that is supposed to be low-impact, seems strange dragging in kotlin but what do I know. The thrift dependency is a problem when it doesn't agree w/ what we ship. Yeah, add to thirdparty I'd say to be safe sir. Shout if issues. > [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty > - > > Key: HBASE-23811 > URL: https://issues.apache.org/jira/browse/HBASE-23811 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > JaegerTracing pulls in lots of dependencies. Some, like libthrift (0.13.0) > conflicts the one ships in HBase (0.12.0). > Additionally, not everyone may want to use Jaeger. > I propose to shade JaegerTracing and its dependencies into an uber jar, place > it as a hbase-thirdparty artifact. As an added benefit, this makes the > management of tracers in the HBase's dependency tree much easier. Finally, we > can follow the same suit and provide Zipkin tracer support in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack opened a new pull request #1147: HBASE-23814 Add null checks and logging to misc set of tests
saintstack opened a new pull request #1147: HBASE-23814 Add null checks and logging to misc set of tests URL: https://github.com/apache/hbase/pull/1147 * TestFullLogReconstruction log the server we've chosen to expire and then note where we starting counting rows * TestAsyncTableScanException use a define for row counts; count 100 instead of 1000 and see if helps * TestRawAsyncTableLimitedScanWithFilter check connection was made before closing it in tearDown * TestLogsCleaner use single mod time. Make it for sure less than now in case test runs all in the same millisecond (would cause test fail) * TestReplicationBase test table is non-null before closing in tearDown This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23765) Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests
[
https://issues.apache.org/jira/browse/HBASE-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032735#comment-17032735
]
Nick Dimiduk commented on HBASE-23765:
--
{noformat}
vagrant@bionic:~/repos/hbase$ cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
127.0.1.1 ubuntu-bionic ubuntu-bionic
{noformat}
> Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests
> -
>
> Key: HBASE-23765
> URL: https://issues.apache.org/jira/browse/HBASE-23765
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Josh Elser
>Priority: Minor
>
> [~ndimiduk] gave an ask over on HBASE-23760 to change some of the
> Hadoop-level configuration properties around secure cluster setup from
> localhost to 127.0.0.1. He and [~bharathv] have been chasing some issues with
> ZooKeeper and not having a resolution of localhost to 127.0.0.1.
> Before I start making a change, how sure are we that this is an issue?
> Assuming that it's the nightlies that we see these on, how about we make a
> change to increase the krb5 and spnego debugging to see if we aren't
> resolving names properly?
> There might be a debug property for DNS lookups in Java too maybe?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23814) Add null checks and logging to misc set of tests
Michael Stack created HBASE-23814: - Summary: Add null checks and logging to misc set of tests Key: HBASE-23814 URL: https://issues.apache.org/jira/browse/HBASE-23814 Project: HBase Issue Type: Bug Reporter: Michael Stack I've been studying unit tests of late. A few are failing but then the output is missing a detail or shutdown complains of NPE because startup didn't succeed. Here are super minor items I've been carrying around that I'd like to land. They do not change the function of tests (there is an attempt at a fix of TestLogsCleaner). * TestFullLogReconstruction log the server we've chosen to expire and then note where we starting counting rows * TestAsyncTableScanException use a define for row counts; count 100 instead of 1000 and see if helps * TestRawAsyncTableLimitedScanWithFilter check connection was made before closing it in tearDown * TestLogsCleaner use single mod time. Make it for sure less than now in case test runs all in the same millisecond (would cause test fail) * TestReplicationBase test table is non-null before closing in tearDown -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23765) Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests
[ https://issues.apache.org/jira/browse/HBASE-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032731#comment-17032731 ] Nick Dimiduk commented on HBASE-23765: -- Does not repro with JDK8. > Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests > - > > Key: HBASE-23765 > URL: https://issues.apache.org/jira/browse/HBASE-23765 > Project: HBase > Issue Type: Task > Components: test >Reporter: Josh Elser >Priority: Minor > > [~ndimiduk] gave an ask over on HBASE-23760 to change some of the > Hadoop-level configuration properties around secure cluster setup from > localhost to 127.0.0.1. He and [~bharathv] have been chasing some issues with > ZooKeeper and not having a resolution of localhost to 127.0.0.1. > Before I start making a change, how sure are we that this is an issue? > Assuming that it's the nightlies that we see these on, how about we make a > change to increase the krb5 and spnego debugging to see if we aren't > resolving names properly? > There might be a debug property for DNS lookups in Java too maybe? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] brfrn169 commented on a change in pull request #1114: HBASE-23146 Support CheckAndMutate with multi conditions
brfrn169 commented on a change in pull request #1114: HBASE-23146 Support
CheckAndMutate with multi conditions
URL: https://github.com/apache/hbase/pull/1114#discussion_r376649523
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTable.java
##
@@ -233,6 +234,28 @@
*/
CheckAndMutateBuilder checkAndMutate(byte[] row, byte[] family);
+ /**
+ * Atomically checks if a row matches the specified filter. If it does, it
adds the
+ * Put/Delete/RowMutations.
+ *
+ * Use the returned {@link CheckAndMutateBuilder} to construct your request
and then execute it.
+ * This is a fluent style API, the code is like:
+ *
+ *
+ *
+ * table.checkAndMutate(row).ifMatches(filter).thenPut(put)
+ * .thenAccept(succ -> {
+ * if (succ) {
+ * System.out.println("Check and put succeeded");
+ * } else {
+ * System.out.println("Check and put failed");
+ * }
+ * });
+ *
+ *
+ */
+ CheckAndMutateBuilder checkAndMutate(byte[] row);
Review comment:
@Apache9
One of the reason why I decided the design of the new API here is that I
wanted to keep the compatibility of the existing API. This patch also involves
coprocessor changes, and I didn't want to change the existing methods and I
just wanted to add new methods.
If you still think the API is strange, I can change it.
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] brfrn169 commented on a change in pull request #1114: HBASE-23146 Support CheckAndMutate with multi conditions
brfrn169 commented on a change in pull request #1114: HBASE-23146 Support
CheckAndMutate with multi conditions
URL: https://github.com/apache/hbase/pull/1114#discussion_r376649523
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTable.java
##
@@ -233,6 +234,28 @@
*/
CheckAndMutateBuilder checkAndMutate(byte[] row, byte[] family);
+ /**
+ * Atomically checks if a row matches the specified filter. If it does, it
adds the
+ * Put/Delete/RowMutations.
+ *
+ * Use the returned {@link CheckAndMutateBuilder} to construct your request
and then execute it.
+ * This is a fluent style API, the code is like:
+ *
+ *
+ *
+ * table.checkAndMutate(row).ifMatches(filter).thenPut(put)
+ * .thenAccept(succ -> {
+ * if (succ) {
+ * System.out.println("Check and put succeeded");
+ * } else {
+ * System.out.println("Check and put failed");
+ * }
+ * });
+ *
+ *
+ */
+ CheckAndMutateBuilder checkAndMutate(byte[] row);
Review comment:
@Apache9
One of the reasons why I decided the design of the new API here is that I
wanted to keep the compatibility of the existing API. This patch also involves
coprocessor changes, and I didn't want to change the existing methods and I
just wanted to add new methods.
If you still think the API is strange, I can change it.
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-23765) Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests
[
https://issues.apache.org/jira/browse/HBASE-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032729#comment-17032729
]
Nick Dimiduk commented on HBASE-23765:
--
Okay! I couldn't repro this at the beginning of the week and was resigned to
giving up. By trying once more, now it's failing again.
{noformat}
vagrant@bionic:~/repos/hbase$ JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 mvn
clean install -DskipTests
...
[INFO]
[INFO] BUILD SUCCESS
[INFO]
vagrant@bionic:~/repos/hbase$ JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 mvn
-pl hbase-rest test -Dtest=TestSecureRESTServer
...
[INFO] Running org.apache.hadoop.hbase.rest.TestSecureRESTServer
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 34.795
s <<< FAILURE! - in org.apache.hadoop.hbase.rest.TestSecureRESTServer
[ERROR] org.apache.hadoop.hbase.rest.TestSecureRESTServer Time elapsed: 34.787
s <<< ERROR!
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
setup connection for hbase/localh...@example.com to localhost/127.0.0.1:33483;
Host Details : local host is: "bionic/10.0.2.15
"; destination host is: "localhost":33483;
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
Caused by: java.io.IOException: Couldn't setup connection for
hbase/localh...@example.com to localhost/127.0.0.1:33483
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
Caused by: javax.security.sasl.SaslException: GSS initiate failed
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism
level: Message stream modified (41) - Message stream modified)
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
Caused by: sun.security.krb5.KrbException: Message stream modified (41) -
Message stream modified
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected
value (906)
at
org.apache.hadoop.hbase.rest.TestSecureRESTServer.setupServer(TestSecureRESTServer.java:193)
{noformat}
> Use 127.0.0.1 instead of localhost when setting up Kerberos-related tests
> -
>
> Key: HBASE-23765
> URL: https://issues.apache.org/jira/browse/HBASE-23765
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Josh Elser
>Priority: Minor
>
> [~ndimiduk] gave an ask over on HBASE-23760 to change some of the
> Hadoop-level configuration properties around secure cluster setup from
> localhost to 127.0.0.1. He and [~bharathv] have been chasing some issues with
> ZooKeeper and not having a resolution of localhost to 127.0.0.1.
> Before I start making a change, how sure are we that this is an issue?
> Assuming that it's the nightlies that we see these on, how about we make a
> change to increase the krb5 and spnego debugging to see if we aren't
> resolving names properly?
> There might be a debug property for DNS lookups in Java too maybe?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23781) Remove deprecated createTableDescriptor(String) from HBaseTestingUtility
[
https://issues.apache.org/jira/browse/HBASE-23781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Hentschel updated HBASE-23781:
--
Status: Patch Available (was: In Progress)
> Remove deprecated createTableDescriptor(String) from HBaseTestingUtility
>
>
> Key: HBASE-23781
> URL: https://issues.apache.org/jira/browse/HBASE-23781
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
>
> {{createTableDescriptor(String)}} in {{HBaseTestingUtility}} was deprecated
> back in 2.0.0 and should be removed for 3.0.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] HorizonNet opened a new pull request #1146: HBASE-23781 Removed deprecated createTableDescriptor(String) from HBaseTestingUtility
HorizonNet opened a new pull request #1146: HBASE-23781 Removed deprecated createTableDescriptor(String) from HBaseTestingUtility URL: https://github.com/apache/hbase/pull/1146 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23811) [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032727#comment-17032727
]
Wei-Chiu Chuang commented on HBASE-23811:
-
For JaegerTracing 1.1.0, there are:
{noformat}
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hbase-jaeger ---
[INFO] org.apache.hbase.thirdparty:hbase-jaeger:jar:3.2.1-SNAPSHOT
[INFO] +- io.jaegertracing:jaeger-client:jar:1.1.0:compile
[INFO] | +- io.jaegertracing:jaeger-core:jar:1.1.0:compile
[INFO] | | +- io.opentracing:opentracing-api:jar:0.33.0:compile
[INFO] | | +- io.opentracing:opentracing-util:jar:0.33.0:compile
[INFO] | | | \- io.opentracing:opentracing-noop:jar:0.33.0:compile
[INFO] | | \- com.google.code.gson:gson:jar:2.8.6:compile
[INFO] | \- io.jaegertracing:jaeger-tracerresolver:jar:1.1.0:compile
[INFO] | \-
io.opentracing.contrib:opentracing-tracerresolver:jar:0.1.8:compile
[INFO] +- io.jaegertracing:jaeger-thrift:jar:1.1.0:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.28:compile
[INFO] | \- com.squareup.okhttp3:okhttp:jar:4.2.2:compile
[INFO] | +- com.squareup.okio:okio:jar:2.2.2:compile
[INFO] | \- org.jetbrains.kotlin:kotlin-stdlib:jar:1.3.50:compile
[INFO] |+- org.jetbrains.kotlin:kotlin-stdlib-common:jar:1.3.50:compile
[INFO] |\- org.jetbrains:annotations:jar:13.0:compile
[INFO] \- org.apache.thrift:libthrift:jar:0.13.0:compile
[INFO]+- org.apache.httpcomponents:httpclient:jar:4.5.6:compile
[INFO]| +- commons-logging:commons-logging:jar:1.2:compile
[INFO]| \- commons-codec:commons-codec:jar:1.10:compile
[INFO]+- org.apache.httpcomponents:httpcore:jar:4.4.1:compile
[INFO]\- javax.annotation:javax.annotation-api:jar:1.3.2:compile
{noformat}
My uber jar removes a bunch of these because hbase already ships with these
dependencies and works ok so far:
{noformat}
org.apache.httpcomponents:httpclient
commons-logging:commons-logging
commons-codec:commons-codec
org.apache.httpcomponents:httpcore
javax.annotation:javax.annotation-api
org.slf4j:slf4j-api
{noformat}
I suspect hbase-thirdparty is a better home for this uber jar but i'm happy to
relocate as long as it makes sense.
> [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty
> -
>
> Key: HBASE-23811
> URL: https://issues.apache.org/jira/browse/HBASE-23811
> Project: HBase
> Issue Type: Sub-task
> Components: thirdparty
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
>
> JaegerTracing pulls in lots of dependencies. Some, like libthrift (0.13.0)
> conflicts the one ships in HBase (0.12.0).
> Additionally, not everyone may want to use Jaeger.
> I propose to shade JaegerTracing and its dependencies into an uber jar, place
> it as a hbase-thirdparty artifact. As an added benefit, this makes the
> management of tracers in the HBase's dependency tree much easier. Finally, we
> can follow the same suit and provide Zipkin tracer support in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack opened a new pull request #1145: HBASE-23813 Get surefire version from parent pom rather than define i…
saintstack opened a new pull request #1145: HBASE-23813 Get surefire version from parent pom rather than define i… URL: https://github.com/apache/hbase/pull/1145 …t in hbase-archetypes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack opened a new pull request #1144: Hbase 23813
saintstack opened a new pull request #1144: Hbase 23813 URL: https://github.com/apache/hbase/pull/1144 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23813) Get surefire version from parent pom rather than define it in hbase-archetypes
[
https://issues.apache.org/jira/browse/HBASE-23813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032726#comment-17032726
]
Michael Stack commented on HBASE-23813:
---
Just does this:
{code}
diff --git a/hbase-archetypes/hbase-client-project/pom.xml
b/hbase-archetypes/hbase-client-project/pom.xml
index 8327e9ab95..4cce0003fe 100644
--- a/hbase-archetypes/hbase-client-project/pom.xml
+++ b/hbase-archetypes/hbase-client-project/pom.xml
@@ -37,8 +37,6 @@
UTF-8
${compileSource}
${compileSource}
-2.19
-4.12
diff --git a/hbase-archetypes/hbase-shaded-client-project/pom.xml
b/hbase-archetypes/hbase-shaded-client-project/pom.xml
index eef7934401..a10de2c5d1 100644
--- a/hbase-archetypes/hbase-shaded-client-project/pom.xml
+++ b/hbase-archetypes/hbase-shaded-client-project/pom.xml
@@ -37,8 +37,6 @@
UTF-8
${compileSource}
${compileSource}
-2.19
-4.12
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/Replicati
{code}
> Get surefire version from parent pom rather than define it in hbase-archetypes
> --
>
> Key: HBASE-23813
> URL: https://issues.apache.org/jira/browse/HBASE-23813
> Project: HBase
> Issue Type: Bug
>Reporter: Michael Stack
>Priority: Trivial
>
> Minor, minor issue. Remove version specification when should be inheriting
> from parent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23813) Get surefire version from parent pom rather than define it in hbase-archetypes
Michael Stack created HBASE-23813: - Summary: Get surefire version from parent pom rather than define it in hbase-archetypes Key: HBASE-23813 URL: https://issues.apache.org/jira/browse/HBASE-23813 Project: HBase Issue Type: Bug Reporter: Michael Stack Minor, minor issue. Remove version specification when should be inheriting from parent. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] markrmiller commented on a change in pull request #1116: HBASE-23783: Address tests writing and reading SSL/Security files in …
markrmiller commented on a change in pull request #1116: HBASE-23783: Address
tests writing and reading SSL/Security files in …
URL: https://github.com/apache/hbase/pull/1116#discussion_r376645511
##
File path: pom.xml
##
@@ -729,6 +729,8 @@
${surefire.skipFirstPart}
${surefire.firstPartForkCount}
false
+${surefire.reportsDirectory}
+${surefire.tempDir}
Review comment:
It was initially just meant to be harmless - it allows you to override via
system property but does not yet try to manage tmp dirs internally. I'll file a
small follow up issue to look at that. Just wanted fast, safe way to override
without patching.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-23812) [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did not replicate enough rows expected:<10> but was:<7>
[
https://issues.apache.org/jira/browse/HBASE-23812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032724#comment-17032724
]
Michael Stack commented on HBASE-23812:
---
Attached patch pushed on branch-2. Leaving issue open to see if this is the fix.
> [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did
> not replicate enough rows expected:<10> but was:<7>
> -
>
> Key: HBASE-23812
> URL: https://issues.apache.org/jira/browse/HBASE-23812
> Project: HBase
> Issue Type: Bug
> Components: flakies
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments:
> 0001-HBASE-23812-Flakey-Test-TestReplicator-testReplicato.patch
>
>
> I've seen this fail. Here is the latest:
> {code}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 21.044 s <<< FAILURE! - in
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator
> [ERROR]
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors
> Time elapsed: 7.139 s <<< FAILURE!
> java.lang.AssertionError: We did not replicate enough rows expected:<10> but
> was:<7>
>at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors(TestReplicator.java:158)
> {code}
> In test logs there are a few (in groups of three) of these:
> {code}
> 4247 2020-02-07 13:50:29,070 WARN
> [RS_REFRESH_PEER-regionserver/localhost:0-1.replicationSource,testReplicatorWithErrors.replicationSource.shipperlocalhost%2C49531%2C1581112194639,testReplicatorWithErrors]
> regionserver. ReplicationSourceShipper(223):
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$FailureInjectingReplicationEndpointForTest
> threw unknown exception:
> 4248 java.lang.ClassCastException:
> org.apache.hbase.thirdparty.com.google.protobuf.ServiceException cannot be
> cast to java.io.IOException
> 4249 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.parallelReplicate(HBaseInterClusterReplicationEndpoint.java:361)
> 4250 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:404)
> 4251 at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$ReplicationEndpointForTest.replicate(TestReplicator.java:228)
> 4252 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:188)
> 4253 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:118)
> {code}
> Let me fix this and see if it makes TestReplicator stable again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23812) [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did not replicate enough rows expected:<10> but was:<7>
[
https://issues.apache.org/jira/browse/HBASE-23812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-23812:
--
Attachment: 0001-HBASE-23812-Flakey-Test-TestReplicator-testReplicato.patch
> [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did
> not replicate enough rows expected:<10> but was:<7>
> -
>
> Key: HBASE-23812
> URL: https://issues.apache.org/jira/browse/HBASE-23812
> Project: HBase
> Issue Type: Bug
> Components: flakies
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments:
> 0001-HBASE-23812-Flakey-Test-TestReplicator-testReplicato.patch
>
>
> I've seen this fail. Here is the latest:
> {code}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 21.044 s <<< FAILURE! - in
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator
> [ERROR]
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors
> Time elapsed: 7.139 s <<< FAILURE!
> java.lang.AssertionError: We did not replicate enough rows expected:<10> but
> was:<7>
>at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors(TestReplicator.java:158)
> {code}
> In test logs there are a few (in groups of three) of these:
> {code}
> 4247 2020-02-07 13:50:29,070 WARN
> [RS_REFRESH_PEER-regionserver/localhost:0-1.replicationSource,testReplicatorWithErrors.replicationSource.shipperlocalhost%2C49531%2C1581112194639,testReplicatorWithErrors]
> regionserver. ReplicationSourceShipper(223):
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$FailureInjectingReplicationEndpointForTest
> threw unknown exception:
> 4248 java.lang.ClassCastException:
> org.apache.hbase.thirdparty.com.google.protobuf.ServiceException cannot be
> cast to java.io.IOException
> 4249 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.parallelReplicate(HBaseInterClusterReplicationEndpoint.java:361)
> 4250 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:404)
> 4251 at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$ReplicationEndpointForTest.replicate(TestReplicator.java:228)
> 4252 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:188)
> 4253 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:118)
> {code}
> Let me fix this and see if it makes TestReplicator stable again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23812) [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did not replicate enough rows expected:<10> but was:<7>
[
https://issues.apache.org/jira/browse/HBASE-23812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032722#comment-17032722
]
Michael Stack commented on HBASE-23812:
---
Fixing the class cast issue, i see these are artificially injected errors. They
are usually handled? Elsewhere? I see in this flakies run that the
ClassCastExceptions are present:
https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/5370/testReport/junit/org.apache.hadoop.hbase.replication.regionserver/TestReplicator/testReplicatorWithErrors/
Let me push what I have and leave this open to see if it helps.
> [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did
> not replicate enough rows expected:<10> but was:<7>
> -
>
> Key: HBASE-23812
> URL: https://issues.apache.org/jira/browse/HBASE-23812
> Project: HBase
> Issue Type: Bug
> Components: flakies
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments:
> 0001-HBASE-23812-Flakey-Test-TestReplicator-testReplicato.patch
>
>
> I've seen this fail. Here is the latest:
> {code}
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 21.044 s <<< FAILURE! - in
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator
> [ERROR]
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors
> Time elapsed: 7.139 s <<< FAILURE!
> java.lang.AssertionError: We did not replicate enough rows expected:<10> but
> was:<7>
>at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors(TestReplicator.java:158)
> {code}
> In test logs there are a few (in groups of three) of these:
> {code}
> 4247 2020-02-07 13:50:29,070 WARN
> [RS_REFRESH_PEER-regionserver/localhost:0-1.replicationSource,testReplicatorWithErrors.replicationSource.shipperlocalhost%2C49531%2C1581112194639,testReplicatorWithErrors]
> regionserver. ReplicationSourceShipper(223):
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$FailureInjectingReplicationEndpointForTest
> threw unknown exception:
> 4248 java.lang.ClassCastException:
> org.apache.hbase.thirdparty.com.google.protobuf.ServiceException cannot be
> cast to java.io.IOException
> 4249 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.parallelReplicate(HBaseInterClusterReplicationEndpoint.java:361)
> 4250 at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:404)
> 4251 at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicator$ReplicationEndpointForTest.replicate(TestReplicator.java:228)
> 4252 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:188)
> 4253 at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:118)
> {code}
> Let me fix this and see if it makes TestReplicator stable again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23810) License checker for hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-23810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032720#comment-17032720 ] Michael Stack commented on HBASE-23810: --- Do you mean apache-rat check or do you mean something else [~weichiu] ? (If former, run w/ the -Prelease profile enabled). > License checker for hbase-thirdparty > > > Key: HBASE-23810 > URL: https://issues.apache.org/jira/browse/HBASE-23810 > Project: HBase > Issue Type: Wish > Components: thirdparty >Reporter: Wei-Chiu Chuang >Priority: Minor > > I'm playing with hbase-thirdparty artifacts, and realized it doesn't perform > license check like the core hbase do. > This is a wish list to add that support some day. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23811) [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032716#comment-17032716 ] Michael Stack commented on HBASE-23811: --- Oh, one thought is that perhaps this belongs in hbase-operator-tools rather than in thirdparty? Thirdparty is always on the runtime classpath though; perhaps that is what you want. > [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty > - > > Key: HBASE-23811 > URL: https://issues.apache.org/jira/browse/HBASE-23811 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > JaegerTracing pulls in lots of dependencies. Some, like libthrift (0.13.0) > conflicts the one ships in HBase (0.12.0). > Additionally, not everyone may want to use Jaeger. > I propose to shade JaegerTracing and its dependencies into an uber jar, place > it as a hbase-thirdparty artifact. As an added benefit, this makes the > management of tracers in the HBase's dependency tree much easier. Finally, we > can follow the same suit and provide Zipkin tracer support in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-23811) [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032712#comment-17032712 ] Michael Stack edited comment on HBASE-23811 at 2/7/20 10:35 PM: We should probably upgrade our libthrift. What other dependencies does it bring in [~weichiu]? Thirdparty generally is those popular libs we'd like to be up-to-date on; protobuf, guava, etc. We shade them so we can update w/o having to worry about the version upstream sports. Let me know if you want to run an hbase-thirdparty release... Can do that for you sir. was (Author: stack): We should probably upgrade our libthrift. What other dependencies does it bring in [~weichiu]? Let me know if you want to run an hbase-thirdparty release... Can do that for you sir. > [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty > - > > Key: HBASE-23811 > URL: https://issues.apache.org/jira/browse/HBASE-23811 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > JaegerTracing pulls in lots of dependencies. Some, like libthrift (0.13.0) > conflicts the one ships in HBase (0.12.0). > Additionally, not everyone may want to use Jaeger. > I propose to shade JaegerTracing and its dependencies into an uber jar, place > it as a hbase-thirdparty artifact. As an added benefit, this makes the > management of tracers in the HBase's dependency tree much easier. Finally, we > can follow the same suit and provide Zipkin tracer support in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23811) [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032712#comment-17032712 ] Michael Stack commented on HBASE-23811: --- We should probably upgrade our libthrift. What other dependencies does it bring in [~weichiu]? Let me know if you want to run an hbase-thirdparty release... Can do that for you sir. > [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty > - > > Key: HBASE-23811 > URL: https://issues.apache.org/jira/browse/HBASE-23811 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > JaegerTracing pulls in lots of dependencies. Some, like libthrift (0.13.0) > conflicts the one ships in HBase (0.12.0). > Additionally, not everyone may want to use Jaeger. > I propose to shade JaegerTracing and its dependencies into an uber jar, place > it as a hbase-thirdparty artifact. As an added benefit, this makes the > management of tracers in the HBase's dependency tree much easier. Finally, we > can follow the same suit and provide Zipkin tracer support in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] esteban commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
esteban commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#discussion_r376638630 ## File path: hbase-shell/src/main/ruby/shell/commands/merge_region.rb ## @@ -33,16 +33,18 @@ def help Examples: - hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME' - hbase> merge_region 'FULL_REGIONNAME', 'FULL_REGIONNAME', true + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME'] + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ...] + hbase> merge_region ['FULL_REGIONNAME', 'FULL_REGIONNAME', 'FULL_REGIONNAME', ...], true - hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME' - hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', true + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME'] + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ...] + hbase> merge_region ['ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', ...], true Review comment: As my previous comment, if we keep support of the syntax for adding only two regions and multiple regions we should keep both examples. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] esteban commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
esteban commented on a change in pull request #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#discussion_r376638342 ## File path: hbase-shell/src/main/ruby/hbase/admin.rb ## @@ -535,14 +535,20 @@ def move(encoded_region_name, server = nil) end #-- -# Merge two regions -def merge_region(region_a_name, region_b_name, force) - @admin.mergeRegionsAsync( -region_a_name.to_java_bytes, -region_b_name.to_java_bytes, -java.lang.Boolean.valueOf(force) +# Merge multiple regions +def merge_region(regions = [], force) Review comment: Can we add support for the previous syntax? and not just when an array is used? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1143: Backport HBASE-15519, HBASE-23364, HBASE-23802 to branch-2.2
Apache-HBase commented on issue #1143: Backport HBASE-15519, HBASE-23364, HBASE-23802 to branch-2.2 URL: https://github.com/apache/hbase/pull/1143#issuecomment-583644663 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ branch-2.2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 39s | branch-2.2 passed | | +1 :green_heart: | compile | 1m 30s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 1m 57s | branch-2.2 passed | | +1 :green_heart: | shadedjars | 4m 27s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | branch-2.2 passed | | +0 :ok: | spotbugs | 3m 15s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 8s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 18s | the patch passed | | +1 :green_heart: | compile | 1m 30s | the patch passed | | +1 :green_heart: | javac | 1m 30s | the patch passed | | -1 :x: | checkstyle | 1m 30s | hbase-server: The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 24s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 16m 46s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 1m 3s | the patch passed | | +1 :green_heart: | findbugs | 4m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 32s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 0m 39s | hbase-hadoop2-compat in the patch passed. | | -1 :x: | unit | 230m 8s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 1m 11s | The patch does not generate ASF License warnings. | | | | 295m 27s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestAdmin1 | | | hadoop.hbase.replication.multiwal.TestReplicationSyncUpToolWithMultipleWAL | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1143/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1143 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux e12b4e794ce5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1143/out/precommit/personality/provided.sh | | git revision | branch-2.2 / 4094275cd0 | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1143/1/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1143/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1143/1/testReport/ | | Max. process+thread count | 4674 (vs. ulimit of 1) | | modules | C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1143/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23812) [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did not replicate enough rows expected:<10> but was:<7>
Michael Stack created HBASE-23812:
-
Summary: [Flakey Test] TestReplicator#testReplicatorWithErrors:
AssertionError: We did not replicate enough rows expected:<10> but was:<7>
Key: HBASE-23812
URL: https://issues.apache.org/jira/browse/HBASE-23812
Project: HBase
Issue Type: Bug
Components: flakies
Reporter: Michael Stack
Assignee: Michael Stack
Fix For: 3.0.0, 2.3.0
I've seen this fail. Here is the latest:
{code}
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.044
s <<< FAILURE! - in
org.apache.hadoop.hbase.replication.regionserver.TestReplicator
[ERROR]
org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors
Time elapsed: 7.139 s <<< FAILURE!
java.lang.AssertionError: We did not replicate enough rows expected:<10> but
was:<7>
at
org.apache.hadoop.hbase.replication.regionserver.TestReplicator.testReplicatorWithErrors(TestReplicator.java:158)
{code}
In test logs there are a few (in groups of three) of these:
{code}
4247 2020-02-07 13:50:29,070 WARN
[RS_REFRESH_PEER-regionserver/localhost:0-1.replicationSource,testReplicatorWithErrors.replicationSource.shipperlocalhost%2C49531%2C1581112194639,testReplicatorWithErrors]
regionserver. ReplicationSourceShipper(223):
org.apache.hadoop.hbase.replication.regionserver.TestReplicator$FailureInjectingReplicationEndpointForTest
threw unknown exception:
4248 java.lang.ClassCastException:
org.apache.hbase.thirdparty.com.google.protobuf.ServiceException cannot be cast
to java.io.IOException
4249 at
org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.parallelReplicate(HBaseInterClusterReplicationEndpoint.java:361)
4250 at
org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:404)
4251 at
org.apache.hadoop.hbase.replication.regionserver.TestReplicator$ReplicationEndpointForTest.replicate(TestReplicator.java:228)
4252 at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:188)
4253 at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:118)
{code}
Let me fix this and see if it makes TestReplicator stable again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1137: HBASE-23804: Fix default master addr hostname in master registry
ndimiduk commented on a change in pull request #1137: HBASE-23804: Fix default master addr hostname in master registry URL: https://github.com/apache/hbase/pull/1137#discussion_r376624361 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java ## @@ -66,4 +68,24 @@ public static String getDefaultHost(String strInterface, String nameserver) return org.apache.hadoop.net.DNS.getDefaultHost(strInterface, nameserver); } } + + /** + * Get the configured hostname for master/regionserver. Gets the default hostname if not specified + * in the configuration. + * @param conf Configuration to look up. + * @param isMaster True if master hostname needs to be looked up and false for regionserver. + */ + public static String getHostname(Configuration conf, boolean isMaster) Review comment: Since this is now in a more public space, and since we have more service types than just "master" and "regionserver", perhaps the `boolean isMaster` should become `String serviceName`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1137: HBASE-23804: Fix default master addr hostname in master registry
ndimiduk commented on a change in pull request #1137: HBASE-23804: Fix default master addr hostname in master registry URL: https://github.com/apache/hbase/pull/1137#discussion_r376626826 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java ## @@ -66,4 +68,24 @@ public static String getDefaultHost(String strInterface, String nameserver) return org.apache.hadoop.net.DNS.getDefaultHost(strInterface, nameserver); } } + + /** + * Get the configured hostname for master/regionserver. Gets the default hostname if not specified + * in the configuration. + * @param conf Configuration to look up. + * @param isMaster True if master hostname needs to be looked up and false for regionserver. + */ + public static String getHostname(Configuration conf, boolean isMaster) Review comment: This method has a number of commits sprinkled into its recent history: HBASE-12954, HBASE-12956, HBASE-13481. They appear related to property binding to `0.0.0.0` and to a configuration with multiple IP's on a network interface. How do we test changes in this part of our code? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1137: HBASE-23804: Fix default master addr hostname in master registry
ndimiduk commented on a change in pull request #1137: HBASE-23804: Fix default master addr hostname in master registry URL: https://github.com/apache/hbase/pull/1137#discussion_r376628904 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java ## @@ -181,7 +181,11 @@ /** Configuration key for the list of master host:ports **/ public static final String MASTER_ADDRS_KEY = "hbase.masters"; - public static final String MASTER_ADDRS_DEFAULT = "localhost:" + DEFAULT_MASTER_PORT; Review comment: Head's up, this file is `InterfaceAudience.Public`, which means removals have to go through a full release deprecation cycle. From the sub-section titled "Client API compatibility" in [the book](http://hbase.apache.org/book.html#hbase.versioning.post10): > An API needs to be deprecated for a whole major version before we will change/remove it. > * An example: An API was deprecated in 2.0.1 and will be marked for deletion in 4.0.0. On the other hand, an API deprecated in 2.0.0 can be removed in 3.0.0. You'll need to retain the hold field -- annotate it as deprecated and add a java doc explaining that the other method could be used in its place. You can file a follow-on ticket with a fixVersion of 4.0.0 as a task to delete the field once we get there (and maybe leave a TODO referencing that ticket). Sadly this constant is a field not a method, which means we have no way to logging a WARN for clients who make use of it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1137: HBASE-23804: Fix default master addr hostname in master registry
ndimiduk commented on a change in pull request #1137: HBASE-23804: Fix default master addr hostname in master registry URL: https://github.com/apache/hbase/pull/1137#discussion_r376631649 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java ## @@ -181,7 +181,11 @@ /** Configuration key for the list of master host:ports **/ public static final String MASTER_ADDRS_KEY = "hbase.masters"; - public static final String MASTER_ADDRS_DEFAULT = "localhost:" + DEFAULT_MASTER_PORT; + // key to the config parameter of server hostname + // the specification of server hostname is optional. The hostname should be resolvable from + // both master and region server + public static final String RS_HOSTNAME_KEY = "hbase.regionserver.hostname"; Review comment: Don't add these here. In [the book](http://hbase.apache.org/book.html#hbase_default_configurations), they're considered "expert" configuration params (and the `master` version appears to not be documented at all). Instead, add them to the `DNS` class, from which they'll be used. Also, it's `InterfaceAudience.Private` so we have more maintenance/refactoring flexibility there. And don't lose their `InterfaceAudience.LimitedPrivate` annotations when you move them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23804) Fix default master addr hostname in master registry
[ https://issues.apache.org/jira/browse/HBASE-23804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032694#comment-17032694 ] Nick Dimiduk commented on HBASE-23804: -- [~clayb] do you have a moment to weigh in on the change included in this PR? I'm curious how we can automate testing this part of the code as it relates to HBASE-12954. Thanks. > Fix default master addr hostname in master registry > --- > > Key: HBASE-23804 > URL: https://issues.apache.org/jira/browse/HBASE-23804 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 3.0.0, 2.3.0, 1.6.0, HBASE-18095 >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Currently, master RPC server (*not* info server) always binds to the address > endpoint to which the default hostname of the server resolves to. However, > master registry picks the default end point to connect to as > "localhost:16000" when "hbase.masters" are not configured. This is leading to > a mismatch because the server may not be listening on the loopback address. > This is a problem only in the scripts (single proc/pseudo distributed modes) > because these are the cases in which "hbase.masters" is not populated by > default. > The fix is to pick the service endpoint the same way the RPC server does it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23805) [Flakey Test] TestRaceBetweenSCPAndDTP
[ https://issues.apache.org/jira/browse/HBASE-23805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-23805. --- Hadoop Flags: Reviewed Assignee: Michael Stack Resolution: Fixed Thanks for the review [~binlijin] Pushed to branch-2 and master. > [Flakey Test] TestRaceBetweenSCPAndDTP > -- > > Key: HBASE-23805 > URL: https://issues.apache.org/jira/browse/HBASE-23805 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: > 0001-HBASE-23805-Flakey-Test-TestRaceBetweenSCPAndDTP.patch, Screen Shot > 2020-02-05 at 9.59.18 PM.png > > > Waits a second and then goes to grab an SCP that may not have started yet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23811) [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032656#comment-17032656 ] Wei-Chiu Chuang commented on HBASE-23811: - I tried to create a new Maven module in HBase, and created a shaded artifact org.apache.hbase:hbase-jaeger. While it worked and I was able to run tests from command line, IntelliJ doesn't load the shaded artifact if it is within the same project. Moving the artifact to hbase-thirdparty solved this problem. Not sure if this is the purpose of the hbase-thirdparty repo, but I'm happy now. > [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty > - > > Key: HBASE-23811 > URL: https://issues.apache.org/jira/browse/HBASE-23811 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > JaegerTracing pulls in lots of dependencies. Some, like libthrift (0.13.0) > conflicts the one ships in HBase (0.12.0). > Additionally, not everyone may want to use Jaeger. > I propose to shade JaegerTracing and its dependencies into an uber jar, place > it as a hbase-thirdparty artifact. As an added benefit, this makes the > management of tracers in the HBase's dependency tree much easier. Finally, we > can follow the same suit and provide Zipkin tracer support in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23811) [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty
Wei-Chiu Chuang created HBASE-23811: --- Summary: [OpenTracing] Add shaded JaegerTracing tracer to hbase-thirdparty Key: HBASE-23811 URL: https://issues.apache.org/jira/browse/HBASE-23811 Project: HBase Issue Type: Sub-task Components: thirdparty Reporter: Wei-Chiu Chuang Assignee: Wei-Chiu Chuang JaegerTracing pulls in lots of dependencies. Some, like libthrift (0.13.0) conflicts the one ships in HBase (0.12.0). Additionally, not everyone may want to use Jaeger. I propose to shade JaegerTracing and its dependencies into an uber jar, place it as a hbase-thirdparty artifact. As an added benefit, this makes the management of tracers in the HBase's dependency tree much easier. Finally, we can follow the same suit and provide Zipkin tracer support in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1116: HBASE-23783: Address tests writing and reading SSL/Security files in …
ndimiduk commented on a change in pull request #1116: HBASE-23783: Address
tests writing and reading SSL/Security files in …
URL: https://github.com/apache/hbase/pull/1116#discussion_r376602767
##
File path: pom.xml
##
@@ -729,6 +729,8 @@
${surefire.skipFirstPart}
${surefire.firstPartForkCount}
false
+${surefire.reportsDirectory}
+${surefire.tempDir}
Review comment:
Hey @markrmiller I don't think this variable is defined anywhere. Missing
some other change someplace?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (HBASE-23810) License checker for hbase-thirdparty
Wei-Chiu Chuang created HBASE-23810: --- Summary: License checker for hbase-thirdparty Key: HBASE-23810 URL: https://issues.apache.org/jira/browse/HBASE-23810 Project: HBase Issue Type: Wish Components: thirdparty Reporter: Wei-Chiu Chuang I'm playing with hbase-thirdparty artifacts, and realized it doesn't perform license check like the core hbase do. This is a wish list to add that support some day. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032642#comment-17032642 ] Michael Stack commented on HBASE-23779: --- So editing extras gets my -T into the mix. Hurray! But ouch! The h2 and h3 unit test pages are just cut-off... unfinished. Haven't seen that before. https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2/2456/artifact/output-jdk8-hadoop3/patch-unit-root.txt Trying again https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2/2458/ > Up the default fork count to make builds complete faster; make count relative > to CPU count > -- > > Key: HBASE-23779 > URL: https://issues.apache.org/jira/browse/HBASE-23779 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: addendum2.patch, test_yetus_934.0.patch > > > Tests take a long time. Our fork count running all tests are conservative -- > 1 (small) for first part and 5 for second part (medium and large). Rather > than hardcoding we should set the fork count to be relative to machine size. > Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box. > Looking up at jenkins, it seems like the boxes are 24 cores... at least going > by my random survey. The load reported on a few seems low though this not > representative (looking at machine/uptime). > More parallelism willl probably mean more test failure. Let me take a look > see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23198) Documentation and release notes
[ https://issues.apache.org/jira/browse/HBASE-23198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23198: Priority: Blocker (was: Major) > Documentation and release notes > --- > > Key: HBASE-23198 > URL: https://issues.apache.org/jira/browse/HBASE-23198 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > > Document all the changes: algorithms, new configuration options, obsolete > configurations, upgrade procedure and possibility of downgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23198) Documentation and release notes
[ https://issues.apache.org/jira/browse/HBASE-23198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23198: Component/s: mob documentation community > Documentation and release notes > --- > > Key: HBASE-23198 > URL: https://issues.apache.org/jira/browse/HBASE-23198 > Project: HBase > Issue Type: Sub-task > Components: community, documentation, mob >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > > Document all the changes: algorithms, new configuration options, obsolete > configurations, upgrade procedure and possibility of downgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23802) Remove unnecessary Configuration instantiation in LossyAccounting
[
https://issues.apache.org/jira/browse/HBASE-23802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032626#comment-17032626
]
Hudson commented on HBASE-23802:
Results for branch branch-2
[build #2456 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2456/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2456//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2456//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2456//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove unnecessary Configuration instantiation in LossyAccounting
> -
>
> Key: HBASE-23802
> URL: https://issues.apache.org/jira/browse/HBASE-23802
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
>
> Mighty [~stack] pointed out over on HBASE-23801 that maybe {{LossyCounting}}
> doesn't need to be instantiating a {{Configuration}} instance in the first
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23568) Improve Threading of Replication
[
https://issues.apache.org/jira/browse/HBASE-23568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032597#comment-17032597
]
HBase QA commented on HBASE-23568:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
7s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m
33s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
6s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 37s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 11s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestReplicationSmallTestsSync |
| | hadoop.hbase.replication.TestReplicationDroppedTables |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-934/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hbase/pull/934 |
| JIRA Issue | HBASE-23568 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 11b47d871ce9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-934/out/precommit/personality/provided.sh
|
| git revision | master / 93c5e7691f |
| Default Java | 1.8.0_181 |
| unit |
[GitHub] [hbase] Apache-HBase commented on issue #934: HBASE-23568: Improve Threading of Replication
Apache-HBase commented on issue #934: HBASE-23568: Improve Threading of Replication URL: https://github.com/apache/hbase/pull/934#issuecomment-583557925 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 7s | master passed | | +1 :green_heart: | compile | 0m 59s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +1 :green_heart: | shadedjars | 5m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | | +0 :ok: | spotbugs | 4m 33s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 32s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 35s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 37s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | | +1 :green_heart: | findbugs | 4m 51s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 105m 11s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 170m 55s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.replication.TestReplicationSmallTestsSync | | | hadoop.hbase.replication.TestReplicationDroppedTables | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-934/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/934 | | JIRA Issue | HBASE-23568 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 11b47d871ce9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-934/out/precommit/personality/provided.sh | | git revision | master / 93c5e7691f | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-934/7/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-934/7/testReport/ | | Max. process+thread count | 6457 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-934/7/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
bharathv commented on a change in pull request #1141: HBASE-23808 [Flakey Test]
TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r376536908
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -128,41 +142,50 @@ public void
testMasterShutdownBeforeStartingAnyRegionServer() throws Exception {
conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, 1);
// Start the cluster
-final HBaseTestingUtility util = new HBaseTestingUtility(conf);
-util.startMiniDFSCluster(3);
-util.startMiniZKCluster();
-util.createRootDir();
-final LocalHBaseCluster cluster =
-new LocalHBaseCluster(conf, NUM_MASTERS, NUM_RS, HMaster.class,
-MiniHBaseCluster.MiniHBaseClusterRegionServer.class);
-final int MASTER_INDEX = 0;
-final MasterThread master = cluster.getMasters().get(MASTER_INDEX);
-master.start();
-LOG.info("Called master start on " + master.getName());
-Thread shutdownThread = new Thread("Shutdown-Thread") {
- @Override
- public void run() {
-LOG.info("Before call to shutdown master");
-try (Connection connection = createConnection(util); Admin admin =
connection.getAdmin()) {
- admin.shutdown();
-} catch (Exception e) {
- LOG.info("Error while calling Admin.shutdown, which is expected: " +
e.getMessage());
+LocalHBaseCluster cluster = null;
+try {
+ htu = new HBaseTestingUtility(conf);
+ htu.startMiniDFSCluster(3);
+ htu.startMiniZKCluster();
+ htu.createRootDir();
+ cluster = new LocalHBaseCluster(conf, NUM_MASTERS, NUM_RS, HMaster.class,
+MiniHBaseCluster.MiniHBaseClusterRegionServer.class);
+ final int MASTER_INDEX = 0;
+ final MasterThread master = cluster.getMasters().get(MASTER_INDEX);
+ master.start();
+ LOG.info("Called master start on " + master.getName());
+ final LocalHBaseCluster finalCluster = cluster;
+ Thread shutdownThread = new Thread("Shutdown-Thread") {
+@Override
+public void run() {
+ LOG.info("Before call to shutdown master");
+ try (Connection connection = createConnection(htu); Admin admin =
connection.getAdmin()) {
+admin.shutdown();
Review comment:
No, not after adding this check
https://github.com/apache/hbase/blob/d110c08dce482ef3161294ab931d2559d8e57fca/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java#L141
This check basically works around the actual problem without fixing it.
Ideally you could do the same because that check is anyway applied in the
master after the branch merge.
Fwiw, I think this race is exposed after committing HBASE-23764, because
that speeds up the connections.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-23802) Remove unnecessary Configuration instantiation in LossyAccounting
[
https://issues.apache.org/jira/browse/HBASE-23802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032563#comment-17032563
]
Nick Dimiduk commented on HBASE-23802:
--
Affects version on HBASE-23364 is set correctly – it does not impact branch-2.2
or branch-2.1.
> Remove unnecessary Configuration instantiation in LossyAccounting
> -
>
> Key: HBASE-23802
> URL: https://issues.apache.org/jira/browse/HBASE-23802
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
>
> Mighty [~stack] pointed out over on HBASE-23801 that maybe {{LossyCounting}}
> doesn't need to be instantiating a {{Configuration}} instance in the first
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[
https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032555#comment-17032555
]
HBase QA commented on HBASE-23779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HBASE-23779 does not apply to master. Rebase required? Wrong
Branch? See
https://yetus.apache.org/documentation/in-progress/precommit-patchnames for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-23779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12992918/addendum2.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1124/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Up the default fork count to make builds complete faster; make count relative
> to CPU count
> --
>
> Key: HBASE-23779
> URL: https://issues.apache.org/jira/browse/HBASE-23779
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: addendum2.patch, test_yetus_934.0.patch
>
>
> Tests take a long time. Our fork count running all tests are conservative --
> 1 (small) for first part and 5 for second part (medium and large). Rather
> than hardcoding we should set the fork count to be relative to machine size.
> Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.
> Looking up at jenkins, it seems like the boxes are 24 cores... at least going
> by my random survey. The load reported on a few seems low though this not
> representative (looking at machine/uptime).
> More parallelism willl probably mean more test failure. Let me take a look
> see.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-23779: -- Attachment: addendum2.patch > Up the default fork count to make builds complete faster; make count relative > to CPU count > -- > > Key: HBASE-23779 > URL: https://issues.apache.org/jira/browse/HBASE-23779 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: addendum2.patch, test_yetus_934.0.patch > > > Tests take a long time. Our fork count running all tests are conservative -- > 1 (small) for first part and 5 for second part (medium and large). Rather > than hardcoding we should set the fork count to be relative to machine size. > Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box. > Looking up at jenkins, it seems like the boxes are 24 cores... at least going > by my random survey. The load reported on a few seems low though this not > representative (looking at machine/uptime). > More parallelism willl probably mean more test failure. Let me take a look > see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032550#comment-17032550 ] stack commented on HBASE-23779: --- Added addendum as patch here. > Up the default fork count to make builds complete faster; make count relative > to CPU count > -- > > Key: HBASE-23779 > URL: https://issues.apache.org/jira/browse/HBASE-23779 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: addendum2.patch, test_yetus_934.0.patch > > > Tests take a long time. Our fork count running all tests are conservative -- > 1 (small) for first part and 5 for second part (medium and large). Rather > than hardcoding we should set the fork count to be relative to machine size. > Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box. > Looking up at jenkins, it seems like the boxes are 24 cores... at least going > by my random survey. The load reported on a few seems low though this not > representative (looking at machine/uptime). > More parallelism willl probably mean more test failure. Let me take a look > see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[
https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032548#comment-17032548
]
stack commented on HBASE-23779:
---
New addendum. See below. Pushed on Master only for now. Kicked off
https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2/2456/
Lets see. This addendum comes of discussion over in
https://github.com/apache/yetus/pull/86
{code}
commit ef380e0a2e3ba138391968126fd0e9d5c910df88 (HEAD -> m, origin/master,
origin/HEAD)
Author: stack
Date: Fri Feb 7 09:40:51 2020 -0800
HBASE-23779 Up the default fork count; make count relative to CPU count
(#1108)
ADDENDUM: Refactor that comes of discussion up on
https://github.com/apache/yetus/pull/86
because what I committed originally, and amended in a subsequent
ADDENDUM is not taking effect.
diff --git a/dev-support/hbase-personality.sh b/dev-support/hbase-personality.sh
index 169f19abcc..7f961c0e1e 100755
--- a/dev-support/hbase-personality.sh
+++ b/dev-support/hbase-personality.sh
@@ -82,14 +82,6 @@ function personality_globals
# Override the maven options
MAVEN_OPTS="${MAVEN_OPTS:-"-Xms4G -Xmx4G"}"
- # Pass maven a -T argument. Should make it run faster. Pass conservative
value.
- # Default is one thread. 0.5C on an apache box of 24 cores and 2 executors
should
- # make for 6 threads? Lets see. Setting this here for yetus to pick up. See
- #
https://yetus.apache.org/documentation/0.11.1/precommit-advanced/#global-definitions
- # See below for more on -T:
- #
https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
- MAVEN_ARGS=("-T0.5C" "${MAVEN_ARGS[@]}")
-
# Yetus 0.7.0 enforces limits. Default proclimit is 1000.
# Up it. See HBASE-19902 for how we arrived at this number.
#shellcheck disable=SC2034
@@ -148,7 +140,11 @@ function personality_modules
clear_personality_queue
- extra="-DHBasePatchProcess"
+ # Pass maven a -T argument. Should make it run faster. Pass conservative
value.
+ # Default is one thread. 0.5C on an apache box of 24 cores and 2 executors
should
+ # make for 6 threads? Lets see. See below for more on -T:
+ #
https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
+ extra="-T0.5C -DHBasePatchProcess"
if [[ "${PATCH_BRANCH}" = branch-1* ]]; then
extra="${extra} -Dhttps.protocols=TLSv1.2"
fi
{code}
> Up the default fork count to make builds complete faster; make count relative
> to CPU count
> --
>
> Key: HBASE-23779
> URL: https://issues.apache.org/jira/browse/HBASE-23779
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: test_yetus_934.0.patch
>
>
> Tests take a long time. Our fork count running all tests are conservative --
> 1 (small) for first part and 5 for second part (medium and large). Rather
> than hardcoding we should set the fork count to be relative to machine size.
> Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.
> Looking up at jenkins, it seems like the boxes are 24 cores... at least going
> by my random survey. The load reported on a few seems low though this not
> representative (looking at machine/uptime).
> More parallelism willl probably mean more test failure. Let me take a look
> see.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on issue #1143: Backport HBASE-15519, HBASE-23364, HBASE-23802 to branch-2.2
ndimiduk commented on issue #1143: Backport HBASE-15519, HBASE-23364, HBASE-23802 to branch-2.2 URL: https://github.com/apache/hbase/pull/1143#issuecomment-583522170 Reviewers: see associated discussion on https://issues.apache.org/jira/browse/HBASE-23802. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23802) Remove unnecessary Configuration instantiation in LossyAccounting
[
https://issues.apache.org/jira/browse/HBASE-23802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032547#comment-17032547
]
Nick Dimiduk commented on HBASE-23802:
--
Backport to branch-2.2 is on [https://github.com/apache/hbase/pull/1143.] I'm
less convinced that back porting the new feature is the correct thing to do.
Seems like we do want HBASE-23364 everywhere though.
Backport to branch-2.1 is even more complicated because it's missing yet
another feature: per-table metrics, HBASE-15728.
> Remove unnecessary Configuration instantiation in LossyAccounting
> -
>
> Key: HBASE-23802
> URL: https://issues.apache.org/jira/browse/HBASE-23802
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
>
> Mighty [~stack] pointed out over on HBASE-23801 that maybe {{LossyCounting}}
> doesn't need to be instantiating a {{Configuration}} instance in the first
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk opened a new pull request #1143: Backport HBASE-15519, HBASE-23364, HBASE-23802 to branch-2.2
ndimiduk opened a new pull request #1143: Backport HBASE-15519, HBASE-23364, HBASE-23802 to branch-2.2 URL: https://github.com/apache/hbase/pull/1143 This PR brings back HBASE-15519 and HBASE-23364 as prerequisites to HBASE-23802. They do not cherry-pick cleanly because of the presence of hbtop on branch-2; I had to unwind some minor complications that appear to arrive via HBASE-23065. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23802) Remove unnecessary Configuration instantiation in LossyAccounting
[
https://issues.apache.org/jira/browse/HBASE-23802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032529#comment-17032529
]
Nick Dimiduk commented on HBASE-23802:
--
HBASE-15519 looks like a new feature, I think that should be skipped for a
released line, correct?
HBASE-23364 seems to be a legitimate bug, so that should be back ported.
> Remove unnecessary Configuration instantiation in LossyAccounting
> -
>
> Key: HBASE-23802
> URL: https://issues.apache.org/jira/browse/HBASE-23802
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
>
> Mighty [~stack] pointed out over on HBASE-23801 that maybe {{LossyCounting}}
> doesn't need to be instantiating a {{Configuration}} instance in the first
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk merged pull request #1140: Backport of HBASE-23802 for branch-1
ndimiduk merged pull request #1140: Backport of HBASE-23802 for branch-1 URL: https://github.com/apache/hbase/pull/1140 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1142: HBASE-23809 The RSGroup shell test is missing
Apache-HBase commented on issue #1142: HBASE-23809 The RSGroup shell test is missing URL: https://github.com/apache/hbase/pull/1142#issuecomment-583496733 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 9s | master passed | | +1 :green_heart: | compile | 0m 24s | master passed | | +1 :green_heart: | checkstyle | 0m 13s | master passed | | +1 :green_heart: | shadedjars | 5m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 14s | master passed | | +0 :ok: | spotbugs | 5m 29s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 0s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 36s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | checkstyle | 0m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 49s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 12s | the patch passed | | +1 :green_heart: | findbugs | 0m 0s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 6m 59s | hbase-shell in the patch failed. | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 56m 54s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestRSGroupShell | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1142 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8d9a2b5c9a0f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1142/out/precommit/personality/provided.sh | | git revision | master / 93c5e7691f | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/1/artifact/out/patch-unit-hbase-shell.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/1/testReport/ | | Max. process+thread count | 3854 (vs. ulimit of 1) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1142/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[
https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032527#comment-17032527
]
Nick Dimiduk commented on HBASE-23779:
--
[~busbey] any objection if I modify the Jenkins script as follows?
{noformat}
diff --git a/dev-support/PreCommit-HBase-Build.sh
b/dev-support/PreCommit-HBase-Build.sh
index f1494978ea..798b829c3d 100644
--- a/dev-support/PreCommit-HBase-Build.sh
+++ b/dev-support/PreCommit-HBase-Build.sh
@@ -82,8 +82,8 @@ else
tar xvpf yetus.tar.gz
prerelease_dirs=(${TEST_FRAMEWORK}/${YETUS_PRERELEASE_GITHUB/\//-}-*)
fi
- TESTPATCHBIN=${prerelease_dirs[0]}/precommit/test-patch.sh
- TESTPATCHLIB=${prerelease_dirs[0]}/precommit
+ TESTPATCHBIN=${prerelease_dirs[0]}/precommit/src/main/shell/test-patch.sh
+ TESTPATCHLIB=${prerelease_dirs[0]}/precommit/src/main/shell
fi
if [[ "true" = "${DEBUG}" ]]; then
{noformat}
> Up the default fork count to make builds complete faster; make count relative
> to CPU count
> --
>
> Key: HBASE-23779
> URL: https://issues.apache.org/jira/browse/HBASE-23779
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: test_yetus_934.0.patch
>
>
> Tests take a long time. Our fork count running all tests are conservative --
> 1 (small) for first part and 5 for second part (medium and large). Rather
> than hardcoding we should set the fork count to be relative to machine size.
> Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.
> Looking up at jenkins, it seems like the boxes are 24 cores... at least going
> by my random survey. The load reported on a few seems low though this not
> representative (looking at machine/uptime).
> More parallelism willl probably mean more test failure. Let me take a look
> see.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[
https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032526#comment-17032526
]
Nick Dimiduk commented on HBASE-23779:
--
Okay, I cannot test my yetus patch using our precommit because the layout of
the repo artifact generated by GitHub doesn't match that script's expectations.
Looks like {{test-patch.sh}} was moved since this job was authored, it's now
under {{precommit/src/main/shell}}.
{noformat}
08:34:51 + prerelease_dirs=(${TEST_FRAMEWORK}/${YETUS_PRERELEASE_GITHUB/\//-}-*)
08:34:51 +
TESTPATCHBIN=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/test_framework/ndimiduk-yetus-7d6ff39/precommit/test-patch.sh
08:34:51 +
TESTPATCHLIB=/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/test_framework/ndimiduk-yetus-7d6ff39/precommit
08:34:51 + [[ true = \t\r\u\e ]]
08:34:51 + ls -l
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/test_framework/ndimiduk-yetus-7d6ff39/precommit/test-patch.sh
08:34:51 ls: cannot access
'/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/test_framework/ndimiduk-yetus-7d6ff39/precommit/test-patch.sh':
No such file or directory
08:34:52 Build step 'Execute shell' marked build as failure
{noformat}
> Up the default fork count to make builds complete faster; make count relative
> to CPU count
> --
>
> Key: HBASE-23779
> URL: https://issues.apache.org/jira/browse/HBASE-23779
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: test_yetus_934.0.patch
>
>
> Tests take a long time. Our fork count running all tests are conservative --
> 1 (small) for first part and 5 for second part (medium and large). Rather
> than hardcoding we should set the fork count to be relative to machine size.
> Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.
> Looking up at jenkins, it seems like the boxes are 24 cores... at least going
> by my random survey. The load reported on a few seems low though this not
> representative (looking at machine/uptime).
> More parallelism willl probably mean more test failure. Let me take a look
> see.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032518#comment-17032518 ] Nick Dimiduk commented on HBASE-23779: -- That didn't use my yetus repo. Trying again, https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/1122/ > Up the default fork count to make builds complete faster; make count relative > to CPU count > -- > > Key: HBASE-23779 > URL: https://issues.apache.org/jira/browse/HBASE-23779 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: test_yetus_934.0.patch > > > Tests take a long time. Our fork count running all tests are conservative -- > 1 (small) for first part and 5 for second part (medium and large). Rather > than hardcoding we should set the fork count to be relative to machine size. > Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box. > Looking up at jenkins, it seems like the boxes are 24 cores... at least going > by my random survey. The load reported on a few seems low though this not > representative (looking at machine/uptime). > More parallelism willl probably mean more test failure. Let me take a look > see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] wchevreuil commented on issue #934: HBASE-23568: Improve Threading of Replication
wchevreuil commented on issue #934: HBASE-23568: Improve Threading of Replication URL: https://github.com/apache/hbase/pull/934#issuecomment-583485512 Just saw a couple of replication unit tests failures on the last build. Could it be anyhow related to this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23779) Up the default fork count to make builds complete faster; make count relative to CPU count
[ https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032517#comment-17032517 ] Michael Stack commented on HBASE-23779: --- Dang... Still not picking up our MAVEN_ARGS https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/1121/consoleText > Up the default fork count to make builds complete faster; make count relative > to CPU count > -- > > Key: HBASE-23779 > URL: https://issues.apache.org/jira/browse/HBASE-23779 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: test_yetus_934.0.patch > > > Tests take a long time. Our fork count running all tests are conservative -- > 1 (small) for first part and 5 for second part (medium and large). Rather > than hardcoding we should set the fork count to be relative to machine size. > Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box. > Looking up at jenkins, it seems like the boxes are 24 cores... at least going > by my random survey. The load reported on a few seems low though this not > representative (looking at machine/uptime). > More parallelism willl probably mean more test failure. Let me take a look > see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test]
TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r376481912
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -128,41 +142,50 @@ public void
testMasterShutdownBeforeStartingAnyRegionServer() throws Exception {
conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, 1);
// Start the cluster
-final HBaseTestingUtility util = new HBaseTestingUtility(conf);
-util.startMiniDFSCluster(3);
-util.startMiniZKCluster();
-util.createRootDir();
-final LocalHBaseCluster cluster =
-new LocalHBaseCluster(conf, NUM_MASTERS, NUM_RS, HMaster.class,
-MiniHBaseCluster.MiniHBaseClusterRegionServer.class);
-final int MASTER_INDEX = 0;
-final MasterThread master = cluster.getMasters().get(MASTER_INDEX);
-master.start();
-LOG.info("Called master start on " + master.getName());
-Thread shutdownThread = new Thread("Shutdown-Thread") {
- @Override
- public void run() {
-LOG.info("Before call to shutdown master");
-try (Connection connection = createConnection(util); Admin admin =
connection.getAdmin()) {
- admin.shutdown();
-} catch (Exception e) {
- LOG.info("Error while calling Admin.shutdown, which is expected: " +
e.getMessage());
+LocalHBaseCluster cluster = null;
+try {
+ htu = new HBaseTestingUtility(conf);
+ htu.startMiniDFSCluster(3);
+ htu.startMiniZKCluster();
+ htu.createRootDir();
+ cluster = new LocalHBaseCluster(conf, NUM_MASTERS, NUM_RS, HMaster.class,
+MiniHBaseCluster.MiniHBaseClusterRegionServer.class);
+ final int MASTER_INDEX = 0;
+ final MasterThread master = cluster.getMasters().get(MASTER_INDEX);
+ master.start();
+ LOG.info("Called master start on " + master.getName());
+ final LocalHBaseCluster finalCluster = cluster;
+ Thread shutdownThread = new Thread("Shutdown-Thread") {
+@Override
+public void run() {
+ LOG.info("Before call to shutdown master");
+ try (Connection connection = createConnection(htu); Admin admin =
connection.getAdmin()) {
+admin.shutdown();
Review comment:
I think this is the source of the instability I've seen in this test, but I
cannot reproduce it reliably.
You see this test failing reliably on your branch?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] belugabehr commented on a change in pull request #934: HBASE-23568: Improve Threading of Replication
belugabehr commented on a change in pull request #934: HBASE-23568: Improve
Threading of Replication
URL: https://github.com/apache/hbase/pull/934#discussion_r376474316
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -314,31 +328,59 @@ private void initializeWALEntryFilter(UUID
peerClusterId) {
this.walEntryFilter = new ChainWALEntryFilter(filters);
}
- private void tryStartNewShipper(String walGroupId,
PriorityBlockingQueue queue) {
-workerThreads.compute(walGroupId, (key, value) -> {
- if (value != null) {
-if (LOG.isDebugEnabled()) {
- LOG.debug(
- "{} Someone has beat us to start a worker thread for wal group
{}",
- logPeerId(), key);
-}
-return value;
- } else {
-if (LOG.isDebugEnabled()) {
- LOG.debug("{} Starting up worker for wal group {}", logPeerId(),
key);
+ /**
+ * Synchronized method so that only one item is inserted at a time. Should
be the only place that
+ * insertions are performed on the workerThreads data structure.
+ *
+ * @param walGroupId The WAL group ID
+ * @param queue The queue of paths to process
+ */
+ private synchronized void tryStartNewShipper(final String walGroupId,
+ final PriorityBlockingQueue queue) {
+
+if (workerThreads.containsKey(walGroupId)) {
Review comment:
Yup. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] belugabehr commented on a change in pull request #934: HBASE-23568: Improve Threading of Replication
belugabehr commented on a change in pull request #934: HBASE-23568: Improve
Threading of Replication
URL: https://github.com/apache/hbase/pull/934#discussion_r376471772
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -585,50 +622,24 @@ public void terminate(String reason, Exception cause,
boolean clearMetrics, bool
initThread.interrupt();
Threads.shutdown(initThread, this.sleepForRetries);
}
-Collection workers = workerThreads.values();
-for (ReplicationSourceShipper worker : workers) {
- worker.stopWorker();
- if(worker.entryReader != null) {
-worker.entryReader.setReaderRunning(false);
- }
-}
-for (ReplicationSourceShipper worker : workers) {
- if (worker.isAlive() || worker.entryReader.isAlive()) {
-try {
- // Wait worker to stop
- Thread.sleep(this.sleepForRetries);
-} catch (InterruptedException e) {
- LOG.info("{} Interrupted while waiting {} to stop", logPeerId(),
worker.getName());
- Thread.currentThread().interrupt();
-}
-// If worker still is alive after waiting, interrupt it
-if (worker.isAlive()) {
- worker.interrupt();
-}
-// If entry reader is alive after waiting, interrupt it
-if (worker.entryReader.isAlive()) {
- worker.entryReader.interrupt();
-}
- }
+this.workerThreads.values().forEach(worker -> worker.stopWorker());
Review comment:
Nah, Workers are assigned a reader. It is the worker's responsibility to
close the reader when it is safe to do so.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
