[GitHub] [hbase] Apache-HBase commented on pull request #4294: HBASE-25624 Bound LoadBalancer's RegionLocationFinder cache
Apache-HBase commented on pull request #4294: URL: https://github.com/apache/hbase/pull/4294#issuecomment-1084095911 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 47s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 41s | master passed | | +1 :green_heart: | compile | 4m 10s | master passed | | +1 :green_heart: | checkstyle | 1m 19s | master passed | | +1 :green_heart: | spotbugs | 3m 6s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 29s | the patch passed | | +1 :green_heart: | compile | 4m 4s | the patch passed | | -0 :warning: | javac | 2m 27s | hbase-server generated 4 new + 189 unchanged - 4 fixed = 193 total (was 193) | | -0 :warning: | checkstyle | 0m 14s | hbase-common: The patch generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) | | -0 :warning: | checkstyle | 0m 8s | hbase-balancer: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | hadoopcheck | 5m 2s | The patch causes 10 errors with Hadoop v3.2.2. | | +1 :green_heart: | spotbugs | 3m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 35m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4294 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 00c33dd40ce6 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 828ea9187e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/3/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt | | hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/3/artifact/yetus-general-check/output/patch-javac-3.2.2.txt | | Max. process+thread count | 69 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-balancer hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4299: HBASE-26885 The TRSP should not go on when it get a bogus server name…
Apache-HBase commented on pull request #4299: URL: https://github.com/apache/hbase/pull/4299#issuecomment-1084091422 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 32s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | shadedjars | 3m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 37s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 3m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 187m 19s | hbase-server in the patch passed. | | | | 204m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4299 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ce531f0e4737 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 828ea9187e | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/2/testReport/ | | Max. process+thread count | 3118 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4223: HBASE-26838 Junit jar is not included in the hbase tar ball, causing …
Apache-HBase commented on pull request #4223: URL: https://github.com/apache/hbase/pull/4223#issuecomment-1084071219 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 3s | master passed | | +1 :green_heart: | compile | 5m 58s | master passed | | +0 :ok: | refguide | 1m 37s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 7s | the patch passed | | +1 :green_heart: | compile | 5m 54s | the patch passed | | +1 :green_heart: | javac | 5m 54s | the patch passed | | +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 1m 36s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 11m 37s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 37m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4223 | | Optional Tests | dupname asflicense shellcheck shelldocs javac hadoopcheck xml compile refguide | | uname | Linux 8f910519edb0 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 828ea9187e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4223/6/yetus-general-check/output/branch-site/book.html | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4223/6/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 138 (vs. ulimit of 3) | | modules | C: hbase-assembly . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/6/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4223: HBASE-26838 Junit jar is not included in the hbase tar ball, causing …
Apache-HBase commented on pull request #4223: URL: https://github.com/apache/hbase/pull/4223#issuecomment-1084052941 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 23s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 14s | master passed | | +1 :green_heart: | compile | 1m 33s | master passed | | +1 :green_heart: | shadedjars | 3m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 12s | the patch passed | | +1 :green_heart: | compile | 1m 32s | the patch passed | | +1 :green_heart: | javac | 1m 32s | the patch passed | | +1 :green_heart: | shadedjars | 3m 40s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 1m 23s | root generated 9 new + 49 unchanged - 9 fixed = 58 total (was 58) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 326m 16s | root in the patch passed. | | | | 348m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4223 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6c7b2055a399 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 828ea9187e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javadoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/5/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/5/testReport/ | | Max. process+thread count | 4699 (vs. ulimit of 3) | | modules | C: hbase-assembly . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4294: HBASE-25624 Bound LoadBalancer's RegionLocationFinder cache
Apache-HBase commented on pull request #4294: URL: https://github.com/apache/hbase/pull/4294#issuecomment-1084043350 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4294: HBASE-25624 Bound LoadBalancer's RegionLocationFinder cache
Apache-HBase commented on pull request #4294: URL: https://github.com/apache/hbase/pull/4294#issuecomment-1084042142 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 48s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 43s | master passed | | +1 :green_heart: | compile | 4m 19s | master passed | | +1 :green_heart: | checkstyle | 1m 20s | master passed | | +1 :green_heart: | spotbugs | 3m 5s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 23s | root in the patch failed. | | -1 :x: | compile | 1m 29s | hbase-server in the patch failed. | | -0 :warning: | javac | 1m 29s | hbase-server in the patch failed. | | -0 :warning: | checkstyle | 0m 13s | hbase-common: The patch generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) | | -0 :warning: | checkstyle | 0m 7s | hbase-balancer: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | hadoopcheck | 1m 30s | The patch causes 14 errors with Hadoop v3.1.2. | | -1 :x: | hadoopcheck | 3m 3s | The patch causes 14 errors with Hadoop v3.2.2. | | -1 :x: | hadoopcheck | 4m 38s | The patch causes 14 errors with Hadoop v3.3.1. | | -1 :x: | spotbugs | 0m 25s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 28m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4294 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 67a8a65df492 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 828ea9187e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt | | hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/artifact/yetus-general-check/output/patch-javac-3.2.2.txt | | hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/artifact/yetus-general-check/output/patch-javac-3.3.1.txt | | spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/artifact/yetus-general-check/output/patch-spotbugs-hbase-server.txt | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-balancer hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4294/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
[GitHub] [hbase] xicm commented on a change in pull request #4294: HBASE-25624 Bound LoadBalancer's RegionLocationFinder cache
xicm commented on a change in pull request #4294:
URL: https://github.com/apache/hbase/pull/4294#discussion_r839132102
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionHDFSBlockLocationFinder.java
##
@@ -102,8 +108,28 @@ public HDFSBlocksDistribution load(RegionInfo key) throws
Exception {
* @return A new Cache.
*/
private LoadingCache createCache() {
-return CacheBuilder.newBuilder().expireAfterWrite(CACHE_TIME,
TimeUnit.MILLISECONDS)
- .build(loader);
+
+long maximumWeight = getConf().getLong(
+ BALANCER_REGIONBLOCKLOCATION_CACHE_MAX_SIZE, 0);
+
+if (maximumWeight > 0) {
+ return CacheBuilder.newBuilder()
+.expireAfterWrite(CACHE_TIME, TimeUnit.MILLISECONDS)
+.weigher(
+ new Weigher() {
+@Override
+public int weigh(RegionInfo regionInfo, HDFSBlocksDistribution
hdfsBlocksDistribution) {
+ return (int) (regionInfo.heapSize() +
+hdfsBlocksDistribution.heapSize());
Review comment:
Calculating the heap involves too many changes, shall we limit the cache
with maximumSize?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] xicm commented on a change in pull request #4294: HBASE-25624 Bound LoadBalancer's RegionLocationFinder cache
xicm commented on a change in pull request #4294:
URL: https://github.com/apache/hbase/pull/4294#discussion_r837190831
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionHDFSBlockLocationFinder.java
##
@@ -102,8 +109,28 @@ public HDFSBlocksDistribution load(RegionInfo key) throws
Exception {
* @return A new Cache.
*/
private LoadingCache createCache() {
-return CacheBuilder.newBuilder().expireAfterWrite(CACHE_TIME,
TimeUnit.MILLISECONDS)
- .build(loader);
+
+long maximumWeight = getConf().getLong(
+ BALANCER_REGIONBLOCKLOCATION_CACHE_MAX_SIZE, 0);
+
+if (maximumWeight > 0) {
+ return CacheBuilder.newBuilder()
+.expireAfterWrite(CACHE_TIME, TimeUnit.MILLISECONDS)
+.weigher(
+ new Weigher() {
+@Override
+public int weigh(RegionInfo regionInfo, HDFSBlocksDistribution
hdfsBlocksDistribution) {
+ return (int)(ObjectSizeCalculator.getObjectSize(regionInfo) +
Review comment:
I use ObjectSizeCalculator.getObjectSize to estimate the size of the
Object in the cache, Is there a better way？
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionHDFSBlockLocationFinder.java
##
@@ -102,8 +109,28 @@ public HDFSBlocksDistribution load(RegionInfo key) throws
Exception {
* @return A new Cache.
*/
private LoadingCache createCache() {
-return CacheBuilder.newBuilder().expireAfterWrite(CACHE_TIME,
TimeUnit.MILLISECONDS)
- .build(loader);
+
+long maximumWeight = getConf().getLong(
+ BALANCER_REGIONBLOCKLOCATION_CACHE_MAX_SIZE, 0);
+
+if (maximumWeight > 0) {
+ return CacheBuilder.newBuilder()
+.expireAfterWrite(CACHE_TIME, TimeUnit.MILLISECONDS)
+.weigher(
+ new Weigher() {
+@Override
+public int weigh(RegionInfo regionInfo, HDFSBlocksDistribution
hdfsBlocksDistribution) {
+ return (int)(ObjectSizeCalculator.getObjectSize(regionInfo) +
Review comment:
ObjectSizeCalculator is not supported on JDK11, Do we have a suggested
approach？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4223: HBASE-26838 Junit jar is not included in the hbase tar ball, causing …
Apache-HBase commented on pull request #4223: URL: https://github.com/apache/hbase/pull/4223#issuecomment-1084018159 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 17s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 40s | master passed | | +1 :green_heart: | compile | 1m 54s | master passed | | +1 :green_heart: | shadedjars | 3m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 9s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 35s | the patch passed | | +1 :green_heart: | compile | 1m 54s | the patch passed | | +1 :green_heart: | javac | 1m 54s | the patch passed | | +1 :green_heart: | shadedjars | 3m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 8s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 246m 18s | root in the patch passed. | | | | 271m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4223 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ae3e366d4156 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 828ea9187e | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/5/testReport/ | | Max. process+thread count | 4460 (vs. ulimit of 3) | | modules | C: hbase-assembly . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26898) Cannot rebuild a cluster from an existing root directory
[
https://issues.apache.org/jira/browse/HBASE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiangJun He resolved HBASE-26898.
-
Resolution: Resolved
> Cannot rebuild a cluster from an existing root directory
>
>
> Key: HBASE-26898
> URL: https://issues.apache.org/jira/browse/HBASE-26898
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 3.0.0-alpha-2
>Reporter: LiangJun He
>Assignee: LiangJun He
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> When I tested to rebuild an HBase cluster, and the rootdir was configured as
> a existed directory (the directory was generated by another HBase cluster of
> the same version), I found the following error message：
> {code:java}
> java.net.UnknownHostException: Call to address=worker-1.cluster-xxx:16020
> failed on local exception: java.net.UnknownHostException:
> worker-1.cluster-xxx:16020 could not be resolved
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:234)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:93)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:424)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:419)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:119)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:134)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.lambda$sendRequest$4(NettyRpcConnection.java:351)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
> at
> org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.UnknownHostException: worker-1.cluster-xxx:16020 could
> not be resolved
> at
> org.apache.hadoop.hbase.ipc.RpcConnection.getRemoteInetAddress(RpcConnection.java:192)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.connect(NettyRpcConnection.java:275)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:78)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$4.run(NettyRpcConnection.java:325)
> at
> org.apache.hadoop.hbase.ipc.HBaseRpcControllerImpl.notifyOnCancel(HBaseRpcControllerImpl.java:262)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest0(NettyRpcConnection.java:308)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.lambda$sendRequest$4(NettyRpcConnection.java:349)
> {code}
> Eventually, I fail to create the cluster.
> But for cloud environments, this operation is a common scenario（Rebuild a
> cluster from an existing rootdir directory）
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4299: HBASE-26885 The TRSP should not go on when it get a bogus server name…
Apache-HBase commented on pull request #4299: URL: https://github.com/apache/hbase/pull/4299#issuecomment-1084008332 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 8s | master passed | | +1 :green_heart: | compile | 2m 13s | master passed | | +1 :green_heart: | checkstyle | 0m 35s | master passed | | +1 :green_heart: | spotbugs | 1m 15s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 9s | the patch passed | | +1 :green_heart: | compile | 2m 14s | the patch passed | | +1 :green_heart: | javac | 2m 14s | the patch passed | | +1 :green_heart: | checkstyle | 0m 36s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 14s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 1m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 8s | The patch does not generate ASF License warnings. | | | | 29m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4299 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux bcf3aa58e247 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 828ea9187e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26898) Cannot rebuild a cluster from an existing root directory
[
https://issues.apache.org/jira/browse/HBASE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17515025#comment-17515025
]
LiangJun He commented on HBASE-26898:
-
I tested with patch from
[HBASE-26245|https://issues.apache.org/jira/browse/HBASE-26245], if we stop the
masters of the old cluster first, and then stop the regionservers, we can
successfully rebuild the new cluster from the existing rootdir, otherwise some
tests show that the new cluster there are some pending procedures, the new
clusters may still hang.
> Cannot rebuild a cluster from an existing root directory
>
>
> Key: HBASE-26898
> URL: https://issues.apache.org/jira/browse/HBASE-26898
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 3.0.0-alpha-2
>Reporter: LiangJun He
>Assignee: LiangJun He
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> When I tested to rebuild an HBase cluster, and the rootdir was configured as
> a existed directory (the directory was generated by another HBase cluster of
> the same version), I found the following error message：
> {code:java}
> java.net.UnknownHostException: Call to address=worker-1.cluster-xxx:16020
> failed on local exception: java.net.UnknownHostException:
> worker-1.cluster-xxx:16020 could not be resolved
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:234)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:93)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:424)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:419)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:119)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:134)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.lambda$sendRequest$4(NettyRpcConnection.java:351)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
> at
> org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.UnknownHostException: worker-1.cluster-xxx:16020 could
> not be resolved
> at
> org.apache.hadoop.hbase.ipc.RpcConnection.getRemoteInetAddress(RpcConnection.java:192)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.connect(NettyRpcConnection.java:275)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:78)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$4.run(NettyRpcConnection.java:325)
> at
> org.apache.hadoop.hbase.ipc.HBaseRpcControllerImpl.notifyOnCancel(HBaseRpcControllerImpl.java:262)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest0(NettyRpcConnection.java:308)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.lambda$sendRequest$4(NettyRpcConnection.java:349)
> {code}
> Eventually, I fail to create the cluster.
> But for cloud environments, this operation is a common scenario（Rebuild a
> cluster from an existing rootdir directory）
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache9 commented on a change in pull request #4136: HBASE-26245 Store region server list in master local region
Apache9 commented on a change in pull request #4136: URL: https://github.com/apache/hbase/pull/4136#discussion_r839117090 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java ## @@ -964,7 +969,8 @@ private void finishActiveMasterInitialization(MonitoredTask status) throws IOExc this.regionServerTracker.upgrade( procsByType.getOrDefault(ServerCrashProcedure.class, Collections.emptyList()).stream() .map(p -> (ServerCrashProcedure) p).map(p -> p.getServerName()).collect(Collectors.toSet()), - walManager.getLiveServersFromWALDir(), walManager.getSplittingServersFromWALDir()); + Sets.union(rsListStorage.getAll(), walManager.getLiveServersFromWALDir()), Review comment: This is for rolling upgrading, where we have nothing in the master local region yet. I think we could remove the union in 4.0.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26898) Cannot rebuild a cluster from an existing root directory
[
https://issues.apache.org/jira/browse/HBASE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17515022#comment-17515022
]
LiangJun He commented on HBASE-26898:
-
My test environment is:
Old cluster: WAL on the old HDFS cluster, rootdir on OSS, zk-state on old
zk-Cluster
New cluster: WAL on new HDFS cluster, rootdir on OSS(the same directory as the
old cluster), zk-state on new zk-Cluster
Only the rootdir data(on OSS) is the same between the two clusters.
> Cannot rebuild a cluster from an existing root directory
>
>
> Key: HBASE-26898
> URL: https://issues.apache.org/jira/browse/HBASE-26898
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 3.0.0-alpha-2
>Reporter: LiangJun He
>Assignee: LiangJun He
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> When I tested to rebuild an HBase cluster, and the rootdir was configured as
> a existed directory (the directory was generated by another HBase cluster of
> the same version), I found the following error message：
> {code:java}
> java.net.UnknownHostException: Call to address=worker-1.cluster-xxx:16020
> failed on local exception: java.net.UnknownHostException:
> worker-1.cluster-xxx:16020 could not be resolved
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:234)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:93)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:424)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:419)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:119)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:134)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.lambda$sendRequest$4(NettyRpcConnection.java:351)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
> at
> org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> at
> org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.UnknownHostException: worker-1.cluster-xxx:16020 could
> not be resolved
> at
> org.apache.hadoop.hbase.ipc.RpcConnection.getRemoteInetAddress(RpcConnection.java:192)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.connect(NettyRpcConnection.java:275)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:78)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$4.run(NettyRpcConnection.java:325)
> at
> org.apache.hadoop.hbase.ipc.HBaseRpcControllerImpl.notifyOnCancel(HBaseRpcControllerImpl.java:262)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest0(NettyRpcConnection.java:308)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.lambda$sendRequest$4(NettyRpcConnection.java:349)
> {code}
> Eventually, I fail to create the cluster.
> But for cloud environments, this operation is a common scenario（Rebuild a
> cluster from an existing rootdir directory）
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] apurtell commented on a change in pull request #4136: HBASE-26245 Store region server list in master local region
apurtell commented on a change in pull request #4136:
URL: https://github.com/apache/hbase/pull/4136#discussion_r839096014
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -964,7 +969,8 @@ private void finishActiveMasterInitialization(MonitoredTask
status) throws IOExc
this.regionServerTracker.upgrade(
procsByType.getOrDefault(ServerCrashProcedure.class,
Collections.emptyList()).stream()
.map(p -> (ServerCrashProcedure) p).map(p ->
p.getServerName()).collect(Collectors.toSet()),
- walManager.getLiveServersFromWALDir(),
walManager.getSplittingServersFromWALDir());
+ Sets.union(rsListStorage.getAll(),
walManager.getLiveServersFromWALDir()),
Review comment:
Do we still need to also track, and take a union here of, live servers
via WALManager? Is the WALManager based live server stuff redundant now? Can we
remove it?
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartWithEmptyWALDirectory.java
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import static org.junit.Assert.assertArrayEquals;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.testclassification.MasterTests;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+/**
+ * Simulate the scenario described in HBASE-26245, where we clean the WAL
directory and try to start
Review comment:
This is excellent! Have been bitten by this scenario in test
environments, nice to see a passing test for a solution.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerListStorage.java
##
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.io.IOException;
+import java.util.Set;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * For storing the region server list.
+ *
+ * Mainly be used when restarting master, to load the previous active region
server list.
+ */
+@InterfaceAudience.Private
+public interface RegionServerListStorage {
Review comment:
I think the 'Storage' part has a redundant meaning, let's just call this
`RegionServerList` ?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRegionRegionServerListStorage.java
##
@@ -0,0 +1,112 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ *
[jira] [Commented] (HBASE-26907) Cannot run tests on Hadoop3 with Java11 versions having 4 version number positions
[
https://issues.apache.org/jira/browse/HBASE-26907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17515009#comment-17515009
]
Andrew Kyle Purtell commented on HBASE-26907:
-
bq. We upgrade our minimum hadoop3 support to 3.2.2
This seems the most simple option.
> Cannot run tests on Hadoop3 with Java11 versions having 4 version number
> positions
> --
>
> Key: HBASE-26907
> URL: https://issues.apache.org/jira/browse/HBASE-26907
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Priority: Major
>
> It happened that my JDK version upgraded to 11.0.14.1. Running unit tests
> involving the HDFS mini cluster now fails with a stack trace that ends with
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Invalid Java version 11.0.14.1
>
>
> at org.eclipse.jetty.util.JavaVersion.parseJDK9(JavaVersion.java:71)
>
>
> at org.eclipse.jetty.util.JavaVersion.parse(JavaVersion.java:49)
>
>
> at org.eclipse.jetty.util.JavaVersion.(JavaVersion.java:43)
> {noformat}
> We are using hadoop-3.2.0, which uses jetty-9.3.24. This is a Jetty issue has
> been fixed upstream in Jetty via
> https://github.com/eclipse/jetty.project/issues/2090. Hadoop has upgraded its
> Jetty version to 9.4.20 in HADOOP-16152, which is available as of
> hadoop-3.2.2.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26245) Store region server list in master local region
[ https://issues.apache.org/jira/browse/HBASE-26245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26245: -- Release Note: A typical HBase deployment on cloud is to store the data other than WAL on OSS, and store the WAL data on a special HDFS cluster. A common operation is to rebuild the cluster with fresh new zk cluster and HDFS cluster, with only the old rootdir on OSS. But it requires extra manual steps since we rely on the WAL directory to find out previous live region servers, so we can schedule SCP to bring regions online. After this issue, now it is possible to rebuild the cluster without extra manual steps as we will also store the previous live region servers in master local region. But notice that you'd better stop masters first and then region servers when rebuilding, as some tests show that if there are some pending procedures, the new clusters may still hang. > Store region server list in master local region > --- > > Key: HBASE-26245 > URL: https://issues.apache.org/jira/browse/HBASE-26245 > Project: HBase > Issue Type: Brainstorming > Components: master, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Just a simple idea still need to be polished. > For large clusters, ZooKeeper could be a bottle neck, there are some related > issues to not track region server list as much as possible, but what if we > want to do more, just do not register region server on zk? > I think, zookeeper here is act as something like a service registry, we need > the list to get all the region servers, and also need to know the changes of > the list. > But in fact, we could also kill region server from master side, so the latter > one could be done by a periodical heartbeat check daemon on master. And for > the former one, we could store the list in master local region, so when > master restart, it could setup the region server list by load from master > local region. > And there are mainly two other side effect which are good: > 1. We do not need to list the WAL directory on HDFS to find the previous > region servers, for scheduling SCP. This could make it possible to restart a > new HBase cluster based on only the root directory. > 2. For now, region server needs to register to HMaster first, and then put > its node on ZooKeeper, if it fails between these two actions, there is no way > for HMaster to clean this dead server, as it never expires on ZooKeeper. > There should be a related issue. If we just do not store a node on zk, then > this problem is also gone. > Of course, there will still be lots of other problems, like whether we need > another heartbeat call as reportForDuty is a bit heavy as we also report the > region list, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (HBASE-26617) Use spotless to reduce the pain on fixing checkstyle issues
[
https://issues.apache.org/jira/browse/HBASE-26617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514659#comment-17514659
]
Duo Zhang edited comment on HBASE-26617 at 3/31/22, 12:10 AM:
--
{quote}
Another thought I had recently is we should maybe try to land the "big spotless
cleanup" PRs in master and the various branch-2[x] at roughly the same time.
Otherwise it's going to be a real pain to be cherry-picking commits when the
branches aren't in sync on formatting. But I could be over thinking?
{quote}
No. I agree with you that we'd better land the big PR at almost the same time,
especially, if a patch has already been committed to master, and we want to
backport them to other branches, then we should commit them before landing the
big reformat PR, otherwise it will be a big pain for the contributor or the
committers for cherry-picking...
was (Author: apache9):
{noformat}
Another thought I had recently is we should maybe try to land the "big spotless
cleanup" PRs in master and the various branch-2[x] at roughly the same time.
Otherwise it's going to be a real pain to be cherry-picking commits when the
branches aren't in sync on formatting. But I could be over thinking?
{noformat}
No. I agree with you that we'd better land the big PR at almost the same time,
especially, if a patch has already been committed to master, and we want to
backport them to other branches, then we should commit them before landing the
big reformat PR, otherwise it will be a big pain for the contributor or the
committers for cherry-picking...
> Use spotless to reduce the pain on fixing checkstyle issues
> ---
>
> Key: HBASE-26617
> URL: https://issues.apache.org/jira/browse/HBASE-26617
> Project: HBase
> Issue Type: New Feature
> Components: build, community
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> https://github.com/diffplug/spotless
> Spotless is a tool for reformatting code based on your configuration file, it
> supports the google java format, as well as a eclipse formatter file.
> I think we could use it to format our code, based on a configuration file in
> our repo, and all the developpers, no matter what IDE they use, they can just
> type a mvn spotless:apply before commiting a PR, then everything is fine.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26617) Use spotless to reduce the pain on fixing checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-26617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514994#comment-17514994 ] Duo Zhang commented on HBASE-26617: --- It is because we haven't landed the big reformat PR on any branches yet :) [~ndimiduk] > Use spotless to reduce the pain on fixing checkstyle issues > --- > > Key: HBASE-26617 > URL: https://issues.apache.org/jira/browse/HBASE-26617 > Project: HBase > Issue Type: New Feature > Components: build, community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > https://github.com/diffplug/spotless > Spotless is a tool for reformatting code based on your configuration file, it > supports the google java format, as well as a eclipse formatter file. > I think we could use it to format our code, based on a configuration file in > our repo, and all the developpers, no matter what IDE they use, they can just > type a mvn spotless:apply before commiting a PR, then everything is fine. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26245) Store region server list in master local region
[ https://issues.apache.org/jira/browse/HBASE-26245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514992#comment-17514992 ] Duo Zhang commented on HBASE-26245: --- OK, so let me add this into the release note. Thank you for your test! > Store region server list in master local region > --- > > Key: HBASE-26245 > URL: https://issues.apache.org/jira/browse/HBASE-26245 > Project: HBase > Issue Type: Brainstorming > Components: master, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Just a simple idea still need to be polished. > For large clusters, ZooKeeper could be a bottle neck, there are some related > issues to not track region server list as much as possible, but what if we > want to do more, just do not register region server on zk? > I think, zookeeper here is act as something like a service registry, we need > the list to get all the region servers, and also need to know the changes of > the list. > But in fact, we could also kill region server from master side, so the latter > one could be done by a periodical heartbeat check daemon on master. And for > the former one, we could store the list in master local region, so when > master restart, it could setup the region server list by load from master > local region. > And there are mainly two other side effect which are good: > 1. We do not need to list the WAL directory on HDFS to find the previous > region servers, for scheduling SCP. This could make it possible to restart a > new HBase cluster based on only the root directory. > 2. For now, region server needs to register to HMaster first, and then put > its node on ZooKeeper, if it fails between these two actions, there is no way > for HMaster to clean this dead server, as it never expires on ZooKeeper. > There should be a related issue. If we just do not store a node on zk, then > this problem is also gone. > Of course, there will still be lots of other problems, like whether we need > another heartbeat call as reportForDuty is a bit heavy as we also report the > region list, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26907) Cannot run tests on Hadoop3 with Java11 versions having 4 version number positions
[
https://issues.apache.org/jira/browse/HBASE-26907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514991#comment-17514991
]
Duo Zhang commented on HBASE-26907:
---
So what is the solution? We upgrade our minimum hadoop3 support to 3.2.2? Or we
just tell users you need specify 3.2.2 if you want to run UTs?
> Cannot run tests on Hadoop3 with Java11 versions having 4 version number
> positions
> --
>
> Key: HBASE-26907
> URL: https://issues.apache.org/jira/browse/HBASE-26907
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Priority: Major
>
> It happened that my JDK version upgraded to 11.0.14.1. Running unit tests
> involving the HDFS mini cluster now fails with a stack trace that ends with
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Invalid Java version 11.0.14.1
>
>
> at org.eclipse.jetty.util.JavaVersion.parseJDK9(JavaVersion.java:71)
>
>
> at org.eclipse.jetty.util.JavaVersion.parse(JavaVersion.java:49)
>
>
> at org.eclipse.jetty.util.JavaVersion.(JavaVersion.java:43)
> {noformat}
> We are using hadoop-3.2.0, which uses jetty-9.3.24. This is a Jetty issue has
> been fixed upstream in Jetty via
> https://github.com/eclipse/jetty.project/issues/2090. Hadoop has upgraded its
> Jetty version to 9.4.20 in HADOOP-16152, which is available as of
> hadoop-3.2.2.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4223: HBASE-26838 Junit jar is not included in the hbase tar ball, causing …
Apache-HBase commented on pull request #4223: URL: https://github.com/apache/hbase/pull/4223#issuecomment-1083723206 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 1s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 18s | master passed | | +1 :green_heart: | compile | 6m 5s | master passed | | +0 :ok: | refguide | 2m 5s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 5s | the patch passed | | +1 :green_heart: | compile | 6m 0s | the patch passed | | -0 :warning: | javac | 6m 0s | root generated 1 new + 1471 unchanged - 1 fixed = 1472 total (was 1472) | | +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 1m 36s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 11m 53s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 38m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4223 | | Optional Tests | dupname asflicense shellcheck shelldocs javac hadoopcheck xml compile refguide | | uname | Linux 709fdf305a45 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 828ea9187e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4223/5/yetus-general-check/output/branch-site/book.html | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/5/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4223/5/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 140 (vs. ulimit of 3) | | modules | C: hbase-assembly . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4223/5/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on pull request #4223: HBASE-26838 Junit jar is not included in the hbase tar ball, causing …
wchevreuil commented on pull request #4223: URL: https://github.com/apache/hbase/pull/4223#issuecomment-1083697255 Thanks @joshelser @busbey and @ndimiduk ! Had pushed two additional commits, with updates to the ref-guide, adding mockito lib, and adding the test libs to the client tar ball. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26871) shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
[
https://issues.apache.org/jira/browse/HBASE-26871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-26871.
-
Resolution: Fixed
> shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
> ---
>
> Key: HBASE-26871
> URL: https://issues.apache.org/jira/browse/HBASE-26871
> Project: HBase
> Issue Type: Bug
> Components: integration tests, jenkins, mapreduce
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Duo Zhang
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0, 2.4.12
>
>
> After fixing the logging problem in HBASE-26870, we could see the actual
> error.
> {noformat}
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hadoop-3/bin/hadoop
> --config
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/output-integration/hadoop-3/hbase-conf/
> jar
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar
> importtsv
> -Dimporttsv.columns=HBASE_ROW_KEY,family1:column1,family1:column4,family1:column3
> test:example example/ -libjars
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/htrace-core4-4.1.0-incubating.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-api-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-context-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-semconv-1.0.1-alpha.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.33.jar
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.mapreduce.Driver
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:311)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26871) shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
[
https://issues.apache.org/jira/browse/HBASE-26871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514970#comment-17514970
]
Andrew Kyle Purtell commented on HBASE-26871:
-
Also picked the validation script improvement to branch-2.4.
> shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
> ---
>
> Key: HBASE-26871
> URL: https://issues.apache.org/jira/browse/HBASE-26871
> Project: HBase
> Issue Type: Bug
> Components: integration tests, jenkins, mapreduce
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Duo Zhang
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0, 2.4.12
>
>
> After fixing the logging problem in HBASE-26870, we could see the actual
> error.
> {noformat}
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hadoop-3/bin/hadoop
> --config
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/output-integration/hadoop-3/hbase-conf/
> jar
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar
> importtsv
> -Dimporttsv.columns=HBASE_ROW_KEY,family1:column1,family1:column4,family1:column3
> test:example example/ -libjars
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/htrace-core4-4.1.0-incubating.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-api-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-context-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-semconv-1.0.1-alpha.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.33.jar
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.mapreduce.Driver
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:311)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26871) shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
[
https://issues.apache.org/jira/browse/HBASE-26871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26871:
Fix Version/s: 2.4.12
> shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
> ---
>
> Key: HBASE-26871
> URL: https://issues.apache.org/jira/browse/HBASE-26871
> Project: HBase
> Issue Type: Bug
> Components: integration tests, jenkins, mapreduce
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Duo Zhang
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0, 2.4.12
>
>
> After fixing the logging problem in HBASE-26870, we could see the actual
> error.
> {noformat}
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hadoop-3/bin/hadoop
> --config
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/output-integration/hadoop-3/hbase-conf/
> jar
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar
> importtsv
> -Dimporttsv.columns=HBASE_ROW_KEY,family1:column1,family1:column4,family1:column3
> test:example example/ -libjars
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/htrace-core4-4.1.0-incubating.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-api-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-context-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-semconv-1.0.1-alpha.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.33.jar
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.mapreduce.Driver
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:311)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26903) Bump httpclient from 4.5.3 to 4.5.13
[
https://issues.apache.org/jira/browse/HBASE-26903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26903:
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Bump httpclient from 4.5.3 to 4.5.13
>
>
> Key: HBASE-26903
> URL: https://issues.apache.org/jira/browse/HBASE-26903
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.12
>
>
> Dependabot auto-generated dependency upgrade:
> https://github.com/apache/hbase/pull/4291
> We can't accept the dependabot PR as-is because it causes a unit test
> failure. Bump the dependency and fix the test by hand.
> There is a comment in our POM indicating this is a known issue:
> {noformat}
>
> {noformat}
> Staying back on a version of httpclient with CVE listed vulnerabilities just
> for this isn't a good option.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] apurtell merged pull request #4296: HBASE-26903 Bump httpclient from 4.5.3 to 4.5.13
apurtell merged pull request #4296: URL: https://github.com/apache/hbase/pull/4296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26871) shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
[
https://issues.apache.org/jira/browse/HBASE-26871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514960#comment-17514960
]
Andrew Kyle Purtell commented on HBASE-26871:
-
I'll cherry pick it from there, thanks Sean.
> shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
> ---
>
> Key: HBASE-26871
> URL: https://issues.apache.org/jira/browse/HBASE-26871
> Project: HBase
> Issue Type: Bug
> Components: integration tests, jenkins, mapreduce
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Duo Zhang
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0
>
>
> After fixing the logging problem in HBASE-26870, we could see the actual
> error.
> {noformat}
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hadoop-3/bin/hadoop
> --config
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/output-integration/hadoop-3/hbase-conf/
> jar
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar
> importtsv
> -Dimporttsv.columns=HBASE_ROW_KEY,family1:column1,family1:column4,family1:column3
> test:example example/ -libjars
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/htrace-core4-4.1.0-incubating.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-api-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-context-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-semconv-1.0.1-alpha.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.33.jar
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.mapreduce.Driver
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:311)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26885) The TRSP should not go on when it get a bogus server name from AM
[
https://issues.apache.org/jira/browse/HBASE-26885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514925#comment-17514925
]
Hudson commented on HBASE-26885:
Results for branch branch-2.5
[build #81 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/81/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/81/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/81/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/81/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/81/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The TRSP should not go on when it get a bogus server name from AM
> -
>
> Key: HBASE-26885
> URL: https://issues.apache.org/jira/browse/HBASE-26885
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.12
>
>
> Currently it will submit lots of unnecessary OpenRegionProcedure by retry.
> Related log looks like below, 'localhost,1,1' is the bogus server:
> {code:java}
> 2022-03-22 10:17:48,301 WARN [PEWorker-8]
> assignment.RegionRemoteProcedureBase: Can not add remote operation pid=17952,
> ppid=17951, state=RUNNABLE, locked=true;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure for region
> {ENCODED => 490391c232c7aa13f7e0d50bfe1f7235, NAME =>
> 'TestTable1,002497747,1647568640784.490391c232c7aa13f7e0d50bfe1f7235.',
> STARTKEY => '002497747', ENDKEY => ''} to server
> localhost,1,1, this usually because the server is alread dead, give up and
> mark the procedure as complete, the parent procedure will take care of this.
> org.apache.hadoop.hbase.procedure2.NoServerDispatchException: localhost,1,1;
> pid=17952, ppid=17951, state=RUNNABLE, locked=true;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:168)
> at
> org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:285)
> at
> org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:58)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965)
> 2022-03-22 10:17:48,301 DEBUG [PEWorker-8] procedure2.RootProcedureState: Add
> procedure pid=17952, ppid=17951, state=SUCCESS, locked=true;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure as the 8th
> rollback step {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26902) Bump bcprov-jdk15on from 1.60 to 1.67
[
https://issues.apache.org/jira/browse/HBASE-26902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514926#comment-17514926
]
Hudson commented on HBASE-26902:
Results for branch branch-2.5
[build #81 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/81/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/81/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/81/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/81/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/81/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Bump bcprov-jdk15on from 1.60 to 1.67
> -
>
> Key: HBASE-26902
> URL: https://issues.apache.org/jira/browse/HBASE-26902
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.12
>
>
> Dependabot generated dependency upgrade.
> https://github.com/apache/hbase/pull/4292
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26871) shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
[
https://issues.apache.org/jira/browse/HBASE-26871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514920#comment-17514920
]
Sean Busbey commented on HBASE-26871:
-
I merged the branch-2 change, but have now hit an unrelated logistics of life
thing that's going to delay me from the rest by several hours. If anyone needs
the branch-2.5 or master stuff faster feel free to step in.
> shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
> ---
>
> Key: HBASE-26871
> URL: https://issues.apache.org/jira/browse/HBASE-26871
> Project: HBase
> Issue Type: Bug
> Components: integration tests, jenkins, mapreduce
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Duo Zhang
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0
>
>
> After fixing the logging problem in HBASE-26870, we could see the actual
> error.
> {noformat}
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hadoop-3/bin/hadoop
> --config
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/output-integration/hadoop-3/hbase-conf/
> jar
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar
> importtsv
> -Dimporttsv.columns=HBASE_ROW_KEY,family1:column1,family1:column4,family1:column3
> test:example example/ -libjars
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/htrace-core4-4.1.0-incubating.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-api-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-context-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-semconv-1.0.1-alpha.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.33.jar
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.mapreduce.Driver
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:311)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HBASE-26909) hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes
[ https://issues.apache.org/jira/browse/HBASE-26909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514915#comment-17514915 ] Sean Busbey edited comment on HBASE-26909 at 3/30/22, 7:46 PM: --- that's an important distinction. if we want a refactoring to pull out the duplicate classes, hbase-shaded-mapreduce will need to show a runtime dependency on the hbase-shaded-client-byo-hadoop artifact and not the one that include hadoop classes. the intended use of hbase-shaded-mapreduce is always with an existing hadoop; the hadoop classes should come from that existing install. was (Author: busbey): that's an important distinction. if we want a refactoring to pull out the duplicate classes, hbase-shaded-mapreduce will need to show a runtime dependency on the hbase-shaded-client-byo-hadoop artifact and not the one that include hadoop classes. the intended use of abase-shaded-mapreduce is always with an existing hadoop; the hadoop classes should come from that existing install. > hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes > -- > > Key: HBASE-26909 > URL: https://issues.apache.org/jira/browse/HBASE-26909 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Priority: Major > > We supply 2 primary artifacts for end-users to consume: > * hbase-shaded-client, which is for general use > * hbase-shaded-mapreduce, which is for use when you need to connect to hbase > via mapreduce. For example, TableInputFormat > The problem is that these artifacts expose tons of duplicate classes. One > example (among many) is org.apache.hadoop.hbase.Cell, which appears in both > jars. > This may not be a problem if your projects are always very isolated – either > doing mapreduce, or not. In that case you just depend in the one you need. > Many users might exist in much more complicated environments where > dependencies tend to bleed along more between projects. Here's an > illustration: > * Imagine a project FooService, which includes two modules FooServiceRestWeb > (for the rest http resources) and FooServiceData (which includes DAOs for > accessing data). FooServiceRestWeb depends on FooServiceData to access hbase. > In this case, FooServiceData should depend on hbase-shaded-client. > * Now imagine another project FooPipeline, which has modules > FooPipelineHadoop (with M/R jobs for processing data) and FooPipelineData > (which has some DAOs for accessing data). In this case, FooPipelineData might > depend on hbase-shaded-mapreduce since the context is intended for M/R. > * The problem arises when suddenly we want to include some data from > FooService into our pipeline. The most straightforward way to achieve this is > by depending on FooServiceData, which has all of he DAOs for that data but > also depends on hbase-shaded-client. At this point you have a problem, > because FooPipelineHadoop now depends on both hbase-shaded-mapreduce and > hbase-shaded-client. > (Note, this obviously skirts around potential microservice solutions like > only accessing FooService's data through the API... it's just for > illustration, and it does come up.) > From a plain java perspective, having these 2 jars on the classpath is > somewhat wasteful but not a huge issue since the implementations are all the > same. > From a maven perspective, it's problematic because the maven dependency > plugin will complain about the conflicting classes. > One potential fix is to add exclusions to the FooServiceData dependency, to > avoid pulling in hbase-shaded-client. This works on a one-off basis but is > much more painful in a large and complicated environment where this may come > up hundreds of times. > A better fix in my opinion is to make hbase-shaded-mapreduce depend on > hbase-shaded-client and then only expose the classes that aren't already > exposed by the shaded client. > [~busbey] also mentioned a BOM being a potential solution, but I don't have > experience with that. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26909) hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes
[ https://issues.apache.org/jira/browse/HBASE-26909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514915#comment-17514915 ] Sean Busbey commented on HBASE-26909: - that's an important distinction. if we want a refactoring to pull out the duplicate classes, hbase-shaded-mapreduce will need to show a runtime dependency on the hbase-shaded-client-byo-hadoop artifact and not the one that include hadoop classes. the intended use of abase-shaded-mapreduce is always with an existing hadoop; the hadoop classes should come from that existing install. > hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes > -- > > Key: HBASE-26909 > URL: https://issues.apache.org/jira/browse/HBASE-26909 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Priority: Major > > We supply 2 primary artifacts for end-users to consume: > * hbase-shaded-client, which is for general use > * hbase-shaded-mapreduce, which is for use when you need to connect to hbase > via mapreduce. For example, TableInputFormat > The problem is that these artifacts expose tons of duplicate classes. One > example (among many) is org.apache.hadoop.hbase.Cell, which appears in both > jars. > This may not be a problem if your projects are always very isolated – either > doing mapreduce, or not. In that case you just depend in the one you need. > Many users might exist in much more complicated environments where > dependencies tend to bleed along more between projects. Here's an > illustration: > * Imagine a project FooService, which includes two modules FooServiceRestWeb > (for the rest http resources) and FooServiceData (which includes DAOs for > accessing data). FooServiceRestWeb depends on FooServiceData to access hbase. > In this case, FooServiceData should depend on hbase-shaded-client. > * Now imagine another project FooPipeline, which has modules > FooPipelineHadoop (with M/R jobs for processing data) and FooPipelineData > (which has some DAOs for accessing data). In this case, FooPipelineData might > depend on hbase-shaded-mapreduce since the context is intended for M/R. > * The problem arises when suddenly we want to include some data from > FooService into our pipeline. The most straightforward way to achieve this is > by depending on FooServiceData, which has all of he DAOs for that data but > also depends on hbase-shaded-client. At this point you have a problem, > because FooPipelineHadoop now depends on both hbase-shaded-mapreduce and > hbase-shaded-client. > (Note, this obviously skirts around potential microservice solutions like > only accessing FooService's data through the API... it's just for > illustration, and it does come up.) > From a plain java perspective, having these 2 jars on the classpath is > somewhat wasteful but not a huge issue since the implementations are all the > same. > From a maven perspective, it's problematic because the maven dependency > plugin will complain about the conflicting classes. > One potential fix is to add exclusions to the FooServiceData dependency, to > avoid pulling in hbase-shaded-client. This works on a one-off basis but is > much more painful in a large and complicated environment where this may come > up hundreds of times. > A better fix in my opinion is to make hbase-shaded-mapreduce depend on > hbase-shaded-client and then only expose the classes that aren't already > exposed by the shaded client. > [~busbey] also mentioned a BOM being a potential solution, but I don't have > experience with that. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] busbey merged pull request #4297: HBASE-26871 addendum. use the jar command from JAVA_HOME
busbey merged pull request #4297: URL: https://github.com/apache/hbase/pull/4297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26871) shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
[
https://issues.apache.org/jira/browse/HBASE-26871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514908#comment-17514908
]
Andrew Kyle Purtell commented on HBASE-26871:
-
Approved
> shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
> ---
>
> Key: HBASE-26871
> URL: https://issues.apache.org/jira/browse/HBASE-26871
> Project: HBase
> Issue Type: Bug
> Components: integration tests, jenkins, mapreduce
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Duo Zhang
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0
>
>
> After fixing the logging problem in HBASE-26870, we could see the actual
> error.
> {noformat}
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hadoop-3/bin/hadoop
> --config
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/output-integration/hadoop-3/hbase-conf/
> jar
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar
> importtsv
> -Dimporttsv.columns=HBASE_ROW_KEY,family1:column1,family1:column4,family1:column3
> test:example example/ -libjars
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/htrace-core4-4.1.0-incubating.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-api-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-context-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-semconv-1.0.1-alpha.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.33.jar
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.mapreduce.Driver
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:311)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26909) hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes
[ https://issues.apache.org/jira/browse/HBASE-26909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514903#comment-17514903 ] Bryan Beaudreault commented on HBASE-26909: --- I've been diffing the contents of the jars and actually, hbase-shaded-mapreduce is _not_ a superset of hbase-shaded-client. It lacks all of the hadoop/hdfs classes, for example does not include FileSystem. Instead it's more accurate to say that hbase-shaded-mapreduce is a superset of hbase-shaded-client-byo-hadoop. > hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes > -- > > Key: HBASE-26909 > URL: https://issues.apache.org/jira/browse/HBASE-26909 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Priority: Major > > We supply 2 primary artifacts for end-users to consume: > * hbase-shaded-client, which is for general use > * hbase-shaded-mapreduce, which is for use when you need to connect to hbase > via mapreduce. For example, TableInputFormat > The problem is that these artifacts expose tons of duplicate classes. One > example (among many) is org.apache.hadoop.hbase.Cell, which appears in both > jars. > This may not be a problem if your projects are always very isolated – either > doing mapreduce, or not. In that case you just depend in the one you need. > Many users might exist in much more complicated environments where > dependencies tend to bleed along more between projects. Here's an > illustration: > * Imagine a project FooService, which includes two modules FooServiceRestWeb > (for the rest http resources) and FooServiceData (which includes DAOs for > accessing data). FooServiceRestWeb depends on FooServiceData to access hbase. > In this case, FooServiceData should depend on hbase-shaded-client. > * Now imagine another project FooPipeline, which has modules > FooPipelineHadoop (with M/R jobs for processing data) and FooPipelineData > (which has some DAOs for accessing data). In this case, FooPipelineData might > depend on hbase-shaded-mapreduce since the context is intended for M/R. > * The problem arises when suddenly we want to include some data from > FooService into our pipeline. The most straightforward way to achieve this is > by depending on FooServiceData, which has all of he DAOs for that data but > also depends on hbase-shaded-client. At this point you have a problem, > because FooPipelineHadoop now depends on both hbase-shaded-mapreduce and > hbase-shaded-client. > (Note, this obviously skirts around potential microservice solutions like > only accessing FooService's data through the API... it's just for > illustration, and it does come up.) > From a plain java perspective, having these 2 jars on the classpath is > somewhat wasteful but not a huge issue since the implementations are all the > same. > From a maven perspective, it's problematic because the maven dependency > plugin will complain about the conflicting classes. > One potential fix is to add exclusions to the FooServiceData dependency, to > avoid pulling in hbase-shaded-client. This works on a one-off basis but is > much more painful in a large and complicated environment where this may come > up hundreds of times. > A better fix in my opinion is to make hbase-shaded-mapreduce depend on > hbase-shaded-client and then only expose the classes that aren't already > exposed by the shaded client. > [~busbey] also mentioned a BOM being a potential solution, but I don't have > experience with that. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26902) Bump bcprov-jdk15on from 1.60 to 1.67
[
https://issues.apache.org/jira/browse/HBASE-26902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514897#comment-17514897
]
Hudson commented on HBASE-26902:
Results for branch master
[build #553 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/553/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/553/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/553/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/553/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Bump bcprov-jdk15on from 1.60 to 1.67
> -
>
> Key: HBASE-26902
> URL: https://issues.apache.org/jira/browse/HBASE-26902
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.12
>
>
> Dependabot generated dependency upgrade.
> https://github.com/apache/hbase/pull/4292
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26871) shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
[
https://issues.apache.org/jira/browse/HBASE-26871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514896#comment-17514896
]
Sean Busbey commented on HBASE-26871:
-
I think the posted results on [https://github.com/apache/hbase/pull/4297] show
the addendum working?
Could someone give me a quick review and I'll push to branch-2, branch-2.5 and
forward port the test change to master.
> shaded mapreduce and shaded byo-hadoop client artifacts contains no classes
> ---
>
> Key: HBASE-26871
> URL: https://issues.apache.org/jira/browse/HBASE-26871
> Project: HBase
> Issue Type: Bug
> Components: integration tests, jenkins, mapreduce
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Duo Zhang
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.5.0, 2.6.0
>
>
> After fixing the logging problem in HBASE-26870, we could see the actual
> error.
> {noformat}
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hadoop-3/bin/hadoop
> --config
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/output-integration/hadoop-3/hbase-conf/
> jar
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar
> importtsv
> -Dimporttsv.columns=HBASE_ROW_KEY,family1:column1,family1:column4,family1:column3
> test:example example/ -libjars
> /home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/shaded-clients/hbase-shaded-mapreduce-2.6.0-SNAPSHOT.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/htrace-core4-4.1.0-incubating.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-api-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-context-1.0.1.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/opentelemetry-semconv-1.0.1-alpha.jar:/home/jenkins/jenkins-home/workspace/ase_Nightly_HBASE-26870-branch-2/hbase-client/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.33.jar
> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.mapreduce.Driver
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:311)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514879#comment-17514879 ] Andrew Kyle Purtell commented on HBASE-20312: - [~heliangjun] reassigned to you. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: LiangJun He >Priority: Major > Fix For: 3.0.0-alpha-3 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% > better read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-20312: --- Assignee: LiangJun He (was: Chance Li) > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: LiangJun He >Priority: Major > Fix For: 3.0.0-alpha-3 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% > better read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26909) hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes
[ https://issues.apache.org/jira/browse/HBASE-26909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26909: -- Description: We supply 2 primary artifacts for end-users to consume: * hbase-shaded-client, which is for general use * hbase-shaded-mapreduce, which is for use when you need to connect to hbase via mapreduce. For example, TableInputFormat The problem is that these artifacts expose tons of duplicate classes. One example (among many) is org.apache.hadoop.hbase.Cell, which appears in both jars. This may not be a problem if your projects are always very isolated – either doing mapreduce, or not. In that case you just depend in the one you need. Many users might exist in much more complicated environments where dependencies tend to bleed along more between projects. Here's an illustration: * Imagine a project FooService, which includes two modules FooServiceRestWeb (for the rest http resources) and FooServiceData (which includes DAOs for accessing data). FooServiceRestWeb depends on FooServiceData to access hbase. In this case, FooServiceData should depend on hbase-shaded-client. * Now imagine another project FooPipeline, which has modules FooPipelineHadoop (with M/R jobs for processing data) and FooPipelineData (which has some DAOs for accessing data). In this case, FooPipelineData might depend on hbase-shaded-mapreduce since the context is intended for M/R. * The problem arises when suddenly we want to include some data from FooService into our pipeline. The most straightforward way to achieve this is by depending on FooServiceData, which has all of he DAOs for that data but also depends on hbase-shaded-client. At this point you have a problem, because FooPipelineHadoop now depends on both hbase-shaded-mapreduce and hbase-shaded-client. (Note, this obviously skirts around potential microservice solutions like only accessing FooService's data through the API... it's just for illustration, and it does come up.) >From a plain java perspective, having these 2 jars on the classpath is >somewhat wasteful but not a huge issue since the implementations are all the >same. >From a maven perspective, it's problematic because the maven dependency plugin >will complain about the conflicting classes. One potential fix is to add exclusions to the FooServiceData dependency, to avoid pulling in hbase-shaded-client. This works on a one-off basis but is much more painful in a large and complicated environment where this may come up hundreds of times. A better fix in my opinion is to make hbase-shaded-mapreduce depend on hbase-shaded-client and then only expose the classes that aren't already exposed by the shaded client. [~busbey] also mentioned a BOM being a potential solution, but I don't have experience with that. was: We supply 2 primary artifacts for end-users to consume: * hbase-shaded-client, which is for general use * hbase-shaded-mapreduce, which is for use when you need to connect to hbase via mapreduce. For example, TableInputFormat The problem is that these artifacts expose tons of duplicate classes. One example (among many) is org.apache.hadoop.hbase.Cell, which appears in both jars. This may not be a problem if your projects are always very isolated – either doing mapreduce, or not. In that case you just depend in the one you need. Many users might exist in much more complicated environments where dependencies tend to bleed along more between projects. Here's an illustration: Imagine a project FooService, which includes two modules FooServiceRestWeb (for the rest http resources) and FooServiceData (which includes DAOs for accessing data). FooServiceRestWeb depends on FooServiceData to access hbase. In this case, FooServiceData should depend on hbase-shaded-client. Now imagine another project FooPipeline, which has modules FooPipelineHadoop (with M/R jobs for processing data) and FooPipelineData (which has some DAOs for accessing data). In this case, FooPipelineData might depend on hbase-shaded-mapreduce since the context is intended for M/R. The problem arises when suddenly we want to include some data from FooService into our pipeline. The most straightforward way to achieve this is by depending on FooServiceData, which has all of he DAOs for that data but also depends on hbase-shaded-client. At this point you have a problem, because FooPipelineHadoop now depends on both hbase-shaded-mapreduce and hbase-shaded-client. (Note, this obviously skirts around potential microservice solutions like only accessing FooService's data through the API... it's just for illustration, and it does come up.) >From a plain java perspective, having these 2 jars on the classpath is >somewhat wasteful but not a huge issue since the implementations are all the >same. >From a maven perspective, it's problematic because the maven dependency plugin
[jira] [Created] (HBASE-26909) hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes
Bryan Beaudreault created HBASE-26909: - Summary: hbase-shaded-mapreduce and hbase-shaded-client expose some of the same classes Key: HBASE-26909 URL: https://issues.apache.org/jira/browse/HBASE-26909 Project: HBase Issue Type: Improvement Reporter: Bryan Beaudreault We supply 2 primary artifacts for end-users to consume: * hbase-shaded-client, which is for general use * hbase-shaded-mapreduce, which is for use when you need to connect to hbase via mapreduce. For example, TableInputFormat The problem is that these artifacts expose tons of duplicate classes. One example (among many) is org.apache.hadoop.hbase.Cell, which appears in both jars. This may not be a problem if your projects are always very isolated – either doing mapreduce, or not. In that case you just depend in the one you need. Many users might exist in much more complicated environments where dependencies tend to bleed along more between projects. Here's an illustration: Imagine a project FooService, which includes two modules FooServiceRestWeb (for the rest http resources) and FooServiceData (which includes DAOs for accessing data). FooServiceRestWeb depends on FooServiceData to access hbase. In this case, FooServiceData should depend on hbase-shaded-client. Now imagine another project FooPipeline, which has modules FooPipelineHadoop (with M/R jobs for processing data) and FooPipelineData (which has some DAOs for accessing data). In this case, FooPipelineData might depend on hbase-shaded-mapreduce since the context is intended for M/R. The problem arises when suddenly we want to include some data from FooService into our pipeline. The most straightforward way to achieve this is by depending on FooServiceData, which has all of he DAOs for that data but also depends on hbase-shaded-client. At this point you have a problem, because FooPipelineHadoop now depends on both hbase-shaded-mapreduce and hbase-shaded-client. (Note, this obviously skirts around potential microservice solutions like only accessing FooService's data through the API... it's just for illustration, and it does come up.) >From a plain java perspective, having these 2 jars on the classpath is >somewhat wasteful but not a huge issue since the implementations are all the >same. >From a maven perspective, it's problematic because the maven dependency plugin >will complain about the conflicting classes. One potential fix is to add exclusions to the FooServiceData dependency, to avoid pulling in hbase-shaded-client. This works on a one-off basis but is much more painful in a large and complicated environment where this may come up hundreds of times. A better fix in my opinion is to make hbase-shaded-mapreduce depend on hbase-shaded-client and then only expose the classes that aren't already exposed by the shaded client. [~busbey] also mentioned a BOM being a potential solution, but I don't have experience with that. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4300: HBASE-26906 Remove duplicate dependency declaration (branch-2)
Apache-HBase commented on pull request #4300: URL: https://github.com/apache/hbase/pull/4300#issuecomment-1083468512 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | -1 :x: | mvninstall | 3m 0s | root in branch-2 failed. | | +1 :green_heart: | compile | 1m 53s | branch-2 passed | | -1 :x: | shadedjars | 4m 9s | branch has 13 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 10s | branch-2 passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 2m 43s | root in the patch failed. | | +1 :green_heart: | compile | 1m 46s | the patch passed | | +1 :green_heart: | javac | 1m 46s | the patch passed | | -1 :x: | shadedjars | 4m 13s | patch has 13 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 47s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 360m 33s | root in the patch passed. | | | | 386m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4300 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6fb54fbce383 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / cd09f3a3aa | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-jdk8-hadoop2-check/output/branch-mvninstall-root.txt | | shadedjars | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-jdk8-hadoop2-check/output/branch-shadedjars.txt | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-jdk8-hadoop2-check/output/patch-mvninstall-root.txt | | shadedjars | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-jdk8-hadoop2-check/output/patch-shadedjars.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/testReport/ | | Max. process+thread count | 4669 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-24917) HBase RSGroup: Table region details under RS Groups are in the onder fromat like HBase 1.x
[ https://issues.apache.org/jira/browse/HBASE-24917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Arshad reassigned HBASE-24917: --- Assignee: (was: Mohammad Arshad) > HBase RSGroup: Table region details under RS Groups are in the onder fromat > like HBase 1.x > -- > > Key: HBASE-24917 > URL: https://issues.apache.org/jira/browse/HBASE-24917 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Priority: Major > > HBase RSGroup: Table region details under RS Groups are in the older format > like HBase 1.x. > > in HBase2, Master UI will contain table details > > > > ||Namespace||Name||State||Regions||Description|| > ||OPEN||OPENING||CLOSED||CLOSING||OFFLINE||FAILED||SPLIT||Other|| > but under Regroup, table region details are different which are same as in > HBase1.x > > ||Namespace||Table||Stats||Online Regions||Offline Regions||Failed > Regions||Split Regions||Other Regions||Description|| -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] bbeaudreault commented on pull request #4180: HBASE-26807 Unify CallQueueTooBigException special pause with CallDroppedException
bbeaudreault commented on pull request #4180: URL: https://github.com/apache/hbase/pull/4180#issuecomment-1083438588 Thanks for your review nick! I'll push a commit with the minor tweaks discussed above later today or tomorrow, to give Andrew time to chime in if he has any other feedback to include as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a change in pull request #4180: HBASE-26807 Unify CallQueueTooBigException special pause with CallDroppedException
bbeaudreault commented on a change in pull request #4180:
URL: https://github.com/apache/hbase/pull/4180#discussion_r838814036
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RemoteWithExtrasException.java
##
@@ -58,15 +60,26 @@
}
public RemoteWithExtrasException(String className, String msg, final boolean
doNotRetry) {
-this(className, msg, null, -1, doNotRetry);
+this(className, msg, doNotRetry, false);
+ }
+
+ public RemoteWithExtrasException(String className, String msg, final boolean
doNotRetry,
Review comment:
Yea I think it'd be cool to remove/deprecate this class if we were to
refactor exception handling based on some of the discussion above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a change in pull request #4180: HBASE-26807 Unify CallQueueTooBigException special pause with CallDroppedException
bbeaudreault commented on a change in pull request #4180:
URL: https://github.com/apache/hbase/pull/4180#discussion_r838812984
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionConfiguration.java
##
@@ -44,6 +51,23 @@
public static final int PRIMARY_SCAN_TIMEOUT_MICROSECOND_DEFAULT = 100;
// 1s
public static final String LOG_SCANNER_ACTIVITY =
"hbase.client.log.scanner.activity";
+ /**
+ * Parameter name for client pause when server is overloaded, denoted by
+ * {@link HBaseServerException#isServerOverloaded()}
+ */
+ public static final String HBASE_CLIENT_PAUSE_FOR_SERVER_OVERLOADED =
+"hbase.client.pause.server.overloaded";
+
+ static {
+// This is added where the configs are referenced. It may be too late to
happen before
+// any user _sets_ the old cqtbe config onto a Configuration option. So we
still need
+// to handle checking both properties in parsing below. The benefit of
calling this is
+// that it should still cause Configuration to log a warning if we do end
up falling
+// through to the old deprecated config.
+Configuration.addDeprecation(
Review comment:
No this should be safe, based on the implementation of the call which is
idempotent. But maybe from the perspective of my response to your other comment
I'll just remove it for simplicity in master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bbeaudreault commented on a change in pull request #4180: HBASE-26807 Unify CallQueueTooBigException special pause with CallDroppedException
bbeaudreault commented on a change in pull request #4180:
URL: https://github.com/apache/hbase/pull/4180#discussion_r838812466
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionConfiguration.java
##
@@ -116,6 +142,20 @@
this.writeRpcTimeout = conf.getInt(HConstants.HBASE_RPC_WRITE_TIMEOUT_KEY,
conf.getInt(HConstants.HBASE_RPC_TIMEOUT_KEY,
HConstants.DEFAULT_HBASE_RPC_TIMEOUT));
+
+long pauseMs = conf.getLong(HBASE_CLIENT_PAUSE,
DEFAULT_HBASE_CLIENT_PAUSE);
+long pauseMsForServerOverloaded =
conf.getLong(HBASE_CLIENT_PAUSE_FOR_SERVER_OVERLOADED,
+ conf.getLong(HConstants.HBASE_CLIENT_PAUSE_FOR_CQTBE, pauseMs));
+if (pauseMsForServerOverloaded < pauseMs) {
Review comment:
Ok cool, i wasn't sure what the best way to handle this is. Honestly, i
didn't think about the double-warn, so good catch. What I'll do is remove this
from ConnectionConfiguration here in master. So branch-2x will have this slight
divergence where it needs to have the logic in both classes, in addition to
handling all of the classes in branch-2x that don't exist here anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-26906) Remove duplicate dependency declaration
[
https://issues.apache.org/jira/browse/HBASE-26906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-26906:
-
Status: Patch Available (was: Open)
> Remove duplicate dependency declaration
> ---
>
> Key: HBASE-26906
> URL: https://issues.apache.org/jira/browse/HBASE-26906
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 2.6.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> On branch-2 derivatives, I noticed on PR builds of JDK11+Hadoop3 that running
> tests with {{-pl}} results in a warning from maven
> {noformat}
> [WARNING]
>
>
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.hbase:hbase-build-configuration:pom:2.6.0-SNAPSHOT
>
> [WARNING]
> 'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)'
> must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar ->
> duplica
> te declaration of version ${hadoop-three.version} @
> org.apache.hbase:hbase:2.6.0-SNAPSHOT,
> /Users/ndimiduk/repos/apache/hbase/pom.xml, line 3816, column 23
>
> [WARNING]
>
>
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.hbase:hbase:pom:2.6.0-SNAPSHOT
>
> [WARNING]
> 'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)'
> must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar ->
> duplica
> te declaration of version ${hadoop-three.version} @ line 3816, column 23
>
>
> [WARNING]
>
>
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
>
> [WARNING]
>
>
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
>
> [WARNING]
> {noformat}
> I _think_ they're harmless, but we should clean them up.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26617) Use spotless to reduce the pain on fixing checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-26617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514855#comment-17514855 ] Nick Dimiduk commented on HBASE-26617: -- I'm not sure about all cases, but at least for backporting HBASE-26545, which landed on master but not branches-2 yet, I can apply the cherry-pick from master, then apply spotless, then carry on with updates for the target branch, and it's not too painful. > Use spotless to reduce the pain on fixing checkstyle issues > --- > > Key: HBASE-26617 > URL: https://issues.apache.org/jira/browse/HBASE-26617 > Project: HBase > Issue Type: New Feature > Components: build, community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > https://github.com/diffplug/spotless > Spotless is a tool for reformatting code based on your configuration file, it > supports the google java format, as well as a eclipse formatter file. > I think we could use it to format our code, based on a configuration file in > our repo, and all the developpers, no matter what IDE they use, they can just > type a mvn spotless:apply before commiting a PR, then everything is fine. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] ndimiduk commented on a change in pull request #4180: HBASE-26807 Unify CallQueueTooBigException special pause with CallDroppedException
ndimiduk commented on a change in pull request #4180:
URL: https://github.com/apache/hbase/pull/4180#discussion_r838794337
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionConfiguration.java
##
@@ -44,6 +51,23 @@
public static final int PRIMARY_SCAN_TIMEOUT_MICROSECOND_DEFAULT = 100;
// 1s
public static final String LOG_SCANNER_ACTIVITY =
"hbase.client.log.scanner.activity";
+ /**
+ * Parameter name for client pause when server is overloaded, denoted by
+ * {@link HBaseServerException#isServerOverloaded()}
+ */
+ public static final String HBASE_CLIENT_PAUSE_FOR_SERVER_OVERLOADED =
+"hbase.client.pause.server.overloaded";
+
+ static {
+// This is added where the configs are referenced. It may be too late to
happen before
+// any user _sets_ the old cqtbe config onto a Configuration option. So we
still need
+// to handle checking both properties in parsing below. The benefit of
calling this is
+// that it should still cause Configuration to log a warning if we do end
up falling
+// through to the old deprecated config.
+Configuration.addDeprecation(
Review comment:
Is there any problem from this deprecation being added twice?
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionConfiguration.java
##
@@ -116,6 +142,20 @@
this.writeRpcTimeout = conf.getInt(HConstants.HBASE_RPC_WRITE_TIMEOUT_KEY,
conf.getInt(HConstants.HBASE_RPC_TIMEOUT_KEY,
HConstants.DEFAULT_HBASE_RPC_TIMEOUT));
+
+long pauseMs = conf.getLong(HBASE_CLIENT_PAUSE,
DEFAULT_HBASE_CLIENT_PAUSE);
+long pauseMsForServerOverloaded =
conf.getLong(HBASE_CLIENT_PAUSE_FOR_SERVER_OVERLOADED,
+ conf.getLong(HConstants.HBASE_CLIENT_PAUSE_FOR_CQTBE, pauseMs));
+if (pauseMsForServerOverloaded < pauseMs) {
Review comment:
Okay this is strange. This logic (which I think is good) is replicated
from the async client. However, on master, the sync client is implemented on
top of the async client, so I think this will at best produce the warning
message twice, and otherwise have no effect. Or maybe there are places where
the ConnectionConfiguration is used, even with the sync-over-async
implementation?
It may be that this is someplace where your patches for master and
branch-2[,.*] will differ, because on the others, async and sync client
implementations are completely independent.
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RemoteWithExtrasException.java
##
@@ -58,15 +60,26 @@
}
public RemoteWithExtrasException(String className, String msg, final boolean
doNotRetry) {
-this(className, msg, null, -1, doNotRetry);
+this(className, msg, doNotRetry, false);
+ }
+
+ public RemoteWithExtrasException(String className, String msg, final boolean
doNotRetry,
Review comment:
Oh, it pains me so that this class is `IA.Public` and it extends a
Hadoop class :'(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26908) Remove warnings from meta replicas feature references in the HBase book
[ https://issues.apache.org/jira/browse/HBASE-26908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514828#comment-17514828 ] Huaxiang Sun commented on HBASE-26908: -- We use the client side MetaReplica LoadBalance mode in our production cluster. Since regions does not move frequently in our cluster, so we do not turn on "async wal replication for meta". Just want to be clear. Unless there are other users who has this feature turned on at their cluster. The plan is to run itbll with this config turn on against 2.5.0 candidate and 2.4.11, if that passes, we can remove the warning from the hbase book. > Remove warnings from meta replicas feature references in the HBase book > --- > > Key: HBASE-26908 > URL: https://issues.apache.org/jira/browse/HBASE-26908 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Andor Molnar >Assignee: Andor Molnar >Priority: Major > > Meta replicas is a new feature in HBase 2.4 and mentioned in "Use with > caution" in the docs. Given that the feature and the related "async wal > replication for meta" is actively used in production already, I'd like to > remove these warnings from the docs. > With this change, users will have more confidence in the feature. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] ndimiduk commented on pull request #4300: HBASE-26906 Remove duplicate dependency declaration (branch-2)
ndimiduk commented on pull request #4300: URL: https://github.com/apache/hbase/pull/4300#issuecomment-1083403135 > This is only for branch-2.x? Master is ok? Correct. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514792#comment-17514792 ] LiangJun He commented on HBASE-20312: - [~chancelq] , This issue can be assigned to me to move forward? Thanks. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0-alpha-3 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% > better read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-20158) Enhance regionserver self health check to avoid stale server
[ https://issues.apache.org/jira/browse/HBASE-20158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514790#comment-17514790 ] LiangJun He commented on HBASE-20158: - [~liyu], This issue can be assigned to me to move forward? Thanks. > Enhance regionserver self health check to avoid stale server > > > Key: HBASE-20158 > URL: https://issues.apache.org/jira/browse/HBASE-20158 > Project: HBase > Issue Type: New Feature >Reporter: Yu Li >Assignee: Yu Li >Priority: Major > > Currently we have many good metrics to monitor our cluster status, such as > totalCallTime/processCallTime/queueCallTime etc. But these metrics won't work > if server got stale and the client call timed out, for example during RS > fullgc or there're some bad disk on HDFS and the read IO got stuck. > We also have a periodic health check chore introduced by HBASE-7351 which > allow us to launch some external script periodically to perform some self > detection. However this won't work if the server's system resource has ran > out, for example no new native thread could be created, no new network > connection could be setup, etc. Notice that although no new thread could not > be launched, running thread won't be affected so zookeeper session is still > alive and RS still regarded as alive, but clients cannot access since no new > connection could be setup. > Here we propose a new HealthChecker called DirectHealthChecker. In this new > checker we won't launch any outer script, but picking some regions on the RS > and send some rpc request to itself, regarding the server as unhealthy if the > call failure ratio exceeds some limit, and send the metrics out to our > monitoring system. More details please refer to the coming patch -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4300: HBASE-26906 Remove duplicate dependency declaration (branch-2)
Apache-HBase commented on pull request #4300: URL: https://github.com/apache/hbase/pull/4300#issuecomment-1083339112 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | -1 :x: | mvninstall | 2m 53s | root in branch-2 failed. | | +1 :green_heart: | compile | 1m 51s | branch-2 passed | | -1 :x: | shadedjars | 3m 48s | branch has 13 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 2s | branch-2 passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 2m 30s | root in the patch failed. | | +1 :green_heart: | compile | 1m 47s | the patch passed | | +1 :green_heart: | javac | 1m 47s | the patch passed | | -1 :x: | shadedjars | 3m 47s | patch has 13 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 231m 43s | root in the patch passed. | | | | 256m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4300 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux df126e2326cd 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / cd09f3a3aa | | Default Java | AdoptOpenJDK-11.0.10+9 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-jdk11-hadoop3-check/output/branch-mvninstall-root.txt | | shadedjars | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-jdk11-hadoop3-check/output/branch-shadedjars.txt | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt | | shadedjars | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/testReport/ | | Max. process+thread count | 5913 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26245) Store region server list in master local region
[ https://issues.apache.org/jira/browse/HBASE-26245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514772#comment-17514772 ] LiangJun He commented on HBASE-26245: - In my test, if we stop the masters of the old cluster first, and then stop the regionservers, we can successfully rebuild the new cluster from the existing rootdir. Thanks [~zhangduo] for the patch. > Store region server list in master local region > --- > > Key: HBASE-26245 > URL: https://issues.apache.org/jira/browse/HBASE-26245 > Project: HBase > Issue Type: Brainstorming > Components: master, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Just a simple idea still need to be polished. > For large clusters, ZooKeeper could be a bottle neck, there are some related > issues to not track region server list as much as possible, but what if we > want to do more, just do not register region server on zk? > I think, zookeeper here is act as something like a service registry, we need > the list to get all the region servers, and also need to know the changes of > the list. > But in fact, we could also kill region server from master side, so the latter > one could be done by a periodical heartbeat check daemon on master. And for > the former one, we could store the list in master local region, so when > master restart, it could setup the region server list by load from master > local region. > And there are mainly two other side effect which are good: > 1. We do not need to list the WAL directory on HDFS to find the previous > region servers, for scheduling SCP. This could make it possible to restart a > new HBase cluster based on only the root directory. > 2. For now, region server needs to register to HMaster first, and then put > its node on ZooKeeper, if it fails between these two actions, there is no way > for HMaster to clean this dead server, as it never expires on ZooKeeper. > There should be a related issue. If we just do not store a node on zk, then > this problem is also gone. > Of course, there will still be lots of other problems, like whether we need > another heartbeat call as reportForDuty is a bit heavy as we also report the > region list, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-23970) TestUsersOperationsWithSecureHadoop fails when an existing ticket is present
[
https://issues.apache.org/jira/browse/HBASE-23970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514759#comment-17514759
]
Nick Dimiduk commented on HBASE-23970:
--
I spent some time debugging this, specifically in the context of
{{TestCustomSaslAuthenticationProviderSimpleRpcServer}}. It appears to me that
the HDFS cluster that is initialized for the test is run without specifying a
{{User}}/{{Principal}}. In my case, that means the test file system is
initialized by using the default {{User}} with the default
{{HadoopLoginModule}}, which resolves to the active ticket. For this user, no
name mapping rule is found, and the exception is thrown.
When expire said ticket, this file system is created instead with the local OS
user {{Principal}}. In this case, no mapping is attempted, and file system
initialization proceeds.
The applicable fall-back logic is implemented in
{{o.a.h.security.UserGroupInformation$HadoopLoginModule#commit()}}.
> TestUsersOperationsWithSecureHadoop fails when an existing ticket is present
>
>
> Key: HBASE-23970
> URL: https://issues.apache.org/jira/browse/HBASE-23970
> Project: HBase
> Issue Type: Test
> Components: security, test
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Nick Dimiduk
>Priority: Minor
>
> {{TestUsersOperationsWithSecureHadoop}} makes assumptions about the state of
> the machine environment. When i have an existing ticket issued and active
> (shown via {{klist}}), this test fails. Signing out of all active sessions
> allows the test to again pass.
> {noformat}
> Caused by:
> org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule:
> No rules applied to m...@example.com
> at
> org.apache.hadoop.hbase.security.TestUsersOperationsWithSecureHadoop.testAuthUtilLogin(TestUsersOperationsWithSecureHadoop.java:146)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26896) list_quota_snapshots fails with ‘ERROR NameError: uninitialized constant Shell::Commands::ListQuotaSnapshots::TABLE’
[
https://issues.apache.org/jira/browse/HBASE-26896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi updated HBASE-26896:
--
Fix Version/s: (was: 2.6.0)
> list_quota_snapshots fails with ‘ERROR NameError: uninitialized constant
> Shell::Commands::ListQuotaSnapshots::TABLE’
>
>
> Key: HBASE-26896
> URL: https://issues.apache.org/jira/browse/HBASE-26896
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.5.0, 3.0.0-alpha-3, 2.4.11
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.12
>
>
> The list_quota_snapshots command fails with
> {noformat}
> ERROR NameError: uninitialized constant
> Shell::Commands::ListQuotaSnapshots::TABLE{noformat}
> regardless of the parameters it's called with.
> Apparently, it used to work in HBase 2.2.
> I don't know enough about Ruby to tell why it used to work, and what broke
> this exactly, but using qualified constants fixes the problem.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4301: HBASE-26908 Remove warnings from meta replicas feature references in the HBase book
Apache-HBase commented on pull request #4301: URL: https://github.com/apache/hbase/pull/4301#issuecomment-1083164596 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 39s | master passed | | +0 :ok: | refguide | 1m 59s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 59s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 1m 30s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | | | | 11m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4301/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4301 | | Optional Tests | dupname asflicense refguide | | uname | Linux 98479eb1e048 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7adccd2a19 | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4301/1/yetus-general-check/output/branch-site/book.html | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4301/1/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 47 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4301/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4301: HBASE-26908 Remove warnings from meta replicas feature references in the HBase book
Apache-HBase commented on pull request #4301: URL: https://github.com/apache/hbase/pull/4301#issuecomment-1083156885 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 3m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4301/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4301 | | Optional Tests | | | uname | Linux e3699bc78c8d 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7adccd2a19 | | Max. process+thread count | 39 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4301/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4301: HBASE-26908 Remove warnings from meta replicas feature references in the HBase book
Apache-HBase commented on pull request #4301: URL: https://github.com/apache/hbase/pull/4301#issuecomment-1083156117 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 2m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4301/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4301 | | Optional Tests | | | uname | Linux 84b298cdc493 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7adccd2a19 | | Max. process+thread count | 33 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4301/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26908) Remove warnings from meta replicas feature references in the HBase book
Andor Molnar created HBASE-26908: Summary: Remove warnings from meta replicas feature references in the HBase book Key: HBASE-26908 URL: https://issues.apache.org/jira/browse/HBASE-26908 Project: HBase Issue Type: Task Components: documentation Reporter: Andor Molnar Assignee: Andor Molnar Meta replicas is a new feature in HBase 2.4 and mentioned in "Use with caution" in the docs. Given that the feature and the related "async wal replication for meta" is actively used in production already, I'd like to remove these warnings from the docs. With this change, users will have more confidence in the feature. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26907) Cannot run tests on Hadoop3 with Java11 versions having 4 version number positions
[
https://issues.apache.org/jira/browse/HBASE-26907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514672#comment-17514672
]
Nick Dimiduk commented on HBASE-26907:
--
Large tests are running for me when I use {{-Dhadoop-three.version=3.2.2}}.
> Cannot run tests on Hadoop3 with Java11 versions having 4 version number
> positions
> --
>
> Key: HBASE-26907
> URL: https://issues.apache.org/jira/browse/HBASE-26907
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Priority: Major
>
> It happened that my JDK version upgraded to 11.0.14.1. Running unit tests
> involving the HDFS mini cluster now fails with a stack trace that ends with
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Invalid Java version 11.0.14.1
>
>
> at org.eclipse.jetty.util.JavaVersion.parseJDK9(JavaVersion.java:71)
>
>
> at org.eclipse.jetty.util.JavaVersion.parse(JavaVersion.java:49)
>
>
> at org.eclipse.jetty.util.JavaVersion.(JavaVersion.java:43)
> {noformat}
> We are using hadoop-3.2.0, which uses jetty-9.3.24. This is a Jetty issue has
> been fixed upstream in Jetty via
> https://github.com/eclipse/jetty.project/issues/2090. Hadoop has upgraded its
> Jetty version to 9.4.20 in HADOOP-16152, which is available as of
> hadoop-3.2.2.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26907) Cannot run tests on Hadoop3 with Java11 versions having 4 version number positions
[
https://issues.apache.org/jira/browse/HBASE-26907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-26907:
-
Description:
It happened that my JDK version upgraded to 11.0.14.1. Running unit tests
involving the HDFS mini cluster now fails with a stack trace that ends with
{noformat}
Caused by: java.lang.IllegalArgumentException: Invalid Java version 11.0.14.1
at org.eclipse.jetty.util.JavaVersion.parseJDK9(JavaVersion.java:71)
at org.eclipse.jetty.util.JavaVersion.parse(JavaVersion.java:49)
at org.eclipse.jetty.util.JavaVersion.(JavaVersion.java:43)
{noformat}
We are using hadoop-3.2.0, which uses jetty-9.3.24. This is a Jetty issue has
been fixed upstream in Jetty via
https://github.com/eclipse/jetty.project/issues/2090. Hadoop has upgraded its
Jetty version to 9.4.20 in HADOOP-16152, which is available as of hadoop-3.2.2.
was:
It happened that my JDK version upgraded to 11.0.14.1. Running unit tests
involving the HDFS mini cluster now fails with a stack trace that ends with
{noformat}
Caused by: java.lang.IllegalArgumentException: Invalid Java version 11.0.14.1
at org.eclipse.jetty.util.JavaVersion.parseJDK9(JavaVersion.java:71)
at org.eclipse.jetty.util.JavaVersion.parse(JavaVersion.java:49)
at org.eclipse.jetty.util.JavaVersion.(JavaVersion.java:43)
{noformat}
We are using hadoop-3.2.0, which uses jetty-9.3.24. This is a Jetty issue has
been fixed upstream in Jetty via
https://github.com/eclipse/jetty.project/issues/2090. Hadoop has upgraded its
Jetty version to 9.4 in HADOOP-16152, which is available as of hadoop-3.2.2.
> Cannot run tests on Hadoop3 with Java11 versions having 4 version number
> positions
> --
>
> Key: HBASE-26907
> URL: https://issues.apache.org/jira/browse/HBASE-26907
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Priority: Major
>
> It happened that my JDK version upgraded to 11.0.14.1. Running unit tests
> involving the HDFS mini cluster now fails with a stack trace that ends with
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Invalid Java version 11.0.14.1
>
>
> at org.eclipse.jetty.util.JavaVersion.parseJDK9(JavaVersion.java:71)
>
>
> at org.eclipse.jetty.util.JavaVersion.parse(JavaVersion.java:49)
>
>
> at org.eclipse.jetty.util.JavaVersion.(JavaVersion.java:43)
> {noformat}
> We are using hadoop-3.2.0, which uses jetty-9.3.24. This is a Jetty issue has
> been fixed upstream in Jetty via
> https://github.com/eclipse/jetty.project/issues/2090. Hadoop has upgraded its
> Jetty version to 9.4.20 in HADOOP-16152, which is available as of
> hadoop-3.2.2.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26907) Cannot run tests on Hadoop3 with Java11 versions having 4 version number positions
Nick Dimiduk created HBASE-26907:
Summary: Cannot run tests on Hadoop3 with Java11 versions having 4
version number positions
Key: HBASE-26907
URL: https://issues.apache.org/jira/browse/HBASE-26907
Project: HBase
Issue Type: Task
Components: build
Reporter: Nick Dimiduk
It happened that my JDK version upgraded to 11.0.14.1. Running unit tests
involving the HDFS mini cluster now fails with a stack trace that ends with
{noformat}
Caused by: java.lang.IllegalArgumentException: Invalid Java version 11.0.14.1
at org.eclipse.jetty.util.JavaVersion.parseJDK9(JavaVersion.java:71)
at org.eclipse.jetty.util.JavaVersion.parse(JavaVersion.java:49)
at org.eclipse.jetty.util.JavaVersion.(JavaVersion.java:43)
{noformat}
We are using hadoop-3.2.0, which uses jetty-9.3.24. This is a Jetty issue has
been fixed upstream in Jetty via
https://github.com/eclipse/jetty.project/issues/2090. Hadoop has upgraded its
Jetty version to 9.4 in HADOOP-16152, which is available as of hadoop-3.2.2.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26617) Use spotless to reduce the pain on fixing checkstyle issues
[
https://issues.apache.org/jira/browse/HBASE-26617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514659#comment-17514659
]
Duo Zhang commented on HBASE-26617:
---
{noformat}
Another thought I had recently is we should maybe try to land the "big spotless
cleanup" PRs in master and the various branch-2[x] at roughly the same time.
Otherwise it's going to be a real pain to be cherry-picking commits when the
branches aren't in sync on formatting. But I could be over thinking?
{noformat}
No. I agree with you that we'd better land the big PR at almost the same time,
especially, if a patch has already been committed to master, and we want to
backport them to other branches, then we should commit them before landing the
big reformat PR, otherwise it will be a big pain for the contributor or the
committers for cherry-picking...
> Use spotless to reduce the pain on fixing checkstyle issues
> ---
>
> Key: HBASE-26617
> URL: https://issues.apache.org/jira/browse/HBASE-26617
> Project: HBase
> Issue Type: New Feature
> Components: build, community
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> https://github.com/diffplug/spotless
> Spotless is a tool for reformatting code based on your configuration file, it
> supports the google java format, as well as a eclipse formatter file.
> I think we could use it to format our code, based on a configuration file in
> our repo, and all the developpers, no matter what IDE they use, they can just
> type a mvn spotless:apply before commiting a PR, then everything is fine.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4299: HBASE-26885 The TRSP should not go on when it get a bogus server name…
Apache-HBase commented on pull request #4299: URL: https://github.com/apache/hbase/pull/4299#issuecomment-1083081032 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 51s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | master passed | | +1 :green_heart: | compile | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 4m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | the patch passed | | +1 :green_heart: | compile | 0m 41s | the patch passed | | +1 :green_heart: | javac | 0m 41s | the patch passed | | +1 :green_heart: | shadedjars | 4m 20s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 210m 37s | hbase-server in the patch passed. | | | | 230m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4299 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 71822c231dd9 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7adccd2a19 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/1/testReport/ | | Max. process+thread count | 2469 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4300: HBASE-26906 Remove duplicate dependency declaration (branch-2)
Apache-HBase commented on pull request #4300: URL: https://github.com/apache/hbase/pull/4300#issuecomment-108303 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | -1 :x: | mvninstall | 2m 53s | root in branch-2 failed. | | +1 :green_heart: | compile | 7m 36s | branch-2 passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 2m 32s | root in the patch failed. | | +1 :green_heart: | compile | 7m 26s | the patch passed | | +1 :green_heart: | javac | 7m 26s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | -1 :x: | hadoopcheck | 4m 18s | The patch causes 13 errors with Hadoop v3.1.2. | | -1 :x: | hadoopcheck | 8m 41s | The patch causes 13 errors with Hadoop v3.2.1. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 32m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4300 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile | | uname | Linux d2fa230585ec 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / cd09f3a3aa | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-general-check/output/branch-mvninstall-root.txt | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/artifact/yetus-general-check/output/patch-javac-3.2.1.txt | | Max. process+thread count | 138 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4300/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4300: HBASE-26906 Remove duplicate dependency declaration (branch-2)
Apache9 commented on pull request #4300: URL: https://github.com/apache/hbase/pull/4300#issuecomment-1083074589 This is only for branch-2.x? Master is ok? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4299: HBASE-26885 The TRSP should not go on when it get a bogus server name…
Apache-HBase commented on pull request #4299: URL: https://github.com/apache/hbase/pull/4299#issuecomment-1083063789 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 18s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | master passed | | +1 :green_heart: | compile | 0m 54s | master passed | | +1 :green_heart: | shadedjars | 4m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 4m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 29s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 192m 45s | hbase-server in the patch passed. | | | | 213m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4299 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2ca3bc9a1ee5 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7adccd2a19 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/1/testReport/ | | Max. process+thread count | 3058 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26906) Remove duplicate dependency declaration
[
https://issues.apache.org/jira/browse/HBASE-26906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-26906:
-
Affects Version/s: 2.6.0
> Remove duplicate dependency declaration
> ---
>
> Key: HBASE-26906
> URL: https://issues.apache.org/jira/browse/HBASE-26906
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 2.6.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> On branch-2 derivatives, I noticed on PR builds of JDK11+Hadoop3 that running
> tests with {{-pl}} results in a warning from maven
> {noformat}
> [WARNING]
>
>
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.hbase:hbase-build-configuration:pom:2.6.0-SNAPSHOT
>
> [WARNING]
> 'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)'
> must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar ->
> duplica
> te declaration of version ${hadoop-three.version} @
> org.apache.hbase:hbase:2.6.0-SNAPSHOT,
> /Users/ndimiduk/repos/apache/hbase/pom.xml, line 3816, column 23
>
> [WARNING]
>
>
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.hbase:hbase:pom:2.6.0-SNAPSHOT
>
> [WARNING]
> 'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)'
> must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar ->
> duplica
> te declaration of version ${hadoop-three.version} @ line 3816, column 23
>
>
> [WARNING]
>
>
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
>
> [WARNING]
>
>
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
>
> [WARNING]
> {noformat}
> I _think_ they're harmless, but we should clean them up.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] ndimiduk opened a new pull request #4300: HBASE-26906 Remove duplicate dependency declaration
ndimiduk opened a new pull request #4300: URL: https://github.com/apache/hbase/pull/4300 I chose to keep the declaration that has more exclusions. That decision was arbitrary. The difference was `javax.servlet:servlet-api`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26906) Remove duplicate dependency declaration
Nick Dimiduk created HBASE-26906:
Summary: Remove duplicate dependency declaration
Key: HBASE-26906
URL: https://issues.apache.org/jira/browse/HBASE-26906
Project: HBase
Issue Type: Task
Components: build
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
On branch-2 derivatives, I noticed on PR builds of JDK11+Hadoop3 that running
tests with {{-pl}} results in a warning from maven
{noformat}
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.hbase:hbase-build-configuration:pom:2.6.0-SNAPSHOT
[WARNING]
'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)'
must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar ->
duplica
te declaration of version ${hadoop-three.version} @
org.apache.hbase:hbase:2.6.0-SNAPSHOT,
/Users/ndimiduk/repos/apache/hbase/pom.xml, line 3816, column 23
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.hbase:hbase:pom:2.6.0-SNAPSHOT
[WARNING]
'profiles.profile[hadoop-3.0].dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)'
must be unique: org.apache.hadoop:hadoop-mapreduce-client-app:test-jar ->
duplica
te declaration of version ${hadoop-three.version} @ line 3816, column 23
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten
the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
[WARNING]
{noformat}
I _think_ they're harmless, but we should clean them up.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26617) Use spotless to reduce the pain on fixing checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-26617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514632#comment-17514632 ] Bryan Beaudreault commented on HBASE-26617: --- Another thought I had recently is we should maybe try to land the "big spotless cleanup" PRs in master and the various branch-2[x] at roughly the same time. Otherwise it's going to be a real pain to be cherry-picking commits when the branches aren't in sync on formatting. But I could be over thinking? > Use spotless to reduce the pain on fixing checkstyle issues > --- > > Key: HBASE-26617 > URL: https://issues.apache.org/jira/browse/HBASE-26617 > Project: HBase > Issue Type: New Feature > Components: build, community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > https://github.com/diffplug/spotless > Spotless is a tool for reformatting code based on your configuration file, it > supports the google java format, as well as a eclipse formatter file. > I think we could use it to format our code, based on a configuration file in > our repo, and all the developpers, no matter what IDE they use, they can just > type a mvn spotless:apply before commiting a PR, then everything is fine. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26617) Use spotless to reduce the pain on fixing checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-26617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514629#comment-17514629 ] Bryan Beaudreault commented on HBASE-26617: --- Yea I was thinking maybe it makes sense to wait a week or 2 before doing the big format. In the meantime let people commit spotless cleanups in their existing patches, for the files they are touching. So for my current PRs, once they gets approval I can commit a single "spotless cleanup" commit. The benefit of waiting is that it might flush out more issues like this before we do a giant 100k line commit to reformat everything (like Andrew's branch-2.4 patch is). I agree with Nick that I find his proposed change significantly easier to read – a little more lines of code, but worth it? > Use spotless to reduce the pain on fixing checkstyle issues > --- > > Key: HBASE-26617 > URL: https://issues.apache.org/jira/browse/HBASE-26617 > Project: HBase > Issue Type: New Feature > Components: build, community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > https://github.com/diffplug/spotless > Spotless is a tool for reformatting code based on your configuration file, it > supports the google java format, as well as a eclipse formatter file. > I think we could use it to format our code, based on a configuration file in > our repo, and all the developpers, no matter what IDE they use, they can just > type a mvn spotless:apply before commiting a PR, then everything is fine. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26617) Use spotless to reduce the pain on fixing checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-26617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514625#comment-17514625 ] Duo Zhang commented on HBASE-26617: --- This is what our current eclipse formatter does. I would say we never use the style you proposed here in the past. I've been using google java format, it will format the code like what you proposed here, IIRC. But I've asked on dev list, we think it will cause a big reformat on our code base which is not good. But anyway, since we plan to do a big reformat now, maybe we could just tweak the eclipse formatter file this time to do all the changes we want, before we commit the big reformat patch? > Use spotless to reduce the pain on fixing checkstyle issues > --- > > Key: HBASE-26617 > URL: https://issues.apache.org/jira/browse/HBASE-26617 > Project: HBase > Issue Type: New Feature > Components: build, community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > https://github.com/diffplug/spotless > Spotless is a tool for reformatting code based on your configuration file, it > supports the google java format, as well as a eclipse formatter file. > I think we could use it to format our code, based on a configuration file in > our repo, and all the developpers, no matter what IDE they use, they can just > type a mvn spotless:apply before commiting a PR, then everything is fine. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26617) Use spotless to reduce the pain on fixing checkstyle issues
[
https://issues.apache.org/jira/browse/HBASE-26617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514614#comment-17514614
]
Nick Dimiduk commented on HBASE-26617:
--
Changes like this one sucks. IMO, the latter is more difficult to read. Can we
tweak this?
{noformat}
protected static final MiniClusterRule miniClusterRule =
MiniClusterRule.newBuilder()
-.setMiniClusterOption(StartMiniClusterOption.builder()
- .numWorkers(3)
- .build())
-.build();
+
.setMiniClusterOption(StartMiniClusterOption.builder().numWorkers(3).build()).build();
{noformat}
> Use spotless to reduce the pain on fixing checkstyle issues
> ---
>
> Key: HBASE-26617
> URL: https://issues.apache.org/jira/browse/HBASE-26617
> Project: HBase
> Issue Type: New Feature
> Components: build, community
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> https://github.com/diffplug/spotless
> Spotless is a tool for reformatting code based on your configuration file, it
> supports the google java format, as well as a eclipse formatter file.
> I think we could use it to format our code, based on a configuration file in
> our repo, and all the developpers, no matter what IDE they use, they can just
> type a mvn spotless:apply before commiting a PR, then everything is fine.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26617) Use spotless to reduce the pain on fixing checkstyle issues
[ https://issues.apache.org/jira/browse/HBASE-26617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514612#comment-17514612 ] Nick Dimiduk commented on HBASE-26617: -- It looks like our spotless config is re-inserting line breaks in the imports list, at least on branch-2. I believe this deviates from our checkstyle and editorconfig configurations. Is this intentional? > Use spotless to reduce the pain on fixing checkstyle issues > --- > > Key: HBASE-26617 > URL: https://issues.apache.org/jira/browse/HBASE-26617 > Project: HBase > Issue Type: New Feature > Components: build, community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > https://github.com/diffplug/spotless > Spotless is a tool for reformatting code based on your configuration file, it > supports the google java format, as well as a eclipse formatter file. > I think we could use it to format our code, based on a configuration file in > our repo, and all the developpers, no matter what IDE they use, they can just > type a mvn spotless:apply before commiting a PR, then everything is fine. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4299: HBASE-26885 The TRSP should not go on when it get a bogus server name…
Apache-HBase commented on pull request #4299: URL: https://github.com/apache/hbase/pull/4299#issuecomment-1082821069 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 24s | master passed | | +1 :green_heart: | compile | 2m 17s | master passed | | +1 :green_heart: | checkstyle | 0m 37s | master passed | | +1 :green_heart: | spotbugs | 1m 20s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 8s | the patch passed | | +1 :green_heart: | compile | 2m 13s | the patch passed | | +1 :green_heart: | javac | 2m 13s | the patch passed | | +1 :green_heart: | checkstyle | 0m 35s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 27s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 1m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 30m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4299 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux c513c4cefbc1 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7adccd2a19 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 60 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4299/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26245) Store region server list in master local region
[ https://issues.apache.org/jira/browse/HBASE-26245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514493#comment-17514493 ] LiangJun He commented on HBASE-26245: - I have a related problem [HBASE-26898|https://issues.apache.org/jira/browse/HBASE-26898] , I will test with this patch, verified if it solves my problem. > Store region server list in master local region > --- > > Key: HBASE-26245 > URL: https://issues.apache.org/jira/browse/HBASE-26245 > Project: HBase > Issue Type: Brainstorming > Components: master, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Just a simple idea still need to be polished. > For large clusters, ZooKeeper could be a bottle neck, there are some related > issues to not track region server list as much as possible, but what if we > want to do more, just do not register region server on zk? > I think, zookeeper here is act as something like a service registry, we need > the list to get all the region servers, and also need to know the changes of > the list. > But in fact, we could also kill region server from master side, so the latter > one could be done by a periodical heartbeat check daemon on master. And for > the former one, we could store the list in master local region, so when > master restart, it could setup the region server list by load from master > local region. > And there are mainly two other side effect which are good: > 1. We do not need to list the WAL directory on HDFS to find the previous > region servers, for scheduling SCP. This could make it possible to restart a > new HBase cluster based on only the root directory. > 2. For now, region server needs to register to HMaster first, and then put > its node on ZooKeeper, if it fails between these two actions, there is no way > for HMaster to clean this dead server, as it never expires on ZooKeeper. > There should be a related issue. If we just do not store a node on zk, then > this problem is also gone. > Of course, there will still be lots of other problems, like whether we need > another heartbeat call as reportForDuty is a bit heavy as we also report the > region list, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #4295: HBASE-26901 delete with null columnQualifier occurs NullPointerException when NewVersionBehavior is on
Apache-HBase commented on pull request #4295: URL: https://github.com/apache/hbase/pull/4295#issuecomment-1082737992 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 17s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 5m 25s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | | -0 :warning: | patch | 6m 10s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 5m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 243m 44s | hbase-server in the patch passed. | | | | 267m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4295/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4295 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e95325d458f5 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7adccd2a19 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4295/2/testReport/ | | Max. process+thread count | 2941 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4295/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4295: HBASE-26901 delete with null columnQualifier occurs NullPointerException when NewVersionBehavior is on
Apache-HBase commented on pull request #4295: URL: https://github.com/apache/hbase/pull/4295#issuecomment-1082694997 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 57s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 5s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | shadedjars | 3m 49s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | master passed | | -0 :warning: | patch | 4m 29s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 3m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 197m 18s | hbase-server in the patch passed. | | | | 217m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4295/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4295 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 092f16c95635 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7adccd2a19 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4295/2/testReport/ | | Max. process+thread count | 2576 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4295/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26885) The TRSP should not go on when it get a bogus server name from AM
[
https://issues.apache.org/jira/browse/HBASE-26885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514473#comment-17514473
]
Zheng Wang commented on HBASE-26885:
Add an addendum PR.
We should throw exception instead of return directly, avoid execute frequently.
> The TRSP should not go on when it get a bogus server name from AM
> -
>
> Key: HBASE-26885
> URL: https://issues.apache.org/jira/browse/HBASE-26885
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.12
>
>
> Currently it will submit lots of unnecessary OpenRegionProcedure by retry.
> Related log looks like below, 'localhost,1,1' is the bogus server:
> {code:java}
> 2022-03-22 10:17:48,301 WARN [PEWorker-8]
> assignment.RegionRemoteProcedureBase: Can not add remote operation pid=17952,
> ppid=17951, state=RUNNABLE, locked=true;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure for region
> {ENCODED => 490391c232c7aa13f7e0d50bfe1f7235, NAME =>
> 'TestTable1,002497747,1647568640784.490391c232c7aa13f7e0d50bfe1f7235.',
> STARTKEY => '002497747', ENDKEY => ''} to server
> localhost,1,1, this usually because the server is alread dead, give up and
> mark the procedure as complete, the parent procedure will take care of this.
> org.apache.hadoop.hbase.procedure2.NoServerDispatchException: localhost,1,1;
> pid=17952, ppid=17951, state=RUNNABLE, locked=true;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:168)
> at
> org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:285)
> at
> org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:58)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965)
> 2022-03-22 10:17:48,301 DEBUG [PEWorker-8] procedure2.RootProcedureState: Add
> procedure pid=17952, ppid=17951, state=SUCCESS, locked=true;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure as the 8th
> rollback step {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] bsglz opened a new pull request #4299: HBASE-26885 The TRSP should not go on when it get a bogus server name…
bsglz opened a new pull request #4299: URL: https://github.com/apache/hbase/pull/4299 Throw HBaseIOException here to make the backoff policy take effect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz closed pull request #4298: HBASE-26885 The TRSP should not go on when it get a bogus server name from AM [addendum]
bsglz closed pull request #4298: URL: https://github.com/apache/hbase/pull/4298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz opened a new pull request #4298: HBASE-26885 The TRSP should not go on when it get a bogus server name from AM [addendum]
bsglz opened a new pull request #4298: URL: https://github.com/apache/hbase/pull/4298 Throw HBaseIOException here to make the backoff policy take effect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4297: HBASE-26871 addendum. use the jar command from JAVA_HOME
Apache-HBase commented on pull request #4297: URL: https://github.com/apache/hbase/pull/4297#issuecomment-1082666964 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 53s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 3m 2s | root in branch-2 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 8s | hbase-shaded-check-invariants in the patch passed. | | +1 :green_heart: | unit | 0m 7s | hbase-shaded-with-hadoop-check-invariants in the patch passed. | | | | 8m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4297/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4297 | | Optional Tests | unit | | uname | Linux 836f19b3cef8 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / cd09f3a3aa | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4297/1/artifact/yetus-jdk11-hadoop3-check/output/branch-mvninstall-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4297/1/testReport/ | | Max. process+thread count | 71 (vs. ulimit of 12500) | | modules | C: hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants U: hbase-shaded | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4297/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4297: HBASE-26871 addendum. use the jar command from JAVA_HOME
Apache-HBase commented on pull request #4297: URL: https://github.com/apache/hbase/pull/4297#issuecomment-1082666316 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 51s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 2m 45s | root in branch-2 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 7s | hbase-shaded-check-invariants in the patch passed. | | +1 :green_heart: | unit | 0m 7s | hbase-shaded-with-hadoop-check-invariants in the patch passed. | | | | 7m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4297/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4297 | | Optional Tests | unit | | uname | Linux a76a1e5f92fa 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / cd09f3a3aa | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4297/1/artifact/yetus-jdk8-hadoop2-check/output/branch-mvninstall-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4297/1/testReport/ | | Max. process+thread count | 64 (vs. ulimit of 12500) | | modules | C: hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants U: hbase-shaded | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4297/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #4223: HBASE-26838 Junit jar is not included in the hbase tar ball, causing …
busbey commented on pull request #4223: URL: https://github.com/apache/hbase/pull/4223#issuecomment-1082663937 > Ping @busbey @ndimiduk , considering mine and Josh's latest comments observations, would you still see this approach as -1, or can we go ahead with this solution for now, provided that I add reg-guide/readme comments/javadocs explaining the pre-condition for test libs to be automatically loaded into the CP by hbase script? yeah that's fine by me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4297: HBASE-26871 addendum. use the jar command from JAVA_HOME
Apache-HBase commented on pull request #4297: URL: https://github.com/apache/hbase/pull/4297#issuecomment-1082663386 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 58s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 4s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 2m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4297/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4297 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux ca7acc7d1439 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / cd09f3a3aa | | Max. process+thread count | 34 (vs. ulimit of 12500) | | modules | C: hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants U: hbase-shaded | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4297/1/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26902) Bump bcprov-jdk15on from 1.60 to 1.67
[
https://issues.apache.org/jira/browse/HBASE-26902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514451#comment-17514451
]
Hudson commented on HBASE-26902:
Results for branch branch-2
[build #503 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/503/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/503/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/503/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/503/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/503/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Bump bcprov-jdk15on from 1.60 to 1.67
> -
>
> Key: HBASE-26902
> URL: https://issues.apache.org/jira/browse/HBASE-26902
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.12
>
>
> Dependabot generated dependency upgrade.
> https://github.com/apache/hbase/pull/4292
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26885) The TRSP should not go on when it get a bogus server name from AM
[
https://issues.apache.org/jira/browse/HBASE-26885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514450#comment-17514450
]
Hudson commented on HBASE-26885:
Results for branch branch-2
[build #503 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/503/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/503/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/503/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/503/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/503/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The TRSP should not go on when it get a bogus server name from AM
> -
>
> Key: HBASE-26885
> URL: https://issues.apache.org/jira/browse/HBASE-26885
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.12
>
>
> Currently it will submit lots of unnecessary OpenRegionProcedure by retry.
> Related log looks like below, 'localhost,1,1' is the bogus server:
> {code:java}
> 2022-03-22 10:17:48,301 WARN [PEWorker-8]
> assignment.RegionRemoteProcedureBase: Can not add remote operation pid=17952,
> ppid=17951, state=RUNNABLE, locked=true;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure for region
> {ENCODED => 490391c232c7aa13f7e0d50bfe1f7235, NAME =>
> 'TestTable1,002497747,1647568640784.490391c232c7aa13f7e0d50bfe1f7235.',
> STARTKEY => '002497747', ENDKEY => ''} to server
> localhost,1,1, this usually because the server is alread dead, give up and
> mark the procedure as complete, the parent procedure will take care of this.
> org.apache.hadoop.hbase.procedure2.NoServerDispatchException: localhost,1,1;
> pid=17952, ppid=17951, state=RUNNABLE, locked=true;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:168)
> at
> org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:285)
> at
> org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:58)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965)
> 2022-03-22 10:17:48,301 DEBUG [PEWorker-8] procedure2.RootProcedureState: Add
> procedure pid=17952, ppid=17951, state=SUCCESS, locked=true;
> org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure as the 8th
> rollback step {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
