[GitHub] [hbase] Apache-HBase commented on pull request #2514: HBASE-25166 MobFileCompactionChore is closing the master's shared clu…
Apache-HBase commented on pull request #2514: URL: https://github.com/apache/hbase/pull/2514#issuecomment-710748683 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 0m 56s | master passed | | +1 :green_heart: | shadedjars | 6m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | +1 :green_heart: | compile | 1m 6s | the patch passed | | +1 :green_heart: | javac | 1m 6s | the patch passed | | +1 :green_heart: | shadedjars | 7m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 143m 15s | hbase-server in the patch failed. | | | | 171m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2514/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2514 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c58e5ca8a456 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e6c78f4037 | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2514/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2514/6/testReport/ | | Max. process+thread count | 4828 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2514/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2514: HBASE-25166 MobFileCompactionChore is closing the master's shared clu…
Apache-HBase commented on pull request #2514: URL: https://github.com/apache/hbase/pull/2514#issuecomment-710747257 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 1m 8s | the patch passed | | +1 :green_heart: | javac | 1m 8s | the patch passed | | +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 132m 54s | hbase-server in the patch passed. | | | | 160m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2514/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2514 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8320bc856936 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e6c78f4037 | | Default Java | 2020-01-14 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2514/6/testReport/ | | Max. process+thread count | 4368 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2514/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
Apache-HBase commented on pull request #2445: URL: https://github.com/apache/hbase/pull/2445#issuecomment-710736283 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 1s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 4s | master passed | | +1 :green_heart: | checkstyle | 2m 34s | master passed | | +1 :green_heart: | spotbugs | 19m 17s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | -0 :warning: | checkstyle | 2m 34s | root: The patch generated 76 new + 962 unchanged - 5 fixed = 1038 total (was 967) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 18m 54s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 7m 28s | the patch passed | | -1 :x: | spotbugs | 1m 12s | hbase-client generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) | | -1 :x: | spotbugs | 2m 18s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -1 :x: | spotbugs | 10m 13s | root generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) | ||| _ Other Tests _ | | -1 :x: | asflicense | 1m 26s | The patch generated 2 ASF License warnings. | | | | 92m 28s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-client | | | Nullcheck of other at line 833 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.isOverlap(RegionInfo) At RegionInfo.java:833 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.isOverlap(RegionInfo) At RegionInfo.java:[line 832] | | | Nullcheck of rhs at line 113 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.lambda$static$0(RegionInfo, RegionInfo) At RegionInfo.java:113 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.lambda$static$0(RegionInfo, RegionInfo) At RegionInfo.java:[line 112] | | | Nullcheck of regionA at line 448 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.areAdjacent(RegionInfo, RegionInfo) At RegionInfo.java:448 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.areAdjacent(RegionInfo, RegionInfo) At RegionInfo.java:[line 447] | | FindBugs | module:hbase-server | | | Possible null pointer dereference of region in org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.create(HRegionServer, String, long, boolean, ServerName) Dereferenced at UnassignRegionHandler.java:region in org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.create(HRegionServer, String, long, boolean, ServerName) Dereferenced at UnassignRegionHandler.java:[line 149] | | FindBugs | module:root | | | Nullcheck of other at line 833 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.isOverlap(RegionInfo) At RegionInfo.java:833 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.isOverlap(RegionInfo) At RegionInfo.java:[line 832] | | | Nullcheck of rhs at line 113 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.lambda$static$0(RegionInfo, RegionInfo) At RegionInfo.java:113 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.lambda$static$0(RegionInfo, RegionInfo) At RegionInfo.java:[line 112] | | | Nullcheck of regionA at line 448 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.areAdjacent(RegionInfo, RegionInfo) At RegionInfo.java:448 of value previously dereferenced in org.apache.hadoop.hbase.client.RegionInfo.areAdjacent(RegionInfo, RegionInfo) At RegionInfo.java:[line 447] | | | Possible null pointer dereference of region in org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.create(HRegionServer, String, long, boolean, ServerName) Dereferenced at UnassignRegionHandler.java:region in org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.create(HReg
[GitHub] [hbase] Apache-HBase commented on pull request #2514: HBASE-25166 MobFileCompactionChore is closing the master's shared clu…
Apache-HBase commented on pull request #2514: URL: https://github.com/apache/hbase/pull/2514#issuecomment-710731386 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | master passed | | +1 :green_heart: | checkstyle | 1m 2s | master passed | | +1 :green_heart: | spotbugs | 1m 59s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | -0 :warning: | checkstyle | 1m 2s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 26s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 40m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2514/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2514 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 9042b4c5a0e1 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e6c78f4037 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2514/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2514/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
Apache-HBase commented on pull request #2445: URL: https://github.com/apache/hbase/pull/2445#issuecomment-710731328 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 36s | master passed | | +1 :green_heart: | compile | 3m 5s | master passed | | +1 :green_heart: | shadedjars | 7m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 28s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 30s | the patch passed | | +1 :green_heart: | compile | 3m 1s | the patch passed | | +1 :green_heart: | javac | 3m 1s | the patch passed | | -1 :x: | shadedjars | 4m 39s | patch has 10 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 15s | hbase-balancer generated 2 new + 0 unchanged - 2 fixed = 2 total (was 2) | | -0 :warning: | javadoc | 3m 0s | root generated 2 new + 172 unchanged - 2 fixed = 174 total (was 174) | ||| _ Other Tests _ | | -1 :x: | unit | 16m 1s | root in the patch failed. | | | | 58m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2445 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 213a12227b06 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e6c78f4037 | | Default Java | 2020-01-14 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-balancer.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-root.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/testReport/ | | Max. process+thread count | 780 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-zookeeper hbase-balancer hbase-server hbase-backup . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
Apache-HBase commented on pull request #2445: URL: https://github.com/apache/hbase/pull/2445#issuecomment-710730879 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 43s | master passed | | +1 :green_heart: | compile | 2m 25s | master passed | | +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 23s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | +1 :green_heart: | compile | 2m 21s | the patch passed | | +1 :green_heart: | javac | 2m 21s | the patch passed | | -1 :x: | shadedjars | 4m 16s | patch has 10 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 20s | hbase-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -0 :warning: | javadoc | 0m 13s | hbase-balancer generated 1 new + 0 unchanged - 1 fixed = 1 total (was 1) | | -0 :warning: | javadoc | 2m 5s | root generated 2 new + 31 unchanged - 1 fixed = 33 total (was 32) | ||| _ Other Tests _ | | -1 :x: | unit | 19m 35s | root in the patch failed. | | | | 55m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2445 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9ff64dd85fb3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e6c78f4037 | | Default Java | 1.8.0_232 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-common.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-balancer.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-root.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/testReport/ | | Max. process+thread count | 1093 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-zookeeper hbase-balancer hbase-server hbase-backup . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2445/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506777664
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -152,9 +159,19 @@
private static final int DEFAULT_RIT_STUCK_WARNING_THRESHOLD = 60 * 1000;
public static final String UNEXPECTED_STATE_REGION = "Unexpected state for ";
- private final ProcedureEvent metaAssignEvent = new ProcedureEvent<>("meta
assign");
+ private final ProcedureEvent rootAssignEvent = new ProcedureEvent<>("root
assign");
+ private final ProcedureEvent rootLoadEvent = new ProcedureEvent<>("root
load");
+
private final ProcedureEvent metaLoadEvent = new ProcedureEvent<>("meta
load");
+ private final ConcurrentSkipListMap>
metaAssignEventMap =
Review comment:
This is here so procedures that depend on particular regions of meta to
be available (eg RTSP, ORP, CRP, etc) are only waiting on the necessary regions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506777664
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -152,9 +159,19 @@
private static final int DEFAULT_RIT_STUCK_WARNING_THRESHOLD = 60 * 1000;
public static final String UNEXPECTED_STATE_REGION = "Unexpected state for ";
- private final ProcedureEvent metaAssignEvent = new ProcedureEvent<>("meta
assign");
+ private final ProcedureEvent rootAssignEvent = new ProcedureEvent<>("root
assign");
+ private final ProcedureEvent rootLoadEvent = new ProcedureEvent<>("root
load");
+
private final ProcedureEvent metaLoadEvent = new ProcedureEvent<>("meta
load");
+ private final ConcurrentSkipListMap>
metaAssignEventMap =
Review comment:
This is here so procedures that depend on particular regions of meta to
be available (eg RTSP, ORP, CRP, etc) are only waiting on the necessary
region(s).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r edited a comment on pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r edited a comment on pull request #2445: URL: https://github.com/apache/hbase/pull/2445#issuecomment-710722619 @saintstack , I did a rebase and force pushed the updated branch. I'll address your comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506776199
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
##
@@ -257,46 +327,62 @@ private void cleanupSplitDir(MasterProcedureEnv env) {
// old hbase:meta tenancy on this server; clean these up if any before
trying to remove the
// WAL directory of this server or we will fail. See archiveMetaLog
comment for more details
// on this condition.
-
env.getMasterServices().getMasterWalManager().archiveMetaLog(this.serverName);
+
env.getMasterServices().getMasterWalManager().archiveCatalogLog(this.serverName,
false);
}
splitWALManager.deleteWALDir(serverName);
} catch (IOException e) {
LOG.warn("Remove WAL directory for {} failed, ignore...{}", serverName,
e.getMessage());
}
}
- private boolean isSplittingDone(MasterProcedureEnv env, boolean splitMeta) {
+ private boolean isSplittingDone(MasterProcedureEnv env,
SplitWALManager.SplitType splitType) {
SplitWALManager splitWALManager =
env.getMasterServices().getSplitWALManager();
try {
- int wals = splitWALManager.getWALsToSplit(serverName, splitMeta).size();
- LOG.debug("Check if {} WAL splitting is done? wals={}, meta={}",
serverName, wals, splitMeta);
+ int wals = splitWALManager.getWALsToSplit(serverName, splitType).size();
+ LOG.debug("Check if {} WAL splitting is done? wals={}, SplitType={}",
+serverName, wals, splitType);
return wals == 0;
} catch (IOException e) {
LOG.warn("Get WALs of {} failed, retry...", serverName, e);
return false;
}
}
- private Procedure[] createSplittingWalProcedures(MasterProcedureEnv env,
boolean splitMeta)
+ private Procedure[] createSplittingWalProcedures(MasterProcedureEnv env,
+SplitWALManager.SplitType splitType)
throws IOException {
-LOG.info("Splitting WALs {}, isMeta: {}", this, splitMeta);
+LOG.info("Splitting WALs {}, SplitType: {}", this, splitType);
SplitWALManager splitWALManager =
env.getMasterServices().getSplitWALManager();
-List procedures = splitWALManager.splitWALs(serverName,
splitMeta);
+List procedures = splitWALManager.splitWALs(serverName,
splitType);
return procedures.toArray(new Procedure[procedures.size()]);
}
private boolean filterDefaultMetaRegions() {
if (regionsOnCrashedServer == null) {
return false;
}
-regionsOnCrashedServer.removeIf(this::isDefaultMetaRegion);
+regionsOnCrashedServer.removeIf((x) -> isDefaultMetaRegion(x) ||
isDefaultRootRegion(x));
return !regionsOnCrashedServer.isEmpty();
}
+ private boolean isDefaultRootRegion(RegionInfo hri) {
+return hri.isRootRegion() && RegionReplicaUtil.isDefaultReplica(hri);
+ }
+
private boolean isDefaultMetaRegion(RegionInfo hri) {
return hri.isMetaRegion() && RegionReplicaUtil.isDefaultReplica(hri);
}
+ private void zkCoordinatedSplitRootLogs(MasterProcedureEnv env) throws
IOException {
+LOG.debug("Splitting root WALs {}", this);
+MasterWalManager mwm = env.getMasterServices().getMasterWalManager();
+AssignmentManager am = env.getMasterServices().getAssignmentManager();
+am.getRegionStates().rootLogSplitting(serverName);
+mwm.splitRootLog(serverName);
+am.getRegionStates().rootLogSplit(serverName);
+LOG.debug("Done splitting root WALs {}", this);
+ }
Review comment:
will make sure to update things then.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on pull request #2445: URL: https://github.com/apache/hbase/pull/2445#issuecomment-710722619 @saintstack , I did a rebase and force pushed the updateated branch. I'll address your comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506778311
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -430,11 +553,19 @@ private boolean isCarryingRegion(final ServerName
serverName, final RegionInfo r
return(node != null && serverName.equals(node.getRegionLocation()));
}
- private RegionInfo getMetaForRegion(final RegionInfo regionInfo) {
-//if (regionInfo.isMetaRegion()) return regionInfo;
-// TODO: handle multiple meta. if the region provided is not meta lookup
-// which meta the region belongs to.
-return RegionInfoBuilder.FIRST_META_REGIONINFO;
+ /**
+ * Check hbase:meta is up and ready for reading. For use during Master
startup only.
+ * @return True if meta is UP and online and startup can progress.
Otherwise, meta is not online
+ * and we will hold here until operator intervention.
+ */
+ @VisibleForTesting
+ public boolean waitForMetaOnline() {
Review comment:
This is only during master startup/initialization. We can do as a
follow-up as that'd be a big change. Off-hand: As currently RegionStates is
mainly used after it has read all the contents of meta. That would be a change
in expectation if it would now be possible that RegionStates may not know about
certain regions. Eg it would be possible that SCP will only recover regions
that RegionStates knows about. And we would need a mechanism to cover for when
other regions of meta are loaded down the road.
Alternatively we can just speedup/parallelize meta assignment until that is
no longer good enough then consider what you mentioned? Note also that as part
of startup/initialization. We would need to scan meta to bootstrap
RegionStates. Internally we parallelized the scanning of meta during startup.
I'll remember to add that in this patch too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506779022
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -1926,6 +2225,27 @@ public void markRegionAsMerged(final RegionInfo child,
final ServerName serverNa
}
TableDescriptor td = master.getTableDescriptors().get(child.getTable());
regionStateStore.mergeRegions(child, mergeParents, serverName, td);
+
+//Split meta assignment event
+if (td.isMetaTable()) {
+ ProcedureEvent parentEvent[] = new
ProcedureEvent[mergeParents.length];
+ for (int i=0; i("meta assign: " + child.getRegionNameAsString()));
+
+ //wake the procedures waiting on parent event, the procedures will awaken
+ //and wait on newly created child event
+ for (int i= mergeParents.length-1; i>=1; i--) {
+metaAssignEventMap.remove(mergeParents[i].getStartKey());
+parentEvent[i].wake(getProcedureScheduler());
+ }
+ //for the first key we don't remove since we already replaced it
+ parentEvent[0].wake(getProcedureScheduler());
+}
+
if (shouldAssignFavoredNodes(child)) {
getFavoredNodePromoter().generateFavoredNodesForMergedRegion(child,
mergeParents);
}
Review comment:
That's true today tho? Or I'm missing something? Let me know so I can
see if that scenario is covered in UT.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
##
@@ -257,46 +327,62 @@ private void cleanupSplitDir(MasterProcedureEnv env) {
// old hbase:meta tenancy on this server; clean these up if any before
trying to remove the
// WAL directory of this server or we will fail. See archiveMetaLog
comment for more details
// on this condition.
-
env.getMasterServices().getMasterWalManager().archiveMetaLog(this.serverName);
+
env.getMasterServices().getMasterWalManager().archiveCatalogLog(this.serverName,
false);
}
splitWALManager.deleteWALDir(serverName);
} catch (IOException e) {
LOG.warn("Remove WAL directory for {} failed, ignore...{}", serverName,
e.getMessage());
}
}
- private boolean isSplittingDone(MasterProcedureEnv env, boolean splitMeta) {
+ private boolean isSplittingDone(MasterProcedureEnv env,
SplitWALManager.SplitType splitType) {
SplitWALManager splitWALManager =
env.getMasterServices().getSplitWALManager();
try {
- int wals = splitWALManager.getWALsToSplit(serverName, splitMeta).size();
- LOG.debug("Check if {} WAL splitting is done? wals={}, meta={}",
serverName, wals, splitMeta);
+ int wals = splitWALManager.getWALsToSplit(serverName, splitType).size();
+ LOG.debug("Check if {} WAL splitting is done? wals={}, SplitType={}",
+serverName, wals, splitType);
return wals == 0;
} catch (IOException e) {
LOG.warn("Get WALs of {} failed, retry...", serverName, e);
return false;
}
}
- private Procedure[] createSplittingWalProcedures(MasterProcedureEnv env,
boolean splitMeta)
+ private Procedure[] createSplittingWalProcedures(MasterProcedureEnv env,
+SplitWALManager.SplitType splitType)
throws IOException {
-LOG.info("Splitting WALs {}, isMeta: {}", this, splitMeta);
+LOG.info("Splitting WALs {}, SplitType: {}", this, splitType);
SplitWALManager splitWALManager =
env.getMasterServices().getSplitWALManager();
-List procedures = splitWALManager.splitWALs(serverName,
splitMeta);
+List procedures = splitWALManager.splitWALs(serverName,
splitType);
return procedures.toArray(new Procedure[procedures.size()]);
}
private boolean filterDefaultMetaRegions() {
if (regionsOnCrashedServer == null) {
return false;
}
-regionsOnCrashedServer.removeIf(this::isDefaultMetaRegion);
+regionsOnCrashedServer.removeIf((x) -> isDefaultMetaRegion(x) ||
isDefaultRootRegion(x));
return !regionsOnCrashedServer.isEmpty();
}
+ private boolean isDefaultRootRegion(RegionInfo hri) {
+return hri.isRootRegion() && RegionReplicaUtil.isDefaultReplica(hri);
+ }
+
private boolean isDefaultMetaRegion(RegionInfo hri) {
return hri.isMetaRegion() && RegionReplicaUtil.isDefaultReplica(hri);
}
+ private void zkCoordinatedSplitRootLogs(MasterProcedureEnv env) throws
IOException {
+LOG.debug("Splitting root WALs {}", this);
+MasterWalManager mwm = env.getMasterServices().getMasterWalManager();
+AssignmentManager am = env.getMasterServices().getAssignmentManager();
+am.getRegionStates().rootLogSplitting(serverName);
+mwm.splitRootLog(serverName);
+am.getRegionStates().rootLogSplit(serverName);
+LOG.debug("Done splitting root WALs {}", this);
+ }
Review comment:
thanks will update.
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506778937
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -1440,10 +1625,42 @@ private void
handleRegionOverStuckWarningThreshold(final RegionInfo regionInfo)
//
// TODO: Master load/bootstrap
//
- public void joinCluster() throws IOException {
+ public void joinCluster(boolean loadRoot) throws IOException {
+joinCluster(loadRoot, true, true);
+ }
+
+ @VisibleForTesting
+ public void joinCluster(boolean loadRoot, boolean shouldWaitForRootOnline,
+boolean shouldWaitForMetaOnline)
+throws IOException {
long startTime = System.nanoTime();
LOG.debug("Joining cluster...");
+// FIRST Catalog tables READ
+// The below cannot make progress w/o hbase:meta being online.
Review comment:
this should talk about loadRoot() and loadMeta(). let me change this to
be more generalized to address both code sections?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506778454
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -768,6 +944,15 @@ public void move(RegionInfo regionInfo) throws IOException
{
@VisibleForTesting
static int compare(TransitRegionStateProcedure left,
TransitRegionStateProcedure right) {
+//TODO francis this is broken once we have more meta entries
Review comment:
Yes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506778434
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -511,6 +654,39 @@ public boolean isMetaLoaded() {
return metaLoadEvent.isReady();
}
+
+ /**
+ * @return True if region is online and scannable else false if an error or
shutdown (Otherwise
+ * we just block in here holding up all forward-progess).
Review comment:
I can't take credit for this radical-ness. I just moved this api from
HMaster to here. :-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506778311
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -430,11 +553,19 @@ private boolean isCarryingRegion(final ServerName
serverName, final RegionInfo r
return(node != null && serverName.equals(node.getRegionLocation()));
}
- private RegionInfo getMetaForRegion(final RegionInfo regionInfo) {
-//if (regionInfo.isMetaRegion()) return regionInfo;
-// TODO: handle multiple meta. if the region provided is not meta lookup
-// which meta the region belongs to.
-return RegionInfoBuilder.FIRST_META_REGIONINFO;
+ /**
+ * Check hbase:meta is up and ready for reading. For use during Master
startup only.
+ * @return True if meta is UP and online and startup can progress.
Otherwise, meta is not online
+ * and we will hold here until operator intervention.
+ */
+ @VisibleForTesting
+ public boolean waitForMetaOnline() {
Review comment:
This is only during master startup/initialization. We can do as a
follow-up as that'd be a big change. As currently RegionStates is mainly used
after it has read all the contents of meta. That would be a change in
expectation if it would now be possible that RegionStates may not know about
certain regions. Eg it would be possible that SCP will only recover regions
that RegionStates knows about. And we would need a mechanism to cover for when
other regions of meta are loaded down the road.
Alternatively we can just speedup/parallelize meta assignment until that is
no longer good enough. Note also that as part of startup/initialization. We
would need to scan meta to bootstrap RegionStates. Internally we parallelized
the scanning of meta during startup. I'll remember to add that here too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r50655
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -400,28 +419,132 @@ public boolean isTableDisabled(final TableName
tableName) {
}
//
- // META Helpers
+ // ROOT Helpers
//
- private boolean isMetaRegion(final RegionInfo regionInfo) {
-return regionInfo.isMetaRegion();
+ private boolean isRootRegion(final RegionInfo regionInfo) {
+return regionInfo.isRootRegion();
}
- public boolean isMetaRegion(final byte[] regionName) {
-return getMetaRegionFromName(regionName) != null;
+ public boolean isCarryingRoot(final ServerName serverName) {
+// TODO: handle multiple root
+return isCarryingRegion(serverName, RegionInfoBuilder.ROOT_REGIONINFO);
}
- public RegionInfo getMetaRegionFromName(final byte[] regionName) {
-for (RegionInfo hri: getMetaRegionSet()) {
- if (Bytes.equals(hri.getRegionName(), regionName)) {
-return hri;
- }
+ private RegionInfo getRootForRegion(final RegionInfo regionInfo) {
+//if (regionInfo.isRootRegion()) return regionInfo;
+// TODO: handle multiple root. if the region provided is not root lookup
+// which root the region belongs to.
+return RegionInfoBuilder.ROOT_REGIONINFO;
+ }
+
+ /**
+ * Check hbase:root is up and ready for reading. For use during Master
startup only.
+ * @return True if root is UP and online and startup can progress.
Otherwise, root is not online
+ * and we will hold here until operator intervention.
+ */
+ public boolean waitForRootOnline() {
+return isRegionOnline(RegionInfoBuilder.ROOT_REGIONINFO);
+ }
+
+ private static final Set ROOT_REGION_SET =
+Collections.singleton(RegionInfoBuilder.ROOT_REGIONINFO);
+ public Set getRootRegionSet() {
+return ROOT_REGION_SET;
+ }
+
+ //
+ // ROOT Event(s) helpers
+ //
+ /**
+ * Notice that, this only means the root region is available on a RS, but
the AM may still be
+ * loading the region states from root, so usually you need to check {@link
#isRootLoaded()} first
+ * before checking this method, unless you can make sure that your piece of
code can only be
+ * executed after AM builds the region states.
+ * @see #isRootLoaded()
+ */
+ public boolean isRootAssigned() {
+return rootAssignEvent.isReady();
+ }
+
+ public boolean isRootRegionInTransition() {
+return !isRootAssigned();
+ }
+
+ /**
+ * Notice that this event does not mean the AM has already finished region
state rebuilding. See
+ * the comment of {@link #isRootAssigned()} for more details.
+ * @see #isRootAssigned()
+ */
+ public boolean waitRootAssigned(Procedure proc, RegionInfo regionInfo) {
+return
getRootAssignEvent(getRootForRegion(regionInfo)).suspendIfNotReady(proc);
+ }
+
+ private void setRootAssigned(RegionInfo rootRegionInfo, boolean assigned) {
+assert isRootRegion(rootRegionInfo) : "unexpected non-root region " +
rootRegionInfo;
+if (!RegionReplicaUtil.isDefaultReplica(rootRegionInfo)) {
+ return;
}
-return null;
+ProcedureEvent rootAssignEvent = getRootAssignEvent(rootRegionInfo);
+if (assigned) {
+ LOG.debug("Setting hbase:root region assigned: "+rootRegionInfo);
+ rootAssignEvent.wake(getProcedureScheduler());
+} else {
+ LOG.debug("Setting hbase:root region unassigned: "+rootRegionInfo);
+ rootAssignEvent.suspend();
+}
+ }
+
+ private ProcedureEvent getRootAssignEvent(RegionInfo rootRegionInfo) {
+assert isRootRegion(rootRegionInfo) : "unexpected non-catalog region " +
rootRegionInfo;
+// TODO: handle multiple root.
Review comment:
Yeah only one root. Thanks for confirming that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506777664
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -152,9 +159,19 @@
private static final int DEFAULT_RIT_STUCK_WARNING_THRESHOLD = 60 * 1000;
public static final String UNEXPECTED_STATE_REGION = "Unexpected state for ";
- private final ProcedureEvent metaAssignEvent = new ProcedureEvent<>("meta
assign");
+ private final ProcedureEvent rootAssignEvent = new ProcedureEvent<>("root
assign");
+ private final ProcedureEvent rootLoadEvent = new ProcedureEvent<>("root
load");
+
private final ProcedureEvent metaLoadEvent = new ProcedureEvent<>("meta
load");
+ private final ConcurrentSkipListMap>
metaAssignEventMap =
Review comment:
This is here so procedures that depend on particular regions of meta to
be available (eg SCP, RTSP, etc) are only waiting on the necessary regions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506776199
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
##
@@ -257,46 +327,62 @@ private void cleanupSplitDir(MasterProcedureEnv env) {
// old hbase:meta tenancy on this server; clean these up if any before
trying to remove the
// WAL directory of this server or we will fail. See archiveMetaLog
comment for more details
// on this condition.
-
env.getMasterServices().getMasterWalManager().archiveMetaLog(this.serverName);
+
env.getMasterServices().getMasterWalManager().archiveCatalogLog(this.serverName,
false);
}
splitWALManager.deleteWALDir(serverName);
} catch (IOException e) {
LOG.warn("Remove WAL directory for {} failed, ignore...{}", serverName,
e.getMessage());
}
}
- private boolean isSplittingDone(MasterProcedureEnv env, boolean splitMeta) {
+ private boolean isSplittingDone(MasterProcedureEnv env,
SplitWALManager.SplitType splitType) {
SplitWALManager splitWALManager =
env.getMasterServices().getSplitWALManager();
try {
- int wals = splitWALManager.getWALsToSplit(serverName, splitMeta).size();
- LOG.debug("Check if {} WAL splitting is done? wals={}, meta={}",
serverName, wals, splitMeta);
+ int wals = splitWALManager.getWALsToSplit(serverName, splitType).size();
+ LOG.debug("Check if {} WAL splitting is done? wals={}, SplitType={}",
+serverName, wals, splitType);
return wals == 0;
} catch (IOException e) {
LOG.warn("Get WALs of {} failed, retry...", serverName, e);
return false;
}
}
- private Procedure[] createSplittingWalProcedures(MasterProcedureEnv env,
boolean splitMeta)
+ private Procedure[] createSplittingWalProcedures(MasterProcedureEnv env,
+SplitWALManager.SplitType splitType)
throws IOException {
-LOG.info("Splitting WALs {}, isMeta: {}", this, splitMeta);
+LOG.info("Splitting WALs {}, SplitType: {}", this, splitType);
SplitWALManager splitWALManager =
env.getMasterServices().getSplitWALManager();
-List procedures = splitWALManager.splitWALs(serverName,
splitMeta);
+List procedures = splitWALManager.splitWALs(serverName,
splitType);
return procedures.toArray(new Procedure[procedures.size()]);
}
private boolean filterDefaultMetaRegions() {
if (regionsOnCrashedServer == null) {
return false;
}
-regionsOnCrashedServer.removeIf(this::isDefaultMetaRegion);
+regionsOnCrashedServer.removeIf((x) -> isDefaultMetaRegion(x) ||
isDefaultRootRegion(x));
return !regionsOnCrashedServer.isEmpty();
}
+ private boolean isDefaultRootRegion(RegionInfo hri) {
+return hri.isRootRegion() && RegionReplicaUtil.isDefaultReplica(hri);
+ }
+
private boolean isDefaultMetaRegion(RegionInfo hri) {
return hri.isMetaRegion() && RegionReplicaUtil.isDefaultReplica(hri);
}
+ private void zkCoordinatedSplitRootLogs(MasterProcedureEnv env) throws
IOException {
+LOG.debug("Splitting root WALs {}", this);
+MasterWalManager mwm = env.getMasterServices().getMasterWalManager();
+AssignmentManager am = env.getMasterServices().getAssignmentManager();
+am.getRegionStates().rootLogSplitting(serverName);
+mwm.splitRootLog(serverName);
+am.getRegionStates().rootLogSplit(serverName);
+LOG.debug("Done splitting root WALs {}", this);
+ }
Review comment:
will make sure to update things then.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445: URL: https://github.com/apache/hbase/pull/2445#discussion_r506776132 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java ## @@ -2986,6 +2988,7 @@ public GetMastersResponse getMasters(RpcController rpcController, GetMastersRequ return resp.build(); } + //TODO francis this needs to be get RootLocation now? @Override public GetMetaRegionLocationsResponse getMetaRegionLocations(RpcController rpcController, Review comment: Yeah I have. Looks like I need to having something like this for backward compatibility but I also need to have a getRootregionLocations() one. Should I just rename this and worry about re-adding the existing api for compatibility later? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445: URL: https://github.com/apache/hbase/pull/2445#discussion_r506775857 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java ## @@ -2531,6 +2531,7 @@ public GetTableStateResponse setTableStateInMeta(RpcController controller, * * @return previous states of the regions */ + //TODO francis support root here Review comment: Yeah hbck related stuff I have not finished. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445: URL: https://github.com/apache/hbase/pull/2445#discussion_r506775609 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java ## @@ -124,7 +124,7 @@ import org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure; import org.apache.hadoop.hbase.master.procedure.DisableTableProcedure; import org.apache.hadoop.hbase.master.procedure.EnableTableProcedure; -import org.apache.hadoop.hbase.master.procedure.InitMetaProcedure; +import org.apache.hadoop.hbase.master.procedure.InitRootProcedure; Review comment: Looks like it's still [here](https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/InitMetaProcedure.java)? Maybe you mean MetaBootstrap? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506775438
##
File path:
hbase-common/src/test/java/org/apache/hadoop/hbase/TestCellComparator.java
##
@@ -205,6 +243,37 @@ public void testMetaComparisons2() {
Bytes.toBytes(TableName.META_TABLE_NAME.getNameAsString()+",,1"),
now))) > 0);
}
+ @Test
+ public void testRootComparisons2() {
Review comment:
Yeah I guess we should then. Let me check the legacy 0.94 unit tests.
Would that mean there'd be millions for meta comparator too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506775285
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
##
@@ -110,6 +111,15 @@
@Deprecated
public static final KVComparator META_COMPARATOR = new MetaComparator();
+ /**
+ * A {@link KVComparator} for hbase:root catalog table
+ * {@link KeyValue}s.
+ * @deprecated Use {@link RootCellComparator#ROOT_COMPARATOR} instead.
+ * Deprecated for hbase 2.0, remove for hbase 3.0.
+ */
+ @Deprecated
+ public static final KVComparator ROOT_COMPARATOR = new RootComparator();
Review comment:
This will be removed.
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
##
@@ -1604,6 +1614,79 @@ public static int getDelimiterInReverse(final byte [] b,
final int offset,
return result;
}
+ /**
+ * A {@link KVComparator} for -ROOT- catalog table
+ * {@link KeyValue}s.
+ * @deprecated : {@link RootCellComparator#ROOT_COMPARATOR} to be used.
+ * Deprecated for hbase 2.0, remove for hbase 3.0.
Review comment:
This will be removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506775224
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
##
@@ -300,8 +300,12 @@ public static CellComparator getCellComparator(TableName
tableName) {
* @return CellComparator to use going off the {@code tableName} passed.
*/
public static CellComparator getCellComparator(byte [] tableName) {
-// FYI, TableName.toBytes does not create an array; just returns existing
array pointer.
-return Bytes.equals(tableName, TableName.META_TABLE_NAME.toBytes())?
- MetaCellComparator.META_COMPARATOR: CellComparatorImpl.COMPARATOR;
+if (Bytes.equals(tableName, TableName.ROOT_TABLE_NAME.toBytes())) {
+ return RootCellComparator.ROOT_COMPARATOR;
+}
+if (Bytes.equals(tableName, TableName.META_TABLE_NAME.toBytes())) {
+ return MetaCellComparator.META_COMPARATOR;
+}
+return CellComparatorImpl.COMPARATOR;
Review comment:
yes will consolidate.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506775087
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
##
@@ -158,4 +174,14 @@ default int compareRows(ByteBuffer row, Cell cell) {
* Do not pollute with types other than BBKV if can be helped; the
Comparator will slow.
*/
Comparator getSimpleComparator();
+
+ static CellComparator getComparator(TableName tableName) {
+if (tableName.equals(TableName.ROOT_TABLE_NAME)) {
+ return RootCellComparator.ROOT_COMPARATOR;
+}
+if (tableName.equals(TableName.META_TABLE_NAME)) {
+ return MetaCellComparator.META_COMPARATOR;
+}
+return CellComparatorImpl.COMPARATOR;
Review comment:
No that's just artifact from the PoC iteration will clean it up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506775087
##
File path:
hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
##
@@ -158,4 +174,14 @@ default int compareRows(ByteBuffer row, Cell cell) {
* Do not pollute with types other than BBKV if can be helped; the
Comparator will slow.
*/
Comparator getSimpleComparator();
+
+ static CellComparator getComparator(TableName tableName) {
+if (tableName.equals(TableName.ROOT_TABLE_NAME)) {
+ return RootCellComparator.ROOT_COMPARATOR;
+}
+if (tableName.equals(TableName.META_TABLE_NAME)) {
+ return MetaCellComparator.META_COMPARATOR;
+}
+return CellComparatorImpl.COMPARATOR;
Review comment:
No that's just artifact from the PoC iteration will consolidate the
other one to use this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445: URL: https://github.com/apache/hbase/pull/2445#discussion_r506775008 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java ## @@ -250,6 +260,7 @@ DEFAULT_VALUES.put(PRIORITY, String.valueOf(DEFAULT_PRIORITY)); DEFAULT_VALUES.keySet().stream() .map(s -> new Bytes(Bytes.toBytes(s))).forEach(RESERVED_KEYWORDS::add); +RESERVED_KEYWORDS.add(IS_ROOT_KEY); RESERVED_KEYWORDS.add(IS_META_KEY); Review comment: Yeah... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506774952
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfoBuilder.java
##
@@ -33,6 +33,25 @@
public static final RegionInfo UNDEFINED =
RegionInfoBuilder.newBuilder(TableName.valueOf("__UNDEFINED__")).build();
+ /**
+ * RegionInfo for first root region
+ * You cannot use this builder to make an instance of the {@link
#ROOT_REGIONINFO}.
+ * Just refer to this instance. Also, while the instance is actually a
MutableRI, its type is
+ * just RI so the mutable methods are not available (unless you go casting);
it appears
+ * as immutable (I tried adding Immutable type but it just makes a mess).
+ *
+ * We are using the non-legacy encoding format to reduce the boilerplace code
+ */
+ public static final RegionInfo ROOT_REGIONINFO =
+new MutableRegionInfo(TableName.ROOT_TABLE_NAME,
+ HConstants.EMPTY_START_ROW,
+ HConstants.EMPTY_END_ROW,
+ false,
+ 0,
+ RegionInfo.DEFAULT_REPLICA_ID,
+ false);
Review comment:
That would be in the table schema. Should still be changeable. I didn't
not remove support.
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfoBuilder.java
##
@@ -33,6 +33,25 @@
public static final RegionInfo UNDEFINED =
RegionInfoBuilder.newBuilder(TableName.valueOf("__UNDEFINED__")).build();
+ /**
+ * RegionInfo for first root region
+ * You cannot use this builder to make an instance of the {@link
#ROOT_REGIONINFO}.
+ * Just refer to this instance. Also, while the instance is actually a
MutableRI, its type is
+ * just RI so the mutable methods are not available (unless you go casting);
it appears
+ * as immutable (I tried adding Immutable type but it just makes a mess).
+ *
+ * We are using the non-legacy encoding format to reduce the boilerplace code
+ */
+ public static final RegionInfo ROOT_REGIONINFO =
+new MutableRegionInfo(TableName.ROOT_TABLE_NAME,
+ HConstants.EMPTY_START_ROW,
+ HConstants.EMPTY_END_ROW,
+ false,
+ 0,
+ RegionInfo.DEFAULT_REPLICA_ID,
+ false);
Review comment:
That would be in the table schema? Should still be changeable. I didn't
remove support.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506774887
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfoBuilder.java
##
@@ -33,6 +33,25 @@
public static final RegionInfo UNDEFINED =
RegionInfoBuilder.newBuilder(TableName.valueOf("__UNDEFINED__")).build();
+ /**
+ * RegionInfo for first root region
+ * You cannot use this builder to make an instance of the {@link
#ROOT_REGIONINFO}.
+ * Just refer to this instance. Also, while the instance is actually a
MutableRI, its type is
+ * just RI so the mutable methods are not available (unless you go casting);
it appears
+ * as immutable (I tried adding Immutable type but it just makes a mess).
+ *
+ * We are using the non-legacy encoding format to reduce the boilerplace code
Review comment:
It's basically saying we're not using the legacy regionId == 0 form or
else we'll have to do the same ugliness we did for meta discussed
[here](https://github.com/apache/hbase/pull/2445/files/483d02c8785be9f3884afa05116e0127f38fdd0d#r503639854).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506774233
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfoBuilder.java
##
@@ -33,6 +33,25 @@
public static final RegionInfo UNDEFINED =
RegionInfoBuilder.newBuilder(TableName.valueOf("__UNDEFINED__")).build();
+ /**
+ * RegionInfo for first root region
Review comment:
Ah right I can remove. I was thinking first as there are replicas but
that's not really it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506774118
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##
@@ -2391,42 +2406,91 @@ public void run(Timeout timeout) throws Exception {
return failedFuture(new IllegalArgumentException("Passed region name
can't be null"));
}
try {
+ TableName parentTable;
CompletableFuture> future;
if (RegionInfo.isEncodedRegionName(regionNameOrEncodedRegionName)) {
String encodedName = Bytes.toString(regionNameOrEncodedRegionName);
-if (encodedName.length() < RegionInfo.MD5_HEX_LENGTH) {
- // old format encodedName, should be meta region
+
+//TODO francis do we really need to support encoded name for root?
+boolean isRoot = false;
+for (int i = 0; i< numRootReplicas; i++) {
+ RegionInfo info =
+
RegionReplicaUtil.getRegionInfoForReplica(RegionInfoBuilder.ROOT_REGIONINFO, i);
+ if (Bytes.equals(info.getRegionName(),
regionNameOrEncodedRegionName)) {
+isRoot = true;
+break;
+ }
+}
+if (isRoot) {
future = connection.registry.getMetaRegionLocations()
.thenApply(locs -> Stream.of(locs.getRegionLocations())
.filter(loc ->
loc.getRegion().getEncodedName().equals(encodedName)).findFirst());
+ parentTable = null;
+} else if (encodedName.length() < RegionInfo.MD5_HEX_LENGTH) {
+ future =
ClientMetaTableAccessor.getRegionLocationWithEncodedName(rootTable,
+regionNameOrEncodedRegionName);
+ parentTable = ROOT_TABLE_NAME;
} else {
future =
ClientMetaTableAccessor.getRegionLocationWithEncodedName(metaTable,
regionNameOrEncodedRegionName);
+ parentTable = META_TABLE_NAME;
}
} else {
RegionInfo regionInfo =
CatalogFamilyFormat.parseRegionInfoFromRegionName(regionNameOrEncodedRegionName);
-if (regionInfo.isMetaRegion()) {
+if (regionInfo.isRootRegion()) {
future = connection.registry.getMetaRegionLocations()
.thenApply(locs -> Stream.of(locs.getRegionLocations())
.filter(loc -> loc.getRegion().getReplicaId() ==
regionInfo.getReplicaId())
.findFirst());
+ parentTable = null;
+ //TODO francis it won't reach here once meta is split
+} else if (regionInfo.isMetaRegion()) {
+ parentTable = ROOT_TABLE_NAME;
+ future =
+ClientMetaTableAccessor.getRegionLocation(rootTable,
regionNameOrEncodedRegionName);
} else {
+ parentTable = META_TABLE_NAME;
future =
ClientMetaTableAccessor.getRegionLocation(metaTable,
regionNameOrEncodedRegionName);
}
}
+ final TableName finalParentTable = parentTable;
CompletableFuture returnedFuture = new
CompletableFuture<>();
addListener(future, (location, err) -> {
if (err != null) {
returnedFuture.completeExceptionally(err);
return;
}
if (!location.isPresent() || location.get().getRegion() == null) {
- returnedFuture.completeExceptionally(
-new UnknownRegionException("Invalid region name or encoded region
name: " +
- Bytes.toStringBinary(regionNameOrEncodedRegionName)));
+ if (META_TABLE_NAME.equals(finalParentTable)) {
+if (LOG.isDebugEnabled()) {
Review comment:
I was trying to avoid the Bytes.toString() binary execution. That's
inconsequential?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506773956
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##
@@ -2391,42 +2406,91 @@ public void run(Timeout timeout) throws Exception {
return failedFuture(new IllegalArgumentException("Passed region name
can't be null"));
}
try {
+ TableName parentTable;
CompletableFuture> future;
if (RegionInfo.isEncodedRegionName(regionNameOrEncodedRegionName)) {
String encodedName = Bytes.toString(regionNameOrEncodedRegionName);
-if (encodedName.length() < RegionInfo.MD5_HEX_LENGTH) {
- // old format encodedName, should be meta region
+
+//TODO francis do we really need to support encoded name for root?
+boolean isRoot = false;
+for (int i = 0; i< numRootReplicas; i++) {
+ RegionInfo info =
+
RegionReplicaUtil.getRegionInfoForReplica(RegionInfoBuilder.ROOT_REGIONINFO, i);
Review comment:
There was support for meta region when it was "root". I'm just keeping
that feature parity. Should I skip support?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506773828
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##
@@ -2391,42 +2406,91 @@ public void run(Timeout timeout) throws Exception {
return failedFuture(new IllegalArgumentException("Passed region name
can't be null"));
}
try {
+ TableName parentTable;
CompletableFuture> future;
if (RegionInfo.isEncodedRegionName(regionNameOrEncodedRegionName)) {
String encodedName = Bytes.toString(regionNameOrEncodedRegionName);
-if (encodedName.length() < RegionInfo.MD5_HEX_LENGTH) {
- // old format encodedName, should be meta region
+
+//TODO francis do we really need to support encoded name for root?
Review comment:
Oh this question was whether we wanted to support the encoded name or
root as argument for "regionNameOrEncodedRegionName"? Becuase then I have to do
the succeeding not so nice thing of enumerating all the root replicas and check
if there is a match.
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##
@@ -2391,42 +2406,91 @@ public void run(Timeout timeout) throws Exception {
return failedFuture(new IllegalArgumentException("Passed region name
can't be null"));
}
try {
+ TableName parentTable;
CompletableFuture> future;
if (RegionInfo.isEncodedRegionName(regionNameOrEncodedRegionName)) {
String encodedName = Bytes.toString(regionNameOrEncodedRegionName);
-if (encodedName.length() < RegionInfo.MD5_HEX_LENGTH) {
- // old format encodedName, should be meta region
+
+//TODO francis do we really need to support encoded name for root?
+boolean isRoot = false;
Review comment:
Will rename.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506773341
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
##
@@ -1130,22 +1143,24 @@ public void run(PRESP resp) {
* List all region locations for the specific table.
*/
private CompletableFuture>
getTableHRegionLocations(TableName tableName) {
-if (TableName.META_TABLE_NAME.equals(tableName)) {
+if (ROOT_TABLE_NAME.equals(tableName)) {
CompletableFuture> future = new
CompletableFuture<>();
- addListener(connection.registry.getMetaRegionLocations(), (metaRegions,
err) -> {
+ addListener(connection.registry.getMetaRegionLocations(), (rootRegions,
err) -> {
if (err != null) {
future.completeExceptionally(err);
-} else if (metaRegions == null || metaRegions.isEmpty() ||
- metaRegions.getDefaultRegionLocation() == null) {
+} else if (rootRegions == null || rootRegions.isEmpty() ||
+ rootRegions.getDefaultRegionLocation() == null) {
future.completeExceptionally(new IOException("meta region does not
found"));
} else {
-
future.complete(Collections.singletonList(metaRegions.getDefaultRegionLocation()));
+
future.complete(Collections.singletonList(rootRegions.getDefaultRegionLocation()));
}
});
return future;
} else {
- // For non-meta table, we fetch all locations by scanning hbase:meta
table
- return ClientMetaTableAccessor.getTableHRegionLocations(metaTable,
tableName);
+ // For non-meta table, we fetch all locations by scanning catalog table
Review comment:
Yes this needs updating.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506772798
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MutableRegionInfo.java
##
@@ -206,16 +209,20 @@ public TableName getTable() {
*/
@Override
public boolean containsRange(byte[] rangeStartKey, byte[] rangeEndKey) {
-if (Bytes.compareTo(rangeStartKey, rangeEndKey) > 0) {
+CellComparator comparator = CellComparator.getComparator(tableName);
Review comment:
Yep will do.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445: URL: https://github.com/apache/hbase/pull/2445#discussion_r506772765 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonRootRegionLocator.java ## @@ -174,10 +184,14 @@ private boolean tryComplete(LocateRequest req, CompletableFuture= req.row. Here we split it to endKey == req.row || // (endKey > req.row && startKey < req.row). The two conditions are equal since startKey < // endKey. +KeyValue.KVComparator comparator = getComparator(loc.getRegion().getTable()); byte[] endKey = loc.getRegion().getEndKey(); -int c = Bytes.compareTo(endKey, req.row); +int c = comparator.compareRows(endKey, 0, endKey.length, + req.row,0, req.row.length); Review comment: There are interfaces of cell and bytebuffer in CellComparator so I can use them if the keys are in the appropriate although in this case they are just byte arrays. Did this answer your question? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506772362
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MutableRegionInfo.java
##
@@ -126,7 +127,9 @@ private static int checkReplicaId(int regionId) {
this.replicaId = checkReplicaId(replicaId);
this.offLine = offLine;
this.regionName = RegionInfo.createRegionName(this.tableName,
this.startKey, this.regionId,
- this.replicaId, !this.tableName.equals(TableName.META_TABLE_NAME));
+ this.replicaId,
+ //1 is region id of FIRST_META_REGION_INFO
+ !(this.tableName.equals(TableName.META_TABLE_NAME) && regionId == 1));
Review comment:
Post split meta the region ids will be in the non-legacy format. This
special case here is to handle the primordial meta region which is in legacy
format. So we need to catch if the primordial meta region is being instantiated
here and make sure it is created in the legacy format.
Note I think I need to handle primordial meta region replicas here as well.
(Do I need to support that?)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506771434
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonRootRegionLocator.java
##
@@ -667,4 +689,14 @@ int getNumberOfCachedRegionLocations(TableName tableName) {
}
return
tableCache.cache.values().stream().mapToInt(RegionLocations::numNonNullElements).sum();
}
+
+ private static KeyValue.KVComparator getComparator(TableName tableName) {
Review comment:
Ah no this is an artifact of PoC just to get things working. I will
consolidate this into CellComparator as well as remove usage of KVComparator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506770269
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonRootRegionLocator.java
##
@@ -68,9 +70,9 @@
* The asynchronous locator for regions other than meta.
*/
@InterfaceAudience.Private
-class AsyncNonMetaRegionLocator {
+class AsyncNonRootRegionLocator {
Review comment:
Yes. I'll update the javadoc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506770199
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/NotAllRootRegionsOnlineException.java
##
@@ -25,19 +25,19 @@
* Thrown when an operation requires the root and all meta regions to be online
*/
@InterfaceAudience.Public
-public class NotAllMetaRegionsOnlineException extends DoNotRetryIOException {
+public class NotAllRootRegionsOnlineException extends DoNotRetryIOException {
Review comment:
Yes good point.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506770139
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/ClientMetaTableAccessor.java
##
@@ -193,15 +193,15 @@ private ClientMetaTableAccessor() {
* @param tableName table we're looking for, can be null for getting all
regions
* @param excludeOfflinedSplitParents don't return split parents
* @return the list of regioninfos and server. The return value will be
wrapped by a
Review comment:
Yes it does. Will do.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445: URL: https://github.com/apache/hbase/pull/2445#discussion_r506770075 ## File path: hbase-balancer/src/main/java/org/apache/hadoop/hbase/favored/FavoredNodeAssignmentHelper.java ## @@ -132,7 +132,7 @@ public static void updateMetaWithFavoredNodesInfo( puts.add(put); } } -MetaTableAccessor.putsToMetaTable(connection, puts); +MetaTableAccessor.putsToCatalogTable(connection, TableName.META_TABLE_NAME, puts); Review comment: Yes in places that don't include any real changes. This one is a mistake tho. It should've stuck with putsToMetaTable(). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506769908
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506769592
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506769105
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506768383
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506768145
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506767909
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506767895
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506767689
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506767036
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506767100
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506766854
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506766960
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506766409
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
+ * @param connection connection we're using
+ * @param visitor Visitor invoked against each row in regions family.
+ */
+ public static void fullScanRegions(Connection connection, TableName
catalogTableName,
+final ClientMetaTableAccessor.Visitor visitor) throws IOException {
+scanCa
[GitHub] [hbase] z3n3r commented on a change in pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
z3n3r commented on a change in pull request #2445:
URL: https://github.com/apache/hbase/pull/2445#discussion_r506766297
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/CatalogAccessor.java
##
@@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase;
+
+import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.hadoop.hbase.Cell.Type;
+import org.apache.hadoop.hbase.ClientMetaTableAccessor.QueryType;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Consistency;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Mutation;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.RegionReplicaUtil;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.RowFilter;
+import org.apache.hadoop.hbase.filter.SubstringComparator;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.ExceptionUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hbase.util.PairOfSameType;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Read/write operations on hbase:meta region as well as
assignment information stored
+ * to hbase:meta.
+ *
+ * Some of the methods of this class take ZooKeeperWatcher as a param. The
only reason for this is
+ * when this class is used on client-side (e.g. HBaseAdmin), we want to use
short-lived connection
+ * (opened before each operation, closed right after), while when used on HM
or HRS (like in
+ * AssignmentManager) we want permanent connection.
+ *
+ * HBASE-10070 adds a replicaId to HRI, meaning more than one HRI can be
defined for the same table
+ * range (table, startKey, endKey). For every range, there will be at least
one HRI defined which is
+ * called default replica.
+ *
+ * Meta layout For each table there is single row named for the table
with a 'table' column
+ * family. The column family currently has one column in it, the 'state'
column:
+ *
+ *
+ * table:state => contains table state
+ *
+ *
+ * For the catalog family, see the comments of {@link CatalogFamilyFormat} for
more details.
+ *
+ * TODO: Add rep_barrier for serial replication explanation. See
SerialReplicationChecker.
+ *
+ * The actual layout of meta should be encapsulated inside CatalogAccessor
methods, and should not
+ * leak out of it (through Result objects, etc)
+ * @see CatalogFamilyFormat
+ * @see ClientMetaTableAccessor
+ */
+@InterfaceAudience.Private
+public class CatalogAccessor {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(CatalogAccessor.class);
+ private static final Logger CATALOGLOG =
+LoggerFactory.getLogger("org.apache.hadoop.hbase.CATALOG");
+
+
+
+
+ // Reading operations //
+
+
+ /**
+ * Performs a full scan of hbase:meta for regions.
Review comment:
It should be for the catalog table. Apologies I haven't cleaned up the
comments yet let me start doing that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on
[GitHub] [hbase] Apache-HBase commented on pull request #2548: HBASE-25189 [Metrics] Add checkAndPut and checkAndDelete latency metrics at table level
Apache-HBase commented on pull request #2548: URL: https://github.com/apache/hbase/pull/2548#issuecomment-710536441 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 18s | branch-1 passed | | +1 :green_heart: | compile | 1m 20s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | compile | 1m 23s | branch-1 passed with JDK v1.7.0_272 | | +1 :green_heart: | checkstyle | 2m 14s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 14s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 1m 21s | branch-1 passed with JDK v1.7.0_272 | | +0 :ok: | spotbugs | 2m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 18s | branch-1 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 2s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | compile | 1m 24s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | javac | 1m 24s | the patch passed | | -1 :x: | checkstyle | 1m 35s | hbase-server: The patch generated 6 new + 156 unchanged - 2 fixed = 162 total (was 158) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 3m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 52s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 1m 3s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 1m 21s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | findbugs | 4m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 30s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 0m 41s | hbase-hadoop2-compat in the patch passed. | | -1 :x: | unit | 120m 27s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 1m 17s | The patch does not generate ASF License warnings. | | | | 176m 12s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestMetaCache | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2548/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2548 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux dc72273419a9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2548/out/precommit/personality/provided.sh | | git revision | branch-1 / e066951 | | Default Java | 1.7.0_272 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2548/3/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2548/3/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2548/3/testReport/ | | Max. process+thread count | 3475 (vs. ulimit of 1) | | modules | C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2548/3/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215613#comment-17215613 ] Huaxiang Sun edited comment on HBASE-18070 at 10/16/20, 7:45 PM: - [~apurtell] , We are working on the client side change at this moment. [~stack] has the source side changes committed to HBASE-18070 branch. HBASE-25125 changes is out for review, [~zhangduo]b and [~stack] had some comments. I agreed with Duo that client side change is more important to verify that load balancing mode for meta replica works. Trying to get the first cut for client side change out for review sometime next week, thanks. was (Author: huaxiangsun): [~apurtell] , We are working on the client side change at this moment. [~stack] has the source side changes committed to HBASE-18070 branch. HBASE-25125 changes is out for review, [~zhangduo]b and [~stack] made comments. I agreed with Duo that client side change is more important to verify that load balancing mode for meta replica works. Trying to get the first cut for client side change out for review sometime next week, thanks. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25179) Assert format is incorrect in HFilePerformanceEvaluation class.
[
https://issues.apache.org/jira/browse/HBASE-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215616#comment-17215616
]
Hudson commented on HBASE-25179:
Results for branch branch-2.2
[build #100 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/100/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/100//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/100//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/100//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Assert format is incorrect in HFilePerformanceEvaluation class.
> ---
>
> Key: HBASE-25179
> URL: https://issues.apache.org/jira/browse/HBASE-25179
> Project: HBase
> Issue Type: Improvement
> Components: Performance, test
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 2.2.7
>
>
> [HFilePerformanceEvaluation
> |https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java#L518]
>
> The format of expected and actual is interchanged.
> {code:java}
> PerformanceEvaluationCommons.assertValueSize(c.getValueLength(), ROW_LENGTH);
> {code}
> The first argument should be expected and second should be actual.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] huaxiangsun commented on pull request #2522: HBASE-25215 Create a ReplicationEndPoint for meta/root replica replic…
huaxiangsun commented on pull request #2522: URL: https://github.com/apache/hbase/pull/2522#issuecomment-710482867 > First, the issue number is incorrect, it should be HBASE-25125, not HBASE-25215. Thanks Duo, will correct. > > What I concern here is that, this is only an enhancement right? The RegionReplicaReplicationEndpoint could also be used to replicate meta edits? The only disadvantage is that, the speed will be slowed down if there is a slow replica, but I do not think it is a blocker for meta replica? Could we make the meta replicas feature work first so we can merge HBASE-18070 back and then let's do the enhancement? Yeah, agreed here. Client side work is in progress, that is the reason, review comments from Stack has not been addressed in this week. > And in general, I do not like that we have a specialized implementation for meta replica inside the region replica framework. They are almost the same so they should share the same code base. I think this enhancement could also be applied to normal RegionReplicaReplicationEndpoint, what we need is the config for the max distance between different secondary replicas. For normal region replicas, the value will be small, which means we will soon block the replication if any replicas are slow, to save memory. For meta replicas, we could have a large default value to let the good replicas catch up fast. > Anyway, my biggest concern here is that, this is not a blocker to prevent us merge HBASE-18070 back, as people said we want this feature in 2.4, we'd better focus on other blocker issues, such as how to let client go to secondary meta replicas as much as possible. Feel free to correct me if I'm wrong. I moved your comments in the design doc, we put some thoughts there already, happy to continue the discussion there. https://docs.google.com/document/d/1jJWVc-idHhhgL4KDRpjMsQJKCl_NRaCLGiH3Wqwd3O8/edit#heading=h.8n622lgkwlai This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215613#comment-17215613 ] Huaxiang Sun commented on HBASE-18070: -- [~apurtell] , We are working on the client side change at this moment. [~stack] has the source side changes committed to HBASE-18070 branch. HBASE-25125 changes is out for review, [~zhangduo]b and [~stack] made comments. I agreed with Duo that client side change is more important to verify that load balancing mode for meta replica works. Trying to get the first cut for client side change out for review sometime next week, thanks. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215597#comment-17215597 ] Andrew Kyle Purtell edited comment on HBASE-18070 at 10/16/20, 6:39 PM: [~zhangduo] I think "we" is colleagues of Stack (at Apple) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature. On the 2.4 release discussion thread on dev@ I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. Please let me know if there are objections to either the wait or the commit of this work. was (Author: apurtell): [~zhangduo] I think "we" is colleagues of Stack (at Apple) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature. On the 2.4 release discussion thread on dev@ I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215597#comment-17215597 ] Andrew Kyle Purtell edited comment on HBASE-18070 at 10/16/20, 6:39 PM: [~zhangduo] I think "we" is colleagues of Stack (at Apple) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature. On the 2.4 release discussion thread on dev@ I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. was (Author: apurtell): [~zhangduo] I think "we" is colleagues of Stack (at Apple) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature, that's why we develop them. On the thread I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215601#comment-17215601 ] Andrew Kyle Purtell commented on HBASE-18070: - As 2.4 RM [~huaxiangsun] [~stack] [~zhangduo] if I can help move this along please let me know, I can spend some time. PR review etc. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215597#comment-17215597 ] Andrew Kyle Purtell commented on HBASE-18070: - [~zhangduo] I think "we" is colleagues of Stack (at Apache) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature, that's why we develop them. On the thread I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215597#comment-17215597 ] Andrew Kyle Purtell edited comment on HBASE-18070 at 10/16/20, 6:35 PM: [~zhangduo] I think "we" is colleagues of Stack (at Apple) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature, that's why we develop them. On the thread I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. was (Author: apurtell): [~zhangduo] I think "we" is colleagues of Stack (at Apache) and Huaxiang (?). I might also like to enable META replicas at my employer (Salesforce) someday. Its fine to mention there may be users of a feature, that's why we develop them. On the thread I think we (the HBase community) have agreed I'll serve as RM for 2.4.0 and I'm fine waiting the estimated week or so for this to land. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25032) Wait for region server to become online before adding it to online servers in Master
[ https://issues.apache.org/jira/browse/HBASE-25032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215595#comment-17215595 ] Andrew Kyle Purtell commented on HBASE-25032: - Thanks for the update [~sandeep.guggilam] > Wait for region server to become online before adding it to online servers in > Master > > > Key: HBASE-25032 > URL: https://issues.apache.org/jira/browse/HBASE-25032 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Guggilam >Assignee: Sandeep Guggilam >Priority: Major > > As part of RS start up, RS reports for duty to Master . Master acknowledges > the request and adds it to the onlineServers list for further assigning any > regions to the RS > Once Master acknowledges the reportForDuty and sends back the response, RS > does a bunch of stuff like initializing replication sources etc before > becoming online. However, sometimes there could be an issue with initializing > replication sources when it is unable to connect to peer clusters because of > some kerberos configuration and there would be a delay of around 20 mins in > becoming online. > > Since master considers it online, it tries to assign regions and which fails > with ServerNotRunningYet exception, then the master tries to unassign which > again fails with the same exception leading the region to FAILED_CLOSE state. > > It would be good to have a check to see if the RS is ready to accept the > assignment requests before adding it to online servers list which would > account for any such delays as described above -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25130) Masters in-memory serverHoldings map is not cleared during hbck repair
[
https://issues.apache.org/jira/browse/HBASE-25130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215594#comment-17215594
]
Andrew Kyle Purtell commented on HBASE-25130:
-
Thanks for the update [~sandeep.guggilam]
> Masters in-memory serverHoldings map is not cleared during hbck repair
> --
>
> Key: HBASE-25130
> URL: https://issues.apache.org/jira/browse/HBASE-25130
> Project: HBase
> Issue Type: Bug
>Reporter: Sandeep Guggilam
>Priority: Major
>
> {color:#1d1c1d}Incase of repairing overlaps, hbck essentially calls the
> closeRegion RPC on RS followed by offline RPC on Master to offline all the
> overlap regions that would be merged into a new region. {color}
> {color:#1d1c1d}However the offline RPC doesn’t remove it from the
> serverHoldings map unless the new state is MERGED/SPLIT
> ([https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java#L719])
> b{color}{color:#1d1c1d}ut the new state in this case is OFFLINE. {color}
> {color:#1d1c1d}This is actually intended to match with the META entries and
> would be removed later when the region is online on a different server.
> However, in our case , the region would never be online on a new server,
> hence the region info is never cleared from the map that is used by balancer
> and SCP for incorrect reeassignment.{color}
> {color:#1d1c1d}We might need to tackle this by removing the entries from the
> map when hbck actually deletes{color}{color:#1d1c1d} the meta entries for
> this region which kind of matches the in-memory map’s expectation with the
> META state.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25183) Move more balancer related classes to hbase-balancer
[ https://issues.apache.org/jira/browse/HBASE-25183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215586#comment-17215586 ] Andrew Kyle Purtell edited comment on HBASE-25183 at 10/16/20, 6:19 PM: bq. If I just want to keep politeness I could just leave the issue there, and reply a single sentence every week with very positive words but no actual action. Everyone will think I’m a good person but what is the value to the project? Ok, but I will say this: I think one can be polite and still stick to one's opinions. For example, let's say I want to separate out the balancer to a separate module. Someone else on the project has concerns, but I don't agree, so I say "Yes I hear your concerns and here is why I disagree: ..." . No more no less. And then I would put up my patch to split the balancer out to its own module. :-) During patch review the community will either accept it or not, there may be a technical veto, the technical veto may be judged to be valid. So I would then choose to abide by the rationale for the technical veto and update my patch, or abandon the work and move on to another issue. Or maybe I would succeed in getting the technical veto lifted through continuing polite communication and technical rationale, and all would be good that way. I think this is the Apache model of handling disagreement over code. was (Author: apurtell): bq. If I just want to keep politeness I could just leave the issue there, and reply a single sentence every week with very positive words but no actual action. Everyone will think I’m a good person but what is the value to the project? Ok, but I will say this: I think one can be polite and still stick to one's opinions. For example, let's say I want to separate out the balancer to a separate module. Someone else on the project has concerns, but I don't agree, so I say "Yes I hear your concerns and here is why I disagree: ..." . No more no less. And then I would put up my patch to split the balancer out to its own module. :-) > Move more balancer related classes to hbase-balancer > > > Key: HBASE-25183 > URL: https://issues.apache.org/jira/browse/HBASE-25183 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25183) Move more balancer related classes to hbase-balancer
[ https://issues.apache.org/jira/browse/HBASE-25183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215586#comment-17215586 ] Andrew Kyle Purtell commented on HBASE-25183: - bq. If I just want to keep politeness I could just leave the issue there, and reply a single sentence every week with very positive words but no actual action. Everyone will think I’m a good person but what is the value to the project? Ok, but I will say this: I think one can be polite and still stick to one's opinions. For example, let's say I want to separate out the balancer to a separate module. Someone else on the project has concerns, but I don't agree, so I say "Yes I hear your concerns and here is why I disagree: ..." . No more no less. And then I would put up my patch to split the balancer out to its own module. :-) > Move more balancer related classes to hbase-balancer > > > Key: HBASE-25183 > URL: https://issues.apache.org/jira/browse/HBASE-25183 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25183) Move more balancer related classes to hbase-balancer
[ https://issues.apache.org/jira/browse/HBASE-25183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215579#comment-17215579 ] Andrew Kyle Purtell commented on HBASE-25183: - bq. We have a ClusterMetrics and usually we will use the metrics there to do calculation. I think this is also a very good abstraction. All the developers of the balancer module except the FN balancer are all good at abstraction and tried best to not rely on master too much. I think this could be kept in the future. So in general, I do not think we need to modify two places, just add new metrics to ClusterMetrics and balancer could benefit. That seems reasonable. This is how a write up could help, getting the answers to such questions before they are asked. One observation is ClusterMetrics, as a data structure, is fat and scales up with the size of the cluster. Maybe as part of this work we can break it up into separate messages/structures that can be independently updated. Just a thought. > Move more balancer related classes to hbase-balancer > > > Key: HBASE-25183 > URL: https://issues.apache.org/jira/browse/HBASE-25183 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25183) Move more balancer related classes to hbase-balancer
[ https://issues.apache.org/jira/browse/HBASE-25183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215570#comment-17215570 ] Andrew Kyle Purtell edited comment on HBASE-25183 at 10/16/20, 6:02 PM: I see the conflict has been moved to a thread on private@. I concur that's the best way to proceed, we can keep the discussion here to the technical issues. To me it looks like a misunderstanding (Stack's style is to ask questions, he is not being provocative, he is being genuine with the questioning; but Duo is also entitled to his opinions) and I hope a private mediation will be able to help. was (Author: apurtell): I see the conflict has been moved to a thread on private@. I concur that's the best way to proceed, we can keep the discussion here to the technical issues. > Move more balancer related classes to hbase-balancer > > > Key: HBASE-25183 > URL: https://issues.apache.org/jira/browse/HBASE-25183 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25183) Move more balancer related classes to hbase-balancer
[ https://issues.apache.org/jira/browse/HBASE-25183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215570#comment-17215570 ] Andrew Kyle Purtell commented on HBASE-25183: - I see the conflict has been moved to a thread on private@. I concur that's the best way to proceed, we can keep the discussion here to the technical issues. > Move more balancer related classes to hbase-balancer > > > Key: HBASE-25183 > URL: https://issues.apache.org/jira/browse/HBASE-25183 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25179) Assert format is incorrect in HFilePerformanceEvaluation class.
[
https://issues.apache.org/jira/browse/HBASE-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215533#comment-17215533
]
Hudson commented on HBASE-25179:
Results for branch branch-2.3
[build #84 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/84/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/84/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/84/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/84/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/84/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Assert format is incorrect in HFilePerformanceEvaluation class.
> ---
>
> Key: HBASE-25179
> URL: https://issues.apache.org/jira/browse/HBASE-25179
> Project: HBase
> Issue Type: Improvement
> Components: Performance, test
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 2.2.7
>
>
> [HFilePerformanceEvaluation
> |https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java#L518]
>
> The format of expected and actual is interchanged.
> {code:java}
> PerformanceEvaluationCommons.assertValueSize(c.getValueLength(), ROW_LENGTH);
> {code}
> The first argument should be expected and second should be actual.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] shahrs87 commented on pull request #2551: [HBASE-25179] Fix Assert format in HFilePerformanceEvaluation class.
shahrs87 commented on pull request #2551: URL: https://github.com/apache/hbase/pull/2551#issuecomment-710236415 Thank you @virajjasani for the review and commit ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-22978) Online slow response log
[
https://issues.apache.org/jira/browse/HBASE-22978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215500#comment-17215500
]
Hudson commented on HBASE-22978:
Results for branch branch-1
[build #44 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Online slow response log
>
>
> Key: HBASE-22978
> URL: https://issues.apache.org/jira/browse/HBASE-22978
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Operability, regionserver, shell
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: Screen Shot 2019-10-19 at 2.31.59 AM.png, Screen Shot
> 2019-10-19 at 2.32.54 AM.png, Screen Shot 2019-10-19 at 2.34.11 AM.png,
> Screen Shot 2019-10-19 at 2.36.14 AM.png
>
>
> Today when an individual RPC exceeds a configurable time bound we log a
> complaint by way of the logging subsystem. These log lines look like:
> {noformat}
> 2019-08-30 22:10:36,195 WARN [,queue=15,port=60020] ipc.RpcServer -
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1567203007549,
> "responsesize":6819737,
> "method":"Scan",
> "param":"region { type: REGION_NAME value:
> \"tsdb,\\000\\000\\215\\f)o\\024\\302\\220\\000\\000\\000\\000\\000\\001\\000\\000\\000\\000\\000\\006\\000\\000\\000\\000\\000\\005\\000\\000",
> "processingtimems":28646,
> "client":"10.253.196.215:41116",
> "queuetimems":22453,
> "class":"HRegionServer"}
> {noformat}
> Unfortunately we often truncate the request parameters, like in the above
> example. We do this because the human readable representation is verbose, the
> rate of too slow warnings may be high, and the combination of these things
> can overwhelm the log capture system. The truncation is unfortunate because
> it eliminates much of the utility of the warnings. For example, the region
> name, the start and end keys, and the filter hierarchy are all important
> clues for debugging performance problems caused by moderate to low
> selectivity queries or queries made at a high rate.
> We can maintain an in-memory ring buffer of requests that were judged to be
> too slow in addition to the responseTooSlow logging. The in-memory
> representation can be complete and compressed. A new admin API and shell
> command can provide access to the ring buffer for online performance
> debugging. A modest sizing of the ring buffer will prevent excessive memory
> utilization for a minor performance debugging feature by limiting the total
> number of retained records. There is some chance a high rate of requests will
> cause information on other interesting requests to be overwritten before it
> can be read. This is the nature of a ring buffer and an acceptable trade off.
> The write request types do not require us to retain all information submitted
> in the request. We don't need to retain all key-values in the mutation, which
> may be too large to comfortably retain. We only need a unique set of row
> keys, or even a min/max range, and total counts.
> The consumers of this information will be debugging tools. We can afford to
> apply fast compression to ring buffer entries (if codec support is
> available), something like snappy or zstandard, and decompress on the fly
> when servicing the retrieval API request. This will minimize the impact of
> retaining more information about slow requests than we do today.
> This proposal is for retention of request information only, the same
> information provided by responseTooSlow warnings. Total size of response
> serialization, possibly also total cell or row counts, should be sufficient
> to characterize the response.
> Optionally persist new entries added to the ring buffer into one or more
> files in HDFS in a write-behind manner. If the HDFS writer blocks or falls
> behind and we are unable to persist an entry before it is overwritten, that
>
[jira] [Commented] (HBASE-24718) Generic NamedQueue framework for recent in-memory history (refactor slowlog)
[
https://issues.apache.org/jira/browse/HBASE-24718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215502#comment-17215502
]
Hudson commented on HBASE-24718:
Results for branch branch-1
[build #44 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Generic NamedQueue framework for recent in-memory history (refactor slowlog)
>
>
> Key: HBASE-24718
> URL: https://issues.apache.org/jira/browse/HBASE-24718
> Project: HBase
> Issue Type: Sub-task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
> Attachments: Screen Shot 2020-07-20 at 2.50.34 PM.png
>
>
> As per the discussion on parent jira, we should come up with named queue
> (online ring buffer) to serve recent history for multiple use-cases like
> slowlog, balancer decision, other region activities e.g flush, compaction,
> split, merge etc.
> Since we already have slow/large rpc logs in ring buffer (HBASE-22978), as
> part of this Jira, the proposal is to refactor slowlog provider to get
> generic payload for ring buffer and based on event type (slow_log is the only
> one for now), we can have separate internal in-memory queues.
> After this refactor, it should be relatively simpler to use the same
> framework and create more cases like parent Jira (balancer decision in ring
> buffer).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24665) MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
[
https://issues.apache.org/jira/browse/HBASE-24665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215504#comment-17215504
]
Hudson commented on HBASE-24665:
Results for branch branch-1
[build #44 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
> --
>
> Key: HBASE-24665
> URL: https://issues.apache.org/jira/browse/HBASE-24665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.3.0, 2.1.10, 1.4.14, 2.2.6
>Reporter: wenfeiyi666
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.6
>
>
> when use multiwal, any a wal request roll, all wal will be together roll.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24849) Branch-1 Backport : HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
[
https://issues.apache.org/jira/browse/HBASE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215503#comment-17215503
]
Hudson commented on HBASE-24849:
Results for branch branch-1
[build #44 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Branch-1 Backport : HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when
> one of the WAL needs a roll
> --
>
> Key: HBASE-24849
> URL: https://issues.apache.org/jira/browse/HBASE-24849
> Project: HBase
> Issue Type: Bug
> Components: wal
>Reporter: Anoop Sam John
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25179) Assert format is incorrect in HFilePerformanceEvaluation class.
[
https://issues.apache.org/jira/browse/HBASE-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215498#comment-17215498
]
Hudson commented on HBASE-25179:
Results for branch branch-1
[build #44 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Assert format is incorrect in HFilePerformanceEvaluation class.
> ---
>
> Key: HBASE-25179
> URL: https://issues.apache.org/jira/browse/HBASE-25179
> Project: HBase
> Issue Type: Improvement
> Components: Performance, test
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 2.2.7
>
>
> [HFilePerformanceEvaluation
> |https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java#L518]
>
> The format of expected and actual is interchanged.
> {code:java}
> PerformanceEvaluationCommons.assertValueSize(c.getValueLength(), ROW_LENGTH);
> {code}
> The first argument should be expected and second should be actual.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23935) Backport HBASE-22978(with sub-tasks), HBASE-24718, HBASE-24528 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-23935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215499#comment-17215499
]
Hudson commented on HBASE-23935:
Results for branch branch-1
[build #44 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Backport HBASE-22978(with sub-tasks), HBASE-24718, HBASE-24528 to branch-1
> --
>
> Key: HBASE-23935
> URL: https://issues.apache.org/jira/browse/HBASE-23935
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 1.7.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 1.7.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24528) Improve balancer decision observability
[
https://issues.apache.org/jira/browse/HBASE-24528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215501#comment-17215501
]
Hudson commented on HBASE-24528:
Results for branch branch-1
[build #44 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/44//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Improve balancer decision observability
> ---
>
> Key: HBASE-24528
> URL: https://issues.apache.org/jira/browse/HBASE-24528
> Project: HBase
> Issue Type: New Feature
> Components: Admin, Balancer, Operability, shell, UI
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
> Attachments: Screenshot 2020-08-12 at 11.50.43 PM.png, Screenshot
> 2020-08-15 at 8.08.57 PM.png
>
>
> We provide detailed INFO and DEBUG level logging of balancer decision
> factors, outcome, and reassignment planning, as well as similarly detailed
> logging of the resulting assignment manager activity. However, an operator
> may need to perform online and interactive observation, debugging, or
> performance analysis of current balancer activity. Scraping and correlating
> the many log lines resulting from a balancer execution is labor intensive and
> has a lot of latency (order of ~minutes to acquire and index, order of
> ~minutes to correlate).
> The balancer should maintain a rolling window of history, e.g. the last 100
> region move plans, or last 1000 region move plans submitted to the assignment
> manager. This history should include decision factor details and weights and
> costs. The rsgroups balancer may be able to provide fairly simple decision
> factors, like for example "this table was reassigned to that regionserver
> group". The underlying or vanilla stochastic balancer on the other hand,
> after a walk over random assignment plans, will have considered a number of
> cost functions with various inputs (locality, load, etc.) and multipliers,
> including custom cost functions. We can devise an extensible class structure
> that represents explanations for balancer decisions, and for each region move
> plan that is actually submitted to the assignment manager, we can keep the
> explanations of all relevant decision factors alongside the other details of
> the assignment plan like the region name, and the source and destination
> regionservers.
> This history should be available via API for use by new shell commands and
> admin UI widgets.
> The new shell commands and UI widgets can unpack the representation of
> balancer decision components into human readable output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2548: HBASE-25189 [Metrics] Add checkAndPut and checkAndDelete latency metrics at table level
Apache-HBase commented on pull request #2548: URL: https://github.com/apache/hbase/pull/2548#issuecomment-710124820 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 15s | branch-1 passed | | +1 :green_heart: | compile | 1m 16s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | compile | 1m 25s | branch-1 passed with JDK v1.7.0_272 | | +1 :green_heart: | checkstyle | 2m 15s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 14s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 1m 19s | branch-1 passed with JDK v1.7.0_272 | | +0 :ok: | spotbugs | 3m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 30s | branch-1 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 4s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | compile | 1m 23s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | javac | 1m 23s | the patch passed | | -1 :x: | checkstyle | 1m 35s | hbase-server: The patch generated 2 new + 156 unchanged - 2 fixed = 158 total (was 158) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 2m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 52s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 1m 5s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 1m 21s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | findbugs | 4m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 29s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 0m 41s | hbase-hadoop2-compat in the patch passed. | | +1 :green_heart: | unit | 120m 8s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 1m 19s | The patch does not generate ASF License warnings. | | | | 176m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2548/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2548 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 859e0bd88aca 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2548/out/precommit/personality/provided.sh | | git revision | branch-1 / e066951 | | Default Java | 1.7.0_272 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2548/2/artifact/out/diff-checkstyle-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2548/2/testReport/ | | Max. process+thread count | 3747 (vs. ulimit of 1) | | modules | C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2548/2/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To resp
[GitHub] [hbase] Apache-HBase commented on pull request #2553: BackPort HBASE-25093 the RSGroupBasedLoadBalancer#retainAssignment throws NPE
Apache-HBase commented on pull request #2553: URL: https://github.com/apache/hbase/pull/2553#issuecomment-710023279 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 49s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 41s | branch-2.3 passed | | +1 :green_heart: | compile | 1m 20s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 5m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | compile | 1m 17s | the patch passed | | +1 :green_heart: | javac | 1m 17s | the patch passed | | +1 :green_heart: | shadedjars | 5m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 153m 12s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 3m 25s | hbase-rsgroup in the patch passed. | | | | 186m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2553 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fce0419237f7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / c7f95fffbd | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/testReport/ | | Max. process+thread count | 3759 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-rsgroup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24849) Branch-1 Backport : HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
[ https://issues.apache.org/jira/browse/HBASE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan resolved HBASE-24849. --- Hadoop Flags: Reviewed Resolution: Fixed > Branch-1 Backport : HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when > one of the WAL needs a roll > -- > > Key: HBASE-24849 > URL: https://issues.apache.org/jira/browse/HBASE-24849 > Project: HBase > Issue Type: Bug >Reporter: Anoop Sam John >Assignee: wenfeiyi666 >Priority: Major > Fix For: 1.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24849) Branch-1 Backport : HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
[ https://issues.apache.org/jira/browse/HBASE-24849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-24849: -- Component/s: wal > Branch-1 Backport : HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when > one of the WAL needs a roll > -- > > Key: HBASE-24849 > URL: https://issues.apache.org/jira/browse/HBASE-24849 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Anoop Sam John >Assignee: wenfeiyi666 >Priority: Major > Fix For: 1.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Reidddddd merged pull request #2194: HBASE-24849 Branch-1 Backport : HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
Reidd merged pull request #2194: URL: https://github.com/apache/hbase/pull/2194 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2553: BackPort HBASE-25093 the RSGroupBasedLoadBalancer#retainAssignment throws NPE
Apache-HBase commented on pull request #2553: URL: https://github.com/apache/hbase/pull/2553#issuecomment-710021082 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 49s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 26s | branch-2.3 passed | | +1 :green_heart: | compile | 1m 37s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 6m 7s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-rsgroup in branch-2.3 failed. | | -0 :warning: | javadoc | 0m 38s | hbase-server in branch-2.3 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 8s | the patch passed | | +1 :green_heart: | compile | 1m 35s | the patch passed | | +1 :green_heart: | javac | 1m 35s | the patch passed | | +1 :green_heart: | shadedjars | 7m 6s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 54s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 28s | hbase-rsgroup in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 142m 34s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 3m 21s | hbase-rsgroup in the patch passed. | | | | 181m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2553 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3ce012e9d085 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / c7f95fffbd | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-rsgroup.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-rsgroup.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/testReport/ | | Max. process+thread count | 3815 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-rsgroup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd commented on pull request #2194: HBASE-24849 Branch-1 Backport : HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
Reidd commented on pull request #2194: URL: https://github.com/apache/hbase/pull/2194#issuecomment-710021123 We are cool now, thx for the contribution, this is nice @WenFeiYi This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2541: HBASE-25173 Remove owner related methods in TableDescriptor/TableDescriptorBuilder
Apache-HBase commented on pull request #2541: URL: https://github.com/apache/hbase/pull/2541#issuecomment-710018183 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 17s | master passed | | +1 :green_heart: | compile | 2m 23s | master passed | | +1 :green_heart: | shadedjars | 7m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 59s | the patch passed | | +1 :green_heart: | compile | 2m 9s | the patch passed | | +1 :green_heart: | javac | 2m 9s | the patch passed | | +1 :green_heart: | shadedjars | 7m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 26s | hbase-client in the patch passed. | | -1 :x: | unit | 219m 55s | hbase-server in the patch failed. | | -1 :x: | unit | 4m 38s | hbase-shell in the patch failed. | | +1 :green_heart: | unit | 3m 3s | hbase-endpoint in the patch passed. | | | | 263m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2541 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 978cc541edc8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e6c78f4037 | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-shell.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/2/testReport/ | | Max. process+thread count | 3807 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server hbase-shell hbase-endpoint U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2541: HBASE-25173 Remove owner related methods in TableDescriptor/TableDescriptorBuilder
Apache-HBase commented on pull request #2541: URL: https://github.com/apache/hbase/pull/2541#issuecomment-710012394 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 51s | master passed | | +1 :green_heart: | compile | 2m 32s | master passed | | +1 :green_heart: | shadedjars | 7m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 44s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 5s | the patch passed | | +1 :green_heart: | compile | 2m 58s | the patch passed | | +1 :green_heart: | javac | 2m 58s | the patch passed | | +1 :green_heart: | shadedjars | 8m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hbase-client in the patch passed. | | -1 :x: | unit | 202m 57s | hbase-server in the patch failed. | | -1 :x: | unit | 4m 38s | hbase-shell in the patch failed. | | +1 :green_heart: | unit | 3m 19s | hbase-endpoint in the patch passed. | | | | 251m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2541 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e49c5aebe4f0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e6c78f4037 | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-shell.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/2/testReport/ | | Max. process+thread count | 3450 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server hbase-shell hbase-endpoint U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24628) Region normalizer now respects a rate limit
[
https://issues.apache.org/jira/browse/HBASE-24628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215361#comment-17215361
]
Hudson commented on HBASE-24628:
Results for branch master
[build #96 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/96/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/96/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/96/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/96/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Region normalizer now respects a rate limit
> ---
>
> Key: HBASE-24628
> URL: https://issues.apache.org/jira/browse/HBASE-24628
> Project: HBase
> Issue Type: Improvement
> Components: Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> There's no limits on the normalizer right now. It will iterate through the
> tables one at a time until it's touched all tables. For any table, it
> generates a complete plan and executes that plan in totality.
> It would be nice to allow operators to configure some limits here. Off the
> top of my head, the two metrics that might be interesting are {{split|merge
> actions / hour}} or {{hfile mb volume / hour}}. Either way, we'd need to
> track a little more metadata.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2194: HBASE-24849 Branch-1 Backport : HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
Apache-HBase commented on pull request #2194: URL: https://github.com/apache/hbase/pull/2194#issuecomment-709988778 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 10m 1s | branch-1 passed | | +1 :green_heart: | compile | 0m 46s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | compile | 0m 46s | branch-1 passed with JDK v1.7.0_272 | | +1 :green_heart: | checkstyle | 1m 45s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 0m 44s | branch-1 passed with JDK v1.7.0_272 | | +0 :ok: | spotbugs | 3m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 7s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 4s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | checkstyle | 1m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 3m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 5m 5s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 0m 44s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | findbugs | 3m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 121m 25s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 165m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2194/10/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2194 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux c94d4d5a346c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2194/out/precommit/personality/provided.sh | | git revision | branch-1 / fb25a7d | | Default Java | 1.7.0_272 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_262 /usr/lib/jvm/zulu-7-amd64:1.7.0_272 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2194/10/testReport/ | | Max. process+thread count | 3741 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2194/10/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25179) Assert format is incorrect in HFilePerformanceEvaluation class.
[
https://issues.apache.org/jira/browse/HBASE-25179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215343#comment-17215343
]
Hudson commented on HBASE-25179:
Results for branch branch-2
[build #78 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78//console].
> Assert format is incorrect in HFilePerformanceEvaluation class.
> ---
>
> Key: HBASE-25179
> URL: https://issues.apache.org/jira/browse/HBASE-25179
> Project: HBase
> Issue Type: Improvement
> Components: Performance, test
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 2.2.7
>
>
> [HFilePerformanceEvaluation
> |https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java#L518]
>
> The format of expected and actual is interchanged.
> {code:java}
> PerformanceEvaluationCommons.assertValueSize(c.getValueLength(), ROW_LENGTH);
> {code}
> The first argument should be expected and second should be actual.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24628) Region normalizer now respects a rate limit
[
https://issues.apache.org/jira/browse/HBASE-24628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215341#comment-17215341
]
Hudson commented on HBASE-24628:
Results for branch branch-2
[build #78 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/78//console].
> Region normalizer now respects a rate limit
> ---
>
> Key: HBASE-24628
> URL: https://issues.apache.org/jira/browse/HBASE-24628
> Project: HBase
> Issue Type: Improvement
> Components: Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> There's no limits on the normalizer right now. It will iterate through the
> tables one at a time until it's touched all tables. For any table, it
> generates a complete plan and executes that plan in totality.
> It would be nice to allow operators to configure some limits here. Off the
> top of my head, the two metrics that might be interesting are {{split|merge
> actions / hour}} or {{hfile mb volume / hour}}. Either way, we'd need to
> track a little more metadata.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25192) s390x support for HBase CI
[ https://issues.apache.org/jira/browse/HBASE-25192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215340#comment-17215340 ] Duo Zhang commented on HBASE-25192: --- I think first we could have a nightly build to run on the new hardware? You just need to add the node as the jenkins slave, like the arm node, and I could help setting up a nightly job for master branch only on it first. If everything is fine, we could add other branches as well, and then maybe pre commit? I'm not sure. But there is a risk that, s390x seems to be an IBM specific platform? Then if we meet an error on this platform in the CI system, i.e, when someone post a new PR, then how could the developer fix the problem as he/she does not have a s390x? Thanks. > s390x support for HBase CI > -- > > Key: HBASE-25192 > URL: https://issues.apache.org/jira/browse/HBASE-25192 > Project: HBase > Issue Type: Improvement >Reporter: Aditi Jadhav >Priority: Major > > Hi All, > As I understand that currently CI available for HBase ( to validate any code > updates ) is : https://ci-hadoop.apache.org/job/HBase/ and as can be seen > most of the deployment here is on amd64 and ppc64le. > Wanted to know your views/comments/suggestions for extending CI of HBase on > s390x . > 1) If the community will be willing to add - vm's/slaves ( s390x based ) to > their current above CI. > 2) Use an externally hosted jenkins CI - for running HBase build on s390x and > link the results of the build to the above CI. > Please let me know your thoughts. > Regards, > Aditi -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25192) s390x support for HBase CI
Aditi Jadhav created HBASE-25192: Summary: s390x support for HBase CI Key: HBASE-25192 URL: https://issues.apache.org/jira/browse/HBASE-25192 Project: HBase Issue Type: Improvement Reporter: Aditi Jadhav Hi All, As I understand that currently CI available for HBase ( to validate any code updates ) is : https://ci-hadoop.apache.org/job/HBase/ and as can be seen most of the deployment here is on amd64 and ppc64le. Wanted to know your views/comments/suggestions for extending CI of HBase on s390x . 1) If the community will be willing to add - vm's/slaves ( s390x based ) to their current above CI. 2) Use an externally hosted jenkins CI - for running HBase build on s390x and link the results of the build to the above CI. Please let me know your thoughts. Regards, Aditi -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2553: BackPort HBASE-25093 the RSGroupBasedLoadBalancer#retainAssignment throws NPE
Apache-HBase commented on pull request #2553: URL: https://github.com/apache/hbase/pull/2553#issuecomment-709962546 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 48s | branch-2.3 passed | | +1 :green_heart: | checkstyle | 1m 26s | branch-2.3 passed | | +1 :green_heart: | spotbugs | 2m 45s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | checkstyle | 1m 26s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 28s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 45m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2553 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux a26fc2808aa5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / c7f95fffbd | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-rsgroup U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2553/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
