[GitHub] [hbase] Apache-HBase commented on pull request #2452: HBASE-25071 ReplicationServer support start ReplicationSource internal
Apache-HBase commented on pull request #2452: URL: https://github.com/apache/hbase/pull/2452#issuecomment-721557915 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 22s | HBASE-24666 passed | | +1 :green_heart: | checkstyle | 1m 15s | HBASE-24666 passed | | +1 :green_heart: | spotbugs | 5m 32s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 1 new + 56 unchanged - 0 fixed = 57 total (was 56) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 16m 56s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 1m 39s | the patch passed | | +1 :green_heart: | spotbugs | 5m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 48m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/11/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2452 | | Optional Tests | dupname asflicense cc hbaseprotoc prototool spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 61f8e8dcfd67 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / f67c3dfc5a | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/11/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/11/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2623: HBASE-25240 gson format of RpcServer.logResponse is abnormal
Apache-HBase commented on pull request #2623: URL: https://github.com/apache/hbase/pull/2623#issuecomment-721550119 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 43s | master passed | | +1 :green_heart: | checkstyle | 1m 27s | master passed | | +1 :green_heart: | spotbugs | 2m 41s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | -0 :warning: | checkstyle | 0m 24s | hbase-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 12s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. | | | | 41m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2623 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 2d2085481eaf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f37cd05c32 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2535: HBASE-25116 RegionMonitor support RegionTask count normalize
infraio commented on a change in pull request #2535: URL: https://github.com/apache/hbase/pull/2535#discussion_r517125577 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java ## @@ -1422,6 +1424,10 @@ public RegionMonitor(Connection connection, String[] monitorTargets, boolean use this.configuredReadTableTimeouts = new HashMap<>(configuredReadTableTimeouts); this.configuredWriteTableTimeout = configuredWriteTableTimeout; this.readAllCF = conf.getBoolean(HConstants.HBASE_CANARY_READ_ALL_CF, true); + this.minTaskCount = conf.getInt(HConstants.HBASE_CANARY_REGION_MONITOR_TASK_COUNT_MIN, 1000); Review comment: How to disable this if I don't want normalize? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2535: HBASE-25116 RegionMonitor support RegionTask count normalize
infraio commented on a change in pull request #2535: URL: https://github.com/apache/hbase/pull/2535#discussion_r517125696 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java ## @@ -1504,6 +1504,13 @@ public static final String HBASE_CANARY_READ_RAW_SCAN_KEY = "hbase.canary.read.raw.enabled"; public static final String HBASE_CANARY_READ_ALL_CF = "hbase.canary.read.all.column.famliy"; + + public static final String HBASE_CANARY_REGION_MONITOR_TASK_COUNT_MAX = + "hbase.canary.region.monitor.task.count.max"; Review comment: region.monitor.task => region.task This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2614: HBASE-25216 The client zk syncer should deal with meta replica count …
Apache-HBase commented on pull request #2614: URL: https://github.com/apache/hbase/pull/2614#issuecomment-721540399 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 50s | master passed | | +1 :green_heart: | compile | 1m 26s | master passed | | +1 :green_heart: | shadedjars | 7m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 13s | the patch passed | | +1 :green_heart: | compile | 1m 15s | the patch passed | | +1 :green_heart: | javac | 1m 15s | the patch passed | | +1 :green_heart: | shadedjars | 6m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 132m 28s | hbase-server in the patch failed. | | | | 165m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2614/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2614 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 27d607eddee8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f37cd05c32 | | Default Java | AdoptOpenJDK-11.0.6+10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2614/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2614/4/testReport/ | | Max. process+thread count | 4184 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2614/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2452: HBASE-25071 ReplicationServer support start ReplicationSource internal
Apache-HBase commented on pull request #2452: URL: https://github.com/apache/hbase/pull/2452#issuecomment-721538497 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 59s | HBASE-24666 passed | | +1 :green_heart: | compile | 1m 47s | HBASE-24666 passed | | +1 :green_heart: | shadedjars | 7m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | compile | 1m 47s | the patch passed | | +1 :green_heart: | javac | 1m 47s | the patch passed | | +1 :green_heart: | shadedjars | 7m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 48s | hbase-protocol-shaded in the patch passed. | | -1 :x: | unit | 209m 50s | hbase-server in the patch failed. | | | | 245m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/10/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2452 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux 8a631ab9fd67 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / f67c3dfc5a | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/10/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/10/testReport/ | | Max. process+thread count | 3585 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/10/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2623: HBASE-25240 gson format of RpcServer.logResponse is abnormal
Apache-HBase commented on pull request #2623: URL: https://github.com/apache/hbase/pull/2623#issuecomment-721533741 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 30s | master passed | | +1 :green_heart: | compile | 1m 20s | master passed | | +1 :green_heart: | shadedjars | 6m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | +1 :green_heart: | compile | 1m 19s | the patch passed | | +1 :green_heart: | javac | 1m 19s | the patch passed | | +1 :green_heart: | shadedjars | 6m 26s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 23s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 153m 9s | hbase-server in the patch passed. | | | | 182m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2623 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 110cd1d80804 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f37cd05c32 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/2/testReport/ | | Max. process+thread count | 4851 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2623: HBASE-25240 gson format of RpcServer.logResponse is abnormal
Apache-HBase commented on pull request #2623: URL: https://github.com/apache/hbase/pull/2623#issuecomment-721530666 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 25s | master passed | | +1 :green_heart: | compile | 1m 34s | master passed | | +1 :green_heart: | shadedjars | 6m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 8s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 7s | the patch passed | | +1 :green_heart: | compile | 1m 35s | the patch passed | | +1 :green_heart: | javac | 1m 35s | the patch passed | | +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 48s | hbase-common in the patch passed. | | -1 :x: | unit | 139m 38s | hbase-server in the patch failed. | | | | 172m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2623 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ba9d15e50181 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f37cd05c32 | | Default Java | AdoptOpenJDK-11.0.6+10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/2/testReport/ | | Max. process+thread count | 3800 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2452: HBASE-25071 ReplicationServer support start ReplicationSource internal
Apache-HBase commented on pull request #2452: URL: https://github.com/apache/hbase/pull/2452#issuecomment-721515636 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 15s | HBASE-24666 passed | | +1 :green_heart: | compile | 2m 4s | HBASE-24666 passed | | +1 :green_heart: | shadedjars | 6m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 2m 3s | the patch passed | | +1 :green_heart: | javac | 2m 3s | the patch passed | | +1 :green_heart: | shadedjars | 6m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 58s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 135m 45s | hbase-server in the patch passed. | | | | 167m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/10/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2452 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux 1c6ad0b868ee 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / f67c3dfc5a | | Default Java | 2020-01-14 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/10/testReport/ | | Max. process+thread count | 3992 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/10/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache-HBase commented on pull request #2584: URL: https://github.com/apache/hbase/pull/2584#issuecomment-721514318 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 47s | master passed | | +1 :green_heart: | compile | 1m 22s | master passed | | +1 :green_heart: | shadedjars | 7m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 46s | the patch passed | | +1 :green_heart: | compile | 1m 23s | the patch passed | | +1 :green_heart: | javac | 1m 23s | the patch passed | | +1 :green_heart: | shadedjars | 7m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. | | -1 :x: | unit | 215m 31s | hbase-server in the patch failed. | | | | 247m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2584 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 00958b11cf21 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c98e993b23 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/testReport/ | | Max. process+thread count | 3599 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25242) Add Increment/Append support to RowMutations
Toshihiro Suzuki created HBASE-25242: Summary: Add Increment/Append support to RowMutations Key: HBASE-25242 URL: https://issues.apache.org/jira/browse/HBASE-25242 Project: HBase Issue Type: New Feature Reporter: Toshihiro Suzuki Assignee: Toshihiro Suzuki Fix For: 3.0.0-alpha-1, 2.4.0 Currently, RowMutations supports only Put and Delete. Supporting Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24210) Add Increment, Append and CheckAndMutate support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-24210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225851#comment-17225851 ] Toshihiro Suzuki commented on HBASE-24210: -- It looks like this change will make things too complicated and it has big side effects and incompatible changes. So I will give it up. Closing it as Won't Fix. > Add Increment, Append and CheckAndMutate support to RowMutations > > > Key: HBASE-24210 > URL: https://issues.apache.org/jira/browse/HBASE-24210 > Project: HBase > Issue Type: New Feature >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently, RowMutations supports only Put and Delete. Supporting Increment, > Append and CheckAndMutate in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24996) Support CheckAndMutate in Region.batchMutate()
[ https://issues.apache.org/jira/browse/HBASE-24996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225850#comment-17225850 ] Toshihiro Suzuki commented on HBASE-24996: -- It looks like this change will make things too complicated and it has big side effects and incompatible changes. So I will give it up. Closing it as Won't Fix. > Support CheckAndMutate in Region.batchMutate() > -- > > Key: HBASE-24996 > URL: https://issues.apache.org/jira/browse/HBASE-24996 > Project: HBase > Issue Type: Sub-task >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > After HBASE-24602, Region.batchMutate() supports Put/Delete/Increment/Append > operations, but it doesn't support CheckAndMutate. If we support > CheckAndMutate Region.batchMutate(), we can perform > Put/Delete/Increment/Append/CheckAndMutate operations atomically, which is > very useful for some cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24210) Add Increment, Append and CheckAndMutate support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-24210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki resolved HBASE-24210. -- Resolution: Won't Fix > Add Increment, Append and CheckAndMutate support to RowMutations > > > Key: HBASE-24210 > URL: https://issues.apache.org/jira/browse/HBASE-24210 > Project: HBase > Issue Type: New Feature >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently, RowMutations supports only Put and Delete. Supporting Increment, > Append and CheckAndMutate in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24996) Support CheckAndMutate in Region.batchMutate()
[ https://issues.apache.org/jira/browse/HBASE-24996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki resolved HBASE-24996. -- Resolution: Won't Fix > Support CheckAndMutate in Region.batchMutate() > -- > > Key: HBASE-24996 > URL: https://issues.apache.org/jira/browse/HBASE-24996 > Project: HBase > Issue Type: Sub-task >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > After HBASE-24602, Region.batchMutate() supports Put/Delete/Increment/Append > operations, but it doesn't support CheckAndMutate. If we support > CheckAndMutate Region.batchMutate(), we can perform > Put/Delete/Increment/Append/CheckAndMutate operations atomically, which is > very useful for some cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] brfrn169 edited a comment on pull request #2498: HBASE-24996 Support CheckAndMutate in Region.batchMutate()
brfrn169 edited a comment on pull request #2498: URL: https://github.com/apache/hbase/pull/2498#issuecomment-721511433 It looks like this change will make things too complicated and it has big side effects and incompatible changes. So I will give it up. Closing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] brfrn169 closed pull request #2498: HBASE-24996 Support CheckAndMutate in Region.batchMutate()
brfrn169 closed pull request #2498: URL: https://github.com/apache/hbase/pull/2498 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] brfrn169 commented on pull request #2498: HBASE-24996 Support CheckAndMutate in Region.batchMutate()
brfrn169 commented on pull request #2498: URL: https://github.com/apache/hbase/pull/2498#issuecomment-721511433 It looks like this change will make things too complicated and it has big side effects, so I will give it up. Closing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2614: HBASE-25216 The client zk syncer should deal with meta replica count …
Apache-HBase commented on pull request #2614: URL: https://github.com/apache/hbase/pull/2614#issuecomment-721505951 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 49s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 56s | master passed | | +1 :green_heart: | checkstyle | 1m 32s | master passed | | +1 :green_heart: | spotbugs | 2m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | the patch passed | | +1 :green_heart: | checkstyle | 1m 13s | the patch passed | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 17m 12s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 42m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2614/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2614 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux f28fa2bc8dc2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f37cd05c32 | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2614/4/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2614/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2623: HBASE-25240 gson format of RpcServer.logResponse is abnormal
Apache-HBase commented on pull request #2623: URL: https://github.com/apache/hbase/pull/2623#issuecomment-721498699 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 43s | master passed | | +1 :green_heart: | checkstyle | 1m 32s | master passed | | +1 :green_heart: | spotbugs | 2m 50s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | -0 :warning: | checkstyle | 0m 24s | hbase-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 16s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 4m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 47m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2623 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux c37ac448f1a6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f37cd05c32 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2623/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache-HBase commented on pull request #2584: URL: https://github.com/apache/hbase/pull/2584#issuecomment-721496809 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 4s | master passed | | +1 :green_heart: | compile | 1m 37s | master passed | | +1 :green_heart: | shadedjars | 6m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 34s | the patch passed | | +1 :green_heart: | javac | 1m 34s | the patch passed | | +1 :green_heart: | shadedjars | 6m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 5s | hbase-client in the patch passed. | | -1 :x: | unit | 139m 32s | hbase-server in the patch failed. | | | | 171m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2584 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c2a6cb6a7483 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c98e993b23 | | Default Java | AdoptOpenJDK-11.0.6+10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/testReport/ | | Max. process+thread count | 3947 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2614: HBASE-25216 The client zk syncer should deal with meta replica count …
Apache9 commented on a change in pull request #2614:
URL: https://github.com/apache/hbase/pull/2614#discussion_r517078074
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSeparateClientZKCluster.java
##
@@ -255,7 +260,20 @@ public void testAsyncTable() throws Exception {
Get get = new Get(row);
Result result = table.get(get).get();
LOG.debug("Result: " + Bytes.toString(result.getValue(family,
qualifier)));
- Assert.assertArrayEquals(value, result.getValue(family, qualifier));
+ assertArrayEquals(value, result.getValue(family, qualifier));
+}
+ }
+
+ @Test
+ public void testChangeMetaReplicaCount() throws Exception {
Review comment:
In the test we increase from 1 to 3, and then reduce back to 2, you mean
we should add one more assert to reduce to 1?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2452: HBASE-25071 ReplicationServer support start ReplicationSource internal
Apache-HBase commented on pull request #2452: URL: https://github.com/apache/hbase/pull/2452#issuecomment-721491103 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 18s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 1s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 28s | HBASE-24666 passed | | +1 :green_heart: | checkstyle | 1m 36s | HBASE-24666 passed | | +1 :green_heart: | spotbugs | 6m 33s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | -0 :warning: | checkstyle | 1m 11s | hbase-server: The patch generated 3 new + 56 unchanged - 0 fixed = 59 total (was 56) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 49s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 1m 47s | the patch passed | | +1 :green_heart: | spotbugs | 7m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 62m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2452 | | Optional Tests | dupname asflicense cc hbaseprotoc prototool spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 69b1fc18422e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / f67c3dfc5a | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/10/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2452/10/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2614: HBASE-25216 The client zk syncer should deal with meta replica count …
Apache9 commented on a change in pull request #2614:
URL: https://github.com/apache/hbase/pull/2614#discussion_r517077805
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/zksyncer/ClientZKSyncer.java
##
@@ -145,8 +177,7 @@ private final void setDataForClientZkUntilSuccess(String
node, byte[] data)
LOG.warn(
"Failed to create znode " + node + " due to: " + e.getMessage() +
", will retry later");
}
- } catch (KeeperException.ConnectionLossException
- | KeeperException.SessionExpiredException ee) {
+ } catch (KeeperException.SessionExpiredException see) {
Review comment:
ConnectionLoss just means the connection is broken, the zk library will
reconnect automatically to resume the session if possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-25235) Cleanup the deprecated methods in TimeRange
[ https://issues.apache.org/jira/browse/HBASE-25235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25235: -- Hadoop Flags: Incompatible change,Reviewed Release Note: Removed all the public constructors of TimeRange, thus now we make TimeRange final. Also removed the withinTimeRange(byte[], int) method, just use withinTimeRange(long) instead. > Cleanup the deprecated methods in TimeRange > --- > > Key: HBASE-25235 > URL: https://issues.apache.org/jira/browse/HBASE-25235 > Project: HBase > Issue Type: Sub-task > Components: API, Client >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25235) Cleanup the deprecated methods in TimeRange
[ https://issues.apache.org/jira/browse/HBASE-25235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25235. --- Fix Version/s: 3.0.0-alpha-1 Resolution: Fixed Merged to master. Thanks [~janh] and [~stack] for reviewing. > Cleanup the deprecated methods in TimeRange > --- > > Key: HBASE-25235 > URL: https://issues.apache.org/jira/browse/HBASE-25235 > Project: HBase > Issue Type: Sub-task > Components: API, Client >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #2616: HBASE-25235 Cleanup the deprecated methods in TimeRange
Apache9 merged pull request #2616: URL: https://github.com/apache/hbase/pull/2616 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2616: HBASE-25235 Cleanup the deprecated methods in TimeRange
Apache9 commented on a change in pull request #2616:
URL: https://github.com/apache/hbase/pull/2616#discussion_r517070761
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java
##
@@ -18,24 +18,23 @@
package org.apache.hadoop.hbase.io;
-import org.apache.hadoop.hbase.util.Bytes;
import org.apache.yetus.audience.InterfaceAudience;
/**
* Represents an interval of version timestamps. Presumes timestamps between
* {@link #INITIAL_MIN_TIMESTAMP} and {@link #INITIAL_MAX_TIMESTAMP} only.
Gets freaked out if
* passed a timestamp that is < {@link #INITIAL_MIN_TIMESTAMP},
- *
+ *
* Evaluated according to minStamp = timestamp maxStamp or
[minStamp,maxStamp) in interval
* notation.
- *
+ *
* Can be returned and read by clients. Should not be directly created by
clients. Thus, all
* constructors are purposely @InterfaceAudience.Private.
- *
+ *
* Immutable. Thread-safe.
*/
@InterfaceAudience.Public
-public class TimeRange {
+public final class TimeRange {
Review comment:
Will mention this in release note. We removed all the public
constructors so it should be final...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2616: HBASE-25235 Cleanup the deprecated methods in TimeRange
Apache9 commented on a change in pull request #2616:
URL: https://github.com/apache/hbase/pull/2616#discussion_r517070822
##
File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/Append.java
##
@@ -162,10 +162,9 @@ public Append addColumn(byte[] family, byte[] qualifier,
byte[] value) {
/**
* Add column and value to this Append operation.
- * @param cell
* @return This instance
*/
- @SuppressWarnings("unchecked")
+ @Override
Review comment:
Just some minor fix on checkstyle.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache9 commented on pull request #2584: URL: https://github.com/apache/hbase/pull/2584#issuecomment-721481888 > You've addressed all my concerns, thank you. > > I'm sorry I didn't notice this earlier, but I think it's better if this is merged into the feature branch and we land the whole feature in one pass, rather than part on the branch and part directly into master. It makes sense now why a bunch of my comments seemed to disappear; they were on the earlier PR. > > I'm +1 for this change and prefer it land via the feature branch so that we can stabilize the client + server before merging to master and before back porting to branch-2. Landing this on master is proposed by me as this PR is not related to the server side changes. It can be used in our current code base, without the changes in HBASE-18070. And what's more, the client side code are different between master and branch-2, as on master, we rebuild the sync client on top of async client which makes it much easier to implement this issue but on branch-2, you need to deal with sync client separately. So I suggest we land this on master, and then start backporting to branch-2 ASAP. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] WenFeiYi commented on a change in pull request #2623: HBASE-25240 gson format of RpcServer.logResponse is abnormal
WenFeiYi commented on a change in pull request #2623:
URL: https://github.com/apache/hbase/pull/2623#discussion_r517068885
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/GsonUtil.java
##
@@ -43,7 +43,7 @@ private GsonUtil() {
*/
public static GsonBuilder createGson() {
return new
GsonBuilder().setLongSerializationPolicy(LongSerializationPolicy.STRING)
- .registerTypeAdapter(LongAdder.class, new TypeAdapter() {
+ .disableHtmlEscaping().registerTypeAdapter(LongAdder.class, new
TypeAdapter() {
Review comment:
good idea
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25053) WAL replay should ignore 0-length files
[
https://issues.apache.org/jira/browse/HBASE-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225808#comment-17225808
]
niuyulin commented on HBASE-25053:
--
[~stack] mind help review this pr and see if this issue can be closed
> WAL replay should ignore 0-length files
> ---
>
> Key: HBASE-25053
> URL: https://issues.apache.org/jira/browse/HBASE-25053
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Affects Versions: 2.3.1
>Reporter: Nick Dimiduk
>Assignee: niuyulin
>Priority: Major
>
> I overdrove a small testing cluster, filling HDFS. After cleaning up data to
> bring HBase back up, I noticed all masters -refused to start- abort. Logs
> complain of seeking past EOF. Indeed the last wal file name logged is a
> 0-length file. WAL replay should gracefully skip and clean up such an empty
> file.
> {noformat}
> 2020-09-16 19:51:30,297 ERROR org.apache.hadoop.hbase.master.HMaster: Failed
> to become active master
> java.io.EOFException: Cannot seek after EOF
> at
> org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1448)
> at
> org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:66)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:323)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:305)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:293)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:429)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:4859)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4765)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1014)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:956)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7496)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7454)
> at
> org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:269)
> at
> org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:309)
> at
> org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:949)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2240)
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:622)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25116) RegionMonitor support RegionTask count normalize
[ https://issues.apache.org/jira/browse/HBASE-25116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225806#comment-17225806 ] niuyulin commented on HBASE-25116: -- yes, If the region num is too large , the canary slows the cluster. > RegionMonitor support RegionTask count normalize > > > Key: HBASE-25116 > URL: https://issues.apache.org/jira/browse/HBASE-25116 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > > large count of region task from canary may affect user normal request, > meanwhile if region task is few, the availability monitoring may shake for > occasional exception. > so , if the task count is large , we will randomly trim tasks for each table, > according to the raito of the table region count in whole tasks region count. > If the task count is few, we will repeat tasks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache9 commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r517058850
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java
##
@@ -196,8 +201,44 @@ private boolean tryComplete(LocateRequest req,
CompletableFuture {
+ConnectionConfiguration connConf = new
ConnectionConfiguration(conn.getConfiguration());
Review comment:
There is a `connConf` in AsyncConnectionImpl so we do not need to create
a new one again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache9 commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r517057999
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// replica id where the stale location comes from.
+private final int fromReplicaId;
+
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final int metaReplicaId, final byte[] endKey) {
+ this.fromReplicaId = metaReplicaId;
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public int getFromReplicaId() {
+ return this.fromReplicaId;
+}
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
+.append("endKey", endKey)
+.append("fromReplicaId",
[GitHub] [hbase] ndimiduk commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
ndimiduk commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r517057141
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// replica id where the stale location comes from.
+private final int fromReplicaId;
+
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final int metaReplicaId, final byte[] endKey) {
+ this.fromReplicaId = metaReplicaId;
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public int getFromReplicaId() {
+ return this.fromReplicaId;
+}
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
+.append("endKey", endKey)
+.append("fromReplicaId",
[GitHub] [hbase] ndimiduk commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
ndimiduk commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r517056790
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
Review comment:
It's true that there will possibly be many instances of this class;
considering its memory footprint is wise.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache-HBase commented on pull request #2584: URL: https://github.com/apache/hbase/pull/2584#issuecomment-721466030 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 53s | master passed | | +1 :green_heart: | checkstyle | 1m 38s | master passed | | +1 :green_heart: | spotbugs | 3m 7s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | -0 :warning: | checkstyle | 0m 27s | hbase-client: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2) | | -0 :warning: | checkstyle | 1m 11s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 50s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 46m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2584 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux e25c9afb531b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c98e993b23 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/13/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2576: HBASE-25212 [branch-1] Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2576: URL: https://github.com/apache/hbase/pull/2576#issuecomment-721457938 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 45s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 9m 56s | branch-1 passed | | +1 :green_heart: | compile | 0m 41s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 0m 45s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 2m 1s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 40s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 3m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 1s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 58s | the patch passed | | +1 :green_heart: | compile | 0m 41s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 0m 41s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | checkstyle | 1m 50s | hbase-server: The patch generated 0 new + 812 unchanged - 18 fixed = 812 total (was 830) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 2m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 39s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 30s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 40s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 2m 55s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 97m 23s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | | | | 145m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/13/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2576 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux abd762f42245 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2576/out/precommit/personality/provided.sh | | git revision | branch-1 / 16b91c8 | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/13/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/13/testReport/ | | Max. process+thread count | 4049 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/13/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r517042696
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java
##
@@ -433,9 +474,24 @@ private void locateInMeta(TableName tableName,
LocateRequest req) {
Scan scan = new Scan().withStartRow(metaStartKey).withStopRow(metaStopKey,
true)
.addFamily(HConstants.CATALOG_FAMILY).setReversed(true).setCaching(locatePrefetchLimit)
.setReadType(ReadType.PREAD);
-if (useMetaReplicas) {
- scan.setConsistency(Consistency.TIMELINE);
+
+switch (this.metaReplicaMode) {
+ case LoadBalance:
+int metaReplicaId = this.metaReplicaSelector.select(tableName,
req.row, req.locateType);
+if (metaReplicaId != RegionInfo.DEFAULT_REPLICA_ID) {
+ // If the selector gives a non-primary meta replica region, then go
with it.
+ // Otherwise, just go to primary in non-hedgedRead mode.
+ scan.setConsistency(Consistency.TIMELINE);
+ scan.setReplicaId(metaReplicaId);
+}
+break;
+ case HedgedRead:
+scan.setConsistency(Consistency.TIMELINE);
+break;
+ default:
+// do nothing
Review comment:
Not yet, will file one tomorrow morning.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r517042290
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// replica id where the stale location comes from.
+private final int fromReplicaId;
+
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final int metaReplicaId, final byte[] endKey) {
+ this.fromReplicaId = metaReplicaId;
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public int getFromReplicaId() {
+ return this.fromReplicaId;
+}
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
+.append("endKey", endKey)
+
[GitHub] [hbase] saintstack commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
saintstack commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r517036330
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java
##
@@ -433,9 +474,24 @@ private void locateInMeta(TableName tableName,
LocateRequest req) {
Scan scan = new Scan().withStartRow(metaStartKey).withStopRow(metaStopKey,
true)
.addFamily(HConstants.CATALOG_FAMILY).setReversed(true).setCaching(locatePrefetchLimit)
.setReadType(ReadType.PREAD);
-if (useMetaReplicas) {
- scan.setConsistency(Consistency.TIMELINE);
+
+switch (this.metaReplicaMode) {
+ case LoadBalance:
+int metaReplicaId = this.metaReplicaSelector.select(tableName,
req.row, req.locateType);
+if (metaReplicaId != RegionInfo.DEFAULT_REPLICA_ID) {
+ // If the selector gives a non-primary meta replica region, then go
with it.
+ // Otherwise, just go to primary in non-hedgedRead mode.
+ scan.setConsistency(Consistency.TIMELINE);
+ scan.setReplicaId(metaReplicaId);
+}
+break;
+ case HedgedRead:
+scan.setConsistency(Consistency.TIMELINE);
+break;
+ default:
+// do nothing
Review comment:
Did you file one @huaxiangsun ?
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaMode.java
##
@@ -0,0 +1,64 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * There are two modes with catalog replica support.
+ *
+ *
+ * HEDGED_READ - Client sends requests to the primary region first,
within a
+ * configured amount of time, if there is no response coming
back,
+ * client sends requests to all replica regions and takes the
first
+ * response.
+ *
+ * LOAD_BALANCE - Client sends requests to replica regions in a
round-robin mode,
+ * if results from replica regions are stale, next time,
client sends requests for
+ * these stale locations to the primary region. In this mode,
scan
+ * requests are load balanced across all replica regions.
+ *
+ */
+@InterfaceAudience.Private
+enum CatalogReplicaMode {
+ NONE {
+@Override
+public String toString() {
+ return "None";
+}
+ },
+ HEDGED_READ {
+@Override
+public String toString() {
+ return "HedgedRead";
+}
+ },
+ LOAD_BALANCE {
+@Override
+public String toString() {
+ return "LoadBalance";
+}
+ };
+
+ public static CatalogReplicaMode fromString(final String value) {
+for(CatalogReplicaMode mode : values()) {
+ if (mode.toString().equalsIgnoreCase(value)) {
Review comment:
Good that you ignore case.
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import
[GitHub] [hbase] Apache-HBase commented on pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache-HBase commented on pull request #2584: URL: https://github.com/apache/hbase/pull/2584#issuecomment-721453161 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 16s | master passed | | +1 :green_heart: | compile | 1m 27s | master passed | | +1 :green_heart: | shadedjars | 7m 27s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 5s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 7m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 15s | hbase-client in the patch passed. | | -1 :x: | unit | 206m 31s | hbase-server in the patch failed. | | | | 239m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2584 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f7a4797375bb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1eceab69b5 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/testReport/ | | Max. process+thread count | 3564 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25241) Add integration test for meta replica load balance mode
Huaxiang Sun created HBASE-25241: Summary: Add integration test for meta replica load balance mode Key: HBASE-25241 URL: https://issues.apache.org/jira/browse/HBASE-25241 Project: HBase Issue Type: Sub-task Components: integration tests Reporter: Huaxiang Sun We need to create an integration test which has meta replica load balance mode enabled and make sure its correctness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #251: HBASE-22114 Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
Apache-HBase commented on pull request #251: URL: https://github.com/apache/hbase/pull/251#issuecomment-721445328 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 45s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | -1 :x: | hbaseanti | 0m 0s | The patch appears use Hadoop classification instead of HBase. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 49s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 3s | branch-1 passed | | +1 :green_heart: | compile | 1m 41s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 1m 44s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 7m 56s | branch-1 passed | | +0 :ok: | refguide | 4m 5s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 2m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 18s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 5m 12s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 2m 43s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +0 :ok: | findbugs | 0m 25s | branch/hbase-resource-bundle no findbugs output file (findbugsXml.xml) | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 57s | the patch passed | | -1 :x: | compile | 0m 10s | root in the patch failed with JDK Azul Systems, Inc.-1.8.0_262-b19. | | -1 :x: | javac | 0m 10s | root in the patch failed with JDK Azul Systems, Inc.-1.8.0_262-b19. | | +1 :green_heart: | compile | 1m 43s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 1m 43s | the patch passed | | +1 :green_heart: | checkstyle | 5m 1s | root: The patch generated 0 new + 87 unchanged - 11 fixed = 87 total (was 98) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | xml | 0m 1s | The patch has 5 ill-formed XML file(s). | | +0 :ok: | refguide | 3m 1s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 2m 51s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 38s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | -1 :x: | javadoc | 0m 10s | root in the patch failed with JDK Azul Systems, Inc.-1.8.0_262-b19. | | +1 :green_heart: | javadoc | 3m 43s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | findbugs | 0m 13s | hbase-resource-bundle has no data from findbugs | | -1 :x: | findbugs | 0m 9s | hbase-tinylfu-blockcache in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 147m 0s | root in the patch failed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 250m 24s | | | Reason | Tests | |---:|:--| | XML | Parsing Error(s): | | | hbase-common/src/main/resources/hbase-default.xml | | | hbase-it/pom.xml | | | hbase-resource-bundle/src/main/resources/supplemental-models.xml | | | hbase-tinylfu-blockcache/pom.xml | | | pom.xml | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/251 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide xml | | uname | Linux 3784a751a851 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-agent/workspace/HBase-PreCommit-GitHub-PR_PR-251/out/precommit/personality/provided.sh | | git revision | branch-1 / 16b91c8 | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | |
[GitHub] [hbase] huaxiangsun commented on pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on pull request #2584: URL: https://github.com/apache/hbase/pull/2584#issuecomment-721441167 Upload a new one to address Nick's comments which did not show up at the front page (embedded in the patch). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2574: URL: https://github.com/apache/hbase/pull/2574#issuecomment-721440531 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 0m 57s | master passed | | +1 :green_heart: | shadedjars | 7m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 7m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 206m 8s | hbase-server in the patch failed. | | | | 234m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/11/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2574 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux eab30f3bdf33 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1eceab69b5 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/11/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/11/testReport/ | | Max. process+thread count | 3125 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/11/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2575: HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2575: URL: https://github.com/apache/hbase/pull/2575#issuecomment-721437620 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 31s | branch-2 passed | | +1 :green_heart: | compile | 1m 10s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 25s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed | | +1 :green_heart: | javac | 1m 11s | the patch passed | | +1 :green_heart: | shadedjars | 7m 22s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 194m 7s | hbase-server in the patch passed. | | | | 224m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/12/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2575 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b0f82f165da8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 729518af9d | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/12/testReport/ | | Max. process+thread count | 2604 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/12/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r517023873
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
Review comment:
Will keep it as it is due to scaling concern.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r516974213
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
Review comment:
Sorry, missed this comment as some of them showing up in the main thread
and some of them only shows up in the patch itself. Concern here is that while
Instant provides more user friendly features, object itself takes more
spaces/memory than a long. Scaling concern?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r517022727
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final byte[] endKey) {
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
+.append("endKey", endKey)
+.append("timestamp", timestamp)
+.toString();
+}
+ }
+
+ private final ConcurrentMap>
+staleCache = new ConcurrentHashMap<>();
+ private volatile int numOfReplicas;
+ private final AsyncConnectionImpl conn;
+ private final TableName tableName;
[GitHub] [hbase] Apache-HBase commented on pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache-HBase commented on pull request #2584: URL: https://github.com/apache/hbase/pull/2584#issuecomment-721435677 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 7m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 1m 33s | master passed | | +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 1m 33s | the patch passed | | +1 :green_heart: | javac | 1m 33s | the patch passed | | +1 :green_heart: | shadedjars | 6m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 6s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 138m 48s | hbase-server in the patch passed. | | | | 176m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2584 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2f630c58d008 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1eceab69b5 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/testReport/ | | Max. process+thread count | 4095 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r517020877
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final byte[] endKey) {
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
Review comment:
Thanks! Just learnt something new about the style.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
[jira] [Updated] (HBASE-25212) Optionally abort requests in progress after deciding a region should close
[
https://issues.apache.org/jira/browse/HBASE-25212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25212:
Hadoop Flags: Reviewed
Release Note:
If hbase.regionserver.close.wait.abort is set to true, interrupt RPC handler
threads holding the region close lock.
Until requests in progress can be aborted, wait on the region close lock for a
configurable interval (specified by hbase.regionserver.close.wait.time.ms,
default 6 (1 minute)). If we have failed to acquire the close lock after
this interval elapses, if allowed (also specified by
hbase.regionserver.close.wait.abort), abort the regionserver.
We will attempt to interrupt any running handlers every
hbase.regionserver.close.wait.interval.ms (default 1 (10 seconds)) until
either the close lock is acquired or we reach the maximum wait time.
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Optionally abort requests in progress after deciding a region should close
> --
>
> Key: HBASE-25212
> URL: https://issues.apache.org/jira/browse/HBASE-25212
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
>
> After deciding a region should be closed, the regionserver will set the
> internal region state to closing and wait for all pending requests to
> complete, via a rendezvous on the region lock. In closing state the region
> will not accept any new requests but requests in progress will be allowed to
> complete before the close action takes place. In our production we see
> outlier wait times on this lock in excess of several minutes.
> During close when there are requests in flight the regionserver is subject to
> any conceivable reason for delay, like full scans over large regions,
> expensive filtering hierarchies, bugs, or store level performance problems
> like slow HDFS. The regionserver should interrupt requests in progress to
> facilitate smaller/shorter close times on an opt-in basis.
> Optionally, via configuration parameter -- which would be a system wide
> default set in hbase-site.xml in common practice but could be overridden in
> table schema for per table settings -- interrupt requests in progress holding
> the region lock rather than wait for completion of all operations in flight.
> Send back NotServingRegionException("region is closing") to the clients of
> the interrupted operations, like we do after the write lock is acquired. The
> client will transparently relocate the region data and resubmit the aborted
> requests per normal retry policy. This can be less disruptive than waiting
> for very long times for a region to close in extreme outlier cases (e.g. 50
> minutes). In such extreme cases it is better to abort the regionserver if the
> close lock cannot be acquired in a reasonable amount of time, because the
> region cannot be made available again until it has closed.
> After waiting for all requests to complete then we flush the region's
> memstore and finish the close. The flush portion of the close process is out
> of scope of this proposal. Under normal conditions the flush portion of the
> close completes quickly. It is specifically waits on the close lock that has
> been an occasional issue in our production that causes difficulty achieving
> 99.99% availability.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] apurtell merged pull request #2576: HBASE-25212 [branch-1] Optionally abort requests in progress after deciding a region should close
apurtell merged pull request #2576: URL: https://github.com/apache/hbase/pull/2576 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell merged pull request #2575: HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close
apurtell merged pull request #2575: URL: https://github.com/apache/hbase/pull/2575 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell merged pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
apurtell merged pull request #2574: URL: https://github.com/apache/hbase/pull/2574 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2575: HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2575: URL: https://github.com/apache/hbase/pull/2575#issuecomment-721425126 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | branch-2 passed | | +1 :green_heart: | compile | 0m 57s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 22s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 5m 52s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 158m 54s | hbase-server in the patch passed. | | | | 183m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/12/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2575 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1bbc2a3dbea6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 729518af9d | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/12/testReport/ | | Max. process+thread count | 3398 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/12/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2574: URL: https://github.com/apache/hbase/pull/2574#issuecomment-721423952 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 19s | master passed | | +1 :green_heart: | compile | 1m 7s | master passed | | +1 :green_heart: | shadedjars | 6m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 7m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 150m 38s | hbase-server in the patch passed. | | | | 179m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/11/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2574 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 55e2be5e7a76 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1eceab69b5 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/11/testReport/ | | Max. process+thread count | 3901 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/11/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2576: HBASE-25212 [branch-1] Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2576: URL: https://github.com/apache/hbase/pull/2576#issuecomment-721414618 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 10m 6s | branch-1 passed | | +1 :green_heart: | compile | 0m 42s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 0m 49s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 2m 11s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 43s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 3m 17s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 13s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 5s | the patch passed | | +1 :green_heart: | compile | 0m 43s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 0m 43s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 0m 48s | the patch passed | | -1 :x: | checkstyle | 1m 59s | hbase-server: The patch generated 1 new + 812 unchanged - 18 fixed = 813 total (was 830) | | -1 :x: | whitespace | 0m 0s | The patch 2 line(s) with tabs. | | +1 :green_heart: | shadedjars | 3m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 57s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 43s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 3m 7s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 107m 36s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 152m 55s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.security.visibility.TestVisibilityLabelsWithACL | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/12/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2576 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux ced0fb91338f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2576/out/precommit/personality/provided.sh | | git revision | branch-1 / 16b91c8 | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/12/artifact/out/diff-checkstyle-hbase-server.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/12/artifact/out/whitespace-tabs.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/12/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/12/testReport/ | | Max. process+thread count | 3029 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2576/12/console | | versions | git=1.9.1
[GitHub] [hbase] ndimiduk commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
ndimiduk commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r516988960
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final byte[] endKey) {
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
Review comment:
Here's an example,
https://github.com/apache/hbase/blob/1eceab6/hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationTarget.java#L75-L80
This is an automated message from the Apache Git
[jira] [Commented] (HBASE-25228) Delete dev-support/jenkins_precommit_jira_yetus.sh
[
https://issues.apache.org/jira/browse/HBASE-25228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225712#comment-17225712
]
Hudson commented on HBASE-25228:
Results for branch branch-1.4
[build #18 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/18/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/18//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/18//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/18//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Delete dev-support/jenkins_precommit_jira_yetus.sh
> --
>
> Key: HBASE-25228
> URL: https://issues.apache.org/jira/browse/HBASE-25228
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.7, 2.3.4
>
>
> We no longer accept contributions via patches attached to Jira. As such, this
> script in dev-support can be deleted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25090) CompactionConfiguration logs unrealistic store file sizes
[
https://issues.apache.org/jira/browse/HBASE-25090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225713#comment-17225713
]
Hudson commented on HBASE-25090:
Results for branch branch-1.4
[build #18 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/18/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/18//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/18//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/18//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> CompactionConfiguration logs unrealistic store file sizes
> -
>
> Key: HBASE-25090
> URL: https://issues.apache.org/jira/browse/HBASE-25090
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 2.3.2
>Reporter: Nick Dimiduk
>Assignee: Zhuoyue Huang
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.7, 2.3.4
>
>
> I noticed this in the debug logging of a unit test that makes use of a
> MiniCluster. I'm pretty sure {{8.00 EB}} -- exabytes?? -- is not an accurate
> reporting of the size of these HFiles.
> {noformat}
> 2020-09-22 16:20:57,996 INFO
> [StoreOpener-5e0b426a952230abb65623fc54869833-1]
> compactions.CompactionConfiguration(177): size [128 MB, 8.00 EB, 8.00 EB);
> files [3, 10); ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r516982360
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final byte[] endKey) {
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
+.append("endKey", endKey)
+.append("timestamp", timestamp)
+.toString();
+}
+ }
+
+ private final ConcurrentMap>
+staleCache = new ConcurrentHashMap<>();
+ private volatile int numOfReplicas;
+ private final AsyncConnectionImpl conn;
+ private final TableName tableName;
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r516981858
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final byte[] endKey) {
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
+.append("endKey", endKey)
+.append("timestamp", timestamp)
+.toString();
+}
+ }
+
+ private final ConcurrentMap>
+staleCache = new ConcurrentHashMap<>();
+ private volatile int numOfReplicas;
+ private final AsyncConnectionImpl conn;
+ private final TableName tableName;
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r516979833
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final byte[] endKey) {
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
+.append("endKey", endKey)
+.append("timestamp", timestamp)
+.toString();
+}
+ }
+
+ private final ConcurrentMap>
+staleCache = new ConcurrentHashMap<>();
+ private volatile int numOfReplicas;
+ private final AsyncConnectionImpl conn;
+ private final TableName tableName;
[jira] [Commented] (HBASE-25228) Delete dev-support/jenkins_precommit_jira_yetus.sh
[
https://issues.apache.org/jira/browse/HBASE-25228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225706#comment-17225706
]
Hudson commented on HBASE-25228:
Results for branch branch-2.2
[build #113 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/113/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/113//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/113//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/113//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/113//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Delete dev-support/jenkins_precommit_jira_yetus.sh
> --
>
> Key: HBASE-25228
> URL: https://issues.apache.org/jira/browse/HBASE-25228
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.7, 2.3.4
>
>
> We no longer accept contributions via patches attached to Jira. As such, this
> script in dev-support can be deleted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24859) Optimize in-memory representation of mapreduce TableSplit objects
[
https://issues.apache.org/jira/browse/HBASE-24859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225707#comment-17225707
]
Hudson commented on HBASE-24859:
Results for branch branch-2.2
[build #113 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/113/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/113//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/113//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/113//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/113//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Optimize in-memory representation of mapreduce TableSplit objects
> -
>
> Key: HBASE-24859
> URL: https://issues.apache.org/jira/browse/HBASE-24859
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Affects Versions: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 2.2.7
>Reporter: Sandeep Pal
>Assignee: Sandeep Pal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 2.2.7, 2.3.4
>
> Attachments: Screen Shot 2020-08-26 at 8.44.34 AM.png, hbase-24859.png
>
>
> It has been observed that when the table has too many regions, MR jobs
> consume a lot of memory in the client. This is because we keep the region
> level information in memory and the memory heavy object is TableSplit because
> of the Scan object as a part of it.
> However, it looks like the TableInputFormat for single table doesn't need to
> store the scan object in the TableSplit because we do not use it and all the
> splits are expected to have the exact same scan object. In TableInputFormat
> we use the scan object directly from the MR conf.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r516977060
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final byte[] endKey) {
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
Review comment:
It may be missed due to the reason I mentioned above.
Sorry, I did not find any instance referring "ShortPrefixToStringStyle" in
hbase codebase. Can you provide more detail/example?
This is an automated message
[GitHub] [hbase] Apache-HBase commented on pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache-HBase commented on pull request #2584: URL: https://github.com/apache/hbase/pull/2584#issuecomment-721391478 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 45s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 0s | master passed | | +1 :green_heart: | checkstyle | 1m 41s | master passed | | +1 :green_heart: | spotbugs | 3m 19s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | -0 :warning: | checkstyle | 0m 32s | hbase-client: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2) | | -0 :warning: | checkstyle | 1m 15s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 11s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 4m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 50m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2584 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux ed937601a6f1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1eceab69b5 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 85 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2584/12/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r516974213
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,293 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// timestamp in milliseconds
+private final long timestamp;
Review comment:
Sorry, missed this comment as some of them showing up in the main thread
and some of them only shows up in the patch itself. Concern here is that
Instant provides more user friendly features, object itself takes more
spaces/memory than a long. Scaling concern?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r516969889
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java
##
@@ -444,6 +503,8 @@ private void locateInMeta(TableName tableName,
LocateRequest req) {
@Override
public void onError(Throwable error) {
+// TODO: if it fails, with meta replica load balance, it may try with
another meta
Review comment:
missed that. Let me delete it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on pull request #2584: URL: https://github.com/apache/hbase/pull/2584#issuecomment-721374482 @Apache9 @saintstack @ndimiduk Latest patch uploaded which addresses Duo's latest comments, please help to review, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2574: URL: https://github.com/apache/hbase/pull/2574#issuecomment-721373351 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +1 :green_heart: | spotbugs | 2m 12s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | the patch passed | | +1 :green_heart: | checkstyle | 1m 24s | hbase-server: The patch generated 0 new + 409 unchanged - 2 fixed = 409 total (was 411) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 32s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 47m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/11/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2574 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 2ed8d49a281b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 1eceab69b5 | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2574/11/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584: URL: https://github.com/apache/hbase/pull/2584#discussion_r516953887 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java ## @@ -82,14 +84,18 @@ static String LOCATE_PREFETCH_LIMIT = "hbase.client.locate.prefetch.limit"; private static final int DEFAULT_LOCATE_PREFETCH_LIMIT = 10; + private static final long GET_META_LOCATIONS_TIMEOUT_IN_MILLISECONDS = 2000; // 2 seconds Review comment: A new patch takes "hbase.rpc.read.timeout". If there is a better config, please provide more details. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584:
URL: https://github.com/apache/hbase/pull/2584#discussion_r516953355
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
##
@@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.IntSupplier;
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector implements a simple
catalog replica load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up location,
+ * it first check if the row is the stale location cache. If yes, the location
from
+ * catalog replica is stale, it will go to the primary region to look up
update-to-date location;
+ * otherwise, it will randomly pick up a replica region for lookup. When
clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a
chore.
+ *
+ * It follows a simple algorithm to choose a replica to go:
+ *
+ *
+ * If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a replica region to do lookup.
+ * If the location from the replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * CatalogReplicaLoadBalanceReplicaSimpleSelector.
+ * When client tries to do location lookup, it checks StaleLocationCache
first for rows it
+ * tries to lookup, if entry exists, it will go with primary meta region
to do lookup;
+ * otherwise, it will follow step 1.
+ * A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ *
+ */
+class CatalogReplicaLoadBalanceSimpleSelector implements
+ CatalogReplicaLoadBalanceSelector, Stoppable {
+ private static final Logger LOG =
+LoggerFactory.getLogger(CatalogReplicaLoadBalanceSimpleSelector.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL_IN_MILLISECONDS = 1500;
// 1.5 seconds
+ private final int REFRESH_REPLICA_COUNT_CHORE_INTERVAL_IN_MILLISECONDS =
6; // 1 minute
+
+ /**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+ private static final class StaleLocationCacheEntry {
+// replica id where the stale location comes from.
+private final int fromReplicaId;
+
+// timestamp in milliseconds
+private final long timestamp;
+
+private final byte[] endKey;
+
+StaleLocationCacheEntry(final int metaReplicaId, final byte[] endKey) {
+ this.fromReplicaId = metaReplicaId;
+ this.endKey = endKey;
+ timestamp = EnvironmentEdgeManager.currentTime();
+}
+
+public byte[] getEndKey() {
+ return this.endKey;
+}
+
+public int getFromReplicaId() {
+ return this.fromReplicaId;
+}
+public long getTimestamp() {
+ return this.timestamp;
+}
+
+@Override
+public String toString() {
+ return new ToStringBuilder(this)
+.append("endKey", endKey)
+
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584: URL: https://github.com/apache/hbase/pull/2584#discussion_r516951109 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java ## @@ -82,14 +84,18 @@ static String LOCATE_PREFETCH_LIMIT = "hbase.client.locate.prefetch.limit"; private static final int DEFAULT_LOCATE_PREFETCH_LIMIT = 10; + private static final long GET_META_LOCATIONS_TIMEOUT = 2000; // 2 seconds Review comment: Moving it to "hbase.rpc.read.timeout". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2575: HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close
Apache-HBase commented on pull request #2575: URL: https://github.com/apache/hbase/pull/2575#issuecomment-721369282 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 22s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 9s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 16s | the patch passed | | -0 :warning: | checkstyle | 1m 18s | hbase-server: The patch generated 1 new + 479 unchanged - 2 fixed = 480 total (was 481) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 16s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 38m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/12/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2575 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux ac6fc1e9f2d9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 729518af9d | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/12/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2575/12/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack closed pull request #2547: HBASE-25185 Remove the usage of MetaTableAccessor in hbase-balancer
saintstack closed pull request #2547: URL: https://github.com/apache/hbase/pull/2547 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #2576: HBASE-25212 [branch-1] Optionally abort requests in progress after deciding a region should close
apurtell commented on pull request #2576: URL: https://github.com/apache/hbase/pull/2576#issuecomment-721353246 Last commit on PR branch addresses remaining feedback, going to commit shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #2575: HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close
apurtell commented on pull request #2575: URL: https://github.com/apache/hbase/pull/2575#issuecomment-721353189 Last commit on PR branch addresses remaining feedback, going to commit shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell edited a comment on pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
apurtell edited a comment on pull request #2574: URL: https://github.com/apache/hbase/pull/2574#issuecomment-721352963 Last commit on PR branch addresses remaining feedback, going to commit shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #2574: HBASE-25212 Optionally abort requests in progress after deciding a region should close
apurtell commented on pull request #2574: URL: https://github.com/apache/hbase/pull/2574#issuecomment-721352963 Last patch addresses remaining feedback, going to commit shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-25190) Create an hbase-tool module and move here standalone tools bundled with core
[ https://issues.apache.org/jira/browse/HBASE-25190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack reassigned HBASE-25190: - Assignee: Michael Stack (was: Duo Zhang) Took back my issue and resolved as won't fix for now. Will work on later. > Create an hbase-tool module and move here standalone tools bundled with core > > > Key: HBASE-25190 > URL: https://issues.apache.org/jira/browse/HBASE-25190 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > > HBASE-23933 is about moving balancer classes to a new hbase-balancer module > as an effort at shrinking the size of the hbase-server module -- it is too > big. > This issue is about creating a hbase-tool module into which we'd move > standalone tools that ship with hbase core such as the bulkloadtool and the > canary and all related supporting and test classes. > I do not see this making much of a dent in the size of hbase-server. This > issue is offered as a small improvement. > (Aside, I'd move some favored node 'tool' classes that are currently in limbo > here at least until we rule on whether to finish or remove if this module was > put in place -- see HBASE-25185). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25190) Create an hbase-tool module and move here standalone tools bundled with core
[ https://issues.apache.org/jira/browse/HBASE-25190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25190. --- Resolution: Won't Fix > Create an hbase-tool module and move here standalone tools bundled with core > > > Key: HBASE-25190 > URL: https://issues.apache.org/jira/browse/HBASE-25190 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Michael Stack >Assignee: Duo Zhang >Priority: Major > > HBASE-23933 is about moving balancer classes to a new hbase-balancer module > as an effort at shrinking the size of the hbase-server module -- it is too > big. > This issue is about creating a hbase-tool module into which we'd move > standalone tools that ship with hbase core such as the bulkloadtool and the > canary and all related supporting and test classes. > I do not see this making much of a dent in the size of hbase-server. This > issue is offered as a small improvement. > (Aside, I'd move some favored node 'tool' classes that are currently in limbo > here at least until we rule on whether to finish or remove if this module was > put in place -- see HBASE-25185). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack closed pull request #2552: HBASE-25190 Create an hbase-tool module and move here standalone tool…
saintstack closed pull request #2552: URL: https://github.com/apache/hbase/pull/2552 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25116) RegionMonitor support RegionTask count normalize
[ https://issues.apache.org/jira/browse/HBASE-25116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225631#comment-17225631 ] Michael Stack commented on HBASE-25116: --- So, running the canary slows the cluster so we add a bound per table on how many regions the canary checks? > RegionMonitor support RegionTask count normalize > > > Key: HBASE-25116 > URL: https://issues.apache.org/jira/browse/HBASE-25116 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > > large count of region task from canary may affect user normal request, > meanwhile if region task is few, the availability monitoring may shake for > occasional exception. > so , if the task count is large , we will randomly trim tasks for each table, > according to the raito of the table region count in whole tasks region count. > If the task count is few, we will repeat tasks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24960) reduce invalid subprocedure task
[ https://issues.apache.org/jira/browse/HBASE-24960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225629#comment-17225629 ] Michael Stack commented on HBASE-24960: --- [~Bo Cui] Have you seen the feedback on [https://github.com/apache/hbase/pull/2511] ? Thank you. > reduce invalid subprocedure task > > > Key: HBASE-24960 > URL: https://issues.apache.org/jira/browse/HBASE-24960 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0-alpha-1, 2.2.3 >Reporter: Bo Cui >Assignee: Bo Cui >Priority: Major > > [https://github.com/apache/hbase/blob/047e0618d290a09a4a269b00548fe17691e31787/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/RegionServerSnapshotManager.java#L165] > [https://github.com/apache/hbase/blob/047e0618d290a09a4a269b00548fe17691e31787/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/flush/RegionServerFlushTableProcedureManager.java#L146] > > if involvedRegions is null or empty, rs should skip subprocedure. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225620#comment-17225620
]
Nick Dimiduk commented on HBASE-24749:
--
bq. I have a question on branching, since this feature will be default disable,
is it possible that we direct contribute into master branch (instead of having
a feature branch)? mainly, we're learning how and when one feature branch can
merge into master branch.
I am of the opinion that feature development should always happen on a feature
branch. Never merge to a releasable branch until you have something you're
ready to release. We consider {{master}} a releasable branch.
I do recommend that feature branches be frequently rebased onto their source
branch. In my experience, once/week has been a good cadence.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2602: HBASE-25229 Instantiate BucketCache before RSs create their ephemeral nodes
Apache-HBase commented on pull request #2602: URL: https://github.com/apache/hbase/pull/2602#issuecomment-721302083 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 11m 47s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 9m 59s | branch-1 passed | | +1 :green_heart: | compile | 0m 42s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | compile | 0m 47s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | checkstyle | 1m 59s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | branch-1 passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 42s | branch-1 passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +0 :ok: | spotbugs | 3m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 10s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 5s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | javac | 0m 46s | the patch passed | | -1 :x: | checkstyle | 1m 46s | hbase-server: The patch generated 1 new + 65 unchanged - 0 fixed = 66 total (was 65) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 3m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 59s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed with JDK Azul Systems, Inc.-1.8.0_262-b19 | | +1 :green_heart: | javadoc | 0m 43s | the patch passed with JDK Azul Systems, Inc.-1.7.0_272-b10 | | +1 :green_heart: | findbugs | 3m 7s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 120m 12s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 175m 37s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.security.visibility.TestVisibilityLabelsWithACL | | | hadoop.hbase.master.TestVersionUpgrade | | | hadoop.hbase.regionserver.TestSplitTransactionOnCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2602/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2602 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 1bb229e0652b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2602/out/precommit/personality/provided.sh | | git revision | branch-1 / 16b91c8 | | Default Java | Azul Systems, Inc.-1.7.0_272-b10 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:Azul Systems, Inc.-1.8.0_262-b19 /usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_272-b10 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2602/2/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2602/2/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2602/2/testReport/ | | Max. process+thread count | 3547 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2602/2/console | | versions | git=1.9.1 maven=3.0.5
[jira] [Commented] (HBASE-25228) Delete dev-support/jenkins_precommit_jira_yetus.sh
[
https://issues.apache.org/jira/browse/HBASE-25228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225598#comment-17225598
]
Hudson commented on HBASE-25228:
Results for branch branch-2
[build #92 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/92/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/92/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/92/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/92/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/92/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Delete dev-support/jenkins_precommit_jira_yetus.sh
> --
>
> Key: HBASE-25228
> URL: https://issues.apache.org/jira/browse/HBASE-25228
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.7, 2.3.4
>
>
> We no longer accept contributions via patches attached to Jira. As such, this
> script in dev-support can be deleted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225590#comment-17225590
]
Tak-Lon (Stephen) Wu commented on HBASE-24749:
--
Hi guys, we're about to sending out milestones and PRs under this parent JIRA.
I have a question on branching, since this feature will be default disable, is
it possible that we direct contribute into master branch (instead of having a
feature branch)? mainly, we're learning how and when one feature branch can
merge into master branch.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] HorizonNet commented on a change in pull request #2616: HBASE-25235 Cleanup the deprecated methods in TimeRange
HorizonNet commented on a change in pull request #2616:
URL: https://github.com/apache/hbase/pull/2616#discussion_r516851115
##
File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/Append.java
##
@@ -162,10 +162,9 @@ public Append addColumn(byte[] family, byte[] qualifier,
byte[] value) {
/**
* Add column and value to this Append operation.
- * @param cell
* @return This instance
*/
- @SuppressWarnings("unchecked")
+ @Override
Review comment:
Seems to be unrelated to the actual change.
##
File path: hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java
##
@@ -18,24 +18,23 @@
package org.apache.hadoop.hbase.io;
-import org.apache.hadoop.hbase.util.Bytes;
import org.apache.yetus.audience.InterfaceAudience;
/**
* Represents an interval of version timestamps. Presumes timestamps between
* {@link #INITIAL_MIN_TIMESTAMP} and {@link #INITIAL_MAX_TIMESTAMP} only.
Gets freaked out if
* passed a timestamp that is < {@link #INITIAL_MIN_TIMESTAMP},
- *
+ *
* Evaluated according to minStamp = timestamp maxStamp or
[minStamp,maxStamp) in interval
* notation.
- *
+ *
* Can be returned and read by clients. Should not be directly created by
clients. Thus, all
* constructors are purposely @InterfaceAudience.Private.
- *
+ *
* Immutable. Thread-safe.
*/
@InterfaceAudience.Public
-public class TimeRange {
+public final class TimeRange {
Review comment:
Probably this needs to be called out separately in the release notes as
it is not binary-compatible. Shouldn't matter as it is introduced in major
release, but probably worth it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2584: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2584: URL: https://github.com/apache/hbase/pull/2584#discussion_r516851437 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java ## @@ -82,14 +84,18 @@ static String LOCATE_PREFETCH_LIMIT = "hbase.client.locate.prefetch.limit"; private static final int DEFAULT_LOCATE_PREFETCH_LIMIT = 10; + private static final long GET_META_LOCATIONS_TIMEOUT_IN_MILLISECONDS = 2000; // 2 seconds Review comment: I will go with this configuration and upload a new patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25210) RegionInfo.isOffline is now a duplication with RegionInfo.isSplit
[ https://issues.apache.org/jira/browse/HBASE-25210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25210. --- Fix Version/s: 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed Merged to master branch. [~niuyulin] brought up this change on the dev mailing list at [http://apache-hbase.679495.n3.nabble.com/DISCUSS-Deprecate-isOffline-and-isSplitParent-in-RegionInfo-td4108550.html] No objection noted. Thanks for the PR [~niuyulin] > RegionInfo.isOffline is now a duplication with RegionInfo.isSplit > - > > Key: HBASE-25210 > URL: https://issues.apache.org/jira/browse/HBASE-25210 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > The only place, where we set it to true is in splitRegion, and at the same > time we will set split to true. > So in general, I suggest that we deprecated isOffline and isSplitParent in > RegionInfo, only leave the isSplit method. And in RegionInfoBuilder, we > deprecated setOffline and only leave the setSplit method. > This could make our code base cleaner. > And for serialization compatibility, we'd better still keep the split and > offline fields in the actual RegionInfo datastructure for a while. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #2580: HBASE-25210 RegionInfo.isOffline is now a duplication with RegionInfo.isSplit
saintstack merged pull request #2580: URL: https://github.com/apache/hbase/pull/2580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25181) Configure hash algorithm in wrapped encryption keys
[ https://issues.apache.org/jira/browse/HBASE-25181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HBASE-25181: - Description: Currently we are using MD5 hash algorithm to store a hash for encryption keys. This hash is needed to verify the secret key of the subject. (e.g. making sure that the same secret key is used during encrypted HFile / WalFile read and write). The MD5 algorithm is considered weak, and can not be used in some (e.g. FIPS compliant) clusters. However, currently it is not possible to use different hash algorithm, or to disable the whole column family encryption globally on the cluster. In this patch: * I introduce a backward compatible way of specifying the hash algorithm. This enable us to use newer and/or more secure hash algorithms like SHA-384 or SHA-512 (which are FIPS compliant). * I added a configuration parameter to globally enable / disable the column family encryption feature. (enabled by default for backward compatibility). This is handy if someone wants to operate an HBase cluster making sure that uses are only relying on other (e.g. HDFS based) encryption mechanisms. was: Currently we are using MD5 hash algorithm to store a hash for encryption keys. This hash is needed to verify the secret key of the subject. (e.g. making sure that the same secrey key is used during encrypted HFile read and write). The MD5 algorithm is considered weak, and can not be used in some (e.g. FIPS compliant) clusters. In the patch I plan to: * introduce a backward compatible way of specifying the hash algorithm. This enable us to use newer and more secure hash algorithms like SHA-384 or SHA-512 (which are FIPS compliant). * change the algoritm used by the hbase shell to generate secure keys for column family encryption (this is only used for testing schema in the shell, the proper data keys are generated by the Java API, see e.g. HBASE-10951) > Configure hash algorithm in wrapped encryption keys > --- > > Key: HBASE-25181 > URL: https://issues.apache.org/jira/browse/HBASE-25181 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.3.2 >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > > Currently we are using MD5 hash algorithm to store a hash for encryption > keys. This hash is needed to verify the secret key of the subject. (e.g. > making sure that the same secret key is used during encrypted HFile / WalFile > read and write). The MD5 algorithm is considered weak, and can not be used in > some (e.g. FIPS compliant) clusters. However, currently it is not possible to > use different hash algorithm, or to disable the whole column family > encryption globally on the cluster. > In this patch: > * I introduce a backward compatible way of specifying the hash algorithm. > This enable us to use newer and/or more secure hash algorithms like SHA-384 > or SHA-512 (which are FIPS compliant). > * I added a configuration parameter to globally enable / disable the column > family encryption feature. (enabled by default for backward compatibility). > This is handy if someone wants to operate an HBase cluster making sure that > uses are only relying on other (e.g. HDFS based) encryption mechanisms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24200) Upgrade to Yetus 0.12.0
[
https://issues.apache.org/jira/browse/HBASE-24200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225558#comment-17225558
]
Hudson commented on HBASE-24200:
Results for branch branch-2.3
[build #98 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/98/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/98/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/98/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/98/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/98/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Upgrade to Yetus 0.12.0
> ---
>
> Key: HBASE-24200
> URL: https://issues.apache.org/jira/browse/HBASE-24200
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.7, 2.3.4
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.7, 2.3.4
>
>
> A new Yetus release is imminent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25228) Delete dev-support/jenkins_precommit_jira_yetus.sh
[
https://issues.apache.org/jira/browse/HBASE-25228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225559#comment-17225559
]
Hudson commented on HBASE-25228:
Results for branch branch-2.3
[build #98 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/98/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/98/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/98/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/98/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/98/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Delete dev-support/jenkins_precommit_jira_yetus.sh
> --
>
> Key: HBASE-25228
> URL: https://issues.apache.org/jira/browse/HBASE-25228
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.7, 2.3.4
>
>
> We no longer accept contributions via patches attached to Jira. As such, this
> script in dev-support can be deleted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24859) Optimize in-memory representation of mapreduce TableSplit objects
[ https://issues.apache.org/jira/browse/HBASE-24859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada resolved HBASE-24859. -- Fix Version/s: 2.3.4 2.2.7 2.4.0 3.0.0-alpha-1 Resolution: Fixed > Optimize in-memory representation of mapreduce TableSplit objects > - > > Key: HBASE-24859 > URL: https://issues.apache.org/jira/browse/HBASE-24859 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 2.2.7 >Reporter: Sandeep Pal >Assignee: Sandeep Pal >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 2.2.7, 2.3.4 > > Attachments: Screen Shot 2020-08-26 at 8.44.34 AM.png, hbase-24859.png > > > It has been observed that when the table has too many regions, MR jobs > consume a lot of memory in the client. This is because we keep the region > level information in memory and the memory heavy object is TableSplit because > of the Scan object as a part of it. > However, it looks like the TableInputFormat for single table doesn't need to > store the scan object in the TableSplit because we do not use it and all the > splits are expected to have the exact same scan object. In TableInputFormat > we use the scan object directly from the MR conf. -- This message was sent by Atlassian Jira (v8.3.4#803005)
