[jira] [Commented] (HBASE-26012) Improve logging and dequeue logic in DelayQueue
[
https://issues.apache.org/jira/browse/HBASE-26012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369248#comment-17369248
]
Hudson commented on HBASE-26012:
Results for branch branch-2
[build #286 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/286/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/286/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/286/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/286/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/286/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/286//console].
> Improve logging and dequeue logic in DelayQueue
> ---
>
> Key: HBASE-26012
> URL: https://issues.apache.org/jira/browse/HBASE-26012
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.3.5, 2.4.4, 3.0.0-alpha-2
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5
>
>
> In Remote Procedure dispatcher, before submitting (sub)Procedure to Thread
> pool, we enqueue it as DelayedWithTimeout object on DelayQueue.
> TimeoutExecutorThread keeps dequeuing elements from this DelayQueue and
> submit the Procedure to the threadpool. The expiration of DelayedWithTimeout
> is determined by getDelay(TimeUnit):
> {code:java}
> @Override
> public long getDelay(final TimeUnit unit) {
> return DelayedUtil.getRemainingTime(unit, getTimeout());
> }
> {code}
> {code:java}
> /**
>* @return Time remaining as milliseconds.
>*/
> public static long getRemainingTime(final TimeUnit resultUnit, final long
> timeout) {
> final long currentTime = EnvironmentEdgeManager.currentTime();
> if (currentTime >= timeout) {
> return 0;
> }
> return resultUnit.convert(timeout - currentTime, TimeUnit.MILLISECONDS);
> }
> {code}
> Hence, in order for the elements to get dequeued on time, it is necessary
> that EnvironmentEdgeManager.currentTime() returns the current time in millis.
> As part of unit test, if we use our own custom EnvironmentEdge and inject it
> using EnvironmentEdgeManager.injectEdge before creating any tables, it is
> possible that we continue returning same value (based on custom impl) with
> EnvironmentEdgeManager.currentTime(). If that is the case, getRemainingTime
> as mentioned above, will never return 0 and hence, the procedure wrapped in
> DelayedWithTimeout might never be dequeued from DelayQueue because it's delay
> will not expire.
> As of today, our system goes in hanging state while waiting for table regions
> to be available (as mentioned above, DelayedWithTimeout object never gets
> dequeued from DelayQueue).
> Thread dump might show something like this consistently:
> {code:java}
> "ProcedureDispatcherTimeoutThread" #319 daemon prio=5 os_prio=31
> tid=0x7fcaf0cae800 nid=0x21d03 waiting on condition [0x700019293000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0007225a0090> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:223)
> at
> org.apache.hadoop.hbase.procedure2.util.DelayedUtil.takeWithoutInterrupt(DelayedUtil.java:82)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher$TimeoutExecutorThread.run(RemoteProcedureDispatcher.java:314)
>Locked ownable synchronizers:
> - None
> {code}
> Although running into situation like this is not likely possible unless
> custom EnvironmentEdge is used as mentioned above, we should improve our
> dequeue logic as well as log
[GitHub] [hbase] mnpoonia commented on a change in pull request #3372: HBASE-25986 set default value of normalization enabled from hbase site
mnpoonia commented on a change in pull request #3372: URL: https://github.com/apache/hbase/pull/3372#discussion_r658492987 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptor.java ## @@ -275,11 +275,11 @@ /** * Check if normalization enable flag of the table is true. If flag is false - * then no region normalizer won't attempt to normalize this table. + * then region normalizer won't attempt to normalize this table. * - * @return true if region normalization is enabled for this table + * @return value of NORMALIZATION_ENABLED key for this table if present else return defaultValue */ - boolean isNormalizationEnabled(); + boolean isNormalizationEnabled(boolean defaulValue); Review comment: That make sense. I was trying to define a logical definition about what will override the other. So currently we have a default value for table descriptor. What i am trying to do is remove it from the default list and define default at hbase-site level. So going forward - either this property will be present or not. and if not present then hbase-site value will be used and if present then the value defined by table descriptor will be used. Let me fix this pattern in code and make it clear. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3426: HBASE-26032 Make HRegion.getStores() an O(1) operation
Apache-HBase commented on pull request #3426: URL: https://github.com/apache/hbase/pull/3426#issuecomment-868207569 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 53s | master passed | | +1 :green_heart: | compile | 1m 28s | master passed | | +1 :green_heart: | shadedjars | 8m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 2s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 8m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 143m 13s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 2m 9s | hbase-examples in the patch passed. | | | | 177m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3426/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3426 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2e4810b8a05a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3426/1/testReport/ | | Max. process+thread count | 3257 (vs. ulimit of 3) | | modules | C: hbase-server hbase-examples U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3426/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3425: HBASE-25991 Do compaction on compaction server
Apache-HBase commented on pull request #3425: URL: https://github.com/apache/hbase/pull/3425#issuecomment-868197934 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-25714 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 19s | HBASE-25714 passed | | +1 :green_heart: | compile | 3m 26s | HBASE-25714 passed | | +1 :green_heart: | checkstyle | 1m 14s | HBASE-25714 passed | | +1 :green_heart: | spotbugs | 2m 12s | HBASE-25714 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 3m 22s | the patch passed | | -0 :warning: | javac | 3m 22s | hbase-server generated 3 new + 190 unchanged - 3 fixed = 193 total (was 193) | | +1 :green_heart: | checkstyle | 1m 11s | hbase-server: The patch generated 0 new + 91 unchanged - 9 fixed = 91 total (was 100) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 54s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 51m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3425 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4ecb4cd4d2f1 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-25714 / da0fa3000e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/2/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3419: HBASE-26027 The calling of HTable.batch blocked at AsyncRequestFuture…
Apache-HBase commented on pull request #3419: URL: https://github.com/apache/hbase/pull/3419#issuecomment-868190353 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 23s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | branch-2 passed | | +1 :green_heart: | compile | 1m 4s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 34s | branch-2 passed | | +1 :green_heart: | spotbugs | 1m 12s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | checkstyle | 0m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 10s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 1m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 34m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3419 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 361bb0e7b14d 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 7e6b66c867 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3419: HBASE-26027 The calling of HTable.batch blocked at AsyncRequestFuture…
Apache-HBase commented on pull request #3419: URL: https://github.com/apache/hbase/pull/3419#issuecomment-868188322 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 24s | branch-2 passed | | +1 :green_heart: | compile | 0m 29s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | shadedjars | 6m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 51s | hbase-client in the patch passed. | | | | 28m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3419 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bcd49c28cf8d 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 7e6b66c867 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/2/testReport/ | | Max. process+thread count | 238 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3419: HBASE-26027 The calling of HTable.batch blocked at AsyncRequestFuture…
Apache-HBase commented on pull request #3419: URL: https://github.com/apache/hbase/pull/3419#issuecomment-868188252 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | branch-2 passed | | +1 :green_heart: | compile | 0m 32s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 0m 32s | the patch passed | | +1 :green_heart: | javac | 0m 32s | the patch passed | | +1 :green_heart: | shadedjars | 6m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 49s | hbase-client in the patch passed. | | | | 28m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3419 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 978d5dd5e2d9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 7e6b66c867 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/2/testReport/ | | Max. process+thread count | 300 (vs. ulimit of 12500) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3419/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3421: HBASE-26026 HBase Write may be stuck forever when using CompactingMem…
Apache-HBase commented on pull request #3421: URL: https://github.com/apache/hbase/pull/3421#issuecomment-868182661 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 50s | master passed | | +1 :green_heart: | compile | 4m 4s | master passed | | +1 :green_heart: | checkstyle | 1m 22s | master passed | | +1 :green_heart: | spotbugs | 2m 33s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 55s | the patch passed | | +1 :green_heart: | compile | 4m 12s | the patch passed | | +1 :green_heart: | javac | 4m 12s | the patch passed | | -0 :warning: | checkstyle | 1m 17s | hbase-server: The patch generated 2 new + 74 unchanged - 0 fixed = 76 total (was 74) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 45s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 55m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3421 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 68f6eb9a51c8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3421/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
Apache-HBase commented on pull request #3356: URL: https://github.com/apache/hbase/pull/3356#issuecomment-868179471 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 37s | master passed | | +1 :green_heart: | compile | 1m 20s | master passed | | +1 :green_heart: | shadedjars | 8m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 27s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 131m 41s | hbase-server in the patch passed. | | | | 170m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3356 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4a765f78efed 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/testReport/ | | Max. process+thread count | 3770 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
Apache-HBase commented on pull request #3356: URL: https://github.com/apache/hbase/pull/3356#issuecomment-868177680 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 1m 33s | master passed | | +1 :green_heart: | shadedjars | 8m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 15s | the patch passed | | +1 :green_heart: | compile | 1m 33s | the patch passed | | +1 :green_heart: | javac | 1m 33s | the patch passed | | +1 :green_heart: | shadedjars | 8m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 6m 10s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 124m 50s | hbase-server in the patch passed. | | | | 164m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3356 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 09f285184e01 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/testReport/ | | Max. process+thread count | 3524 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369212#comment-17369212 ] Sean Busbey commented on HBASE-26031: - You can see here where all the executors got saturated, we built up a queue, and it’s been working through it: https://ci-hadoop.apache.org/label/hbase-staging/load-statistics Will check in the morning how they actually did. > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: image-2021-06-24-16-14-03-721.png > > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] dingwei-2017 commented on pull request #3423: HBASE-26017 fix pe tool totalRows exceed maximum of int
dingwei-2017 commented on pull request #3423: URL: https://github.com/apache/hbase/pull/3423#issuecomment-868172281 @busbey thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] dingwei-2017 commented on pull request #3418: HBASE-26016 fix pretty tool print header missing last indexblock and bloomchunk
dingwei-2017 commented on pull request #3418: URL: https://github.com/apache/hbase/pull/3418#issuecomment-868171329 @busbey already done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26028) The view as json page shows exception when using TinyLfuBlockCache
[ https://issues.apache.org/jira/browse/HBASE-26028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-26028: --- Attachment: HBASE-26028-beforepatch.jpg HBASE-26028-afterpatch.jpg > The view as json page shows exception when using TinyLfuBlockCache > -- > > Key: HBASE-26028 > URL: https://issues.apache.org/jira/browse/HBASE-26028 > Project: HBase > Issue Type: Bug > Components: UI >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Attachments: HBASE-26028-afterpatch.jpg, HBASE-26028-beforepatch.jpg > > > Some variable in TinyLfuBlockCache should be marked as transient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23349) Low refCount preventing archival of compacted away files
[
https://issues.apache.org/jira/browse/HBASE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369205#comment-17369205
]
Chengliang commented on HBASE-23349:
Is the issue already fixed in 1.7.0 released on 2021/06/12? It is still marked
as UNRESOLVED.
I faced exactly the same error.
{code:java}
regionserver.HStore - Can't archive compacted file
hdfs://{{root-dir}}/hbase/data/default/t1/12a9e1112e0371955b3db8d3ebb2d298/cf1/73b72f5ddfce4a34a9e01afe7b83c1f9
because of either isCompactedAway=true or file has reference,
isReferencedInReads=true, refCount=1, skipping for now.{code}
Thanks you so much for the clarification.
Best & Regards,
Larry
> Low refCount preventing archival of compacted away files
>
>
> Key: HBASE-23349
> URL: https://issues.apache.org/jira/browse/HBASE-23349
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.6.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
>
> We have observed that refCount on compacted away store files as low as 1 is
> prevent archival.
> {code:java}
> regionserver.HStore - Can't archive compacted file
> hdfs://{{root-dir}}/hbase/data/default/t1/12a9e1112e0371955b3db8d3ebb2d298/cf1/73b72f5ddfce4a34a9e01afe7b83c1f9
> because of either isCompactedAway=true or file has reference,
> isReferencedInReads=true, refCount=1, skipping for now.
> {code}
> We should come up with core code (run as part of discharger thread)
> gracefully resolve reader lock issue by resetting ongoing scanners to start
> pointing to new store files instead of compacted away store files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3426: HBASE-26032 Make HRegion.getStores() an O(1) operation
Apache-HBase commented on pull request #3426: URL: https://github.com/apache/hbase/pull/3426#issuecomment-868166641 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 51s | master passed | | +1 :green_heart: | compile | 3m 52s | master passed | | +1 :green_heart: | checkstyle | 1m 25s | master passed | | +1 :green_heart: | spotbugs | 2m 47s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | +1 :green_heart: | compile | 3m 46s | the patch passed | | +1 :green_heart: | javac | 3m 46s | the patch passed | | -0 :warning: | checkstyle | 1m 7s | hbase-server: The patch generated 2 new + 249 unchanged - 0 fixed = 251 total (was 249) | | -0 :warning: | checkstyle | 0m 15s | hbase-examples: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 5s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 8s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 51m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3426/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3426 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux c5a62929da1b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3426/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3426/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-examples.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server hbase-examples U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3426/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26026) HBase Write may be stuck forever when using CompactingMemStore
[
https://issues.apache.org/jira/browse/HBASE-26026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated HBASE-26026:
-
Description:
Sometimes I observed that HBase Write might be stuck in my hbase cluster which
enabling {{CompactingMemStore}}. I have simulated the problem by unit test in
my PR.
The problem is caused by {{CompactingMemStore.checkAndAddToActiveSize}} :
{code:java}
425 private boolean checkAndAddToActiveSize(MutableSegment currActive, Cell
cellToAdd,
426 MemStoreSizing memstoreSizing) {
427if (shouldFlushInMemory(currActive, cellToAdd, memstoreSizing)) {
428 if (currActive.setInMemoryFlushed()) {
429flushInMemory(currActive);
430if (setInMemoryCompactionFlag()) {
431 // The thread is dispatched to do in-memory compaction in the
background
..
}
{code}
In line 427, if {{currActive.getDataSize}} adding the size of {{cellToAdd}}
exceeds {{CompactingMemStore.inmemoryFlushSize}}, then {{currActive}} should
be flushed, {{MutableSegment.setInMemoryFlushed()}} is invoked in above line
428 ：
{code:java}
public boolean setInMemoryFlushed() {
return flushed.compareAndSet(false, true);
}
{code}
After set {{currActive.flushed}} to true, in above line 429
{{flushInMemory(currActive)}} invokes
{{CompactingMemStore.pushActiveToPipeline}} :
{code:java}
protected void pushActiveToPipeline(MutableSegment currActive) {
if (!currActive.isEmpty()) {
pipeline.pushHead(currActive);
resetActive();
}
}
{code}
In above {{CompactingMemStore.pushActiveToPipeline}} method , if the
{{currActive.cellSet}} is empty, then nothing is done. Due to concurrent
writes and because we first add cell size to {{currActive.getDataSize}} and
then actually add cell to {{currActive.cellSet}}, it is possible that
{{currActive.getDataSize}} could not accommodate {{cellToAdd}} but
{{currActive.cellSet}} is still empty if pending writes which not yet add cells
to {{currActive.cellSet}}.
So if the {{currActive.cellSet}} is empty now, then no {{ActiveSegment}} is
created, and new writes still continue target to {{currActive}}, but
{{currActive.flushed}} is true, {{currActive}} could not enter
{{flushInMemory(currActive)}} again，and new {{ActiveSegment}} could not be
created forever ! In the end all writes would be stuck.
In my opinion , once {{currActive.flushed}} is set true, it could not continue
use as {{ActiveSegment}} , and because of concurrent pending writes, only after
{{currActive.updatesLock.writeLock()}} is acquired(i.e.
{{currActive.waitForUpdates}} is called) in
{{CompactingMemStore.inMemoryCompaction}} ,we can safely say {{currActive}} is
empty or not.
My fix is remove the {{if (!currActive.isEmpty())}} check here and left the
check to background {{InMemoryCompactionRunnable}} after
{{currActive.waitForUpdates}} is called. An alternative fix is we use
synchronization mechanism in {{checkAndAddToActiveSize}} method to prevent all
writes , wait for all pending write completed(i.e. currActive.waitForUpdates is
called) and if {{currActive}} is still empty ,then we set
{{currActive.flushed}} back to false,but I am not inclined to use so heavy
synchronization in write path, and I think we would better maintain lockless
implementation for {{CompactingMemStore.add}} method just as now and
{{currActive.waitForUpdates}} would better be left in background
{{InMemoryCompactionRunnable}}.
was:
Sometimes I observed that HBase Write might be stuck in my hbase cluster which
enabling {{CompactingMemStore}}. I have simulated the problem by unit test in
my PR.
The problem is caused by {{CompactingMemStore.checkAndAddToActiveSize}} :
{code:java}
425 private boolean checkAndAddToActiveSize(MutableSegment currActive, Cell
cellToAdd,
426 MemStoreSizing memstoreSizing) {
427if (shouldFlushInMemory(currActive, cellToAdd, memstoreSizing)) {
428 if (currActive.setInMemoryFlushed()) {
429flushInMemory(currActive);
430if (setInMemoryCompactionFlag()) {
431 // The thread is dispatched to do in-memory compaction in the
background
..
}
{code}
In line 427, if {{currActive.getDataSize}} adding the size of {{cellToAdd}}
exceeds {{CompactingMemStore.inmemoryFlushSize}}, then {{currActive}} should
be flushed, {{MutableSegment.setInMemoryFlushed()}} is invoked in above line
428 ：
{code:java}
public boolean setInMemoryFlushed() {
return flushed.compareAndSet(false, true);
}
{code}
After set {{currActive.flushed}} to true, in above line 429
{{flushInMemory(currActive)}} invokes
{{CompactingMemStore.pushActiveToPipeline}} :
{code:java}
protected void pushActiveToPipeline(MutableSegment currActive) {
if (!currActive.isEmpty()) {
pipeline.pushHead(currActive);
resetActive();
}
}
{code}
In above {{CompactingMemStore.pushActiveToPipeline}} method , if the
[jira] [Updated] (HBASE-26027) The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
[
https://issues.apache.org/jira/browse/HBASE-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Wang updated HBASE-26027:
---
Description:
The batch api of HTable contains a param named results to store result or
exception, its type is Object[].
If user pass an array with other type, eg:
org.apache.hadoop.hbase.client.Result, and we need to put an exception into it
by some reason, then the ArrayStoreException will occur in
AsyncRequestFutureImpl.updateResult, then the
AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
actionsInProgress again and again, forever.
It is better to add an cutoff calculated by operationTimeout, instead of only
depend on the value of actionsInProgress.
BTW, this issue only for 2.x, since 3.x the implement has refactored.
{code:java}
[ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1
error for test processing localhost,16020,1624343786295
java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO ] [2021/06/22 23:23:10,375] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:20,378] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:30,384] main - #1, waiting for 10 actions to finish
on table:
[INFO ] [2021/06/22 23:23:40,387] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:23:50,397] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:00,400] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:10,408] main - #1, waiting for 10 actions to finish
on table: test
[INFO ] [2021/06/22 23:24:20,413] main - #1, waiting for 10 actions to finish
on table: test
{code}
was:
The batch api of HTable contains a param named results to store result or
exception, its type is Object[].
If user pass an array with other type, eg:
org.apache.hadoop.hbase.client.Result, then the ArrayStoreException will occur
in AsyncRequestFutureImpl.updateResult, then the
AsyncRequestFutureImpl.decActionCounter will be skipped, then in the
AsyncRequestFutureImpl.waitUntilDone we will stuck at here checking the
actionsInProgress again and again, forever.
It is better to add an cutoff calculated by operationTimeout, instead of only
depend on the value of actionsInProgress.
BTW, this issue only for 2.x, since 3.x the implement has refactored.
{code:java}
[ERROR] [2021/06/22 23:23:00,676] hconnection-0x6b927fb-shared-pool3-t1 - id=1
error for test processing localhost,16020,1624343786295
java.lang.ArrayStoreException: org.apache.hadoop.hbase.DoNotRetryIOException
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.updateResult(AsyncRequestFutureImpl.java:1242)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.trySetResultSimple(AsyncRequestFutureImpl.java:1087)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.setError(AsyncRequestFutureImpl.java:1021)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.manageError(AsyncRequestFutureImpl.java:683)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.receiveGlobalFailure(AsyncRequestFutureImpl.java:716)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1500(AsyncRequestFutureImpl.java:69)
at
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:219)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
[GitHub] [hbase] jojochuang opened a new pull request #3426: HBASE-26032 Make HRegion.getStores() an O(1) operation
jojochuang opened a new pull request #3426: URL: https://github.com/apache/hbase/pull/3426 See the corresponding jira HBASE-26032 for the details and the visualvm output. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369185#comment-17369185 ] Sean Busbey commented on HBASE-26031: - Hopefully the builds will not just be on one machine. Looking at the load for the label should show all the executors getting used > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: image-2021-06-24-16-14-03-721.png > > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26032) Make HRegion.getStores() an O(1) operation
[ https://issues.apache.org/jira/browse/HBASE-26032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HBASE-26032: Description: This is a relatively minor issue, but I did spot HRegion.getStores() popping up in my profiler. Checking the code, I realized that HRegion.getStores() allocates a new array list in it, converting the Collection<> to List<>. But it also makes it an O( n ) in space and time complexity. This conversion appears mostly unnecessary, because we only iterate the stores in production code, and so the new ArrayList object is thrown away immediately. Only in a number of test code where we index into the stores. I suggest we should return the stores object directly, an O( 1 ) operation. was: This is a relatively minor issue, but I did spot HRegion.getStores() popping up in my profiler. Checking the code, I realized that HRegion.getStores() allocates a new array list in it, converting the Collection<> to List<>. But it also makes it an O( n ) in space and time complexity. This conversion appears mostly unnecessary, because we only iterate the stores in production code, and so the new ArrayList object is thrown away immediately. Only in a number of test code where we index into the stores. > Make HRegion.getStores() an O(1) operation > -- > > Key: HBASE-26032 > URL: https://issues.apache.org/jira/browse/HBASE-26032 > Project: HBase > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: Screen Shot 2021-06-24 at 3.56.33 PM.png > > > This is a relatively minor issue, but I did spot HRegion.getStores() popping > up in my profiler. > Checking the code, I realized that HRegion.getStores() allocates a new array > list in it, converting the Collection<> to List<>. But it also makes it an O( > n ) in space and time complexity. > This conversion appears mostly unnecessary, because we only iterate the > stores in production code, and so the new ArrayList object is thrown away > immediately. Only in a number of test code where we index into the stores. > I suggest we should return the stores object directly, an O( 1 ) operation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369183#comment-17369183 ] Sean Busbey commented on HBASE-26031: - Yeah I aborted the first one because I needed to allow concurrent builds > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: image-2021-06-24-16-14-03-721.png > > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26032) Make HRegion.getStores() an O(1) operation
Wei-Chiu Chuang created HBASE-26032: --- Summary: Make HRegion.getStores() an O(1) operation Key: HBASE-26032 URL: https://issues.apache.org/jira/browse/HBASE-26032 Project: HBase Issue Type: Improvement Reporter: Wei-Chiu Chuang Assignee: Wei-Chiu Chuang Attachments: Screen Shot 2021-06-24 at 3.56.33 PM.png This is a relatively minor issue, but I did spot HRegion.getStores() popping up in my profiler. Checking the code, I realized that HRegion.getStores() allocates a new array list in it, converting the Collection<> to List<>. But it also makes it an O( n ) in space and time complexity. This conversion appears mostly unnecessary, because we only iterate the stores in production code, and so the new ArrayList object is thrown away immediately. Only in a number of test code where we index into the stores. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26032) Make HRegion.getStores() an O(1) operation
[ https://issues.apache.org/jira/browse/HBASE-26032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HBASE-26032: Priority: Minor (was: Major) > Make HRegion.getStores() an O(1) operation > -- > > Key: HBASE-26032 > URL: https://issues.apache.org/jira/browse/HBASE-26032 > Project: HBase > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Minor > Attachments: Screen Shot 2021-06-24 at 3.56.33 PM.png > > > This is a relatively minor issue, but I did spot HRegion.getStores() popping > up in my profiler. > Checking the code, I realized that HRegion.getStores() allocates a new array > list in it, converting the Collection<> to List<>. But it also makes it an O( > n ) in space and time complexity. > This conversion appears mostly unnecessary, because we only iterate the > stores in production code, and so the new ArrayList object is thrown away > immediately. Only in a number of test code where we index into the stores. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26012) Improve logging and dequeue logic in DelayQueue
[
https://issues.apache.org/jira/browse/HBASE-26012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369175#comment-17369175
]
Hudson commented on HBASE-26012:
Results for branch branch-2.4
[build #150 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve logging and dequeue logic in DelayQueue
> ---
>
> Key: HBASE-26012
> URL: https://issues.apache.org/jira/browse/HBASE-26012
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.3.5, 2.4.4, 3.0.0-alpha-2
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 2.4.5
>
>
> In Remote Procedure dispatcher, before submitting (sub)Procedure to Thread
> pool, we enqueue it as DelayedWithTimeout object on DelayQueue.
> TimeoutExecutorThread keeps dequeuing elements from this DelayQueue and
> submit the Procedure to the threadpool. The expiration of DelayedWithTimeout
> is determined by getDelay(TimeUnit):
> {code:java}
> @Override
> public long getDelay(final TimeUnit unit) {
> return DelayedUtil.getRemainingTime(unit, getTimeout());
> }
> {code}
> {code:java}
> /**
>* @return Time remaining as milliseconds.
>*/
> public static long getRemainingTime(final TimeUnit resultUnit, final long
> timeout) {
> final long currentTime = EnvironmentEdgeManager.currentTime();
> if (currentTime >= timeout) {
> return 0;
> }
> return resultUnit.convert(timeout - currentTime, TimeUnit.MILLISECONDS);
> }
> {code}
> Hence, in order for the elements to get dequeued on time, it is necessary
> that EnvironmentEdgeManager.currentTime() returns the current time in millis.
> As part of unit test, if we use our own custom EnvironmentEdge and inject it
> using EnvironmentEdgeManager.injectEdge before creating any tables, it is
> possible that we continue returning same value (based on custom impl) with
> EnvironmentEdgeManager.currentTime(). If that is the case, getRemainingTime
> as mentioned above, will never return 0 and hence, the procedure wrapped in
> DelayedWithTimeout might never be dequeued from DelayQueue because it's delay
> will not expire.
> As of today, our system goes in hanging state while waiting for table regions
> to be available (as mentioned above, DelayedWithTimeout object never gets
> dequeued from DelayQueue).
> Thread dump might show something like this consistently:
> {code:java}
> "ProcedureDispatcherTimeoutThread" #319 daemon prio=5 os_prio=31
> tid=0x7fcaf0cae800 nid=0x21d03 waiting on condition [0x700019293000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0007225a0090> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:223)
> at
> org.apache.hadoop.hbase.procedure2.util.DelayedUtil.takeWithoutInterrupt(DelayedUtil.java:82)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher$TimeoutExecutorThread.run(RemoteProcedureDispatcher.java:314)
>Locked ownable synchronizers:
> - None
> {code}
> Although running into situation like this is not likely possible unless
> custom EnvironmentEdge is used as mentioned above, we should improve our
> dequeue logic as well as log important message to show where we are stuck.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26013) Get operations readRows metrics becomes zero after HBASE-25677
[
https://issues.apache.org/jira/browse/HBASE-26013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369173#comment-17369173
]
Hudson commented on HBASE-26013:
Results for branch branch-2.4
[build #150 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Get operations readRows metrics becomes zero after HBASE-25677
> --
>
> Key: HBASE-26013
> URL: https://issues.apache.org/jira/browse/HBASE-26013
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 1.7.1, 2.4.5
>
>
> After HBASE-25677, Server+table counters on each scan are extracted from
> #nextRaw to rsServices scan. In this case, the get operation will not count
> the read rows. So that the readRows metrics becomes zero. Should add counter
> in metricsUpdateForGet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25677) Server+table counters on each scan #nextRaw invocation becomes a bottleneck when heavy load
[
https://issues.apache.org/jira/browse/HBASE-25677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369174#comment-17369174
]
Hudson commented on HBASE-25677:
Results for branch branch-2.4
[build #150 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/150/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Server+table counters on each scan #nextRaw invocation becomes a bottleneck
> when heavy load
> ---
>
> Key: HBASE-25677
> URL: https://issues.apache.org/jira/browse/HBASE-25677
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 2.3.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.3.5, 2.4.3
>
>
> On a heavily loaded server mostly doing reads/scan, I saw that 90+% of
> handlers were BLOCKED in this fashion in thread dumps:
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=117,queue=17,port=16020" #161 daemon
> prio=5 os_prio=0 tid=0x7f748757f000 nid=0x73e9 waiting for monitor entry
> [0x7f74783e]
> java.lang.Thread.State: BLOCKED (on object monitor)
>at
> java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
>- waiting to lock <0x7f7647e3cc38> (a
> java.util.concurrent.ConcurrentHashMap$Node)
>at
> org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.getOrCreateTableMeter(MetricsTableQueryMeterImpl.java:80)
>at
> org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.updateTableReadQueryMeter(MetricsTableQueryMeterImpl.java:90)
>at
> org.apache.hadoop.hbase.regionserver.RegionServerTableMetrics.updateTableReadQueryMeter(RegionServerTableMetrics.java:89)
>at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServer.updateReadQueryMeter(MetricsRegionServer.java:274)
>at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6742)
>at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3319)
>- locked <0x7f896c0165a0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
>at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3566)
>at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858)
>at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393)
>at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
>at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
>at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> {code}
> It kept up for good periods of time.
> I saw it to a leser extent on other servers, with less load.
> These RS had 400+ Regions a good few of which were serving out scan reads;
> the server was doing ~1M hits a second. In this scenario, I saw the above
> bottleneck.
> Looking at it, it came in w/ when the parent issue feature was added. There
> are these read counts and then there were also write counts. The write counts
> are mostly batch-based. Let me do same thing here for the read update the
> central server+table count after scan is done rather than per invocation of
> #nextRaw.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
Apache-HBase commented on pull request #3356: URL: https://github.com/apache/hbase/pull/3356#issuecomment-868124227 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 35s | master passed | | +1 :green_heart: | compile | 3m 40s | master passed | | +1 :green_heart: | checkstyle | 1m 16s | master passed | | +1 :green_heart: | spotbugs | 2m 31s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 3m 37s | the patch passed | | -0 :warning: | javac | 0m 31s | hbase-balancer generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) | | -0 :warning: | checkstyle | 0m 14s | hbase-balancer: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 18m 7s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 50m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3356 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8f1c3a697430 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/7/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] clarax commented on pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on pull request #3356: URL: https://github.com/apache/hbase/pull/3356#issuecomment-868101875 Rebase from master to solve the test failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] clarax commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
clarax commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-868101481 @ndimiduk @Apache9 Good to go? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
Apache-HBase commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-868100262 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | master passed | | +1 :green_heart: | compile | 0m 29s | master passed | | +1 :green_heart: | checkstyle | 0m 14s | master passed | | +1 :green_heart: | spotbugs | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 0m 30s | the patch passed | | -0 :warning: | javac | 0m 30s | hbase-balancer generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) | | -0 :warning: | checkstyle | 0m 13s | hbase-balancer: The patch generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 49s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 0m 45s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 40m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3415 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4ae3c7616ade 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/6/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-balancer U: hbase-balancer | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/6/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
Apache-HBase commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-868098651 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 0m 22s | master passed | | +1 :green_heart: | shadedjars | 8m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 11s | the patch passed | | +1 :green_heart: | compile | 0m 22s | the patch passed | | +1 :green_heart: | javac | 0m 22s | the patch passed | | +1 :green_heart: | shadedjars | 8m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 8m 13s | hbase-balancer in the patch passed. | | | | 36m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3415 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bea4020a84a1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/6/testReport/ | | Max. process+thread count | 313 (vs. ulimit of 3) | | modules | C: hbase-balancer U: hbase-balancer | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
Apache-HBase commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-868098191 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | master passed | | +1 :green_heart: | compile | 0m 21s | master passed | | +1 :green_heart: | shadedjars | 8m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 0m 20s | the patch passed | | +1 :green_heart: | javac | 0m 20s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 42s | hbase-balancer in the patch passed. | | | | 34m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3415 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3af058c458c7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/6/testReport/ | | Max. process+thread count | 354 (vs. ulimit of 3) | | modules | C: hbase-balancer U: hbase-balancer | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3415/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-26031: -- Attachment: image-2021-06-24-16-14-03-721.png > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: image-2021-06-24-16-14-03-721.png > > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369149#comment-17369149 ] Michael Stack commented on HBASE-26031: --- Is this 5 builds running concurrently on one machine? !image-2021-06-24-16-14-03-721.png! > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369148#comment-17369148 ] Michael Stack commented on HBASE-26031: --- Aborted by you [~busbey] ? > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25990) Add donated buildbots for jenkins
[ https://issues.apache.org/jira/browse/HBASE-25990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369146#comment-17369146 ] Michael Stack commented on HBASE-25990: --- The 5 machines hbase11-hbase-15 now show as buildbots here [https://ci-hadoop.apache.org/computer/] Our [~busbey] is testing they actually work just like the others See HBASE-26031 > Add donated buildbots for jenkins > - > > Key: HBASE-25990 > URL: https://issues.apache.org/jira/browse/HBASE-25990 > Project: HBase > Issue Type: Task > Components: build >Reporter: Michael Stack >Priority: Major > Attachments: Screen Shot 2021-06-22 at 1.43.12 PM.png > > > This issue is for keeping notes on how to add a donated buildbot to our > apache build. > My employer donated budget (I badly under-estimated cost but whatever...). > This issue is about adding 5 GCP nodes. > There is this page up on apache on donating machines for build > https://infra.apache.org/hosting-external-agent.html It got me some of the > ways... at least as far as the bit about mailing root@a.o(nada). > At [~zhangduo]'s encouragement -- he has been this route already adding in > the xiaomi donation -- I filed a JIRA after deploying a machine on GCP, > INFRA-21973. > I then reached out on slack and the gentleman Gavin MacDonald picked up the > task. > He told me run this script on all hosts after making edits (comment out line > #39 where we set hostname -- doesn't work): > https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/agent-install.sh > (For more context on the above script and as a good backgrounder, see the > nice C* page on how to do this setup: > https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.md) > After doing the above, I had to do a visudo on each host to add a line for an > infra account to allow passwordless access. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25990) Add donated buildbots for jenkins
[ https://issues.apache.org/jira/browse/HBASE-25990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25990: -- Attachment: Screen Shot 2021-06-22 at 1.43.12 PM.png > Add donated buildbots for jenkins > - > > Key: HBASE-25990 > URL: https://issues.apache.org/jira/browse/HBASE-25990 > Project: HBase > Issue Type: Task > Components: build >Reporter: Michael Stack >Priority: Major > Attachments: Screen Shot 2021-06-22 at 1.43.12 PM.png > > > This issue is for keeping notes on how to add a donated buildbot to our > apache build. > My employer donated budget (I badly under-estimated cost but whatever...). > This issue is about adding 5 GCP nodes. > There is this page up on apache on donating machines for build > https://infra.apache.org/hosting-external-agent.html It got me some of the > ways... at least as far as the bit about mailing root@a.o(nada). > At [~zhangduo]'s encouragement -- he has been this route already adding in > the xiaomi donation -- I filed a JIRA after deploying a machine on GCP, > INFRA-21973. > I then reached out on slack and the gentleman Gavin MacDonald picked up the > task. > He told me run this script on all hosts after making edits (comment out line > #39 where we set hostname -- doesn't work): > https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/agent-install.sh > (For more context on the above script and as a good backgrounder, see the > nice C* page on how to do this setup: > https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.md) > After doing the above, I had to do a visudo on each host to add a line for an > infra account to allow passwordless access. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25990) Add donated buildbots for jenkins
[ https://issues.apache.org/jira/browse/HBASE-25990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369145#comment-17369145 ] Michael Stack commented on HBASE-25990: --- Update. Took a while for the infra steps to go through. See the associated infra ticket (w/ some offline pinging on backchannels up on slack). I've been updating the infra ticket on state transitions. Its important to move it from 'user' to 'infra' ownership which is done via this 'button' which I didn't see at first. See below. !Screen Shot 2021-06-22 at 1.43.12 PM.png! > Add donated buildbots for jenkins > - > > Key: HBASE-25990 > URL: https://issues.apache.org/jira/browse/HBASE-25990 > Project: HBase > Issue Type: Task > Components: build >Reporter: Michael Stack >Priority: Major > Attachments: Screen Shot 2021-06-22 at 1.43.12 PM.png > > > This issue is for keeping notes on how to add a donated buildbot to our > apache build. > My employer donated budget (I badly under-estimated cost but whatever...). > This issue is about adding 5 GCP nodes. > There is this page up on apache on donating machines for build > https://infra.apache.org/hosting-external-agent.html It got me some of the > ways... at least as far as the bit about mailing root@a.o(nada). > At [~zhangduo]'s encouragement -- he has been this route already adding in > the xiaomi donation -- I filed a JIRA after deploying a machine on GCP, > INFRA-21973. > I then reached out on slack and the gentleman Gavin MacDonald picked up the > task. > He told me run this script on all hosts after making edits (comment out line > #39 where we set hostname -- doesn't work): > https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/agent-install.sh > (For more context on the above script and as a good backgrounder, see the > nice C* page on how to do this setup: > https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.md) > After doing the above, I had to do a visudo on each host to add a line for an > infra account to allow passwordless access. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369141#comment-17369141 ] Michael Stack commented on HBASE-26031: --- Thanks [~busbey] > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369098#comment-17369098 ] Sean Busbey commented on HBASE-26031: - queued several builds to saturate the nodes. > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[
https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369097#comment-17369097
]
Hudson commented on HBASE-26031:
Results for branch HBASE-26031
[build #1 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26031/1/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26031/1/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26031/1/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26031/1/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26031/1//console].
> Validate nightly builds run on new ci workers hbase10-hbase15
> -
>
> Key: HBASE-26031
> URL: https://issues.apache.org/jira/browse/HBASE-26031
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> Per slack, asf infra has finished adding in nodes hbase10-hbase15 to
> ci-hadoop.
> make sure they can run nightly.
> # Set labels for all these node to "hbase-staging"
> # Push a feature branch off of current HEAD that updates the agent labels to
> use "hbase-staging"
> # trigger a bunch of runs. make sure *something* runs on each of the nodes
> # Set labels for the nodes to "hbase"
> # delete feature branch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
Apache-HBase commented on pull request #3356: URL: https://github.com/apache/hbase/pull/3356#issuecomment-867971504 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | compile | 1m 22s | master passed | | +1 :green_heart: | shadedjars | 8m 22s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 8m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 11s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 131m 47s | hbase-server in the patch passed. | | | | 171m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3356 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 28e532ecfd93 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/testReport/ | | Max. process+thread count | 3431 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369093#comment-17369093 ] Sean Busbey commented on HBASE-26031: - Nightly branch filter updated to include this feature branch. https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26031/ > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369092#comment-17369092 ] Sean Busbey commented on HBASE-26031: - branch is up: https://github.com/apache/hbase/tree/HBASE-26031 > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
Apache-HBase commented on pull request #3356: URL: https://github.com/apache/hbase/pull/3356#issuecomment-867969037 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 20s | master passed | | +1 :green_heart: | compile | 1m 33s | master passed | | +1 :green_heart: | shadedjars | 8m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 33s | the patch passed | | +1 :green_heart: | compile | 1m 38s | the patch passed | | +1 :green_heart: | javac | 1m 38s | the patch passed | | +1 :green_heart: | shadedjars | 8m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 6m 5s | hbase-balancer in the patch passed. | | -1 :x: | unit | 125m 9s | hbase-server in the patch failed. | | | | 166m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3356 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ba9cb446b97d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/testReport/ | | Max. process+thread count | 3302 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
busbey commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r658290872
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -303,54 +302,54 @@ private String getBalanceReason(double total, double
sumMultiplier) {
boolean needsBalance(TableName tableName, BalancerClusterState cluster) {
ClusterLoadState cs = new ClusterLoadState(cluster.clusterState);
if (cs.getNumServers() < MIN_SERVER_BALANCE) {
- if (LOG.isDebugEnabled()) {
-LOG.debug("Not running balancer because only " + cs.getNumServers()
+ LOG.info("Not running balancer because only " + cs.getNumServers()
+ " active regionserver(s)");
- }
sendRejectionReasonToRingBuffer(() -> "The number of RegionServers " +
cs.getNumServers() +
" < MIN_SERVER_BALANCE(" + MIN_SERVER_BALANCE + ")", null);
return false;
}
if (areSomeRegionReplicasColocated(cluster)) {
+ LOG.info("Running balancer because at least one server hosts replicas of
the same region.");
return true;
}
if (idleRegionServerExist(cluster)){
+ LOG.info("Running balancer because cluster has idle server(s).");
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
+}
+if (sumMultiplier <= 0)
+{
+ LOG.error("At least one cost function needs a multiplier > 0");
+ return false;
Review comment:
I agree that we should not list all of the cost functions. How about if
we just include a note like "the default cost functions should have configs
named similar to hbase.master.balancer.stochastic.regionCountCost with
_regionCount_ replaced with the specific cost function name."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369081#comment-17369081 ] Sean Busbey commented on HBASE-26031: - Updated the relevant node labels just using the jenkins UI. (e.g. https://ci-hadoop.apache.org/computer/hbase15/configure ). should probably write this stuff up so any of the committers can do it. > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369078#comment-17369078 ] Sean Busbey commented on HBASE-26031: - fyi [~stack] I'm doing this now-ish > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-26031: Description: Per slack, asf infra has finished adding in nodes hbase10-hbase15 to ci-hadoop. make sure they can run nightly. # Set labels for all these node to "hbase-staging" # Push a feature branch off of current HEAD that updates the agent labels to use "hbase-staging" # trigger a bunch of runs. make sure *something* runs on each of the nodes # Set labels for the nodes to "hbase" # delete feature branch was: Per slack, asf infra has finished adding in nodes hbase10-hbase15 to ci-hadoop. make sure they can run nightly. > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-26031 started by Sean Busbey. --- > Validate nightly builds run on new ci workers hbase10-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
Sean Busbey created HBASE-26031: --- Summary: Validate nightly builds run on new ci workers hbase10-hbase15 Key: HBASE-26031 URL: https://issues.apache.org/jira/browse/HBASE-26031 Project: HBase Issue Type: Task Components: test Reporter: Sean Busbey Assignee: Sean Busbey Per slack, asf infra has finished adding in nodes hbase10-hbase15 to ci-hadoop. make sure they can run nightly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
Apache-HBase commented on pull request #3356: URL: https://github.com/apache/hbase/pull/3356#issuecomment-867905822 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 52s | master passed | | +1 :green_heart: | compile | 3m 58s | master passed | | +1 :green_heart: | checkstyle | 1m 20s | master passed | | +1 :green_heart: | spotbugs | 2m 45s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | +1 :green_heart: | compile | 3m 57s | the patch passed | | -0 :warning: | javac | 0m 33s | hbase-balancer generated 1 new + 17 unchanged - 0 fixed = 18 total (was 17) | | -0 :warning: | checkstyle | 0m 15s | hbase-balancer: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 18m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 23s | The patch does not generate ASF License warnings. | | | | 52m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3356 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux a6d9e820410e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/artifact/yetus-general-check/output/diff-compile-javac-hbase-balancer.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-balancer.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-balancer hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3356/6/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
clarax commented on a change in pull request #3415: URL: https://github.com/apache/hbase/pull/3415#discussion_r658223242 ## File path: hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java ## @@ -125,7 +125,7 @@ private int stepsPerRegion = 800; private long maxRunningTime = 30 * 1000 * 1; // 30 seconds. private int numRegionLoadsToRemember = 15; - private float minCostNeedBalance = 0.05f; + private float minCostNeedBalance = 0.025f; Review comment: Yes, as explained in the jira and the pr, we have to lower it for consistent user experience. the old code almost always return max for table skew that inflates the total cost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369057#comment-17369057
]
Andrew Kyle Purtell edited comment on HBASE-26021 at 6/24/21, 7:24 PM:
---
[~vjasani] [~reidchan] [~zhangduo]
This is what I would recommend as the least bad option (all options are bad...)
# Withdraw the 1.7.0 release immediately. Delete from the mirrors.
# Apply the fix. I like your simple fix posted on this issue above [~vjasani]
but since we backported a change it makes more sense to backport its revert
too:
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
. However, if someone has already upgraded to 1.7.0, has this ship sailed? Do
we need your fix on this issue instead? In any case we clean up the issue.
# Release 1.7.0.1.
The vote and review of the 1.7.0.1 RC should be fast given there have been few
commits since 1.7.0.
When we release 1.7.0.1 we can state in the release notes that 1.7.0 had this
bug and anyone who upgraded to it who wants to upgrade to 2.x must upgrade to
1.7.0.1 first.
was (Author: apurtell):
[~vjasani] [~reidchan] [~zhangduo]
This is what I would recommend as the least bad option (all options are bad...)
# Withdraw the 1.7.0 release immediately. Delete from the mirrors.
# Apply the fix. I like your simple fix posted on this issue above [~vjasani]
but since we backported a change it makes more sense to backport its revert
too:
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
# Release 1.7.0.1.
The vote and review of the 1.7.0.1 RC should be fast given there have been few
commits since 1.7.0.
When we release 1.7.0.1 we can state in the release notes that 1.7.0 had this
bug and anyone who upgraded to it who wants to upgrade to 2.x must upgrade to
1.7.0.1 first.
> HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase meta*" and "*\n hbase namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
>
[GitHub] [hbase] clarax commented on a change in pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
clarax commented on a change in pull request #3415: URL: https://github.com/apache/hbase/pull/3415#discussion_r658223242 ## File path: hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java ## @@ -125,7 +125,7 @@ private int stepsPerRegion = 800; private long maxRunningTime = 30 * 1000 * 1; // 30 seconds. private int numRegionLoadsToRemember = 15; - private float minCostNeedBalance = 0.05f; + private float minCostNeedBalance = 0.025f; Review comment: Yes, as explained in the jira and the pr, we have to lower it for constant user experience. the old code almost always return max for table skew that inflates the total cost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369057#comment-17369057
]
Andrew Kyle Purtell edited comment on HBASE-26021 at 6/24/21, 7:23 PM:
---
[~vjasani] [~reidchan] [~zhangduo]
This is what I would recommend as the least bad option (all options are bad...)
# Withdraw the 1.7.0 release immediately. Delete from the mirrors.
# Apply the fix. I like your simple fix posted on this issue above [~vjasani]
but since we backported a change it makes more sense to backport its revert
too:
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
# Release 1.7.0.1.
The vote and review of the 1.7.0.1 RC should be fast given there have been few
commits since 1.7.0.
When we release 1.7.0.1 we can state in the release notes that 1.7.0 had this
bug and anyone who upgraded to it who wants to upgrade to 2.x must upgrade to
1.7.0.1 first.
was (Author: apurtell):
[~vjasani] [~reidchan] [~zhangduo]
This is what I would recommend as the least bad option (all options are bad...)
# Withdraw the 1.7.0 release immediately. Delete from the mirrors.
# Apply the fix.
# Release 1.7.0.1.
The vote and review of the 1.7.0.1 RC should be fast given there have been few
commits since 1.7.0.
When we release 1.7.0.1 we can state in the release notes that 1.7.0 had this
bug and anyone who upgraded to it who wants to upgrade to 2.x must upgrade to
1.7.0.1 first.
> HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase meta*" and "*\n hbase namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
>
[GitHub] [hbase] clarax commented on a change in pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
clarax commented on a change in pull request #3415:
URL: https://github.com/apache/hbase/pull/3415#discussion_r658222382
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/CostFunction.java
##
@@ -89,13 +89,14 @@ protected void regionMoved(int region, int oldServer, int
newServer) {
* @return The scaled value.
*/
protected static double scale(double min, double max, double value) {
-if (max <= min || value <= min) {
+if (max <= min || value <= min
+ || Math.abs(max - min) <= 0.01 || Math.abs(value - min) <= 0.01) {
Review comment:
Let me create a COST_EPSILON because I have seen quite wide range of
precision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369057#comment-17369057
]
Andrew Kyle Purtell commented on HBASE-26021:
-
[~vjasani] [~reidchan] [~zhangduo]
This is what I would recommend as the least bad option (all options are bad...)
# Withdraw the 1.7.0 release immediately. Delete from the mirrors.
# Apply the fix.
# Release 1.7.0.1.
The vote and review of the 1.7.0.1 RC should be fast given there have been few
commits since 1.7.0.
When we release 1.7.0.1 we can state in the release notes that 1.7.0 had this
bug and anyone who upgraded to it who wants to upgrade to 2.x must upgrade to
1.7.0.1 first.
> HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase meta*" and "*\n hbase namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R^Dmeta
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
>
[GitHub] [hbase] clarax commented on a change in pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
clarax commented on a change in pull request #3415:
URL: https://github.com/apache/hbase/pull/3415#discussion_r658220514
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -106,4 +88,32 @@ private static double getSum(double[] stats) {
}
return total;
}
+
+ /**
+ * Return the min skew of distribution
+ */
+ public static double getMinSkew(double total, double numServers) {
Review comment:
It is to convert the input from integer to double for computation in the
function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
clarax commented on a change in pull request #3415:
URL: https://github.com/apache/hbase/pull/3415#discussion_r658219316
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -106,4 +88,32 @@ private static double getSum(double[] stats) {
}
return total;
}
+
+ /**
+ * Return the min skew of distribution
+ */
+ public static double getMinSkew(double total, double numServers) {
Review comment:
yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on a change in pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
clarax commented on a change in pull request #3415:
URL: https://github.com/apache/hbase/pull/3415#discussion_r658207914
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -106,4 +88,32 @@ private static double getSum(double[] stats) {
}
return total;
}
+
+ /**
+ * Return the min skew of distribution
+ */
+ public static double getMinSkew(double total, double numServers) {
+double mean = total / numServers;
+// It's possible that there aren't enough regions to go around
+double min;
+if (numServers > total) {
Review comment:
This is the case when we have more nodes than regions, we will have
nodes without regions and it is balanced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] clarax commented on pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
clarax commented on pull request #3415: URL: https://github.com/apache/hbase/pull/3415#issuecomment-867879858 "values like minCostNeedBalance below are at 0.00 precision, so I think we should have an epsilon of at least 0.000 precision." Actually not really. this is aggregated deviation before scaling or divided by multiplier so it is at the precision of close to 1 or 0.1. But it is good to go lower to 0.001. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368776#comment-17368776
]
Viraj Jasani edited comment on HBASE-26021 at 6/24/21, 6:47 PM:
Here are HBase proto messages:
{code:java}
/**
* Table Schema
* Inspired by the rest TableSchema
*/
message TableSchema {
optional TableName table_name = 1;
repeated BytesBytesPair attributes = 2;
repeated ColumnFamilySchema column_families = 3;
repeated NameStringPair configuration = 4;
}
/** Denotes state of the table */
message TableState {
// Table's current state
enum State {
ENABLED = 0;
DISABLED = 1;
DISABLING = 2;
ENABLING = 3;
}
// This is the table's state.
required State state = 1;
required TableName table = 2;
optional uint64 timestamp = 3;
}
/** On HDFS representation of table state. */
message TableDescriptor {
required TableSchema schema = 1;
optional TableState.State state = 2 [ default = ENABLED ];
}
{code}
Serialization in FSTableDescriptors:
{code:java}
private static void writeTD(final FileSystem fs, final Path p, final
TableDescriptor htd)
throws IOException {
FSDataOutputStream out = fs.create(p, false);
try {
// We used to write this file out as a serialized HTD Writable followed by
two '\n's and then
// the toString version of HTD. Now we just write out the pb serialization.
out.write(htd.toByteArray());
} finally {
out.close();
}
}
{code}
Before this patch, we used to serialize "HTableDescriptor" in place of
"TableDescriptor" in above method (as [~apurtell] mentioned).
TableDescriptor/HTableDescriptor's toByteArray():
{code:java}
/**
* @return This instance serialized with pb with pb magic prefix
* @see #parseFrom(byte[])
*/
public byte [] toByteArray() {
return ProtobufUtil.prependPBMagic(convert().toByteArray());
}
{code}
Here is the main difference in both convert():
HTableDescriptor#convert converts TD to HBaseProtos.TableSchema (mentioned in
first code block above) whereas TableDescriptor#convert converts TD to
HBaseProtos.TableDescriptor (mentioned in first code block above). As per Proto
message definitions, HBaseProtos.TableDescriptor contains 2 fields:
HBaseProtos.TableSchema and HBaseProtos.TableState.State, so state is the
additional field.
HBase 2 deserializes only HBaseProtos.TableSchema as per this code:
{code:java}
/**
* @param bytes A pb serialized {@link ModifyableTableDescriptor} instance
* with pb magic prefix
* @return An instance of {@link ModifyableTableDescriptor} made from
* bytes
* @throws DeserializationException
* @see #toByteArray()
*/
private static TableDescriptor parseFrom(final byte[] bytes)
throws DeserializationException {
if (!ProtobufUtil.isPBMagicPrefix(bytes)) {
throw new DeserializationException("Expected PB encoded
ModifyableTableDescriptor");
}
int pblen = ProtobufUtil.lengthOfPBMagic();
HBaseProtos.TableSchema.Builder builder =
HBaseProtos.TableSchema.newBuilder();
try {
ProtobufUtil.mergeFrom(builder, bytes, pblen, bytes.length - pblen);
return ProtobufUtil.toTableDescriptor(builder.build());
} catch (IOException e) {
throw new DeserializationException(e);
}
}
{code}
HBase 2's HBase proto does not even have TableDescriptor message, this is odd.
This means that HBASE-7767 introduced HBaseProtos.TableDescriptor and later it
was reverted as well and now it's no longer being used.
Let me dig in why this was removed and for what reason. [~apurtell] [~bharathv]
Although we have released 1.7, but we might not be able to continue with Proto
incompatibility, specifically when HBase 1 has new TD class with it's own Proto
definition but on the other hand HBase 2 has it removed. (Edit: more details in
following comments)
-This brings the question of whether we should really have Protobuf
compatibility guidelines just like source comaptibility guidelines we follow
for backward compatible releases.-
{quote}Unless it actually prevents an upgrade from 1.6, which moots the release
and we will have to withdraw it.
{quote}
Let me test this as well today if possible (else tomorrow) but based on the
code, it seems we should not have any problem with this. This
[commit|https://github.com/apache/hbase/commit/431b8a5383b894381583bbb9ceef5911911b705c]
takes care of backward compatibility.
I assume deserialization should fail from 1.6 to 1.7 and 1.7 has backward
compat code present after DeserializationException is caught:
{code:java}
private static TableDescriptor readTableDescriptor(FileSystem fs, FileStatus
status,
boolean rewritePb) throws IOException {
int len = Ints.checkedCast(status.getLen());
byte [] content = new byte[len];
FSDataInputStream fsDataInputStream = fs.open(status.getPath());
try {
fsDataInputStream.readFully(content);
} finally {
fsDataInputStream.close();
}
TableDescriptor td =
[GitHub] [hbase] ndimiduk commented on a change in pull request #3415: HBASE-25739 TableSkewCostFunction need to use aggregated deviation
ndimiduk commented on a change in pull request #3415:
URL: https://github.com/apache/hbase/pull/3415#discussion_r658136085
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -106,4 +88,32 @@ private static double getSum(double[] stats) {
}
return total;
}
+
+ /**
+ * Return the min skew of distribution
+ */
+ public static double getMinSkew(double total, double numServers) {
Review comment:
Why are the input arguments `double`? Can there be a fractional amount
of either of these quantities?
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/BalancerClusterState.java
##
@@ -301,17 +307,28 @@ public String getRack(ServerName server) {
for (int i = 0; i < regionIndexToServerIndex.length; i++) {
if (regionIndexToServerIndex[i] >= 0) {
numRegionsPerServerPerTable[regionIndexToServerIndex[i]][regionIndexToTableIndex[i]]++;
+numRegionsPerTable[regionIndexToTableIndex[i]]++;
}
}
-numMaxRegionsPerTable = new int[numTables];
+// Avoid repeated computation for planning
+meanRegionsPerTable = new double[numTables];
+regionSkewByTable = new double[numTables];
+maxRegionSkewByTable = new double[numTables];
+minRegionSkewByTable = new double[numTables];
+
+for (int i = 0; i < numTables; i++) {
+ meanRegionsPerTable[i] = Double.valueOf(numRegionsPerTable[i]) /
numServers;
+ minRegionSkewByTable[i] +=
DoubleArrayCost.getMinSkew(numRegionsPerTable[i], numServers);
+ maxRegionSkewByTable[i] +=
DoubleArrayCost.getMaxSkew(numRegionsPerTable[i], numServers);
+}
+
for (int[] aNumRegionsPerServerPerTable : numRegionsPerServerPerTable) {
- for (tableIndex = 0; tableIndex < aNumRegionsPerServerPerTable.length;
tableIndex++) {
-if (aNumRegionsPerServerPerTable[tableIndex] >
numMaxRegionsPerTable[tableIndex]) {
- numMaxRegionsPerTable[tableIndex] =
aNumRegionsPerServerPerTable[tableIndex];
-}
+ for (int tableIdx = 0; tableIdx < aNumRegionsPerServerPerTable.length;
tableIdx++) {
+regionSkewByTable[tableIdx] +=
Math.abs(aNumRegionsPerServerPerTable[tableIdx]
+ - meanRegionsPerTable[tableIdx]);
}
-}
+ }
Review comment:
nit: white space.
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -106,4 +88,32 @@ private static double getSum(double[] stats) {
}
return total;
}
+
+ /**
+ * Return the min skew of distribution
+ */
+ public static double getMinSkew(double total, double numServers) {
Review comment:
Is `total` the "total number of regions in the cluster"?
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/DoubleArrayCost.java
##
@@ -106,4 +88,32 @@ private static double getSum(double[] stats) {
}
return total;
}
+
+ /**
+ * Return the min skew of distribution
+ */
+ public static double getMinSkew(double total, double numServers) {
+double mean = total / numServers;
+// It's possible that there aren't enough regions to go around
+double min;
+if (numServers > total) {
Review comment:
should this be `>=` ?
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -240,7 +240,8 @@ protected void loadConf(Configuration conf) {
curFunctionCosts = new double[costFunctions.size()];
tempFunctionCosts = new double[costFunctions.size()];
-LOG.info("Loaded config; maxSteps=" + maxSteps + ", stepsPerRegion=" +
stepsPerRegion +
+LOG.info("Loaded config; maxSteps=" + maxSteps + " ,runMaxSteps=" +
runMaxSteps,
Review comment:
nit: white space.
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/CostFunction.java
##
@@ -89,13 +89,14 @@ protected void regionMoved(int region, int oldServer, int
newServer) {
* @return The scaled value.
*/
protected static double scale(double min, double max, double value) {
-if (max <= min || value <= min) {
+if (max <= min || value <= min
+ || Math.abs(max - min) <= 0.01 || Math.abs(value - min) <= 0.01) {
Review comment:
values like `minCostNeedBalance` below are at 0.00 precision, so I think
we should have an epsilon of at least 0.000 precision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25966) Fix typo in NOTICE.vm
[
https://issues.apache.org/jira/browse/HBASE-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369000#comment-17369000
]
Hudson commented on HBASE-25966:
Results for branch branch-2
[build #285 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Fix typo in NOTICE.vm
> -
>
> Key: HBASE-25966
> URL: https://issues.apache.org/jira/browse/HBASE-25966
> Project: HBase
> Issue Type: Bug
> Components: build, community
>Affects Versions: 3.0.0-alpha-1, 1.4.0, 2.3.0, 1.7.0, 2.4.0, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> I noticed there's a typo in
> {{hbase-resource-bundle/src/main/resources/META-INF/NOTICE.vm}}:
> "bundled-boostrap" instead of "bundled-bootstrap".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26020) Split TestWALEntryStream.testDifferentCounts out
[
https://issues.apache.org/jira/browse/HBASE-26020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368999#comment-17368999
]
Hudson commented on HBASE-26020:
Results for branch branch-2
[build #285 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Split TestWALEntryStream.testDifferentCounts out
>
>
> Key: HBASE-26020
> URL: https://issues.apache.org/jira/browse/HBASE-26020
> Project: HBase
> Issue Type: Improvement
> Components: Replication, test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> It consumes too much time and may cause the whole UT to timeout.
> And in fact, it should be implemented as parameterized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25677) Server+table counters on each scan #nextRaw invocation becomes a bottleneck when heavy load
[
https://issues.apache.org/jira/browse/HBASE-25677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369002#comment-17369002
]
Hudson commented on HBASE-25677:
Results for branch branch-2
[build #285 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Server+table counters on each scan #nextRaw invocation becomes a bottleneck
> when heavy load
> ---
>
> Key: HBASE-25677
> URL: https://issues.apache.org/jira/browse/HBASE-25677
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 2.3.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.3.5, 2.4.3
>
>
> On a heavily loaded server mostly doing reads/scan, I saw that 90+% of
> handlers were BLOCKED in this fashion in thread dumps:
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=117,queue=17,port=16020" #161 daemon
> prio=5 os_prio=0 tid=0x7f748757f000 nid=0x73e9 waiting for monitor entry
> [0x7f74783e]
> java.lang.Thread.State: BLOCKED (on object monitor)
>at
> java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
>- waiting to lock <0x7f7647e3cc38> (a
> java.util.concurrent.ConcurrentHashMap$Node)
>at
> org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.getOrCreateTableMeter(MetricsTableQueryMeterImpl.java:80)
>at
> org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.updateTableReadQueryMeter(MetricsTableQueryMeterImpl.java:90)
>at
> org.apache.hadoop.hbase.regionserver.RegionServerTableMetrics.updateTableReadQueryMeter(RegionServerTableMetrics.java:89)
>at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServer.updateReadQueryMeter(MetricsRegionServer.java:274)
>at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6742)
>at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3319)
>- locked <0x7f896c0165a0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
>at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3566)
>at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858)
>at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393)
>at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
>at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
>at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> {code}
> It kept up for good periods of time.
> I saw it to a leser extent on other servers, with less load.
> These RS had 400+ Regions a good few of which were serving out scan reads;
> the server was doing ~1M hits a second. In this scenario, I saw the above
> bottleneck.
> Looking at it, it came in w/ when the parent issue feature was added. There
> are these read counts and then there were also write counts. The write counts
> are mostly batch-based. Let me do same thing here for the read update the
> central server+table count after scan is done rather than per invocation of
> #nextRaw.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26013) Get operations readRows metrics becomes zero after HBASE-25677
[
https://issues.apache.org/jira/browse/HBASE-26013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369001#comment-17369001
]
Hudson commented on HBASE-26013:
Results for branch branch-2
[build #285 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Get operations readRows metrics becomes zero after HBASE-25677
> --
>
> Key: HBASE-26013
> URL: https://issues.apache.org/jira/browse/HBASE-26013
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 1.7.1, 2.4.5
>
>
> After HBASE-25677, Server+table counters on each scan are extracted from
> #nextRaw to rsServices scan. In this case, the get operation will not count
> the read rows. So that the readRows metrics becomes zero. Should add counter
> in metricsUpdateForGet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26019) Remove reflections used in HBaseConfiguration.getPassword()
[
https://issues.apache.org/jira/browse/HBASE-26019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368998#comment-17368998
]
Hudson commented on HBASE-26019:
Results for branch branch-2
[build #285 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/285//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Remove reflections used in HBaseConfiguration.getPassword()
> ---
>
> Key: HBASE-26019
> URL: https://issues.apache.org/jira/browse/HBASE-26019
> Project: HBase
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> HBaseConfiguration.getPassword() uses Hadoop API Configuration.getPassword().
> The API was added in Hadoop 2.6.0. Reflection was used to access the API.
> It's time to remove the reflection and invoke the API directly. (HBase 3.0 as
> well as 2.x too)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368973#comment-17368973
]
Viraj Jasani edited comment on HBASE-26021 at 6/24/21, 5:11 PM:
Based on the history, this
[commit|https://github.com/apache/hbase/commit/dec0ec229070465ce5a1f7381366a82278149f06]
(HBASE-13016) cleaned up TableState.State from HBaseProto.TableDescriptor and
finally this
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
removed TableDescriptor itself from HBaseProtos (HBASE-15467: Remove 1.x/2.0
TableDescriptor incompatibility). And as part of this commit itself, entire
TableDescriptor class also has been removed.
[~apurtell] [~bharathv] Thoughts on backporting
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
to branch-1? It's bit weird to resolve this. We backported something to
branch-1 which itself was reverted from branch-2 in order to maintain 1.x/2.x
compatibility.
was (Author: vjasani):
Based on the history, this
[commit|https://github.com/apache/hbase/commit/dec0ec229070465ce5a1f7381366a82278149f06]
(HBASE-13016) cleaned up TableState.State from HBaseProto.TableDescriptor and
finally this
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
removed TableDescriptor itself from HBaseProtos (HBASE-15467: Remove 1.x/2.0
TableDescriptor incompatibility). And as part of this commit itself, entire
TableDescriptor class also has been removed.
[~apurtell] [~bharathv] Thoughts on backporting
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
to branch-1? It's bit weird to resolve this. We backported something to
branch-1 which itself was removed from branch-2 in order to maintain 1.x/2.x
compatibility.
> HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase meta*" and "*\n hbase namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
>
[jira] [Comment Edited] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368973#comment-17368973
]
Viraj Jasani edited comment on HBASE-26021 at 6/24/21, 5:09 PM:
Based on the history, this
[commit|https://github.com/apache/hbase/commit/dec0ec229070465ce5a1f7381366a82278149f06]
(HBASE-13016) cleaned up TableState.State from HBaseProto.TableDescriptor and
finally this
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
removed TableDescriptor itself from HBaseProtos (HBASE-15467: Remove 1.x/2.0
TableDescriptor incompatibility). And as part of this commit itself, entire
TableDescriptor class also has been removed.
[~apurtell] [~bharathv] Thoughts on backporting
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
to branch-1? It's bit weird to resolve this. We backported something to
branch-1 which itself was removed from branch-2 in order to maintain 1.x/2.x
compatibility.
was (Author: vjasani):
Based on the history, this
[commit|https://github.com/apache/hbase/commit/dec0ec229070465ce5a1f7381366a82278149f06]
(HBASE-13016) cleaned up TableState.State from HBaseProto.TableDescriptor and
finally this
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
removed TableDescriptor itself from HBaseProtos (HBASE-15467: Remove 1.x/2.0
TableDescriptor incompatibility).
> HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase meta*" and "*\n hbase namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
>
[jira] [Commented] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368973#comment-17368973
]
Viraj Jasani commented on HBASE-26021:
--
Based on the history, this
[commit|https://github.com/apache/hbase/commit/dec0ec229070465ce5a1f7381366a82278149f06]
(HBASE-13016) cleaned up TableState.State from HBaseProto.TableDescriptor and
finally this
[commit|https://github.com/apache/hbase/commit/bdb0cc8808af7c3d08af4a506f34b8341726b58e]
removed TableDescriptor itself from HBaseProtos (HBASE-15467: Remove 1.x/2.0
TableDescriptor incompatibility).
> HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase meta*" and "*\n hbase namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R^Dmeta
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
>
[jira] [Assigned] (HBASE-26030) hbase-cleanup.sh did not clean the wal dir if hbase.wal.dir configured individually
[ https://issues.apache.org/jira/browse/HBASE-26030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mokai reassigned HBASE-26030: - Assignee: mokai > hbase-cleanup.sh did not clean the wal dir if hbase.wal.dir configured > individually > --- > > Key: HBASE-26030 > URL: https://issues.apache.org/jira/browse/HBASE-26030 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 2.2.3 >Reporter: mokai >Assignee: mokai >Priority: Major > > If 'hbase.wal.dir' and 'hbase.rootdir' are configured to different filesystem > or directories, hbase clean script will clean the directory of > 'hbase.rootdir' only, the directory of 'hbase.wal.dir' will be left. > The left direcotries 'WAls,' 'oldWALs', and recovered.edits under region dir > should be cleaned also. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25739) TableSkewCostFunction need to use aggregated deviation
[ https://issues.apache.org/jira/browse/HBASE-25739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368964#comment-17368964 ] Nick Dimiduk commented on HBASE-25739: -- bq. Because of the fix, the default 0.05 minCostNeedBalance will not quite work. As a gap-stopper before I check in auto-tuning threshold, should I just reduce the default value? So people won't be caught off guard? The broken TableSkewCostFunction artificially inflate the total cost. So if the fix is in and we don't change threshold, people will be badly surprised that balancer gets stuck. This sounds like a case where we have to implement both changes together, nor neither of them. In that case, we have to leave them both out of any patch releases. > TableSkewCostFunction need to use aggregated deviation > -- > > Key: HBASE-25739 > URL: https://issues.apache.org/jira/browse/HBASE-25739 > Project: HBase > Issue Type: Sub-task > Components: Balancer, master >Reporter: Clara Xiong >Assignee: Clara Xiong >Priority: Major > Attachments: > TEST-org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerBalanceCluster.xml, > > org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerBalanceCluster.txt > > > TableSkewCostFunction uses the sum of the max deviation region per server for > all tables as the measure of unevenness. It doesn't work in a very common > scenario in operations. Say we have 100 regions on 50 nodes, two on each. We > add 50 new nodes and they have 0 each. The max deviation from the mean is 1, > compared to 99 in the worst case scenario of 100 regions on a single server. > The normalized cost is 1/99 = 0.011 < default threshold of 0.05. Balancer > wouldn't move. The proposal is to use aggregated deviation of the count per > region server to detect this scenario, generating a cost of 100/198 = 0.5 in > this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #2596: Backport "HBASE-24419 Normalizer merge plans should consider more than 2 region…" to branch-2
ndimiduk commented on a change in pull request #2596:
URL: https://github.com/apache/hbase/pull/2596#discussion_r658109266
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -315,35 +316,60 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
* towards target average or target region count.
*/
private List computeMergeNormalizationPlans(final
NormalizeContext ctx) {
-if (ctx.getTableRegions().size() < minRegionCount) {
+if (isEmpty(ctx.getTableRegions()) || ctx.getTableRegions().size() <
minRegionCount) {
LOG.debug("Table {} has {} regions, required min number of regions for
normalizer to run"
+ " is {}, not computing merge plans.", ctx.getTableName(),
ctx.getTableRegions().size(),
minRegionCount);
return Collections.emptyList();
}
-final double avgRegionSizeMb = ctx.getAverageRegionSizeMb();
+final long avgRegionSizeMb = (long) ctx.getAverageRegionSizeMb();
+if (avgRegionSizeMb < mergeMinRegionSizeMb) {
Review comment:
I haven't done a JIRA/git audit recently, but the normalizer on master
and branch-2 should be identical. branch-2.4 has most of the big improvements
that I'm aware of, including support for rate-limiting.
Be advised that its settings for merging are still very coarse. I've been
using it recently on large tables containing large regions and find that the
2x-off-average is not practical. It needs more tuning and possibly new settings
in order to work well in a large ranges of table topologies.
Depending on how old of an HBase you're targeting, there was a bug where it
would always force-merge, which could cause errors in meta. I advise against
running normalizer before 2.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25677) Server+table counters on each scan #nextRaw invocation becomes a bottleneck when heavy load
[
https://issues.apache.org/jira/browse/HBASE-25677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368954#comment-17368954
]
Hudson commented on HBASE-25677:
Results for branch branch-2.3
[build #243 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/243/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/243/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/243/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/243/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/243/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Server+table counters on each scan #nextRaw invocation becomes a bottleneck
> when heavy load
> ---
>
> Key: HBASE-25677
> URL: https://issues.apache.org/jira/browse/HBASE-25677
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 2.3.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0, 2.3.5, 2.4.3
>
>
> On a heavily loaded server mostly doing reads/scan, I saw that 90+% of
> handlers were BLOCKED in this fashion in thread dumps:
> {code}
> "RpcServer.default.FPBQ.Fifo.handler=117,queue=17,port=16020" #161 daemon
> prio=5 os_prio=0 tid=0x7f748757f000 nid=0x73e9 waiting for monitor entry
> [0x7f74783e]
> java.lang.Thread.State: BLOCKED (on object monitor)
>at
> java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
>- waiting to lock <0x7f7647e3cc38> (a
> java.util.concurrent.ConcurrentHashMap$Node)
>at
> org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.getOrCreateTableMeter(MetricsTableQueryMeterImpl.java:80)
>at
> org.apache.hadoop.hbase.regionserver.MetricsTableQueryMeterImpl.updateTableReadQueryMeter(MetricsTableQueryMeterImpl.java:90)
>at
> org.apache.hadoop.hbase.regionserver.RegionServerTableMetrics.updateTableReadQueryMeter(RegionServerTableMetrics.java:89)
>at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServer.updateReadQueryMeter(MetricsRegionServer.java:274)
>at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6742)
>at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3319)
>- locked <0x7f896c0165a0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
>at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3566)
>at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44858)
>at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393)
>at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
>at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
>at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> {code}
> It kept up for good periods of time.
> I saw it to a leser extent on other servers, with less load.
> These RS had 400+ Regions a good few of which were serving out scan reads;
> the server was doing ~1M hits a second. In this scenario, I saw the above
> bottleneck.
> Looking at it, it came in w/ when the parent issue feature was added. There
> are these read counts and then there were also write counts. The write counts
> are mostly batch-based. Let me do same thing here for the read update the
> central server+table count after scan is done rather than per invocation of
> #nextRaw.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26013) Get operations readRows metrics becomes zero after HBASE-25677
[
https://issues.apache.org/jira/browse/HBASE-26013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368953#comment-17368953
]
Hudson commented on HBASE-26013:
Results for branch branch-2.3
[build #243 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/243/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/243/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/243/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/243/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/243/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Get operations readRows metrics becomes zero after HBASE-25677
> --
>
> Key: HBASE-26013
> URL: https://issues.apache.org/jira/browse/HBASE-26013
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.6, 1.7.1, 2.4.5
>
>
> After HBASE-25677, Server+table counters on each scan are extracted from
> #nextRaw to rsServices scan. In this case, the get operation will not count
> the read rows. So that the readRows metrics becomes zero. Should add counter
> in metricsUpdateForGet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] clarax commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
clarax commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r658088094
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -303,54 +302,54 @@ private String getBalanceReason(double total, double
sumMultiplier) {
boolean needsBalance(TableName tableName, BalancerClusterState cluster) {
ClusterLoadState cs = new ClusterLoadState(cluster.clusterState);
if (cs.getNumServers() < MIN_SERVER_BALANCE) {
- if (LOG.isDebugEnabled()) {
-LOG.debug("Not running balancer because only " + cs.getNumServers()
+ LOG.info("Not running balancer because only " + cs.getNumServers()
+ " active regionserver(s)");
- }
sendRejectionReasonToRingBuffer(() -> "The number of RegionServers " +
cs.getNumServers() +
" < MIN_SERVER_BALANCE(" + MIN_SERVER_BALANCE + ")", null);
return false;
}
if (areSomeRegionReplicasColocated(cluster)) {
+ LOG.info("Running balancer because at least one server hosts replicas of
the same region.");
return true;
}
if (idleRegionServerExist(cluster)){
+ LOG.info("Running balancer because cluster has idle server(s).");
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
+}
+if (sumMultiplier <= 0)
+{
+ LOG.error("At least one cost function needs a multiplier > 0");
+ return false;
Review comment:
I am afraid it would be too wordy. It is well documented, logged and
used already. The enhanced logging logs enough about cost functions and their
multipliers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (HBASE-26030) hbase-cleanup.sh did not clean the wal dir if hbase.wal.dir configured individually
mokai created HBASE-26030: - Summary: hbase-cleanup.sh did not clean the wal dir if hbase.wal.dir configured individually Key: HBASE-26030 URL: https://issues.apache.org/jira/browse/HBASE-26030 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 2.2.3 Reporter: mokai If 'hbase.wal.dir' and 'hbase.rootdir' are configured to different filesystem or directories, hbase clean script will clean the directory of 'hbase.rootdir' only, the directory of 'hbase.wal.dir' will be left. The left direcotries 'WAls,' 'oldWALs', and recovered.edits under region dir should be cleaned also. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 closed pull request #3424: HBASE-26029 It is not reliable to use nodeDeleted event to track regi…
Apache9 closed pull request #3424: URL: https://github.com/apache/hbase/pull/3424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #3356: HBASE-25973 Balancer should explain progress in a better way in log
busbey commented on a change in pull request #3356:
URL: https://github.com/apache/hbase/pull/3356#discussion_r658053232
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -303,54 +302,54 @@ private String getBalanceReason(double total, double
sumMultiplier) {
boolean needsBalance(TableName tableName, BalancerClusterState cluster) {
ClusterLoadState cs = new ClusterLoadState(cluster.clusterState);
if (cs.getNumServers() < MIN_SERVER_BALANCE) {
- if (LOG.isDebugEnabled()) {
-LOG.debug("Not running balancer because only " + cs.getNumServers()
+ LOG.info("Not running balancer because only " + cs.getNumServers()
+ " active regionserver(s)");
- }
sendRejectionReasonToRingBuffer(() -> "The number of RegionServers " +
cs.getNumServers() +
" < MIN_SERVER_BALANCE(" + MIN_SERVER_BALANCE + ")", null);
return false;
}
if (areSomeRegionReplicasColocated(cluster)) {
+ LOG.info("Running balancer because at least one server hosts replicas of
the same region.");
return true;
}
if (idleRegionServerExist(cluster)){
+ LOG.info("Running balancer because cluster has idle server(s).");
return true;
}
+sumMultiplier = 0.0f;
double total = 0.0;
-float sumMultiplier = 0.0f;
for (CostFunction c : costFunctions) {
float multiplier = c.getMultiplier();
- if (multiplier <= 0) {
-LOG.trace("{} not needed because multiplier is <= 0",
c.getClass().getSimpleName());
-continue;
- }
+ double cost = c.cost();
if (!c.isNeeded()) {
LOG.trace("{} not needed", c.getClass().getSimpleName());
continue;
}
+ total += cost * multiplier;
sumMultiplier += multiplier;
- total += c.cost() * multiplier;
+}
+if (sumMultiplier <= 0)
+{
+ LOG.error("At least one cost function needs a multiplier > 0");
+ return false;
Review comment:
give a pointer in this message about what configs they should be looking
for
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] busbey commented on pull request #3199: fix(docs): corrects typo in project documentation
busbey commented on pull request #3199: URL: https://github.com/apache/hbase/pull/3199#issuecomment-867725708 Hello! As a project we use the ASF Jira instance to track work. Please create an account and let us know what it is. https://issues.apache.org/jira/projects/HBASE/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26017) hbase performance evaluation tool could not support datasize more than 2048g
[ https://issues.apache.org/jira/browse/HBASE-26017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-26017: Status: Patch Available (was: Open) > hbase performance evaluation tool could not support datasize more than 2048g > - > > Key: HBASE-26017 > URL: https://issues.apache.org/jira/browse/HBASE-26017 > Project: HBase > Issue Type: Bug > Components: PE >Affects Versions: 2.4.4, 2.3.2, 2.1.0 >Reporter: dingwei2019 >Assignee: dingwei2019 >Priority: Minor > > in our daily test, we may hope to test more datasize than 2048g, when we set > --size more than 2048g, pe print abnormal message like this: > [TestClient-1] hbase.PerformanceEvaluation: Start class > org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest at offset > -1138166308 for -21474836 rows > > this is due to variable totalRows in TestOptions defined by int(-2147483648 > --- 2147483647), One GB is 1048576(1024*1024) by default. The max value of > totalRow is 2147483647, in this condition, we may write not larger than > 2147483647/1048576 = 2047.999G. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-26017) hbase performance evaluation tool could not support datasize more than 2048g
[ https://issues.apache.org/jira/browse/HBASE-26017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-26017: --- Assignee: dingwei2019 > hbase performance evaluation tool could not support datasize more than 2048g > - > > Key: HBASE-26017 > URL: https://issues.apache.org/jira/browse/HBASE-26017 > Project: HBase > Issue Type: Bug > Components: PE >Affects Versions: 2.1.0, 2.3.2, 2.4.4 >Reporter: dingwei2019 >Assignee: dingwei2019 >Priority: Minor > > in our daily test, we may hope to test more datasize than 2048g, when we set > --size more than 2048g, pe print abnormal message like this: > [TestClient-1] hbase.PerformanceEvaluation: Start class > org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest at offset > -1138166308 for -21474836 rows > > this is due to variable totalRows in TestOptions defined by int(-2147483648 > --- 2147483647), One GB is 1048576(1024*1024) by default. The max value of > totalRow is 2147483647, in this condition, we may write not larger than > 2147483647/1048576 = 2047.999G. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] busbey commented on pull request #3418: fix pretty tool print header missing last indexblock and bloomchunk
busbey commented on pull request #3418: URL: https://github.com/apache/hbase/pull/3418#issuecomment-867716724 Hi! The HBase project uses jira to track work on the project. Please file a problem description here: http://issues.apache.org/jira/browse/HBASE and then edit your PR and commit subject line to reference the issue id. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3425: HBASE-25991 Do compaction on compaction server
Apache-HBase commented on pull request #3425: URL: https://github.com/apache/hbase/pull/3425#issuecomment-867698555 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-25714 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | HBASE-25714 passed | | +1 :green_heart: | compile | 1m 1s | HBASE-25714 passed | | +1 :green_heart: | shadedjars | 7m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | HBASE-25714 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 7m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 177m 38s | hbase-server in the patch failed. | | | | 208m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3425 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c395792e22c0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-25714 / da0fa3000e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/testReport/ | | Max. process+thread count | 3814 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3425: HBASE-25991 Do compaction on compaction server
Apache-HBase commented on pull request #3425: URL: https://github.com/apache/hbase/pull/3425#issuecomment-867692443 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-25714 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 41s | HBASE-25714 passed | | +1 :green_heart: | compile | 1m 16s | HBASE-25714 passed | | +1 :green_heart: | shadedjars | 7m 54s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | HBASE-25714 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 31s | the patch passed | | +1 :green_heart: | compile | 1m 16s | the patch passed | | +1 :green_heart: | javac | 1m 16s | the patch passed | | +1 :green_heart: | shadedjars | 7m 57s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 166m 53s | hbase-server in the patch passed. | | | | 199m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3425 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ced734610386 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-25714 / da0fa3000e | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/testReport/ | | Max. process+thread count | 3826 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3424: HBASE-26029 It is not reliable to use nodeDeleted event to track regi…
Apache-HBase commented on pull request #3424: URL: https://github.com/apache/hbase/pull/3424#issuecomment-867653029 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 15s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 9m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 13s | the patch passed | | +1 :green_heart: | compile | 1m 21s | the patch passed | | +1 :green_heart: | javac | 1m 21s | the patch passed | | +1 :green_heart: | shadedjars | 9m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 43s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 190m 58s | hbase-server in the patch passed. | | | | 226m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3424/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3424 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cef1af150cd5 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3424/1/testReport/ | | Max. process+thread count | 2330 (vs. ulimit of 3) | | modules | C: hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3424/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26020) Split TestWALEntryStream.testDifferentCounts out
[
https://issues.apache.org/jira/browse/HBASE-26020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368841#comment-17368841
]
Hudson commented on HBASE-26020:
Results for branch master
[build #331 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Split TestWALEntryStream.testDifferentCounts out
>
>
> Key: HBASE-26020
> URL: https://issues.apache.org/jira/browse/HBASE-26020
> Project: HBase
> Issue Type: Improvement
> Components: Replication, test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> It consumes too much time and may cause the whole UT to timeout.
> And in fact, it should be implemented as parameterized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26019) Remove reflections used in HBaseConfiguration.getPassword()
[
https://issues.apache.org/jira/browse/HBASE-26019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368840#comment-17368840
]
Hudson commented on HBASE-26019:
Results for branch master
[build #331 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove reflections used in HBaseConfiguration.getPassword()
> ---
>
> Key: HBASE-26019
> URL: https://issues.apache.org/jira/browse/HBASE-26019
> Project: HBase
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> HBaseConfiguration.getPassword() uses Hadoop API Configuration.getPassword().
> The API was added in Hadoop 2.6.0. Reflection was used to access the API.
> It's time to remove the reflection and invoke the API directly. (HBase 3.0 as
> well as 2.x too)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25966) Fix typo in NOTICE.vm
[
https://issues.apache.org/jira/browse/HBASE-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368842#comment-17368842
]
Hudson commented on HBASE-25966:
Results for branch master
[build #331 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix typo in NOTICE.vm
> -
>
> Key: HBASE-25966
> URL: https://issues.apache.org/jira/browse/HBASE-25966
> Project: HBase
> Issue Type: Bug
> Components: build, community
>Affects Versions: 3.0.0-alpha-1, 1.4.0, 2.3.0, 1.7.0, 2.4.0, 2.5.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> I noticed there's a typo in
> {{hbase-resource-bundle/src/main/resources/META-INF/NOTICE.vm}}:
> "bundled-boostrap" instead of "bundled-bootstrap".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25934) Add username for RegionScannerHolder
[
https://issues.apache.org/jira/browse/HBASE-25934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368839#comment-17368839
]
Hudson commented on HBASE-25934:
Results for branch master
[build #331 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/331/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add username for RegionScannerHolder
>
>
> Key: HBASE-25934
> URL: https://issues.apache.org/jira/browse/HBASE-25934
> Project: HBase
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> This JIRA[HBASE-25542|https://issues.apache.org/jira/browse/HBASE-25542] has
> added part of the client information before, we can also add username for
> RegionScannerHolder.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
[ https://issues.apache.org/jira/browse/HBASE-26029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368830#comment-17368830 ] Duo Zhang commented on HBASE-26029: --- After talkling with [~sunxin] offline, we think that the most clear way to fix this, is to add a step for SCP to assign the replication queue of the dead server to different region servers. > It is not reliable to use nodeDeleted event to track region server's death > -- > > Key: HBASE-26029 > URL: https://issues.apache.org/jira/browse/HBASE-26029 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > > When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, > where a region server up and down between two sync requests, then we can not > know the death of the region server. > https://github.com/apache/hbase/pull/3405#discussion_r656720923 > This is a valid point, and when thinking of a solution, I noticed that, the > current zk iplementation has the same problem. Notice that, a watcher on zk > can only be triggered once, so after zk triggers the watcher, and before you > set a new watcher, it is possible that a region server is up and down, and > you will miss the nodeDeleted event for this region server. > I think, the general approach here, which could works for both master based > and zk based replication tracker is that, we should not rely on the tracker > to tell you which region server is dead. Instead, we just provide the list of > live regionservers, and the upper layer should compare this list with the > expected list(for replication, the list should be gotten by listing > replicators), to detect the dead region servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3424: HBASE-26029 It is not reliable to use nodeDeleted event to track regi…
Apache-HBase commented on pull request #3424: URL: https://github.com/apache/hbase/pull/3424#issuecomment-867605188 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 26s | master passed | | +1 :green_heart: | compile | 1m 32s | master passed | | +1 :green_heart: | shadedjars | 8m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 0s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 10s | the patch passed | | +1 :green_heart: | compile | 1m 31s | the patch passed | | +1 :green_heart: | javac | 1m 31s | the patch passed | | +1 :green_heart: | shadedjars | 8m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 32s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 124m 5s | hbase-server in the patch passed. | | | | 157m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3424/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3424 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b72871d70cd3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d11dc81721 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3424/1/testReport/ | | Max. process+thread count | 3934 (vs. ulimit of 3) | | modules | C: hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3424/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368796#comment-17368796
]
Viraj Jasani edited comment on HBASE-26021 at 6/24/21, 12:13 PM:
-
Here is the patch that resolves this (de)serialization issue, I have confirmed
this. The only thing I am worried about is what else could be compromised with
Proto incompatibilities with missing Proto message.
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
index c275f00c72..a2a9b0df0f 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
@@ -96,7 +96,8 @@ public class TableDescriptor {
* @see #parseFrom(byte[])
*/
public byte [] toByteArray() {
- return ProtobufUtil.prependPBMagic(convert().toByteArray());
+ return ProtobufUtil
+ .prependPBMagic(hTableDescriptor.convert().toByteArray());
}
{code}
This patch works for HBase 1.7 as well, because as I mentioned above, hbase-1
takes care of backward compatibility by catching DeserializationException and
trying to deserialize to old TD i.e HTableDescriptor. Confirmed this as well.
What I believe we should consider path forward for this issue:
# Apply this patch and consider rolling out 1.8 release for anyone looking
forward to upgrade from HBase 1 to 2.
# Thoughts on catching any other missing edge cases of HBASE-7767 backport. I
tried generating more traffic in my local testing but it might not be enough.
And I think as long as backport helps MasterRegistry's work to land on
branch-1, we are good and we might not want to consider rework on it.
Edit: From the code, I can see HBaseProtos.TableDescriptor is not used anywhere
else other than the use-case we are focusing on. Other than introducing
TableDescriptor and TableState in HBase proto, I don't see any changes in proto
of big concern. The patch also has changes in Master and Zookeeper protos but
they don't seem problematic from their usage.
I think we should be good with the above patch once we have enough +1 and we
can soon plan for new release from branch-1.
was (Author: vjasani):
Here is the patch that resolves this (de)serialization issue, I have confirmed
this. The only thing I am worried about is what else could be compromised with
Proto incompatibilities with missing Proto message.
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
index c275f00c72..a2a9b0df0f 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
@@ -96,7 +96,8 @@ public class TableDescriptor {
* @see #parseFrom(byte[])
*/
public byte [] toByteArray() {
- return ProtobufUtil.prependPBMagic(convert().toByteArray());
+ return ProtobufUtil
+ .prependPBMagic(hTableDescriptor.convert().toByteArray());
}
{code}
This patch works for HBase 1.7 as well, because as I mentioned above, hbase-1
takes care of backward compatibility by catching DeserializationException and
trying to deserialize to old TD i.e HTableDescriptor. Confirmed this as well.
What I believe we should consider path forward for this issue:
# Apply this patch and consider rolling out 1.8 release for anyone looking
forward to upgrade from HBase 1 to 2.
# Thoughts on catching any missing edge cases of HBASE-7767 backport. I think
as long as backport helps MasterRegistry's work to land on branch-1, we are
good and we might not want to consider rework on it. I am already running PE as
of now in my local testing to generate more traffic and add millions of rows
and scanning them, but this might not be enough and we could plan for rigorous
testing on big cluster.
Edit: From the code, I can see HBaseProtos.TableDescriptor is not used anywhere
else other than the usecase we are focusing on. Other than introducing
TableDescriptor and TableState in HBase proto, I don't see any changes in proto
of big concern. The patch also has changes in Master and Zookeeper protos but
they don't seem problematic from their usage.
I think we should be good with the above patch once we have enough +1 and we
can soon plan for new release from branch-1.
> HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
>
[GitHub] [hbase] Apache-HBase commented on pull request #3425: HBASE-25991 Do compaction on compaction server
Apache-HBase commented on pull request #3425: URL: https://github.com/apache/hbase/pull/3425#issuecomment-867585444 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-25714 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | HBASE-25714 passed | | +1 :green_heart: | compile | 3m 13s | HBASE-25714 passed | | +1 :green_heart: | checkstyle | 1m 6s | HBASE-25714 passed | | +1 :green_heart: | spotbugs | 2m 8s | HBASE-25714 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 3m 13s | the patch passed | | +1 :green_heart: | javac | 3m 13s | the patch passed | | -0 :warning: | checkstyle | 1m 6s | hbase-server: The patch generated 36 new + 98 unchanged - 2 fixed = 134 total (was 100) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 18m 4s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | -1 :x: | spotbugs | 2m 17s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 47m 10s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | org.apache.hadoop.hbase.compactionserver.CompactionTask defines compareTo(CompactionTask) and uses Object.equals() At CompactionTask.java:Object.equals() At CompactionTask.java:[lines 54-58] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3425 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux dff38e646e74 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-25714 / da0fa3000e | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/artifact/yetus-general-check/output/whitespace-eol.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3425/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368796#comment-17368796
]
Viraj Jasani edited comment on HBASE-26021 at 6/24/21, 12:04 PM:
-
Here is the patch that resolves this (de)serialization issue, I have confirmed
this. The only thing I am worried about is what else could be compromised with
Proto incompatibilities with missing Proto message.
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
index c275f00c72..a2a9b0df0f 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
@@ -96,7 +96,8 @@ public class TableDescriptor {
* @see #parseFrom(byte[])
*/
public byte [] toByteArray() {
- return ProtobufUtil.prependPBMagic(convert().toByteArray());
+ return ProtobufUtil
+ .prependPBMagic(hTableDescriptor.convert().toByteArray());
}
{code}
This patch works for HBase 1.7 as well, because as I mentioned above, hbase-1
takes care of backward compatibility by catching DeserializationException and
trying to deserialize to old TD i.e HTableDescriptor. Confirmed this as well.
What I believe we should consider path forward for this issue:
# Apply this patch and consider rolling out 1.8 release for anyone looking
forward to upgrade from HBase 1 to 2.
# Thoughts on catching any missing edge cases of HBASE-7767 backport. I think
as long as backport helps MasterRegistry's work to land on branch-1, we are
good and we might not want to consider rework on it. I am already running PE as
of now in my local testing to generate more traffic and add millions of rows
and scanning them, but this might not be enough and we could plan for rigorous
testing on big cluster.
Edit: From the code, I can see HBaseProtos.TableDescriptor is not used anywhere
else other than the usecase we are focusing on. Other than introducing
TableDescriptor and TableState in HBase proto, I don't see any changes in proto
of big concern. The patch also has changes in Master and Zookeeper protos but
they don't seem problematic from their usage.
I think we should be good with the above patch once we have enough +1 and we
can soon plan for new release from branch-1.
was (Author: vjasani):
Here is the patch that resolves this (de)serialization issue, I have confirmed
this. The only thing I am worried about is what else could be compromised with
Proto incompatibilities with missing Proto message.
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
index c275f00c72..a2a9b0df0f 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
@@ -96,7 +96,8 @@ public class TableDescriptor {
* @see #parseFrom(byte[])
*/
public byte [] toByteArray() {
- return ProtobufUtil.prependPBMagic(convert().toByteArray());
+ return ProtobufUtil
+ .prependPBMagic(hTableDescriptor.convert().toByteArray());
}
{code}
This patch works for HBase 1.7 as well, because as I mentioned above, hbase-1
takes care of backward compatibility by catching DeserializationException and
trying to deserialize to old TD i.e HTableDescriptor. Confirmed this as well.
What I believe we should consider path forward for this issue:
# Apply this patch and consider rolling out 1.8 release for anyone looking
forward to upgrade from HBase 1 to 2.
# Thoughts on catching any missing edge cases of HBASE-7767 backport. I think
as long as backport helps MasterRegistry's work to land on branch-1, we are
good but looks like we requires rigorous testing with this patch. I am already
running pe as of now in my local testing to generate more traffic and add
millions of rows and scanning them, but this might not be enough.
> HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase meta*" and "*\n hbase
[jira] [Updated] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated HBASE-26021:
-
Description:
As of today, if we bring up HBase cluster using branch-1 and upgrade to
branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
seems to be producing "*\n hbase meta*" and "*\n hbase namespace*"
{code:java}
2021-06-22 00:05:56,611 INFO
[RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
2021-06-22 00:05:56,648 INFO
[RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
regionserver.RSRpcServices: Open
hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
2021-06-22 00:05:56,759 ERROR
[RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025] ipc.RpcServer:
Unexpected throwable object
java.lang.IllegalArgumentException: Illegal character <
> at 0. Namespaces may only contain 'alphanumeric characters' from any language
> and digits:
^Ehbase^R namespace
at
org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
at
org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
at org.apache.hadoop.hbase.TableName.(TableName.java:348)
at
org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
at
org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
at
org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
at
org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
2021-06-22 00:05:56,759 ERROR
[RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025] ipc.RpcServer:
Unexpected throwable object
java.lang.IllegalArgumentException: Illegal character <
> at 0. Namespaces may only contain 'alphanumeric characters' from any language
> and digits:
^Ehbase^R^Dmeta
at
org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
at
org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
at org.apache.hadoop.hbase.TableName.(TableName.java:348)
at
org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
at
org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
at
org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
at
org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
[jira] [Updated] (HBASE-26021) HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated HBASE-26021:
-
Summary: HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
(was: HBase 1.7 to 2.4 upgrade issue)
> HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
> --
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.7.0, 2.4.4
>Reporter: Viraj Jasani
>Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase:\n meta*" and "*\n hbase:\n namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R^Dmeta
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
>
