[GitHub] [hbase] ragarkar commented on pull request #5210: HBASE-27820: HBase is not starting due to Jersey library conflicts wi…
ragarkar commented on PR #5210: URL: https://github.com/apache/hbase/pull/5210#issuecomment-1539450137 Tests are failing with the following stacktrace: [ERROR] org.apache.hadoop.hbase.mapreduce.TestHBaseMRTestingUtility.testMRYarnConfigsPopulation Time elapsed: 0.214 s <<< ERROR! java.lang.NoClassDefFoundError: javax/ws/rs/WebApplicationException at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:711) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:706) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:719) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at org.apache.hadoop.yarn.server.MiniYARNCluster.setNonHARMConfigurationWithEphemeralPorts(MiniYARNCluster.java:338) at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:306) at org.apache.hadoop.mapreduce.v2.MiniMRYarnCluster.serviceInit(MiniMRYarnCluster.java:211) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:80) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:188) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:176) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:168) at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:129) at org.apache.hadoop.hbase.HBaseTestingUtil.startMiniMapReduceCluster(HBaseTestingUtil.java:2352) at org.apache.hadoop.hbase.HBaseTestingUtil.startMiniMapReduceCluster(HBaseTestingUtil.java:2285) at org.apache.hadoop.hbase.mapreduce.TestHBaseMRTestingUtility.testMRYarnConfigsPopulation(TestHBaseMRTestingUtility.java:64) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: javax.ws.rs.WebApplicationException at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at
[jira] [Commented] (HBASE-27844) changed type names to avoid conflicts with built-in types
[
https://issues.apache.org/jira/browse/HBASE-27844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720789#comment-17720789
]
Hudson commented on HBASE-27844:
Results for branch branch-2
[build #807 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/807/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/807/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/807/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/807/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/807/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> changed type names to avoid conflicts with built-in types
> -
>
> Key: HBASE-27844
> URL: https://issues.apache.org/jira/browse/HBASE-27844
> Project: HBase
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.5.4
>Reporter: ruanhui
>Assignee: ruanhui
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.5, 2.4.18
>
>
> Some compilers will resolve Builder to java.lang.Thread.Builder instead of
> Builder in pb and cause compilation failure. We should try to avoid conflicts
> with built-in class names.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-27844) changed type names to avoid conflicts with built-in types
[
https://issues.apache.org/jira/browse/HBASE-27844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720772#comment-17720772
]
Hudson commented on HBASE-27844:
Results for branch branch-2.4
[build #557 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/557/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/557/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/557/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/557/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/557/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> changed type names to avoid conflicts with built-in types
> -
>
> Key: HBASE-27844
> URL: https://issues.apache.org/jira/browse/HBASE-27844
> Project: HBase
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.5.4
>Reporter: ruanhui
>Assignee: ruanhui
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.5, 2.4.18
>
>
> Some compilers will resolve Builder to java.lang.Thread.Builder instead of
> Builder in pb and cause compilation failure. We should try to avoid conflicts
> with built-in class names.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-27849) Move stable pointer to 2.5.x
[ https://issues.apache.org/jira/browse/HBASE-27849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720743#comment-17720743 ] Duo Zhang commented on HBASE-27849: --- Committed via 'svn ci'. It should take sometime before we can see the changes on download.a.o. Will resolve the issue once downloads.a.o is updated. Thanks. > Move stable pointer to 2.5.x > > > Key: HBASE-27849 > URL: https://issues.apache.org/jira/browse/HBASE-27849 > Project: HBase > Issue Type: Task > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27849) Move stable pointer to 2.5.x
[
https://issues.apache.org/jira/browse/HBASE-27849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720742#comment-17720742
]
Duo Zhang commented on HBASE-27849:
---
This is the diff, which I plan to commit.
{noformat}
Index: HEADER.html
===
--- HEADER.html (版本 61745)
+++ HEADER.html (工作副本)
@@ -5,7 +5,7 @@
mirror site, not from www.apache.org.
We suggest downloading the current stable release.
-The 2.4.z series is the current stable release line, it supercedes earlier
release lines
+The 2.5.z series is the current stable release line, it supercedes earlier
release lines
If you are a user of HBase 1.y and can not yet upgrade to the stable
release line, we suggest downloading the latest 1.y minor release.
Note that:
0.96 was EOM'd September 2014; 0.94 and 0.98 were EOM'd April 2017
Index: stable
===
--- stable (版本 61745)
+++ stable (工作副本)
@@ -1 +1 @@
-link 2.4.17
\ No newline at end of file
+link 2.5.4
\ No newline at end of file
{noformat}
> Move stable pointer to 2.5.x
>
>
> Key: HBASE-27849
> URL: https://issues.apache.org/jira/browse/HBASE-27849
> Project: HBase
> Issue Type: Task
> Components: community
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27849) Move stable pointer to 2.5.x
Duo Zhang created HBASE-27849: - Summary: Move stable pointer to 2.5.x Key: HBASE-27849 URL: https://issues.apache.org/jira/browse/HBASE-27849 Project: HBase Issue Type: Task Components: community Reporter: Duo Zhang Assignee: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27109) Move replication queue storage from zookeeper to a separated HBase table
[
https://issues.apache.org/jira/browse/HBASE-27109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720720#comment-17720720
]
Hudson commented on HBASE-27109:
Results for branch HBASE-27109/table_based_rqs
[build #73 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/73/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/73/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/73/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/73/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move replication queue storage from zookeeper to a separated HBase table
>
>
> Key: HBASE-27109
> URL: https://issues.apache.org/jira/browse/HBASE-27109
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> This is a more specific issue based on the works which are already done in
> HBASE-15867.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #5202: HBASE-27109 Move replication queue storage from zookeeper to a separated HBase table
Apache-HBase commented on PR #5202: URL: https://github.com/apache/hbase/pull/5202#issuecomment-1539171024 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 4s | master passed | | +1 :green_heart: | compile | 1m 30s | master passed | | +1 :green_heart: | shadedjars | 4m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 15s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 48s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | shadedjars | 4m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 396m 37s | root in the patch passed. | | | | 426m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5202/10/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5202 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 88fd58b40833 5.4.0-1099-aws #107~18.04.1-Ubuntu SMP Fri Mar 17 16:49:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d49df8b7a9 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5202/10/testReport/ | | Max. process+thread count | 4759 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-replication hbase-procedure hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5202/10/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5226: [Draft] HBASE-27798: Client side should back off based on wait interval in RpcThrottlingException
Apache-HBase commented on PR #5226: URL: https://github.com/apache/hbase/pull/5226#issuecomment-1539100106 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | branch-2 passed | | +1 :green_heart: | compile | 0m 52s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 18s | branch-2 passed | | +1 :green_heart: | spotless | 0m 59s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 2s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | checkstyle | 0m 23s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 21m 45s | Patch does not cause any errors with Hadoop 2.10.2 or 3.2.4 3.3.4. | | +1 :green_heart: | spotless | 0m 47s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | | | | 40m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5226/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5226 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 55c3076e228c 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d89bb1cfd8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 85 (vs. ulimit of 3) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5226/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5226: [Draft] HBASE-27798: Client side should back off based on wait interval in RpcThrottlingException
Apache-HBase commented on PR #5226: URL: https://github.com/apache/hbase/pull/5226#issuecomment-1539095000 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 18s | branch-2 passed | | +1 :green_heart: | compile | 0m 29s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 45s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | shadedjars | 5m 22s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 8m 0s | hbase-client in the patch passed. | | | | 34m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5226/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5226 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1fd2f8e39575 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d89bb1cfd8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5226/1/testReport/ | | Max. process+thread count | 196 (vs. ulimit of 3) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5226/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5226: [Draft] HBASE-27798: Client side should back off based on wait interval in RpcThrottlingException
Apache-HBase commented on PR #5226: URL: https://github.com/apache/hbase/pull/5226#issuecomment-1539092248 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 20s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 42s | branch-2 passed | | +1 :green_heart: | compile | 0m 28s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | shadedjars | 5m 23s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 52s | hbase-client in the patch passed. | | | | 31m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5226/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5226 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9030f8bf348d 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / d89bb1cfd8 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5226/1/testReport/ | | Max. process+thread count | 170 (vs. ulimit of 3) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5226/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] rmdmattingly commented on a diff in pull request #5226: [Draft] HBASE-27798: Client side should back off based on wait interval in RpcThrottlingException
rmdmattingly commented on code in PR #5226:
URL: https://github.com/apache/hbase/pull/5226#discussion_r1187897189
##
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerImpl.java:
##
@@ -140,21 +141,30 @@ public T callWithRetries(RetryingCallable callable,
int callTimeout)
if (tries >= maxAttempts - 1) {
throw new RetriesExhaustedException(tries, exceptions);
}
-// If the server is dead, we need to wait a little before retrying, to
give
-// a chance to the regions to be moved
-// get right pause time, start by RETRY_BACKOFF[0] * pauseBase, where
pauseBase might be
-// special when encountering an exception indicating the server is
overloaded.
-// see #HBASE-17114 and HBASE-26807
-long pauseBase =
- HBaseServerException.isServerOverloaded(t) ?
pauseForServerOverloaded : pause;
-expectedSleep = callable.sleep(pauseBase, tries);
-
-// If, after the planned sleep, there won't be enough time left, we
stop now.
-long duration = singleCallDuration(expectedSleep);
-if (duration > callTimeout) {
- String msg = "callTimeout=" + callTimeout + ", callDuration=" +
duration + ": "
-+ t.getMessage() + " " +
callable.getExceptionMessageAdditionalDetail();
- throw (SocketTimeoutException) new
SocketTimeoutException(msg).initCause(t);
+if (t instanceof RpcThrottlingException) {
+ RpcThrottlingException rpcThrottlingException =
(RpcThrottlingException) t;
+ expectedSleep = rpcThrottlingException.getWaitInterval();
+ if (LOG.isDebugEnabled()) {
+LOG.debug("Sleeping for {}ms after catching
RpcThrottlingException", expectedSleep,
+ rpcThrottlingException);
+ }
+} else {
+ // If the server is dead, we need to wait a little before retrying,
to give
+ // a chance to the regions to be moved
+ // get right pause time, start by RETRY_BACKOFF[0] * pauseBase,
where pauseBase might be
+ // special when encountering an exception indicating the server is
overloaded.
+ // see #HBASE-17114 and HBASE-26807
+ long pauseBase =
+HBaseServerException.isServerOverloaded(t) ?
pauseForServerOverloaded : pause;
+ expectedSleep = callable.sleep(pauseBase, tries);
+
+ // If, after the planned sleep, there won't be enough time left, we
stop now.
+ long duration = singleCallDuration(expectedSleep);
+ if (duration > callTimeout) {
+String msg = "callTimeout=" + callTimeout + ", callDuration=" +
duration + ": "
+ + t.getMessage() + " " +
callable.getExceptionMessageAdditionalDetail();
+throw (SocketTimeoutException) new
SocketTimeoutException(msg).initCause(t);
+ }
Review Comment:
Moving this check inside of the `else` block is pretty consequential, and
I'm not sure which way we'd prefer.
If we leave this logic outside of the `if (t instanceof RTE) else` block
then significant wait intervals — those which would extend past the call
timeout — may be totally ignored. We'd instead just throw quickly which would
probably manifest as retries, and consequently sort of produce the opposite of
a back off.
Alternatively, by isolating this logic to only non-RTE exceptions, we're
more likely to respect the back off requested — even if it will inevitably
result in the timeout being exceeded.
Maybe we want to calculate the min of the requested wait interval and the
duration remaining prior to timeout, and somewhat guarantee that we wait for
that? I'm open to other suggestions too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] rmdmattingly opened a new pull request, #5226: [Draft] HBASE-27798: Client side should back off based on wait interval in RpcThrottlingException
rmdmattingly opened a new pull request, #5226: URL: https://github.com/apache/hbase/pull/5226 The RpcThrottlingException tells the client how much to back off, but right now the recommendation is ignored. This PR introduces logic that respects said back off recommendation. I've tested the synchronous client implementation on a cluster at my day job. Here is the log output from a single client thread that's being throttled by a strict quota: https://gist.github.com/rmdmattingly/4c8e63bec16cfa2e4324ff6590a5f8ea. It demonstrates that the thread is backing off for the given wait interval. I'm very open to adding more explicit unit testing of this behavior, and/or gating behind some new Configuration option, and would appreciate any suggestions regarding where said tests/config should live. This also didn't apply particularly cleanly to the master branch, but I'm particularly interested in getting this feature on branch-2. How should I approach this? I'm open to figuring out an equivalent solution against master. @bbeaudreault -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5202: HBASE-27109 Move replication queue storage from zookeeper to a separated HBase table
Apache-HBase commented on PR #5202: URL: https://github.com/apache/hbase/pull/5202#issuecomment-1539032451 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 48s | master passed | | +1 :green_heart: | compile | 2m 6s | master passed | | +1 :green_heart: | shadedjars | 4m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 41s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 17s | the patch passed | | +1 :green_heart: | compile | 1m 43s | the patch passed | | +1 :green_heart: | javac | 1m 43s | the patch passed | | +1 :green_heart: | shadedjars | 4m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 262m 52s | root in the patch passed. | | | | 295m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5202/10/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5202 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4ee3b571432e 5.4.0-1101-aws #109~18.04.1-Ubuntu SMP Mon Apr 24 20:40:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d49df8b7a9 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5202/10/testReport/ | | Max. process+thread count | 4821 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-replication hbase-procedure hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5202/10/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-27795) Define RPC API for cache cleaning
[ https://issues.apache.org/jira/browse/HBASE-27795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shanmukha Haripriya Kota reassigned HBASE-27795: Assignee: Shanmukha Haripriya Kota > Define RPC API for cache cleaning > - > > Key: HBASE-27795 > URL: https://issues.apache.org/jira/browse/HBASE-27795 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Shanmukha Haripriya Kota >Priority: Major > > We should add an RPC API to allow for a "limited manual" cache cleaning. If > hbase.rs.evictblocksonclose is set to false, blocks may hang in the cache > upon regions move between RSes. > The method at the RS level, should compare the files from its online regions > against the files in the prefetch list file, evicting blocks from the files > in the prefetch list file that are not in any of the online regions for the > given RS. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HBASE-27794) Tooling for parsing/reading the prefetch files list file
[ https://issues.apache.org/jira/browse/HBASE-27794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shanmukha Haripriya Kota reassigned HBASE-27794: Assignee: Shanmukha Haripriya Kota > Tooling for parsing/reading the prefetch files list file > > > Key: HBASE-27794 > URL: https://issues.apache.org/jira/browse/HBASE-27794 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Shanmukha Haripriya Kota >Priority: Minor > > The content of the file defined by hbase.prefetch.file.list.path is encoded. > It would be nice to have some extra tool for properly parsing it and print > the list in human readable format, for easy of troubleshooting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #5202: HBASE-27109 Move replication queue storage from zookeeper to a separated HBase table
Apache-HBase commented on PR #5202: URL: https://github.com/apache/hbase/pull/5202#issuecomment-1538764551 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 22s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 1s | Shelldocs was not available. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 10s | master passed | | +1 :green_heart: | compile | 5m 56s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +0 :ok: | refguide | 2m 49s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotless | 0m 54s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 15m 4s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 23s | the patch passed | | +1 :green_heart: | compile | 6m 12s | the patch passed | | +1 :green_heart: | cc | 6m 12s | the patch passed | | -0 :warning: | javac | 6m 12s | root generated 9 new + 899 unchanged - 11 fixed = 908 total (was 910) | | -0 :warning: | checkstyle | 1m 21s | root: The patch generated 6 new + 70 unchanged - 1 fixed = 76 total (was 71) | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 4s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 2m 55s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 14m 31s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. | | +1 :green_heart: | hbaseprotoc | 5m 50s | the patch passed | | +1 :green_heart: | spotless | 0m 44s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 15m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. | | | | 94m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5202/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5202 | | Optional Tests | dupname asflicense spotless shellcheck shelldocs javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool xml refguide | | uname | Linux 398204467fbc 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d49df8b7a9 | | Default Java | Eclipse Adoptium-11.0.17+8 | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-5202/10/yetus-general-check/output/branch-site/book.html | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5202/10/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5202/10/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-5202/10/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 175 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-replication hbase-procedure hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5202/10/console | | versions | git=2.34.1 maven=3.8.6 shellcheck=0.8.0 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For
[jira] [Commented] (HBASE-27760) Release 2.5.4
[ https://issues.apache.org/jira/browse/HBASE-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720612#comment-17720612 ] Andrew Kyle Purtell commented on HBASE-27760: - Thanks, well that explains it for sure. My apologies. > Release 2.5.4 > - > > Key: HBASE-27760 > URL: https://issues.apache.org/jira/browse/HBASE-27760 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Assignee: Andrew Kyle Purtell >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27844) changed type names to avoid conflicts with built-in types
[ https://issues.apache.org/jira/browse/HBASE-27844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-27844. --- Fix Version/s: 2.6.0 3.0.0-alpha-4 2.5.5 2.4.18 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.4+. Thanks [~frostruan] for contributing! > changed type names to avoid conflicts with built-in types > - > > Key: HBASE-27844 > URL: https://issues.apache.org/jira/browse/HBASE-27844 > Project: HBase > Issue Type: Improvement > Components: build >Affects Versions: 2.5.4 >Reporter: ruanhui >Assignee: ruanhui >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.5, 2.4.18 > > > Some compilers will resolve Builder to java.lang.Thread.Builder instead of > Builder in pb and cause compilation failure. We should try to avoid conflicts > with built-in class names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27760) Release 2.5.4
[ https://issues.apache.org/jira/browse/HBASE-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720552#comment-17720552 ] Duo Zhang commented on HBASE-27760: --- Seems you accidentally removed 2.5.4 while you wanted to remove 2.5.3... [~apurtell] Check 'svn diff -r 61294:61295'... > Release 2.5.4 > - > > Key: HBASE-27760 > URL: https://issues.apache.org/jira/browse/HBASE-27760 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Assignee: Andrew Kyle Purtell >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27760) Release 2.5.4
[ https://issues.apache.org/jira/browse/HBASE-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720549#comment-17720549 ] Andrew Kyle Purtell commented on HBASE-27760: - What happened? Accidental removal? svn repo issue? > Release 2.5.4 > - > > Key: HBASE-27760 > URL: https://issues.apache.org/jira/browse/HBASE-27760 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Assignee: Andrew Kyle Purtell >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27109) Move replication queue storage from zookeeper to a separated HBase table
[ https://issues.apache.org/jira/browse/HBASE-27109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-27109: -- Release Note: We introduced a table based replication queue storage in this issue. The queue data will stored in hbase:replication table. This is the last piece of persistent data on zookeeper. So after this change, we are OK to clean up all the data on zookeeper, as now they are all transient, a cluster restarting can fix everything. The data structure has been changed a bit as now we only support an offset for a WAL group instead of storing all the WAL files for a WAL group. Please see the replication internals section in our ref guide for more details. To break the cyclic dependency issue, i.e, creating a new WAL writer requires writing to replication queue storage first but with table based replication queue storage, you first need a WAL writer when you want to update to table, now we will not record a queue when creating a new WAL writer instance. The downside for this change it that, the logic for claiming queue and WAL cleaner are much more complicated. See AssignReplicationQueuesProcedure and ReplicationLogCleaner for more details if you have interest. Notice that, we will use a separated WAL provider for hbase:replication table, so you will see a new WAL file for the region server which holds the hbase:replication table. If we do not do this, the update to hbase:replication table will also generate some WAL edits in the WAL file we need to track in replication, and then lead to more updates to hbase:replication table since we have advanced the replication offset. In this way we will generate a lot of garbage in our WAL file, even if we write nothing to the cluster. So a separated WAL provider which is not tracked by replication is necessary here. The data migration will be done automatically during rolling upgrading, of course the migration via a full cluster restart is also supported, but please make sure you restart master with new code first. The replication peers will be disabled during the migration and no claiming queue will be scheduled at the same time. So you may see a lot of unfinished SCPs during the migration but do not worry, it will not block the normal failover, all regions will be assigned. The replication peers will be enabled again after the migration is done, no manual operations needed. The ReplicationSyncUp tool is also affected. The goal of this tool is to replicate data to peer cluster while the source cluster is down. But if we store the replication queue data in a hbase table, it is impossible for us to get the newest data if the source cluster is down. So here we choose to read from the region directory directly to load all the replication queue data in memory, and do the sync up work. We may loss the newest data so in this way we need to replicate more data but it will not affect correctness. was: We introduced a table based replication queue storage in this issue. The queue data will stored in hbase:replication table. The data structure has been changed a bit as now we only support an offset for a WAL group instead of storing all the WAL files for a WAL group. Please see the replication internals section in our ref guide for more details. To break the cyclic dependency issue, i.e, creating a new WAL writer requires writing to replication queue storage first but with table based replication queue storage, you first need a WAL writer when you want to update to table, now we will not record a queue when creating a new WAL writer instance. The downside for this change it that, the logic for claiming queue and WAL cleaner are much more complicated. See AssignReplicationQueuesProcedure and ReplicationLogCleaner for more details if you have interest. Notice that, we will use a separated WAL provider for hbase:replication table, so you will see a new WAL file for the region server which holds the hbase:replication table. If we do not do this, the update to hbase:replication table will also generate some WAL edits in the WAL file we need to track in replication, and then lead to more updates to hbase:replication table since we have advanced the replication offset. In this way we will generate a lot of garbage in our WAL file, even if we write nothing to the cluster. So a separated WAL provider which is not tracked by replication is necessary here. The data migration will be done automatically during rolling upgrading, of course the migration via a full cluster restart is also supported, but please make sure you restart master with new code first. The replication peers will be disabled during the migration and no claiming queue will be scheduled at the same time. So you may see a lot of unfinished SCPs during the migration but do not worry, it will not block the normal failover, all regions will be assigned. The
[jira] [Updated] (HBASE-27109) Move replication queue storage from zookeeper to a separated HBase table
[ https://issues.apache.org/jira/browse/HBASE-27109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-27109: -- Release Note: We introduced a table based replication queue storage in this issue. The queue data will stored in hbase:replication table. The data structure has been changed a bit as now we only support an offset for a WAL group instead of storing all the WAL files for a WAL group. Please see the replication internals section in our ref guide for more details. To break the cyclic dependency issue, i.e, creating a new WAL writer requires writing to replication queue storage first but with table based replication queue storage, you first need a WAL writer when you want to update to table, now we will not record a queue when creating a new WAL writer instance. The downside for this change it that, the logic for claiming queue and WAL cleaner are much more complicated. See AssignReplicationQueuesProcedure and ReplicationLogCleaner for more details if you have interest. Notice that, we will use a separated WAL provider for hbase:replication table, so you will see a new WAL file for the region server which holds the hbase:replication table. If we do not do this, the update to hbase:replication table will also generate some WAL edits in the WAL file we need to track in replication, and then lead to more updates to hbase:replication table since we have advanced the replication offset. In this way we will generate a lot of garbage in our WAL file, even if we write nothing to the cluster. So a separated WAL provider which is not tracked by replication is necessary here. The data migration will be done automatically during rolling upgrading, of course the migration via a full cluster restart is also supported, but please make sure you restart master with new code first. The replication peers will be disabled during the migration and no claiming queue will be scheduled at the same time. So you may see a lot of unfinished SCPs during the migration but do not worry, it will not block the normal failover, all regions will be assigned. The replication peers will be enabled again after the migration is done, no manual operations needed. The ReplicationSyncUp tool is also affected. The goal of this tool is to replicate data to peer cluster while the source cluster is down. But if we store the replication queue data in a hbase table, it is impossible for us to get the newest data if the source cluster is down. So here we choose to read from the region directory directly to load all the replication queue data in memory, and do the sync up work. We may loss the newest data so in this way we need to replicate more data but it will not affect correctness. > Move replication queue storage from zookeeper to a separated HBase table > > > Key: HBASE-27109 > URL: https://issues.apache.org/jira/browse/HBASE-27109 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > This is a more specific issue based on the works which are already done in > HBASE-15867. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27844) changed type names to avoid conflicts with built-in types
[ https://issues.apache.org/jira/browse/HBASE-27844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720512#comment-17720512 ] ruanhui commented on HBASE-27844: - already done. Thanks Duo. > changed type names to avoid conflicts with built-in types > - > > Key: HBASE-27844 > URL: https://issues.apache.org/jira/browse/HBASE-27844 > Project: HBase > Issue Type: Improvement > Components: build >Affects Versions: 2.5.4 >Reporter: ruanhui >Assignee: ruanhui >Priority: Minor > > Some compilers will resolve Builder to java.lang.Thread.Builder instead of > Builder in pb and cause compilation failure. We should try to avoid conflicts > with built-in class names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [hbase] Apache9 merged pull request #5223: HBASE-27844 changed type names to avoid conflicts with built-in types
Apache9 merged PR #5223: URL: https://github.com/apache/hbase/pull/5223 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27844) changed type names to avoid conflicts with built-in types
[ https://issues.apache.org/jira/browse/HBASE-27844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720508#comment-17720508 ] Duo Zhang commented on HBASE-27844: --- I think we'd better also report an issue for the Kona JDK? > changed type names to avoid conflicts with built-in types > - > > Key: HBASE-27844 > URL: https://issues.apache.org/jira/browse/HBASE-27844 > Project: HBase > Issue Type: Improvement > Components: build >Affects Versions: 2.5.4 >Reporter: ruanhui >Assignee: ruanhui >Priority: Minor > > Some compilers will resolve Builder to java.lang.Thread.Builder instead of > Builder in pb and cause compilation failure. We should try to avoid conflicts > with built-in class names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27109) Move replication queue storage from zookeeper to a separated HBase table
[
https://issues.apache.org/jira/browse/HBASE-27109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720487#comment-17720487
]
Hudson commented on HBASE-27109:
Results for branch HBASE-27109/table_based_rqs
[build #72 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/72/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/72/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/72/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/72/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move replication queue storage from zookeeper to a separated HBase table
>
>
> Key: HBASE-27109
> URL: https://issues.apache.org/jira/browse/HBASE-27109
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> This is a more specific issue based on the works which are already done in
> HBASE-15867.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] chrajeshbabu commented on pull request #5167: HBASE-27698 Migrate meta locations from zookeeper to master data may …
chrajeshbabu commented on PR #5167: URL: https://github.com/apache/hbase/pull/5167#issuecomment-1538183542 @Apache9 I have tried your suggestion of creating put entry of meta table in master region which gets filled in regionstates and helps to avoid calling init meta procedure there is problem with this approach. Actually meta assignment called only two places 1) during server crash procedure when the region server went down is carrying meta this is not the case during the express upgrade cases 2) during meta initialisation once after the fs layout creation step is done. By adding the just meta entry in the master region without knowing the location, won't reach any of the above flows and leading to meta region hanging in transition forever. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27760) Release 2.5.4
[ https://issues.apache.org/jira/browse/HBASE-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720470#comment-17720470 ] Nick Dimiduk commented on HBASE-27760: -- I believe it's fixed. > Release 2.5.4 > - > > Key: HBASE-27760 > URL: https://issues.apache.org/jira/browse/HBASE-27760 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Assignee: Andrew Kyle Purtell >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27760) Release 2.5.4
[ https://issues.apache.org/jira/browse/HBASE-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720460#comment-17720460 ] Nick Dimiduk commented on HBASE-27760: -- I don't see 2.5.4 in the downloads directory, https://downloads.apache.org/hbase/. Indeed, `svn ls https://dist.apache.org/repos/dist/release/hbase/` shows 2.5.3. `svn ls https://dist.apache.org/repos/dist/dev/hbase/` shows an empty directory. I see that the release artifacts are still present in svn history. Trying to restore and fix. > Release 2.5.4 > - > > Key: HBASE-27760 > URL: https://issues.apache.org/jira/browse/HBASE-27760 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Assignee: Andrew Kyle Purtell >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
