[jira] [Created] (HBASE-26595) Put up 4.0.1RC0
Duo Zhang created HBASE-26595: - Summary: Put up 4.0.1RC0 Key: HBASE-26595 URL: https://issues.apache.org/jira/browse/HBASE-26595 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26594) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 4.0.1
Duo Zhang created HBASE-26594: - Summary: [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 4.0.1 Key: HBASE-26594 URL: https://issues.apache.org/jira/browse/HBASE-26594 Project: HBase Issue Type: Sub-task Components: thirdparty Reporter: Duo Zhang Fix For: thirdparty-4.0.1 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26593) [hbase-thirdparty] Set version as 4.0.1 in prep for first RC
Duo Zhang created HBASE-26593: - Summary: [hbase-thirdparty] Set version as 4.0.1 in prep for first RC Key: HBASE-26593 URL: https://issues.apache.org/jira/browse/HBASE-26593 Project: HBase Issue Type: Sub-task Components: build, pom, thirdparty Reporter: Duo Zhang Fix For: thirdparty-4.0.1 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #70: HBASE-26592 Fix the broken shaded protobuf module
Apache-HBase commented on pull request #70: URL: https://github.com/apache/hbase-thirdparty/pull/70#issuecomment-996452388 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 46s | master passed | | +1 :green_heart: | compile | 0m 9s | master passed | | +1 :green_heart: | javadoc | 0m 8s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 10s | the patch passed | | +1 :green_heart: | compile | 0m 9s | the patch passed | | +1 :green_heart: | javac | 0m 9s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 0m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 9s | hbase-shaded-protobuf in the patch passed. | | +1 :green_heart: | asflicense | 0m 6s | The patch does not generate ASF License warnings. | | | | 3m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-70/1/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/70 | | Optional Tests | dupname asflicense javac javadoc unit xml compile | | uname | Linux 0eff5a9cf227 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / c81cf5c | | Default Java | Oracle Corporation-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-70/1/testReport/ | | Max. process+thread count | 39 (vs. ulimit of 1000) | | modules | C: hbase-shaded-protobuf U: hbase-shaded-protobuf | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-70/1/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26592) Fix the broken shaded protobuf module
[ https://issues.apache.org/jira/browse/HBASE-26592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26592: -- Priority: Blocker (was: Major) > Fix the broken shaded protobuf module > - > > Key: HBASE-26592 > URL: https://issues.apache.org/jira/browse/HBASE-26592 > Project: HBase > Issue Type: Sub-task > Components: hbase-thirdparty, Protobufs >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: thirdparty-3.5.2, thirdparty-4.0.1 > > > First, we should include the proto files in the final uber jar. > Second, we should not pull in the protobuf-java dependency as we always > include everything in the uber jar. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26592) Fix the broken shaded protobuf module
[ https://issues.apache.org/jira/browse/HBASE-26592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26592: -- Fix Version/s: thirdparty-3.5.2 > Fix the broken shaded protobuf module > - > > Key: HBASE-26592 > URL: https://issues.apache.org/jira/browse/HBASE-26592 > Project: HBase > Issue Type: Sub-task > Components: hbase-thirdparty, Protobufs >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.5.2, thirdparty-4.0.1 > > > First, we should include the proto files in the final uber jar. > Second, we should not pull in the protobuf-java dependency as we always > include everything in the uber jar. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase-thirdparty] Apache9 commented on pull request #70: HBASE-26592 Fix the broken shaded protobuf module
Apache9 commented on pull request #70: URL: https://github.com/apache/hbase-thirdparty/pull/70#issuecomment-996451596 Remove protobuf-java dependency, we only need to unpack its source so only need the artifact dependency in maven dependency plugin. Add a resources section to include the proto files when generating the final jar. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3958: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2
Apache-HBase commented on pull request #3958: URL: https://github.com/apache/hbase/pull/3958#issuecomment-996444918 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 8m 48s | branch-2 passed | | +1 :green_heart: | compile | 2m 11s | branch-2 passed | | +1 :green_heart: | shadedjars | 11m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 17s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 7m 26s | the patch passed | | +1 :green_heart: | compile | 2m 4s | the patch passed | | +1 :green_heart: | javac | 2m 4s | the patch passed | | +1 :green_heart: | shadedjars | 10m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 239m 13s | hbase-server in the patch passed. | | | | 288m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3958/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3958 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d8f92bffb140 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 555fcf42b0 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3958/1/testReport/ | | Max. process+thread count | 2827 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3958/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3959: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2.5
Apache-HBase commented on pull request #3959: URL: https://github.com/apache/hbase/pull/3959#issuecomment-996442632 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 56s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.5 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 37s | branch-2.5 passed | | +1 :green_heart: | compile | 1m 26s | branch-2.5 passed | | +1 :green_heart: | shadedjars | 8m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | branch-2.5 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 9s | the patch passed | | +1 :green_heart: | compile | 1m 21s | the patch passed | | +1 :green_heart: | javac | 1m 21s | the patch passed | | +1 :green_heart: | shadedjars | 8m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 243m 41s | hbase-server in the patch passed. | | | | 282m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3959/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3959 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1cdb5b90d87d 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.5 / f1add382bd | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3959/1/testReport/ | | Max. process+thread count | 2507 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3959/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3960: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2.4
Apache-HBase commented on pull request #3960: URL: https://github.com/apache/hbase/pull/3960#issuecomment-996438291 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 52s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 21s | branch-2.4 passed | | +1 :green_heart: | compile | 1m 26s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 7m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 50s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 7m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 233m 12s | hbase-server in the patch failed. | | | | 270m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3960/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3960 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 764f67dd4e75 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / d7c3720e52 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3960/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3960/1/testReport/ | | Max. process+thread count | 2436 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3960/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26591) [hbase-thirdparty] Make a 4.0.1 release for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461184#comment-17461184
]
Pankaj Kumar commented on HBASE-26591:
--
Yes it is causing compilation issue. Thanks [~zhangduo]
> [hbase-thirdparty] Make a 4.0.1 release for hbase-thirdparty
>
>
> Key: HBASE-26591
> URL: https://issues.apache.org/jira/browse/HBASE-26591
> Project: HBase
> Issue Type: Umbrella
> Components: community, hbase-thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Copy the content in mailing list here
> {noformat}
> Oh, when backporting to branch-2, I found that there is a serious problem
> with this release.
> We do not include the proto files in protobuf-java jar, I think this is a
> side effect of HBASE-26496.
> And on the master branch, hbase-shaded-protobuf also pulls in protobuf-java
> 3.19.1 dependency so everything is fine. But for branch-2, we will need to
> depend on protobuf 2.5.0 then the compilation is broken.
> Anyway, we should include the proto files in hbase-shaded-protobuf, and also
> we should not pull in the original protobuf-java dependency.
> Let me fix this and do a 4.0.1 release.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26592) Fix the broken shaded protobuf module
Duo Zhang created HBASE-26592: - Summary: Fix the broken shaded protobuf module Key: HBASE-26592 URL: https://issues.apache.org/jira/browse/HBASE-26592 Project: HBase Issue Type: Sub-task Components: hbase-thirdparty, Protobufs Reporter: Duo Zhang Assignee: Duo Zhang Fix For: thirdparty-4.0.1 First, we should include the proto files in the final uber jar. Second, we should not pull in the protobuf-java dependency as we always include everything in the uber jar. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HBASE-26591) [hbase-thirdparty] Make a 4.0.1 release for hbase-thirdparty
[
https://issues.apache.org/jira/browse/HBASE-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang reassigned HBASE-26591:
-
Assignee: Duo Zhang
> [hbase-thirdparty] Make a 4.0.1 release for hbase-thirdparty
>
>
> Key: HBASE-26591
> URL: https://issues.apache.org/jira/browse/HBASE-26591
> Project: HBase
> Issue Type: Umbrella
> Components: community, hbase-thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Copy the content in mailing list here
> {noformat}
> Oh, when backporting to branch-2, I found that there is a serious problem
> with this release.
> We do not include the proto files in protobuf-java jar, I think this is a
> side effect of HBASE-26496.
> And on the master branch, hbase-shaded-protobuf also pulls in protobuf-java
> 3.19.1 dependency so everything is fine. But for branch-2, we will need to
> depend on protobuf 2.5.0 then the compilation is broken.
> Anyway, we should include the proto files in hbase-shaded-protobuf, and also
> we should not pull in the original protobuf-java dependency.
> Let me fix this and do a 4.0.1 release.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26591) [hbase-thirdparty] Make a 4.0.1 release for hbase-thirdparty
Duo Zhang created HBASE-26591:
-
Summary: [hbase-thirdparty] Make a 4.0.1 release for
hbase-thirdparty
Key: HBASE-26591
URL: https://issues.apache.org/jira/browse/HBASE-26591
Project: HBase
Issue Type: Umbrella
Components: community, hbase-thirdparty
Reporter: Duo Zhang
Copy the content in mailing list here
{noformat}
Oh, when backporting to branch-2, I found that there is a serious problem with
this release.
We do not include the proto files in protobuf-java jar, I think this is a side
effect of HBASE-26496.
And on the master branch, hbase-shaded-protobuf also pulls in protobuf-java
3.19.1 dependency so everything is fine. But for branch-2, we will need to
depend on protobuf 2.5.0 then the compilation is broken.
Anyway, we should include the proto files in hbase-shaded-protobuf, and also
we should not pull in the original protobuf-java dependency.
Let me fix this and do a 4.0.1 release.
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26498) [hbase-thirdparty] Make a 4.0.0 release for hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-26498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26498. --- Resolution: Fixed > [hbase-thirdparty] Make a 4.0.0 release for hbase-thirdparty > > > Key: HBASE-26498 > URL: https://issues.apache.org/jira/browse/HBASE-26498 > Project: HBase > Issue Type: Umbrella > Components: community, hbase-thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3940: HBASE-26566 Optimize encodeNumeric in OrderedBytes
Apache-HBase commented on pull request #3940: URL: https://github.com/apache/hbase/pull/3940#issuecomment-996424297 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 25s | master passed | | +1 :green_heart: | compile | 0m 49s | master passed | | +1 :green_heart: | checkstyle | 0m 24s | master passed | | +1 :green_heart: | spotbugs | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | the patch passed | | +1 :green_heart: | compile | 0m 48s | the patch passed | | +1 :green_heart: | javac | 0m 48s | the patch passed | | +1 :green_heart: | checkstyle | 0m 24s | hbase-common: The patch generated 0 new + 39 unchanged - 5 fixed = 39 total (was 44) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 21m 43s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 0m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 44m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3940 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 3e9da9499d9b 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 139f08587a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3940: HBASE-26566 Optimize encodeNumeric in OrderedBytes
Apache-HBase commented on pull request #3940: URL: https://github.com/apache/hbase/pull/3940#issuecomment-996419359 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 47s | master passed | | +1 :green_heart: | compile | 0m 26s | master passed | | +1 :green_heart: | shadedjars | 8m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 32s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | shadedjars | 8m 22s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 8s | hbase-common in the patch passed. | | | | 31m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3940 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b00849640b0f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 139f08587a | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/5/testReport/ | | Max. process+thread count | 312 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3940: HBASE-26566 Optimize encodeNumeric in OrderedBytes
Apache-HBase commented on pull request #3940: URL: https://github.com/apache/hbase/pull/3940#issuecomment-996418831 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 59s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 11s | master passed | | +1 :green_heart: | compile | 0m 26s | master passed | | +1 :green_heart: | shadedjars | 8m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed | | +1 :green_heart: | javac | 0m 25s | the patch passed | | +1 :green_heart: | shadedjars | 8m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 42s | hbase-common in the patch passed. | | | | 30m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3940 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fcd0b2755da8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 139f08587a | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/5/testReport/ | | Max. process+thread count | 339 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3960: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2.4
Apache-HBase commented on pull request #3960: URL: https://github.com/apache/hbase/pull/3960#issuecomment-996405163 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | branch-2.4 passed | | +1 :green_heart: | compile | 1m 0s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 24s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 143m 49s | hbase-server in the patch passed. | | | | 173m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3960/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3960 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 03bf080cf635 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / d7c3720e52 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3960/1/testReport/ | | Max. process+thread count | 3514 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3960/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3958: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2
Apache-HBase commented on pull request #3958: URL: https://github.com/apache/hbase/pull/3958#issuecomment-996405296 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 52s | branch-2 passed | | +1 :green_heart: | compile | 0m 59s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 146m 56s | hbase-server in the patch passed. | | | | 173m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3958/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3958 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d4f3a5a27d50 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 555fcf42b0 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3958/1/testReport/ | | Max. process+thread count | 4332 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3958/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #3910: HBASE-25623 Upgrade hbase-thirdparty dependency to 4.0.0
Apache9 merged pull request #3910: URL: https://github.com/apache/hbase/pull/3910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3951: HBASE-26473 Introduce `db.hbase.container_operations` span attribute
Apache9 commented on a change in pull request #3951:
URL: https://github.com/apache/hbase/pull/3951#discussion_r770366748
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/trace/TableOperationSpanBuilder.java
##
@@ -82,6 +89,72 @@ public TableOperationSpanBuilder setOperation(final
Operation operation) {
return this;
}
+ // `setContainerOperations` perform a recursive descent expansion of all the
operations
+ // contained within the provided "batch" object.
+
+ public TableOperationSpanBuilder setContainerOperations(final RowMutations
mutations) {
+final Operation[] ops = mutations.getMutations()
+ .stream()
+ .flatMap(row -> Stream.concat(Stream.of(valueFrom(row)),
unpackRowOperations(row).stream()))
+ .toArray(Operation[]::new);
+return setContainerOperations(ops);
+ }
+
+ public TableOperationSpanBuilder setContainerOperations(final Row row) {
+final Operation[] ops =
+ Stream.concat(Stream.of(valueFrom(row)),
unpackRowOperations(row).stream())
+ .toArray(Operation[]::new);
+return setContainerOperations(ops);
+ }
+
+ public TableOperationSpanBuilder setContainerOperations(
+final Collection operations
+ ) {
+final Operation[] ops = operations.stream()
+ .flatMap(row -> Stream.concat(Stream.of(valueFrom(row)),
unpackRowOperations(row).stream()))
+ .toArray(Operation[]::new);
+return setContainerOperations(ops);
+ }
+
+ private static Set unpackRowOperations(final Row row) {
+final Set ops = new HashSet<>();
+if (row instanceof CheckAndMutate) {
+ final CheckAndMutate cam = (CheckAndMutate) row;
+ ops.addAll(unpackRowOperations(cam));
+}
+if (row instanceof RowMutations) {
+ final RowMutations mutations = (RowMutations) row;
+ ops.addAll(unpackRowOperations(mutations));
+}
+return ops;
+ }
+
+ private static Set unpackRowOperations(final CheckAndMutate cam) {
+final Set ops = new HashSet<>();
+final Operation op = valueFrom(cam.getAction());
+switch (op) {
+ case BATCH:
+ case CHECK_AND_MUTATE:
+ops.addAll(unpackRowOperations(cam.getAction()));
+break;
+ default:
+ops.add(op);
+}
+return ops;
+ }
+
+ public TableOperationSpanBuilder setContainerOperations(
Review comment:
Do we want to store a key value pair here to also reflect the cout for
each operation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #3952: HBASE-26474 Implement connection-level attributes
Apache9 commented on a change in pull request #3952:
URL: https://github.com/apache/hbase/pull/3952#discussion_r771051291
##
File path:
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableTracing.java
##
@@ -100,7 +99,9 @@
private ClientService.Interface stub;
- private AsyncConnection conn;
+ private User user;
Review comment:
Do we really need to declare it as a class member field? It is only used
in one method?
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionLocator.java
##
@@ -117,18 +123,29 @@ private boolean isMeta(TableName tableName) {
}
}
- private List getRegionName(RegionLocations locs) {
-List names = new ArrayList<>();
-for (HRegionLocation loc : locs.getRegionLocations()) {
- if (loc != null) {
-names.add(loc.getRegion().getRegionNameAsString());
- }
-}
-return names;
+ private static List getRegionNames(RegionLocations locs) {
+if (locs == null) { return Collections.emptyList(); }
Review comment:
We still do not have an consensus on the style so let's keep the old
way? At least for me I do not link to have this style of '{}' block, as if the
line is too long, you still need to make a wrapping and then the code will be
like
```
if () {
xxx
}
```
but not
```
if () {
xxx }
```
Seems not consistent...
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/trace/TableOperationSpanBuilder.java
##
@@ -46,51 +44,61 @@
import org.apache.yetus.audience.InterfaceAudience;
/**
- * Construct {@link io.opentelemetry.api.trace.Span} instances originating from
- * "table operations" -- the verbs in our public API that interact with data
in tables.
+ * Construct {@link Span} instances originating from "table operations" -- the
verbs in our public
+ * API that interact with data in tables.
*/
@InterfaceAudience.Private
-public class TableOperationSpanBuilder implements Supplier {
+public class TableOperationSpanBuilder>
+ extends TableSpanBuilder {
// n.b. The results of this class are tested implicitly by way of the likes
of
// `TestAsyncTableTracing` and friends.
private static final String unknown = "UNKNOWN";
- private TableName tableName;
- private final Map, Object> attributes = new HashMap<>();
+ public TableOperationSpanBuilder(final AsyncConnectionImpl conn) {
+super(conn);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public B self() {
Review comment:
Why we need to have so complicated class hierarchy? A class has a
generic type parameter which is just itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3913: HBASE-26536 Tweak checkstyle LeftCurly config to "nlow"
Apache9 commented on pull request #3913:
URL: https://github.com/apache/hbase/pull/3913#issuecomment-996376122
For me I do not like to have a left '{' and right '}' on the same line,
unless there is nothing inside the '{}'.
I think in the old time, the suggest way to have a single line if is to do
not have '{}', but now the best practise is to always have '{}' for if, else,
while, for, etc, to avoid misusage.
So I suggest we just keep the old style, I do not think this is a blocker?
Just update the formatter and do a format.
And to end the checkstyle war, do you have any interest to introduce the
spotless plugin to our project? It can format all the java files with a ecplise
formatter file, add headers, and could even format files other than java.
WDYT?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-26542) Apply a `package` to test protobuf files
[
https://issues.apache.org/jira/browse/HBASE-26542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461154#comment-17461154
]
Hudson commented on HBASE-26542:
Results for branch branch-2.4
[build #258 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/258/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/258/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/258/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/258/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/258/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Apply a `package` to test protobuf files
>
>
> Key: HBASE-26542
> URL: https://issues.apache.org/jira/browse/HBASE-26542
> Project: HBase
> Issue Type: Test
> Components: Protobufs, test
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 2.5.0, 2.4.9, 2.6.0, 3.0.0-alpha-3
>
>
> This is needed in a couple places in order to test that traces over the IPC
> layer carry correct span names, and it's good hygiene anyway.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3959: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2.5
Apache-HBase commented on pull request #3959: URL: https://github.com/apache/hbase/pull/3959#issuecomment-996354675 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.5 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 0s | branch-2.5 passed | | +1 :green_heart: | compile | 3m 41s | branch-2.5 passed | | +1 :green_heart: | checkstyle | 1m 23s | branch-2.5 passed | | +1 :green_heart: | spotbugs | 2m 31s | branch-2.5 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | the patch passed | | +1 :green_heart: | compile | 3m 35s | the patch passed | | +1 :green_heart: | javac | 3m 35s | the patch passed | | +1 :green_heart: | checkstyle | 1m 17s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 25m 43s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 18s | The patch does not generate ASF License warnings. | | | | 62m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3959/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3959 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux f3b73d1a8f17 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.5 / f1add382bd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 85 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3959/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3958: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2
Apache-HBase commented on pull request #3958: URL: https://github.com/apache/hbase/pull/3958#issuecomment-996346042 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 21s | branch-2 passed | | +1 :green_heart: | compile | 4m 7s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 26s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 49s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 35s | the patch passed | | +1 :green_heart: | compile | 4m 0s | the patch passed | | +1 :green_heart: | javac | 4m 0s | the patch passed | | +1 :green_heart: | checkstyle | 1m 28s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 16m 0s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 55m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3958/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3958 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux cb24d7f11737 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 555fcf42b0 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3958/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3960: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2.4
Apache-HBase commented on pull request #3960: URL: https://github.com/apache/hbase/pull/3960#issuecomment-996342256 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 39s | branch-2.4 passed | | +1 :green_heart: | compile | 3m 15s | branch-2.4 passed | | +1 :green_heart: | checkstyle | 1m 6s | branch-2.4 passed | | +1 :green_heart: | spotbugs | 2m 6s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 18s | the patch passed | | +1 :green_heart: | compile | 3m 14s | the patch passed | | +1 :green_heart: | javac | 3m 14s | the patch passed | | +1 :green_heart: | checkstyle | 1m 5s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 16s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 45m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3960/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3960 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 9d163ff225bb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / d7c3720e52 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3960/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3959: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2.5
Apache-HBase commented on pull request #3959: URL: https://github.com/apache/hbase/pull/3959#issuecomment-996339765 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.5 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 23s | branch-2.5 passed | | +1 :green_heart: | compile | 0m 59s | branch-2.5 passed | | +1 :green_heart: | shadedjars | 7m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | branch-2.5 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 7m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 11m 3s | hbase-server in the patch failed. | | | | 40m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3959/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3959 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 00ab569609be 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.5 / f1add382bd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3959/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3959/1/testReport/ | | Max. process+thread count | 811 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3959/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26542) Apply a `package` to test protobuf files
[ https://issues.apache.org/jira/browse/HBASE-26542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-26542. -- Resolution: Fixed > Apply a `package` to test protobuf files > > > Key: HBASE-26542 > URL: https://issues.apache.org/jira/browse/HBASE-26542 > Project: HBase > Issue Type: Test > Components: Protobufs, test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 2.5.0, 2.4.9, 2.6.0, 3.0.0-alpha-3 > > > This is needed in a couple places in order to test that traces over the IPC > layer carry correct span names, and it's good hygiene anyway. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26542) Apply a `package` to test protobuf files
[ https://issues.apache.org/jira/browse/HBASE-26542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461138#comment-17461138 ] Nick Dimiduk commented on HBASE-26542: -- Addendum pushed to branch-2.4+. Thanks for the quick review, [~psomogyi] > Apply a `package` to test protobuf files > > > Key: HBASE-26542 > URL: https://issues.apache.org/jira/browse/HBASE-26542 > Project: HBase > Issue Type: Test > Components: Protobufs, test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 2.5.0, 2.4.9, 2.6.0, 3.0.0-alpha-3 > > > This is needed in a couple places in order to test that traces over the IPC > layer carry correct span names, and it's good hygiene anyway. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] ndimiduk merged pull request #3960: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2.4
ndimiduk merged pull request #3960: URL: https://github.com/apache/hbase/pull/3960 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #3959: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2.5
ndimiduk merged pull request #3959: URL: https://github.com/apache/hbase/pull/3959 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #3958: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2
ndimiduk merged pull request #3958: URL: https://github.com/apache/hbase/pull/3958 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #3960: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2.4
ndimiduk opened a new pull request #3960: URL: https://github.com/apache/hbase/pull/3960 RpcServer identifies the services it hosts by unqualified service name. Thus, use `getName()` instead of `getFullName()`. See also HBASE-26589. Signed-off-by: Peter Somogyi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #3958: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2
ndimiduk opened a new pull request #3958: URL: https://github.com/apache/hbase/pull/3958 RpcServer identifies the services it hosts by unqualified service name. Thus, use `getName()` instead of `getFullName()`. See also HBASE-26589. Signed-off-by: Peter Somogyi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #3959: Backport "HBASE-26542 Apply a `package` to test protobuf files (addendum)" to branch-2.5
ndimiduk opened a new pull request #3959: URL: https://github.com/apache/hbase/pull/3959 RpcServer identifies the services it hosts by unqualified service name. Thus, use `getName()` instead of `getFullName()`. See also HBASE-26589. Signed-off-by: Peter Somogyi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #3957: HBASE-26542 Apply a `package` to test protobuf files (addendum)
ndimiduk merged pull request #3957: URL: https://github.com/apache/hbase/pull/3957 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #3957: HBASE-26542 Apply a `package` to test protobuf files (addendum)
ndimiduk commented on pull request #3957: URL: https://github.com/apache/hbase/pull/3957#issuecomment-996302802 Test failures are unrelated, and the previously broken test passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26590) Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
[ https://issues.apache.org/jira/browse/HBASE-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461122#comment-17461122 ] Huaxiang Sun commented on HBASE-26590: -- Update, put the fix into a test which does real Table#get(), and still shows there is performance regression, so this is not the only cause. Debugging. > Hbase-client Meta lookup performance regression between hbase-1 and hbase-2 > --- > > Key: HBASE-26590 > URL: https://issues.apache.org/jira/browse/HBASE-26590 > Project: HBase > Issue Type: Improvement > Components: meta >Affects Versions: 2.4.0, 2.5.0, 2.3.7, 2.6.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > One of our users complained higher latency after application upgrades from > hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load > Balance mode during app restart. I reproduced the regression by a test for > meta lookup. > At my test cluster, there are 160k regions for the test table, so there are > 160k entries in meta region. Used one thread to do 1 million meta lookup > against the meta region server. > > ||Version ||Meta Replica Load Balance Enabled||Time || > ||2.4.5-with-fixed||Yes||336458ms|| > ||2.4.5-with-fixed||No||333253ms|| > ||2.4.5||Yes||469980ms|| > ||2.4.5||No||470515ms|| > | *cdh-5.16.2*| *No* | *323412ms*| > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3957: HBASE-26542 Apply a `package` to test protobuf files (addendum)
Apache-HBase commented on pull request #3957: URL: https://github.com/apache/hbase/pull/3957#issuecomment-996272875 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 49s | master passed | | +1 :green_heart: | compile | 1m 32s | master passed | | +1 :green_heart: | shadedjars | 10m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 37s | the patch passed | | +1 :green_heart: | compile | 1m 33s | the patch passed | | +1 :green_heart: | javac | 1m 33s | the patch passed | | +1 :green_heart: | shadedjars | 10m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 230m 54s | hbase-server in the patch passed. | | | | 270m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3957/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3957 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 86987dd0500c 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c93e4572c1 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3957/1/testReport/ | | Max. process+thread count | 3104 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3957/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461120#comment-17461120
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #10 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/10/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/10/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/10/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/10/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26498) [hbase-thirdparty] Make a 4.0.0 release for hbase-thirdparty
[ https://issues.apache.org/jira/browse/HBASE-26498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461116#comment-17461116 ] Nick Dimiduk commented on HBASE-26498: -- This is finished, [~zhangduo]? Thanks! > [hbase-thirdparty] Make a 4.0.0 release for hbase-thirdparty > > > Key: HBASE-26498 > URL: https://issues.apache.org/jira/browse/HBASE-26498 > Project: HBase > Issue Type: Umbrella > Components: community, hbase-thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26523) Upgrade hbase-thirdparty dependency to 4.0.0
[ https://issues.apache.org/jira/browse/HBASE-26523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461114#comment-17461114 ] Nick Dimiduk commented on HBASE-26523: -- [~apurtell] I took the liberty of marking this as a blocker for 2.5.0. > Upgrade hbase-thirdparty dependency to 4.0.0 > > > Key: HBASE-26523 > URL: https://issues.apache.org/jira/browse/HBASE-26523 > Project: HBase > Issue Type: Task > Components: thirdparty >Affects Versions: 2.5.0, 2.6.0, 3.0.0-alpha-3 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26523) Upgrade hbase-thirdparty dependency to 4.0.0
[ https://issues.apache.org/jira/browse/HBASE-26523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-26523: - Affects Version/s: 2.5.0 2.6.0 3.0.0-alpha-3 > Upgrade hbase-thirdparty dependency to 4.0.0 > > > Key: HBASE-26523 > URL: https://issues.apache.org/jira/browse/HBASE-26523 > Project: HBase > Issue Type: Task > Components: thirdparty >Affects Versions: 2.5.0, 2.6.0, 3.0.0-alpha-3 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26523) Upgrade hbase-thirdparty dependency to 4.0.0
[ https://issues.apache.org/jira/browse/HBASE-26523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-26523: - Priority: Blocker (was: Major) > Upgrade hbase-thirdparty dependency to 4.0.0 > > > Key: HBASE-26523 > URL: https://issues.apache.org/jira/browse/HBASE-26523 > Project: HBase > Issue Type: Task > Components: thirdparty >Affects Versions: 2.5.0, 2.6.0, 3.0.0-alpha-3 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase-operator-tools] tsuna commented on pull request #96: HBASE-26571 Upgrade to log4j 2.16.0
tsuna commented on pull request #96: URL: https://github.com/apache/hbase-operator-tools/pull/96#issuecomment-996248327 What's the ETA to cut the 1.2.0 release? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-26590) Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
[ https://issues.apache.org/jira/browse/HBASE-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461066#comment-17461066 ] Huaxiang Sun edited comment on HBASE-26590 at 12/16/21, 10:12 PM: -- It is hard to compare with the master branch, as it saves the fetched extra locations into client's meta cache. was (Author: huaxiangsun): It is hard to compare with the master branch, as it saves the fetched locations into client's meta cache. > Hbase-client Meta lookup performance regression between hbase-1 and hbase-2 > --- > > Key: HBASE-26590 > URL: https://issues.apache.org/jira/browse/HBASE-26590 > Project: HBase > Issue Type: Improvement > Components: meta >Affects Versions: 2.4.0, 2.5.0, 2.3.7, 2.6.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > One of our users complained higher latency after application upgrades from > hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load > Balance mode during app restart. I reproduced the regression by a test for > meta lookup. > At my test cluster, there are 160k regions for the test table, so there are > 160k entries in meta region. Used one thread to do 1 million meta lookup > against the meta region server. > > ||Version ||Meta Replica Load Balance Enabled||Time || > ||2.4.5-with-fixed||Yes||336458ms|| > ||2.4.5-with-fixed||No||333253ms|| > ||2.4.5||Yes||469980ms|| > ||2.4.5||No||470515ms|| > | *cdh-5.16.2*| *No* | *323412ms*| > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26590) Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
[ https://issues.apache.org/jira/browse/HBASE-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461066#comment-17461066 ] Huaxiang Sun commented on HBASE-26590: -- It is hard to compare with the master branch, as it saves the fetched locations into client's meta cache. > Hbase-client Meta lookup performance regression between hbase-1 and hbase-2 > --- > > Key: HBASE-26590 > URL: https://issues.apache.org/jira/browse/HBASE-26590 > Project: HBase > Issue Type: Improvement > Components: meta >Affects Versions: 2.4.0, 2.5.0, 2.3.7, 2.6.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > One of our users complained higher latency after application upgrades from > hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load > Balance mode during app restart. I reproduced the regression by a test for > meta lookup. > At my test cluster, there are 160k regions for the test table, so there are > 160k entries in meta region. Used one thread to do 1 million meta lookup > against the meta region server. > > ||Version ||Meta Replica Load Balance Enabled||Time || > ||2.4.5-with-fixed||Yes||336458ms|| > ||2.4.5-with-fixed||No||333253ms|| > ||2.4.5||Yes||469980ms|| > ||2.4.5||No||470515ms|| > | *cdh-5.16.2*| *No* | *323412ms*| > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3957: HBASE-26542 Apply a `package` to test protobuf files (addendum)
Apache-HBase commented on pull request #3957: URL: https://github.com/apache/hbase/pull/3957#issuecomment-996225516 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 8m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 10s | the patch passed | | +1 :green_heart: | compile | 1m 8s | the patch passed | | +1 :green_heart: | javac | 1m 8s | the patch passed | | +1 :green_heart: | shadedjars | 8m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 151m 6s | hbase-server in the patch failed. | | | | 182m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3957/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3957 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dd3ca6fb0b09 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c93e4572c1 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3957/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3957/1/testReport/ | | Max. process+thread count | 4529 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3957/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26590) Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
[ https://issues.apache.org/jira/browse/HBASE-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-26590: - Affects Version/s: (was: 3.0.0-alpha-1) > Hbase-client Meta lookup performance regression between hbase-1 and hbase-2 > --- > > Key: HBASE-26590 > URL: https://issues.apache.org/jira/browse/HBASE-26590 > Project: HBase > Issue Type: Improvement > Components: meta >Affects Versions: 2.4.0, 2.5.0, 2.3.7, 2.6.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > One of our users complained higher latency after application upgrades from > hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load > Balance mode during app restart. I reproduced the regression by a test for > meta lookup. > At my test cluster, there are 160k regions for the test table, so there are > 160k entries in meta region. Used one thread to do 1 million meta lookup > against the meta region server. > > ||Version ||Meta Replica Load Balance Enabled||Time || > ||2.4.5-with-fixed||Yes||336458ms|| > ||2.4.5-with-fixed||No||333253ms|| > ||2.4.5||Yes||469980ms|| > ||2.4.5||No||470515ms|| > | *cdh-5.16.2*| *No* | *323412ms*| > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26265) Update ref guide to mention the new store file tracker implementations
[ https://issues.apache.org/jira/browse/HBASE-26265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-26265: --- Hadoop Flags: Reviewed > Update ref guide to mention the new store file tracker implementations > -- > > Key: HBASE-26265 > URL: https://issues.apache.org/jira/browse/HBASE-26265 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Wellington Chevreuil >Priority: Major > Fix For: HBASE-26067 > > > For example, when to use these store file trackers. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461046#comment-17461046 ] Josh Elser commented on HBASE-26067: I've just merged the book updates from Wellington. We have no more open tasks here :) Let me put together a VOTE thread (given Duo's suggestion to still do a formal vote). > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26265) Update ref guide to mention the new store file tracker implementations
[ https://issues.apache.org/jira/browse/HBASE-26265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved HBASE-26265. Resolution: Fixed > Update ref guide to mention the new store file tracker implementations > -- > > Key: HBASE-26265 > URL: https://issues.apache.org/jira/browse/HBASE-26265 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Wellington Chevreuil >Priority: Major > Fix For: HBASE-26067 > > > For example, when to use these store file trackers. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26265) Update ref guide to mention the new store file tracker implementations
[ https://issues.apache.org/jira/browse/HBASE-26265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-26265: --- Fix Version/s: HBASE-26067 > Update ref guide to mention the new store file tracker implementations > -- > > Key: HBASE-26265 > URL: https://issues.apache.org/jira/browse/HBASE-26265 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Wellington Chevreuil >Priority: Major > Fix For: HBASE-26067 > > > For example, when to use these store file trackers. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] joshelser merged pull request #3942: HBASE-26265 Update ref guide to mention the new store file tracker im…
joshelser merged pull request #3942: URL: https://github.com/apache/hbase/pull/3942 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3950: HBASE-26540 Implement a tool to verify the lag of region replication
Apache-HBase commented on pull request #3950: URL: https://github.com/apache/hbase/pull/3950#issuecomment-996185958 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-26233 Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 45s | HBASE-26233 passed | | +1 :green_heart: | compile | 1m 50s | HBASE-26233 passed | | +1 :green_heart: | shadedjars | 10m 52s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | HBASE-26233 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 7s | the patch passed | | +1 :green_heart: | compile | 1m 41s | the patch passed | | +1 :green_heart: | javac | 1m 41s | the patch passed | | +1 :green_heart: | shadedjars | 10m 16s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 241m 1s | hbase-server in the patch passed. | | | | 283m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3950/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3950 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f90f95319aa9 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-26233 / 5c18b67c78 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3950/3/testReport/ | | Max. process+thread count | 3404 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3950/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26590) Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
[ https://issues.apache.org/jira/browse/HBASE-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-26590: - Affects Version/s: 2.4.0 2.5.0 2.6.0 > Hbase-client Meta lookup performance regression between hbase-1 and hbase-2 > --- > > Key: HBASE-26590 > URL: https://issues.apache.org/jira/browse/HBASE-26590 > Project: HBase > Issue Type: Improvement > Components: meta >Affects Versions: 3.0.0-alpha-1, 2.4.0, 2.5.0, 2.3.7, 2.6.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > One of our users complained higher latency after application upgrades from > hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load > Balance mode during app restart. I reproduced the regression by a test for > meta lookup. > At my test cluster, there are 160k regions for the test table, so there are > 160k entries in meta region. Used one thread to do 1 million meta lookup > against the meta region server. > > ||Version ||Meta Replica Load Balance Enabled||Time || > ||2.4.5-with-fixed||Yes||336458ms|| > ||2.4.5-with-fixed||No||333253ms|| > ||2.4.5||Yes||469980ms|| > ||2.4.5||No||470515ms|| > | *cdh-5.16.2*| *No* | *323412ms*| > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26590) Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
[
https://issues.apache.org/jira/browse/HBASE-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461031#comment-17461031
]
Huaxiang Sun commented on HBASE-26590:
--
I debugged the code, found that this regression is caused by the following
line. It now asks region server to return 5 rows, which will take more time for
region server to process. This change was introduced in HBASE-20182, which in
most normal cases, the extra 4 rows returned are discarded. The proposed fix is
to revert back to the hbase-1 behavior, i.e, ask for 1 row in meta scan. For
the corner case fixed by HBASE-20182, it will go back to meta region server
couple more times for the correct location.
{code:java}
diff --git
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
b/hbase-
client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
index 9145c55c0a..6039387b6e 100644
---
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
+++
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
@@ -888,7 +888,7 @@ class ConnectionImplementation implements
ClusterConnection, Closeable {
byte[] metaStopKey =
RegionInfo.createRegionName(tableName, HConstants.EMPTY_START_ROW, "",
false);
Scan s = new Scan().withStartRow(metaStartKey).withStopRow(metaStopKey,
true)
- .addFamily(HConstants.CATALOG_FAMILY).setReversed(true).setCaching(5)
+ .addFamily(HConstants.CATALOG_FAMILY).setReversed(true).setCaching(1)
.setReadType(ReadType.PREAD);
switch (this.metaReplicaMode) { {code}
> Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
> ---
>
> Key: HBASE-26590
> URL: https://issues.apache.org/jira/browse/HBASE-26590
> Project: HBase
> Issue Type: Improvement
> Components: meta
>Affects Versions: 3.0.0-alpha-1, 2.3.7
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Major
>
> One of our users complained higher latency after application upgrades from
> hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load
> Balance mode during app restart. I reproduced the regression by a test for
> meta lookup.
> At my test cluster, there are 160k regions for the test table, so there are
> 160k entries in meta region. Used one thread to do 1 million meta lookup
> against the meta region server.
>
> ||Version ||Meta Replica Load Balance Enabled||Time ||
> ||2.4.5-with-fixed||Yes||336458ms||
> ||2.4.5-with-fixed||No||333253ms||
> ||2.4.5||Yes||469980ms||
> ||2.4.5||No||470515ms||
> | *cdh-5.16.2*| *No* | *323412ms*|
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26590) Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
[ https://issues.apache.org/jira/browse/HBASE-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461032#comment-17461032 ] Huaxiang Sun commented on HBASE-26590: -- 2.4.5-with-fixed is the release with the proposed fix. With that, meta lookup time is similar to cdh-5.16.2's. > Hbase-client Meta lookup performance regression between hbase-1 and hbase-2 > --- > > Key: HBASE-26590 > URL: https://issues.apache.org/jira/browse/HBASE-26590 > Project: HBase > Issue Type: Improvement > Components: meta >Affects Versions: 3.0.0-alpha-1, 2.3.7 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > One of our users complained higher latency after application upgrades from > hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load > Balance mode during app restart. I reproduced the regression by a test for > meta lookup. > At my test cluster, there are 160k regions for the test table, so there are > 160k entries in meta region. Used one thread to do 1 million meta lookup > against the meta region server. > > ||Version ||Meta Replica Load Balance Enabled||Time || > ||2.4.5-with-fixed||Yes||336458ms|| > ||2.4.5-with-fixed||No||333253ms|| > ||2.4.5||Yes||469980ms|| > ||2.4.5||No||470515ms|| > | *cdh-5.16.2*| *No* | *323412ms*| > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26590) Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
[ https://issues.apache.org/jira/browse/HBASE-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-26590: - Description: One of our users complained higher latency after application upgrades from hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load Balance mode during app restart. I reproduced the regression by a test for meta lookup. At my test cluster, there are 160k regions for the test table, so there are 160k entries in meta region. Used one thread to do 1 million meta lookup against the meta region server. ||Version ||Meta Replica Load Balance Enabled||Time || ||2.4.5-with-fixed||Yes||336458ms|| ||2.4.5-with-fixed||No||333253ms|| ||2.4.5||Yes||469980ms|| ||2.4.5||No||470515ms|| | *cdh-5.16.2*| *No* | *323412ms*| was: One of our users complained higher latency after application upgrades from hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load Balance mode during app restart. I reproduced the regression by a test for meta lookup. At my test cluster, there are 160k regions for the test table, so there are 160k entries in meta region. Used one thread to do 1 million meta lookup against the meta region server. > Hbase-client Meta lookup performance regression between hbase-1 and hbase-2 > --- > > Key: HBASE-26590 > URL: https://issues.apache.org/jira/browse/HBASE-26590 > Project: HBase > Issue Type: Improvement > Components: meta >Affects Versions: 3.0.0-alpha-1, 2.3.7 > Environment: ||Version ||Meta Replica Load Balance Enabled||Time > || > ||2.4.5-with-fixed||Yes||336458ms|| > ||2.4.5-with-fixed||No||333253ms|| > ||2.4.5||Yes||469980ms|| > ||2.4.5||No||470515ms|| > | *cdh-5.16.2*| *No* | *323412ms*| >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > One of our users complained higher latency after application upgrades from > hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load > Balance mode during app restart. I reproduced the regression by a test for > meta lookup. > At my test cluster, there are 160k regions for the test table, so there are > 160k entries in meta region. Used one thread to do 1 million meta lookup > against the meta region server. > > ||Version ||Meta Replica Load Balance Enabled||Time || > ||2.4.5-with-fixed||Yes||336458ms|| > ||2.4.5-with-fixed||No||333253ms|| > ||2.4.5||Yes||469980ms|| > ||2.4.5||No||470515ms|| > | *cdh-5.16.2*| *No* | *323412ms*| > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HBASE-26590) Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
[ https://issues.apache.org/jira/browse/HBASE-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-26590: - Environment: (was: ||Version ||Meta Replica Load Balance Enabled||Time || ||2.4.5-with-fixed||Yes||336458ms|| ||2.4.5-with-fixed||No||333253ms|| ||2.4.5||Yes||469980ms|| ||2.4.5||No||470515ms|| | *cdh-5.16.2*| *No* | *323412ms*|) > Hbase-client Meta lookup performance regression between hbase-1 and hbase-2 > --- > > Key: HBASE-26590 > URL: https://issues.apache.org/jira/browse/HBASE-26590 > Project: HBase > Issue Type: Improvement > Components: meta >Affects Versions: 3.0.0-alpha-1, 2.3.7 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > One of our users complained higher latency after application upgrades from > hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load > Balance mode during app restart. I reproduced the regression by a test for > meta lookup. > At my test cluster, there are 160k regions for the test table, so there are > 160k entries in meta region. Used one thread to do 1 million meta lookup > against the meta region server. > > ||Version ||Meta Replica Load Balance Enabled||Time || > ||2.4.5-with-fixed||Yes||336458ms|| > ||2.4.5-with-fixed||No||333253ms|| > ||2.4.5||Yes||469980ms|| > ||2.4.5||No||470515ms|| > | *cdh-5.16.2*| *No* | *323412ms*| > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26590) Hbase-client Meta lookup performance regression between hbase-1 and hbase-2
Huaxiang Sun created HBASE-26590: Summary: Hbase-client Meta lookup performance regression between hbase-1 and hbase-2 Key: HBASE-26590 URL: https://issues.apache.org/jira/browse/HBASE-26590 Project: HBase Issue Type: Improvement Components: meta Affects Versions: 2.3.7, 3.0.0-alpha-1 Environment: ||Version ||Meta Replica Load Balance Enabled||Time || ||2.4.5-with-fixed||Yes||336458ms|| ||2.4.5-with-fixed||No||333253ms|| ||2.4.5||Yes||469980ms|| ||2.4.5||No||470515ms|| | *cdh-5.16.2*| *No* | *323412ms*| Reporter: Huaxiang Sun Assignee: Huaxiang Sun One of our users complained higher latency after application upgrades from hbase-1.2 client (CDH-5.16.2) to hbase-2.4.5 client with meta replica Load Balance mode during app restart. I reproduced the regression by a test for meta lookup. At my test cluster, there are 160k regions for the test table, so there are 160k entries in meta region. Used one thread to do 1 million meta lookup against the meta region server. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3957: HBASE-26542 Apply a `package` to test protobuf files (addendum)
Apache-HBase commented on pull request #3957: URL: https://github.com/apache/hbase/pull/3957#issuecomment-996146604 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 31s | master passed | | +1 :green_heart: | compile | 3m 22s | master passed | | +1 :green_heart: | checkstyle | 1m 13s | master passed | | +1 :green_heart: | spotbugs | 2m 16s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | the patch passed | | +1 :green_heart: | compile | 3m 26s | the patch passed | | +1 :green_heart: | javac | 3m 26s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 21m 45s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 2m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 55m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3957/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3957 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4769231ef2bc 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c93e4572c1 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3957/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #3957: HBASE-26542 Apply a `package` to test protobuf files (addendum)
ndimiduk commented on pull request #3957: URL: https://github.com/apache/hbase/pull/3957#issuecomment-996120131 > Your recommended solution in [HBASE-26589](https://issues.apache.org/jira/browse/HBASE-26589) makes sense. Maybe there's an argument to be made in favor of sending smallest messages possible? Perhaps the service should accept the `name` so long as it is unambiguous amongst the registered services. I don't think it likely that we'd host multiple services from different packages. RpcServer startup should probably make some assertions about the services it is hosting, that their unqualified names are unambiguous. As of now, it looks like we have a first-registered-wins scenario. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3950: HBASE-26540 Implement a tool to verify the lag of region replication
Apache-HBase commented on pull request #3950: URL: https://github.com/apache/hbase/pull/3950#issuecomment-996113598 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-26233 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 23s | HBASE-26233 passed | | +1 :green_heart: | compile | 1m 6s | HBASE-26233 passed | | +1 :green_heart: | shadedjars | 8m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | HBASE-26233 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 9m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 163m 53s | hbase-server in the patch failed. | | | | 198m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3950/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3950 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 91365cc63d73 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-26233 / 5c18b67c78 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3950/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3950/3/testReport/ | | Max. process+thread count | 4435 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3950/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #3957: HBASE-26542 Apply a `package` to test protobuf files (addendum)
ndimiduk opened a new pull request #3957: URL: https://github.com/apache/hbase/pull/3957 RpcServer identifies the services it hosts by unqualified service name. Thus, use `getName()` instead of `getFullName()`. See also HBASE-26589. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26589) Protobuf Services should be resolved by their full name
[
https://issues.apache.org/jira/browse/HBASE-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-26589:
-
Description:
I noticed this while investigating test failures resulting from HBASE-26542.
The test in question ({{TestRpcServerSlowConnectionSetup}}) builds a connection
header by populating the {{serviceName}} field with the service descriptor's
{{getFullName()}} method. The value returned by {{getFullName()}} is the
protobuf service's package and name. However, down in {{RpcServer}}, we resolve
the service named by the connection header using the service descriptor's
{{getName()}} method. {{getName()}} returns the protobuf service's name,
without the package.
It is my opinion that this is a bug in the {{RpcServer}}, it should resolve
services using their fully qualified name.
was:
I noticed this while investigating test failures resulting from HBASE-26472.
The test in question ({{TestRpcServerSlowConnectionSetup}}) builds a connection
header by populating the {{serviceName}} field with the service descriptor's
{{getFullName()}} method. The value returned by {{getFullName()}} is the
protobuf service's package and name. However, down in {{RpcServer}}, we resolve
the service named by the connection header using the service descriptor's
{{getName()}} method. {{getName()}} returns the protobuf service's name,
without the package.
It is my opinion that this is a bug in the {{RpcServer}}, it should resolve
services using their fully qualified name.
> Protobuf Services should be resolved by their full name
> ---
>
> Key: HBASE-26589
> URL: https://issues.apache.org/jira/browse/HBASE-26589
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0
>Reporter: Nick Dimiduk
>Priority: Major
>
> I noticed this while investigating test failures resulting from HBASE-26542.
> The test in question ({{TestRpcServerSlowConnectionSetup}}) builds a
> connection header by populating the {{serviceName}} field with the service
> descriptor's {{getFullName()}} method. The value returned by
> {{getFullName()}} is the protobuf service's package and name. However, down
> in {{RpcServer}}, we resolve the service named by the connection header using
> the service descriptor's {{getName()}} method. {{getName()}} returns the
> protobuf service's name, without the package.
> It is my opinion that this is a bug in the {{RpcServer}}, it should resolve
> services using their fully qualified name.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26472) Adhere to semantic conventions regarding table data operations
[ https://issues.apache.org/jira/browse/HBASE-26472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-26472. -- Resolution: Fixed The problem is actually HBASE-26542. > Adhere to semantic conventions regarding table data operations > -- > > Key: HBASE-26472 > URL: https://issues.apache.org/jira/browse/HBASE-26472 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > > Follow the guidance outlined in > https://github.com/open-telemetry/opentelemetry-specification/blob/3e380e2/specification/trace/semantic_conventions/database.dm -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Reopened] (HBASE-26542) Apply a `package` to test protobuf files
[ https://issues.apache.org/jira/browse/HBASE-26542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reopened HBASE-26542: -- This breaks an RPC test. Reopening for addendum. > Apply a `package` to test protobuf files > > > Key: HBASE-26542 > URL: https://issues.apache.org/jira/browse/HBASE-26542 > Project: HBase > Issue Type: Test > Components: Protobufs, test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 2.5.0, 2.4.9, 2.6.0, 3.0.0-alpha-3 > > > This is needed in a couple places in order to test that traces over the IPC > layer carry correct span names, and it's good hygiene anyway. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26589) Protobuf Services should be resolved by their full name
Nick Dimiduk created HBASE-26589:
Summary: Protobuf Services should be resolved by their full name
Key: HBASE-26589
URL: https://issues.apache.org/jira/browse/HBASE-26589
Project: HBase
Issue Type: Task
Components: IPC/RPC
Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0
Reporter: Nick Dimiduk
I noticed this while investigating test failures resulting from HBASE-26472.
The test in question ({{TestRpcServerSlowConnectionSetup}}) builds a connection
header by populating the {{serviceName}} field with the service descriptor's
{{getFullName()}} method. The value returned by {{getFullName()}} is the
protobuf service's package and name. However, down in {{RpcServer}}, we resolve
the service named by the connection header using the service descriptor's
{{getName()}} method. {{getName()}} returns the protobuf service's name,
without the package.
It is my opinion that this is a bug in the {{RpcServer}}, it should resolve
services using their fully qualified name.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26589) Protobuf Services should be resolved by their full name
[
https://issues.apache.org/jira/browse/HBASE-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-26589:
-
Issue Type: Bug (was: Task)
> Protobuf Services should be resolved by their full name
> ---
>
> Key: HBASE-26589
> URL: https://issues.apache.org/jira/browse/HBASE-26589
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0
>Reporter: Nick Dimiduk
>Priority: Major
>
> I noticed this while investigating test failures resulting from HBASE-26472.
> The test in question ({{TestRpcServerSlowConnectionSetup}}) builds a
> connection header by populating the {{serviceName}} field with the service
> descriptor's {{getFullName()}} method. The value returned by
> {{getFullName()}} is the protobuf service's package and name. However, down
> in {{RpcServer}}, we resolve the service named by the connection header using
> the service descriptor's {{getName()}} method. {{getName()}} returns the
> protobuf service's name, without the package.
> It is my opinion that this is a bug in the {{RpcServer}}, it should resolve
> services using their fully qualified name.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3942: HBASE-26265 Update ref guide to mention the new store file tracker im…
Apache-HBase commented on pull request #3942: URL: https://github.com/apache/hbase/pull/3942#issuecomment-996080995 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-26067 Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 3s | HBASE-26067 passed | | +0 :ok: | refguide | 5m 45s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 1s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 5m 17s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 26m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3942/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3942 | | Optional Tests | dupname asflicense refguide | | uname | Linux 3ea6e14b5641 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-26067 / 81c9b8793e | | refguide | https://nightlies.apache.org/hbase/HBase/HBase-PreCommit-GitHub-PR/PR-3942/3/yetus-general-check/output/branch-site/book.html | | refguide | https://nightlies.apache.org/hbase/HBase/HBase-PreCommit-GitHub-PR/PR-3942/3/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 63 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3942/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3942: HBASE-26265 Update ref guide to mention the new store file tracker im…
Apache-HBase commented on pull request #3942: URL: https://github.com/apache/hbase/pull/3942#issuecomment-996063325 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-26067 Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 2m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3942/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3942 | | Optional Tests | | | uname | Linux 13fc112c8371 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-26067 / 81c9b8793e | | Max. process+thread count | 41 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3942/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3942: HBASE-26265 Update ref guide to mention the new store file tracker im…
Apache-HBase commented on pull request #3942: URL: https://github.com/apache/hbase/pull/3942#issuecomment-996062878 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-26067 Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 1m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3942/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3942 | | Optional Tests | | | uname | Linux b37a2f259a19 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-26067 / 81c9b8793e | | Max. process+thread count | 54 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3942/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (HBASE-26472) Adhere to semantic conventions regarding table data operations
[ https://issues.apache.org/jira/browse/HBASE-26472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reopened HBASE-26472: -- Broken test on branch-2.5 looks related. Investigating. > Adhere to semantic conventions regarding table data operations > -- > > Key: HBASE-26472 > URL: https://issues.apache.org/jira/browse/HBASE-26472 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > > Follow the guidance outlined in > https://github.com/open-telemetry/opentelemetry-specification/blob/3e380e2/specification/trace/semantic_conventions/database.dm -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26542) Apply a `package` to test protobuf files
[
https://issues.apache.org/jira/browse/HBASE-26542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460918#comment-17460918
]
Hudson commented on HBASE-26542:
Results for branch branch-2
[build #419 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/419/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/419/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/419/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/419/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/419/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Apply a `package` to test protobuf files
>
>
> Key: HBASE-26542
> URL: https://issues.apache.org/jira/browse/HBASE-26542
> Project: HBase
> Issue Type: Test
> Components: Protobufs, test
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 2.5.0, 2.4.9, 2.6.0, 3.0.0-alpha-3
>
>
> This is needed in a couple places in order to test that traces over the IPC
> layer carry correct span names, and it's good hygiene anyway.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3950: HBASE-26540 Implement a tool to verify the lag of region replication
Apache-HBase commented on pull request #3950: URL: https://github.com/apache/hbase/pull/3950#issuecomment-996021615 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-26233 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 42s | HBASE-26233 passed | | +1 :green_heart: | compile | 4m 19s | HBASE-26233 passed | | +1 :green_heart: | checkstyle | 1m 34s | HBASE-26233 passed | | +1 :green_heart: | spotbugs | 2m 44s | HBASE-26233 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 53s | the patch passed | | +1 :green_heart: | compile | 4m 30s | the patch passed | | +1 :green_heart: | javac | 4m 30s | the patch passed | | +1 :green_heart: | checkstyle | 1m 30s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 29m 3s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 3m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 71m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3950/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3950 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 03a80e95af65 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-26233 / 5c18b67c78 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3950/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26588) Implement a migration tool to help users migrate SFT implementation for a large set of tables
[ https://issues.apache.org/jira/browse/HBASE-26588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460845#comment-17460845 ] Duo Zhang commented on HBASE-26588: --- I think the tool will be straight forward, just call the API in HBASE-26587 to change the SFT implementation for all tables on the cluster. A possible configurable value is the max concurrency, i.e, how many tables can be altered concurrently. > Implement a migration tool to help users migrate SFT implementation for a > large set of tables > - > > Key: HBASE-26588 > URL: https://issues.apache.org/jira/browse/HBASE-26588 > Project: HBase > Issue Type: Sub-task > Components: tooling >Reporter: Duo Zhang >Priority: Major > > It will be very useful for our users who deploy HBase on S3 like systems. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26587) Introduce a new Admin API to change SFT implementation
[ https://issues.apache.org/jira/browse/HBASE-26587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460843#comment-17460843 ] Duo Zhang commented on HBASE-26587: --- I would like to introduce a new procedure, which just schedule two MTP as sub procedures, one for switching from SRC to MIGRATION, and the second for switching from MIGRATION to DST. > Introduce a new Admin API to change SFT implementation > -- > > Key: HBASE-26587 > URL: https://issues.apache.org/jira/browse/HBASE-26587 > Project: HBase > Issue Type: Sub-task > Components: Admin, API >Reporter: Duo Zhang >Priority: Major > > Now changing SFT implementation requires two MTP, and there are also lots of > rules on how to correctly set the MIGRATION SFT, I think we'd better > introduce a special method to implement the complicated logic by code,to make > our users life easier. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] Apache9 commented on pull request #3950: HBASE-26540 Implement a tool to verify the lag of region replication
Apache9 commented on pull request #3950: URL: https://github.com/apache/hbase/pull/3950#issuecomment-995965259 Have already used this tool to compare the region replication lag for HBASE-26233 and master. Please see the comments on HBASE-26487. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26487) Run some tests to verify the new region replication framework
[
https://issues.apache.org/jira/browse/HBASE-26487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460835#comment-17460835
]
Duo Zhang commented on HBASE-26487:
---
And I also executed a 100 rows test for HBASE-26233 branch
The command is
{noformat}
./bin/hbase org.apache.hadoop.hbase.RegionReplicationLagEvaluation -vlen 1024
-r 100
{noformat}
The result are
{noformat}
2021-12-17T00:03:39,046 INFO [main] hbase.RegionReplicationLagEvaluation: Test
finished, min lag 0 ms, max lag 335 ms, mean lag 0 ms
2021-12-17T00:03:39,046 INFO [main] hbase.RegionReplicationLagEvaluation:
25.0% lag: 0 ms
2021-12-17T00:03:39,047 INFO [main] hbase.RegionReplicationLagEvaluation:
50.0% lag: 0 ms
2021-12-17T00:03:39,047 INFO [main] hbase.RegionReplicationLagEvaluation:
75.0% lag: 0 ms
2021-12-17T00:03:39,047 INFO [main] hbase.RegionReplicationLagEvaluation:
90.0% lag: 0 ms
2021-12-17T00:03:39,047 INFO [main] hbase.RegionReplicationLagEvaluation:
95.0% lag: 0 ms
2021-12-17T00:03:39,047 INFO [main] hbase.RegionReplicationLagEvaluation:
98.0% lag: 0 ms
2021-12-17T00:03:39,047 INFO [main] hbase.RegionReplicationLagEvaluation:
99.0% lag: 1 ms
2021-12-17T00:03:39,047 INFO [main] hbase.RegionReplicationLagEvaluation:
99.9% lag: 3 ms
{noformat}
During the test, the region is split, and is also flushed several times. But
the 99.9% lag is still same, and the max lag is only 335 ms.
> Run some tests to verify the new region replication framework
> -
>
> Key: HBASE-26487
> URL: https://issues.apache.org/jira/browse/HBASE-26487
> Project: HBase
> Issue Type: Sub-task
> Components: integration tests, test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Make sure there is no big bugs before merging back.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26487) Run some tests to verify the new region replication framework
[
https://issues.apache.org/jira/browse/HBASE-26487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460826#comment-17460826
]
Duo Zhang commented on HBASE-26487:
---
Use the RegionReplicationLagEvaluation tool introduce in HBASE-26540 to compare
the replication lag for both master and HBASE-26233 branch.
For HBASE-26233, executed the command 3 times
{noformat}
./bin/hbase org.apache.hadoop.hbase.RegionReplicationLagEvaluation -vlen 1024
-r 1
{noformat}
The results are
{noformat}
2021-12-16T23:16:50,280 INFO [main] hbase.RegionReplicationLagEvaluation: Test
finished, min lag 0 ms, max lag 22 ms, mean lag 0 ms
2021-12-16T23:16:50,280 INFO [main] hbase.RegionReplicationLagEvaluation:
25.0% lag: 0 ms
2021-12-16T23:16:50,280 INFO [main] hbase.RegionReplicationLagEvaluation:
50.0% lag: 0 ms
2021-12-16T23:16:50,280 INFO [main] hbase.RegionReplicationLagEvaluation:
75.0% lag: 0 ms
2021-12-16T23:16:50,280 INFO [main] hbase.RegionReplicationLagEvaluation:
90.0% lag: 0 ms
2021-12-16T23:16:50,280 INFO [main] hbase.RegionReplicationLagEvaluation:
95.0% lag: 0 ms
2021-12-16T23:16:50,280 INFO [main] hbase.RegionReplicationLagEvaluation:
98.0% lag: 1 ms
2021-12-16T23:16:50,280 INFO [main] hbase.RegionReplicationLagEvaluation:
99.0% lag: 2 ms
2021-12-16T23:16:50,280 INFO [main] hbase.RegionReplicationLagEvaluation:
99.9% lag: 3 ms
2021-12-16T23:17:20,229 INFO [main] hbase.RegionReplicationLagEvaluation: Test
finished, min lag 0 ms, max lag 7 ms, mean lag 0 ms
2021-12-16T23:17:20,229 INFO [main] hbase.RegionReplicationLagEvaluation:
25.0% lag: 0 ms
2021-12-16T23:17:20,229 INFO [main] hbase.RegionReplicationLagEvaluation:
50.0% lag: 0 ms
2021-12-16T23:17:20,229 INFO [main] hbase.RegionReplicationLagEvaluation:
75.0% lag: 0 ms
2021-12-16T23:17:20,230 INFO [main] hbase.RegionReplicationLagEvaluation:
90.0% lag: 0 ms
2021-12-16T23:17:20,230 INFO [main] hbase.RegionReplicationLagEvaluation:
95.0% lag: 0 ms
2021-12-16T23:17:20,230 INFO [main] hbase.RegionReplicationLagEvaluation:
98.0% lag: 0 ms
2021-12-16T23:17:20,230 INFO [main] hbase.RegionReplicationLagEvaluation:
99.0% lag: 0 ms
2021-12-16T23:17:20,230 INFO [main] hbase.RegionReplicationLagEvaluation:
99.9% lag: 3 ms
2021-12-16T23:19:25,922 INFO [main] hbase.RegionReplicationLagEvaluation: Test
finished, min lag 0 ms, max lag 17 ms, mean lag 0 ms
2021-12-16T23:19:25,922 INFO [main] hbase.RegionReplicationLagEvaluation:
25.0% lag: 0 ms
2021-12-16T23:19:25,923 INFO [main] hbase.RegionReplicationLagEvaluation:
50.0% lag: 0 ms
2021-12-16T23:19:25,923 INFO [main] hbase.RegionReplicationLagEvaluation:
75.0% lag: 0 ms
2021-12-16T23:19:25,923 INFO [main] hbase.RegionReplicationLagEvaluation:
90.0% lag: 0 ms
2021-12-16T23:19:25,923 INFO [main] hbase.RegionReplicationLagEvaluation:
95.0% lag: 0 ms
2021-12-16T23:19:25,923 INFO [main] hbase.RegionReplicationLagEvaluation:
98.0% lag: 0 ms
2021-12-16T23:19:25,923 INFO [main] hbase.RegionReplicationLagEvaluation:
99.0% lag: 0 ms
2021-12-16T23:19:25,923 INFO [main] hbase.RegionReplicationLagEvaluation:
99.9% lag: 3 ms
{noformat}
For master branch, I set the replication.source.sleepforretries to 100(ms)
since the default 1000(ms) will make the test running for a very long time. And
to make the test finish faster, I reduced the number of rows to 1000 instead of
1, and also executed 3 times.
{noformat}
./bin/hbase org.apache.hadoop.hbase.RegionReplicationLagEvaluation -vlen 1024
-r 1000
{noformat}
The results are
{noformat}
2021-12-16T23:38:30,805 INFO [main] hbase.RegionReplicationLagEvaluation: Test
finished, min lag 3 ms, max lag 349 ms, mean lag 101 ms
2021-12-16T23:38:30,805 INFO [main] hbase.RegionReplicationLagEvaluation:
25.0% lag: 14 ms
2021-12-16T23:38:30,805 INFO [main] hbase.RegionReplicationLagEvaluation:
50.0% lag: 98 ms
2021-12-16T23:38:30,805 INFO [main] hbase.RegionReplicationLagEvaluation:
75.0% lag: 184 ms
2021-12-16T23:38:30,806 INFO [main] hbase.RegionReplicationLagEvaluation:
90.0% lag: 187 ms
2021-12-16T23:38:30,806 INFO [main] hbase.RegionReplicationLagEvaluation:
95.0% lag: 188 ms
2021-12-16T23:38:30,806 INFO [main] hbase.RegionReplicationLagEvaluation:
98.0% lag: 190 ms
2021-12-16T23:38:30,806 INFO [main] hbase.RegionReplicationLagEvaluation:
99.0% lag: 192 ms
2021-12-16T23:38:30,806 INFO [main] hbase.RegionReplicationLagEvaluation:
99.9% lag: 200 ms
2021-12-16T23:40:51,459 INFO [main] hbase.RegionReplicationLagEvaluation: Test
finished, min lag 2 ms, max lag 290 ms, mean lag 101 ms
2021-12-16T23:40:51,459 INFO [main] hbase.RegionReplicationLagEvaluation:
25.0% lag: 15 ms
2021-12-16T23:40:51,459 INFO [main] hbase.RegionReplicationLagEvaluation:
50.0% lag: 85 ms
2021-12-16T23:40:51,460 INFO [main] hbase.RegionReplicationLagEvaluation:
75.0% lag: 183 ms
2021-12-16T23:40:51,460 INFO [main] hbase.RegionReplicationLagEvaluation:
90.0% lag: 185 ms
[GitHub] [hbase] Apache9 commented on a change in pull request #3942: HBASE-26265 Update ref guide to mention the new store file tracker im…
Apache9 commented on a change in pull request #3942: URL: https://github.com/apache/hbase/pull/3942#discussion_r770644789 ## File path: src/main/asciidoc/_chapters/store_file_tracking.adoc ## @@ -0,0 +1,175 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +[[storefiletracking]] += Store File Tracking +:doctype: book +:numbered: +:toc: left +:icons: font +:experimental: + +== Overview + +This feature introduces an abstraction layer to track store files still used/needed by store +engines, allowing for plugging different approaches of identifying store +files required by the given store. + +Historically, HBase internals have relied on creating hfiles on temporary directories first, renaming +those files to the actual store directory at operation commit time. That's a simple and convenient +way to separate transient from already finalised files that are ready to serve client reads with data. +This approach works well with strong consistent file systems, but with the popularity of less consistent +file systems, mainly Object Store file systems, dependency on rename operations starts to introduce +performance penalties. Amazon S3 Object Store, in particular, has been the most affected deployment, +due to the its lack of atomic renames, requiring an additional locking layer implemented by HBOSS, +to guarantee consistency and integrity of operations. + +With *Store File Tracking*, decision on where to originally create new hfiles and how to proceed upon +commit is delegated to the specific Store File Tracking implementation. +It can be set at individual Table or Column Family configurations, as well as in processes +*hbase-site.xml* configuration file. + +NOTE: When specified in *hbase_site.xml*, this configuration is also saved into tables configuration +at table creation time. This is to avoid dangerous configuration mismatches between processes, which +could potentially lead to data loss. + +== Available Implementations + +Store File Tracking initial version provides three builtin implementations: + +* DEFAULT +* FILE +* MIGRATION + +### DEFAULT + +As per the name, this is the Store File Tracking implementation used by default when now explicit +configuration has been defined. The DEFAULT tracker implements the standard approach using temporary +directories and renames. + +### FILE + +A file tracker implementation that creates new files straight in the store directory, avoiding the +need for rename operations. It keeps a list of committed hfiles in memory, backed by meta files, in +each store directory. Whenever a new hfile is committed, the list of _tracked files_ in the given +store is updated and a new meta file is written with this list contents, discarding the previous +meta file now containing an out dated list. + +### MIGRATION + +A special implementation to be used when swapping between Store File Tracking implementations on +pre-existing tables that already contain data, and therefore, files being tracked under an specific +logic. + +== Usage + +For fresh deployments that don't yet contain any user data, *FILE* implementation can be just set as +value for *hbase.store.file-tracker.impl* property in global *hbase-site.xml* configuration, prior +to the first hbase start. Omitting this property sets the *DEFAULT* implementation. + +### Switching implementations globally Review comment: Filed HBASE-26586, HBASE-26587 and HBASE-26588 for these things. Will work on them soon once the feature branch is merged :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26588) Implement a migration tool to help users migrate SFT implementation for a large set of tables
Duo Zhang created HBASE-26588: - Summary: Implement a migration tool to help users migrate SFT implementation for a large set of tables Key: HBASE-26588 URL: https://issues.apache.org/jira/browse/HBASE-26588 Project: HBase Issue Type: Sub-task Components: tooling Reporter: Duo Zhang It will be very useful for our users who deploy HBase on S3 like systems. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26587) Introduce a new Admin API to change SFT implementation
Duo Zhang created HBASE-26587: - Summary: Introduce a new Admin API to change SFT implementation Key: HBASE-26587 URL: https://issues.apache.org/jira/browse/HBASE-26587 Project: HBase Issue Type: Sub-task Components: Admin, API Reporter: Duo Zhang Now changing SFT implementation requires two MTP, and there are also lots of rules on how to correctly set the MIGRATION SFT, I think we'd better introduce a special method to implement the complicated logic by code,to make our users life easier. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26586) The MigrateStoreFileTrackerProcedure should not rely on the global config when setting SFT implementation for a table
Duo Zhang created HBASE-26586: - Summary: The MigrateStoreFileTrackerProcedure should not rely on the global config when setting SFT implementation for a table Key: HBASE-26586 URL: https://issues.apache.org/jira/browse/HBASE-26586 Project: HBase Issue Type: Sub-task Components: HFile Reporter: Duo Zhang If a table does not have a SFT implementation in its table descriptor, it means we are upgrading from an older version where SFT is not implemented, so the SFT implementation for the table should always be DEFAULT. A misconfigure here could cause data loss. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26262) Implement a region based store file tracker
[ https://issues.apache.org/jira/browse/HBASE-26262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460808#comment-17460808 ] Josh Elser commented on HBASE-26262: Agree with Wellington. I think avoiding other implementations is a good thing. Let's revisit when there's a technical need to do so :) > Implement a region based store file tracker > --- > > Key: HBASE-26262 > URL: https://issues.apache.org/jira/browse/HBASE-26262 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hbase] wchevreuil commented on a change in pull request #3942: HBASE-26265 Update ref guide to mention the new store file tracker im…
wchevreuil commented on a change in pull request #3942: URL: https://github.com/apache/hbase/pull/3942#discussion_r770631946 ## File path: src/main/asciidoc/_chapters/store_file_tracking.adoc ## @@ -0,0 +1,175 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +[[storefiletracking]] += Store File Tracking +:doctype: book +:numbered: +:toc: left +:icons: font +:experimental: + +== Overview + +This feature introduces an abstraction layer to track store files still used/needed by store +engines, allowing for plugging different approaches of identifying store +files required by the given store. + +Historically, HBase internals have relied on creating hfiles on temporary directories first, renaming +those files to the actual store directory at operation commit time. That's a simple and convenient +way to separate transient from already finalised files that are ready to serve client reads with data. +This approach works well with strong consistent file systems, but with the popularity of less consistent +file systems, mainly Object Store file systems, dependency on rename operations starts to introduce +performance penalties. Amazon S3 Object Store, in particular, has been the most affected deployment, +due to the its lack of atomic renames, requiring an additional locking layer implemented by HBOSS, +to guarantee consistency and integrity of operations. + +With *Store File Tracking*, decision on where to originally create new hfiles and how to proceed upon +commit is delegated to the specific Store File Tracking implementation. +It can be set at individual Table or Column Family configurations, as well as in processes +*hbase-site.xml* configuration file. + +NOTE: When specified in *hbase_site.xml*, this configuration is also saved into tables configuration +at table creation time. This is to avoid dangerous configuration mismatches between processes, which +could potentially lead to data loss. + +== Available Implementations + +Store File Tracking initial version provides three builtin implementations: + +* DEFAULT +* FILE +* MIGRATION + +### DEFAULT + +As per the name, this is the Store File Tracking implementation used by default when now explicit +configuration has been defined. The DEFAULT tracker implements the standard approach using temporary +directories and renames. + +### FILE + +A file tracker implementation that creates new files straight in the store directory, avoiding the +need for rename operations. It keeps a list of committed hfiles in memory, backed by meta files, in +each store directory. Whenever a new hfile is committed, the list of _tracked files_ in the given +store is updated and a new meta file is written with this list contents, discarding the previous +meta file now containing an out dated list. + +### MIGRATION + +A special implementation to be used when swapping between Store File Tracking implementations on +pre-existing tables that already contain data, and therefore, files being tracked under an specific +logic. + +== Usage + +For fresh deployments that don't yet contain any user data, *FILE* implementation can be just set as +value for *hbase.store.file-tracker.impl* property in global *hbase-site.xml* configuration, prior +to the first hbase start. Omitting this property sets the *DEFAULT* implementation. + +### Switching implementations globally Review comment: Ok, my test deployment didn't include HBASE-26263, which would basically cause the global config switch to be ignored. >MigrateStoreFileTrackerProcedure will not always set the SFT impl to default, if we set a MIGRATION store file tracker globally, RollingUpgradeChore will set the SFT implementation to MIGRATION. We could add a validation in StoreFileTrackerFactory to not allow setting MIGRATION when no explicitly SFT config has been set yet? That would prevent this issue. And yeah allowing MIGRATION to be set globally, does require for the MIGRATION SRC and DST be explicitly set to DEFAULT and FILE, respectively. Let me remove this section altogether, and update the next one to emphasise that switches are currently only supported at Table or CF level
[jira] [Created] (HBASE-26585) Add SFT configuration to META table descriptor when creating META
Wellington Chevreuil created HBASE-26585: Summary: Add SFT configuration to META table descriptor when creating META Key: HBASE-26585 URL: https://issues.apache.org/jira/browse/HBASE-26585 Project: HBase Issue Type: Sub-task Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil META doesn't get created via CreateTableProcedure, but rather during master initialisation on cluster's first run, so changes implemented on HBASE-26246 don't take effect for meta table. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26472) Adhere to semantic conventions regarding table data operations
[
https://issues.apache.org/jira/browse/HBASE-26472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460732#comment-17460732
]
Hudson commented on HBASE-26472:
Results for branch branch-2.5
[build #4 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.5/4/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.5/4/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.5/4/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.5/4/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.5/4/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Adhere to semantic conventions regarding table data operations
> --
>
> Key: HBASE-26472
> URL: https://issues.apache.org/jira/browse/HBASE-26472
> Project: HBase
> Issue Type: Sub-task
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> Follow the guidance outlined in
> https://github.com/open-telemetry/opentelemetry-specification/blob/3e380e2/specification/trace/semantic_conventions/database.dm
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26542) Apply a `package` to test protobuf files
[
https://issues.apache.org/jira/browse/HBASE-26542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460733#comment-17460733
]
Hudson commented on HBASE-26542:
Results for branch branch-2.5
[build #4 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.5/4/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.5/4/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.5/4/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.5/4/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.5/4/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Apply a `package` to test protobuf files
>
>
> Key: HBASE-26542
> URL: https://issues.apache.org/jira/browse/HBASE-26542
> Project: HBase
> Issue Type: Test
> Components: Protobufs, test
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 2.5.0, 2.4.9, 2.6.0, 3.0.0-alpha-3
>
>
> This is needed in a couple places in order to test that traces over the IPC
> layer carry correct span names, and it's good hygiene anyway.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HBASE-26566) Optimize encodeNumeric in OrderedBytes
[
https://issues.apache.org/jira/browse/HBASE-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yutong Xiao updated HBASE-26566:
Description:
Found current estimate E step in OrderedBytes is to search step by step in
loops. We can directly calculate the length to move the point and let the time
complexity become to O(1).
>From the comments of the method encodeNumericLarge:
{panel:title=encodeNumericLarge}
Encode the large magnitude floating point number *val* using the key encoding.
The caller guarantees that *val* will be
finite and abs({*}val{*}) >= 1.0.
A floating point value is encoded as an integer exponent *E* and a mantissa
{*}M{*}. The original value is equal to {*}(M * 100^E){*}. *E* is set to the
smallest value possible without making *M* greater than or equal to 1.0.
Each centimal digit of the mantissa is stored in a byte. If the value of the
centimal digit is *X* (hence X>=0 and X<=99) then the byte value will be 2*X+1
for every byte of the mantissa, except for the last byte which will be 2*X+0.
The mantissa must be the minimum number of bytes necessary to represent the
value; trailing *X==0* digits are omitted. This means that the mantissa will
never contain a byte with the value 0x00.
The function could be divided into 4 steps:
# Confirm the sign (Negative or not)
# Normalise the abs({*}val{*}), transformed to *(M * 100^E)*
# Encode *M* by peeling off centimal digit and
# Encode *X*{panel}
At the step 2, we normalise abs(*val*) and determine the exponent *E*.
The current implementation is :
{code:java}
while (abs.compareTo(E32) >= 0 && e <= 350) { abs = abs.movePointLeft(32);
e +=16; }
while (abs.compareTo(E8) >= 0 && e <= 350) { abs = abs.movePointLeft(8);
e+= 4; }
while (abs.compareTo(BigDecimal.ONE) >= 0 && e <= 350) { abs =
abs.movePointLeft(2); e++; }
{code}
Which just move the point of abs(*val*), (which is in form ., where y &
z are digits, e.g. 1000.0001) from right to left step by step, until abs(val) <
1.
In the loop body, the increment of e is always the half of steps the point
moved to left. (step=32, deltaE=16), (step=8, deltaE=4), (step=2, deltaE=1)
As e starts from 0, the value of e must be the half of the total moved steps at
last.
So that we could save the above three loops and calculate the moved length and
e directly.
My implementation:
{code:java}
// This is the number of digits of the Integer part of abs(val) when val >
10
int integerDigits = abs.precision() - abs.scale();
// If length of the Integer part is odd, from the last loop above, we
actually moved one more step forward.
int lengthToMoveLeft = integerDigits % 2 == 0 ? integerDigits :
integerDigits + 1;
e = lengthToMoveLeft / 2;
// The edge condition.
if (e > 350) {
e = 351;
lengthToMoveLeft = 702;
}
abs = abs.movePointLeft(lengthToMoveLeft);
{code}
In this case, we only move the point once. The time complexity is O(1).
Same idea to the method of encodeNumericSmall.
>From the JMH test, (*JmhBenchmark.java & Benchmark.log in attachments*), the
>throughput has an incredible improvement.
At step 3 peeling off M into centimals. Current implementation is:
{code:java}
for (int i = 0; i < 18 && abs.compareTo(BigDecimal.ZERO) != 0; i++) {
abs = abs.movePointRight(2); // Will create new BigDecimal object.
d = abs.intValue();
dst.put((byte) (2 * d + 1));
abs = abs.subtract(BigDecimal.valueOf(d)); // Will create new BigDecimal
object.
}
{code}
Also move the point of big decimal step by step. BigDecimal operations are
costly and create plenty of new BigDecimal objects, which are only used once.
Here I tried to directly encoding the BigDecimal based on the string to avoid
BigDecimal operations.
My implementation:
{code:java}
private static void encodeToCentimal(PositionedByteRange dst, BigDecimal val) {
String stringOfAbs = val.stripTrailingZeros().toPlainString();
String value = stringOfAbs.substring(stringOfAbs.indexOf('.') + 1);
int d;
int maxPrecision = Math.min(MAX_NUM_ENCODE_BYTES * 2, value.length());
for (int i = 0; i < maxPrecision; i += 2) {
d = (value.charAt(i) - '0') * 10;
if (i + 1 < maxPrecision) {
d += (value.charAt(i + 1) - '0');
}
dst.put((byte) (2 * d + 1));
}
}
{code}
>From the JMH test, (*EncodingBenchmark.java & Benchmark-encoding.log*) this
>raises the throughtput of the encoding about 200%.
was:
Found current estimate E step in OrderedBytes is to search step by step in
loops. We can directly calculate the length to move the point and let the time
complexity become to O(1).
>From the comments of the method encodeNumericLarge:
{panel:title=encodeNumericLarge}
Encode the large magnitude floating point number *val* using the key encoding.
The caller guarantees that *val* will be
finite and abs({*}val{*}) >= 1.0.
A floating point value is
[jira] [Commented] (HBASE-26542) Apply a `package` to test protobuf files
[
https://issues.apache.org/jira/browse/HBASE-26542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17460662#comment-17460662
]
Hudson commented on HBASE-26542:
Results for branch master
[build #470 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/470/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/470/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/470/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/470/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Apply a `package` to test protobuf files
>
>
> Key: HBASE-26542
> URL: https://issues.apache.org/jira/browse/HBASE-26542
> Project: HBase
> Issue Type: Test
> Components: Protobufs, test
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 2.5.0, 2.4.9, 2.6.0, 3.0.0-alpha-3
>
>
> This is needed in a couple places in order to test that traces over the IPC
> layer carry correct span names, and it's good hygiene anyway.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hbase] Apache-HBase commented on pull request #3899: HBASE-26488 Memory leak when MemStore retry flushing
Apache-HBase commented on pull request #3899: URL: https://github.com/apache/hbase/pull/3899#issuecomment-995776979 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 42s | master passed | | +1 :green_heart: | compile | 1m 7s | master passed | | +1 :green_heart: | shadedjars | 9m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 29s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 9m 24s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 217m 19s | hbase-server in the patch failed. | | | | 252m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3899 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c9b611abac37 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c93e4572c1 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/testReport/ | | Max. process+thread count | 3126 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3899/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3940: HBASE-26566 Optimize encodeNumeric in OrderedBytes
Apache-HBase commented on pull request #3940: URL: https://github.com/apache/hbase/pull/3940#issuecomment-995773730 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 43s | master passed | | +1 :green_heart: | compile | 0m 57s | master passed | | +1 :green_heart: | checkstyle | 0m 27s | master passed | | +1 :green_heart: | spotbugs | 0m 51s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | checkstyle | 0m 25s | hbase-common: The patch generated 0 new + 39 unchanged - 5 fixed = 39 total (was 44) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 22m 10s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotbugs | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 46m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3940 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 8a7aa0c073a7 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c93e4572c1 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3940: HBASE-26566 Optimize encodeNumeric in OrderedBytes
Apache-HBase commented on pull request #3940: URL: https://github.com/apache/hbase/pull/3940#issuecomment-995764035 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 47s | master passed | | +1 :green_heart: | compile | 0m 28s | master passed | | +1 :green_heart: | shadedjars | 9m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | the patch passed | | +1 :green_heart: | compile | 0m 24s | the patch passed | | +1 :green_heart: | javac | 0m 24s | the patch passed | | +1 :green_heart: | shadedjars | 9m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 57s | hbase-common in the patch passed. | | | | 34m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3940 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0ef58a8aaacc 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c93e4572c1 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/4/testReport/ | | Max. process+thread count | 228 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3940: HBASE-26566 Optimize encodeNumeric in OrderedBytes
Apache-HBase commented on pull request #3940: URL: https://github.com/apache/hbase/pull/3940#issuecomment-995762417 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 34s | master passed | | +1 :green_heart: | compile | 0m 27s | master passed | | +1 :green_heart: | shadedjars | 8m 27s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 33s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed | | +1 :green_heart: | javac | 0m 29s | the patch passed | | +1 :green_heart: | shadedjars | 8m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 5s | hbase-common in the patch passed. | | | | 32m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3940 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a3df46a2eb61 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c93e4572c1 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/4/testReport/ | | Max. process+thread count | 291 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3940/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on pull request #3940: HBASE-26566 Optimize encodeNumeric in OrderedBytes
YutSean commented on pull request #3940: URL: https://github.com/apache/hbase/pull/3940#issuecomment-995717726 Found the function could be optimized one step forward. Has update the code in the latest commit and upload the jmh test results in the jira ticket. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26566) Optimize encodeNumeric in OrderedBytes
[
https://issues.apache.org/jira/browse/HBASE-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yutong Xiao updated HBASE-26566:
Description:
Found current estimate E step in OrderedBytes is to search step by step in
loops. We can directly calculate the length to move the point and let the time
complexity become to O(1).
>From the comments of the method encodeNumericLarge:
{panel:title=encodeNumericLarge}
Encode the large magnitude floating point number *val* using the key encoding.
The caller guarantees that *val* will be
finite and abs({*}val{*}) >= 1.0.
A floating point value is encoded as an integer exponent *E* and a mantissa
{*}M{*}. The original value is equal to {*}(M * 100^E){*}. *E* is set to the
smallest value possible without making *M* greater than or equal to 1.0.
Each centimal digit of the mantissa is stored in a byte. If the value of the
centimal digit is *X* (hence X>=0 and X<=99) then the byte value will be 2*X+1
for every byte of the mantissa, except for the last byte which will be 2*X+0.
The mantissa must be the minimum number of bytes necessary to represent the
value; trailing *X==0* digits are omitted. This means that the mantissa will
never contain a byte with the value 0x00.
The function could be divided into 4 steps:
# Confirm the sign (Negative or not)
# Normalise the abs({*}val{*}), transformed to *(M * 100^E)*
# Encode *M* by peeling off centimal digit and
# Encode *X*{panel}
At the step 2, we normalise abs(*val*) and determine the exponent *E*.
The current implementation is :
{code:java}
while (abs.compareTo(E32) >= 0 && e <= 350) { abs = abs.movePointLeft(32);
e +=16; }
while (abs.compareTo(E8) >= 0 && e <= 350) { abs = abs.movePointLeft(8);
e+= 4; }
while (abs.compareTo(BigDecimal.ONE) >= 0 && e <= 350) { abs =
abs.movePointLeft(2); e++; }
{code}
Which just move the point of abs(*val*), (which is in form ., where y &
z are digits, e.g. 1000.0001) from right to left step by step, until abs(val) <
1.
In the loop body, the increment of e is always the half of steps the point
moved to left. (step=32, deltaE=16), (step=8, deltaE=4), (step=2, deltaE=1)
As e starts from 0, the value of e must be the half of the total moved steps at
last.
So that we could save the above three loops and calculate the moved length and
e directly.
My implementation:
{code:java}
// This is the number of digits of the Integer part of abs(val) when val >
10
int integerDigits = abs.precision() - abs.scale();
// If length of the Integer part is odd, from the last loop above, we
actually moved one more step forward.
int lengthToMoveLeft = integerDigits % 2 == 0 ? integerDigits :
integerDigits + 1;
e = lengthToMoveLeft / 2;
// The edge condition.
if (e > 350) {
e = 351;
lengthToMoveLeft = 702;
}
abs = abs.movePointLeft(lengthToMoveLeft);
{code}
In this case, we only move the point once. The time complexity is O(1).
Same idea to the method of encodeNumericSmall.
>From the JMH test, (*JmhBenchmark.java & Benchmark.log in attachments*), the
>throughput has an incredible improvement.
At step 3 peeling off M into centimals. Current implementation is:
{code:java}
for (int i = 0; i < 18 && abs.compareTo(BigDecimal.ZERO) != 0; i++) {
abs = abs.movePointRight(2); // Will create new BigDecimal object.
d = abs.intValue();
dst.put((byte) (2 * d + 1));
abs = abs.subtract(BigDecimal.valueOf(d)); // Will create new BigDecimal
object.
}
{code}
Also move the point of big decimal step by step. BigDecimal operations are
costly and create plenty of new BigDecimal objects, which are only used once.
Here I tried to directly encoding the BigDecimal based on the string to avoid
BigDecimal operations.
My implementation:
{code:java}
private static void encodeToCentimal(PositionedByteRange dst, BigDecimal val) {
String stringOfAbs = val.stripTrailingZeros().toPlainString();
String value = stringOfAbs.substring(stringOfAbs.indexOf('.') + 1);
int d;
int maxPrecision = Math.min(MAX_NUM_ENCODE_BYTES * 2, value.length());
for (int i = 0; i < maxPrecision; i += 2) {
d = (value.charAt(i) - '0') * 10;
if (i + 1 < maxPrecision) {
d += (value.charAt(i + 1) - '0');
}
dst.put((byte) (2 * d + 1));
}
}
{code}
>From the JMH test, (*EncodingBenchmark.java & Benchmark-encoding.log*) this
>raises the throughtput of the encoding about 200%.
Have did a jmh benchmark. The result and code are in the attachments.
was:
Found current estimate E step in OrderedBytes is to search step by step in
loops. We can directly calculate the length to move the point and let the time
complexity become to O(1).
>From the comments of the method encodeNumericLarge:
{panel:title=encodeNumericLarge}
Encode the large magnitude floating point number *val* using the key encoding.
The caller guarantees that *val*
