[GitHub] [hbase] Apache-HBase commented on pull request #3244: HBASE-25869 WAL value compression
Apache-HBase commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-839431756 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 55s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 21s | master passed | | +1 :green_heart: | compile | 5m 40s | master passed | | +1 :green_heart: | checkstyle | 2m 17s | master passed | | +1 :green_heart: | spotbugs | 7m 12s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 5m 45s | the patch passed | | +1 :green_heart: | cc | 5m 45s | the patch passed | | -0 :warning: | javac | 0m 48s | hbase-common generated 1 new + 158 unchanged - 1 fixed = 159 total (was 159) | | +1 :green_heart: | checkstyle | 1m 45s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 1s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 2m 4s | the patch passed | | -1 :x: | spotbugs | 2m 24s | hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 73m 44s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | Unread field:field be static? At CompressionContext.java:[line 53] | | | Unread field:field be static? At CompressionContext.java:[line 54] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3244 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux 63d74aa98125 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/10/artifact/yetus-general-check/output/diff-compile-javac-hbase-common.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/10/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/10/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3252: HBASE-25862 update reference guide for log4j2.xml
Apache-HBase commented on pull request #3252: URL: https://github.com/apache/hbase/pull/3252#issuecomment-839410714 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 22s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | master passed | | +0 :ok: | refguide | 3m 37s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 3m 44s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 18m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3252 | | Optional Tests | dupname asflicense refguide | | uname | Linux e4d581c9ac23 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/3/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/3/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 65 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3257: HBASE-25817 Use GuavaCache in thrift server hashmap
Apache-HBase commented on pull request #3257: URL: https://github.com/apache/hbase/pull/3257#issuecomment-839410187 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | master passed | | +1 :green_heart: | compile | 0m 57s | master passed | | +1 :green_heart: | checkstyle | 0m 39s | master passed | | +1 :green_heart: | spotbugs | 1m 21s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | -0 :warning: | javac | 0m 58s | hbase-thrift generated 1 new + 34 unchanged - 0 fixed = 35 total (was 34) | | -0 :warning: | checkstyle | 0m 41s | hbase-thrift: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 50s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 1m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 39m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3257 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux fd8c72885d5e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/2/artifact/yetus-general-check/output/diff-compile-javac-hbase-thrift.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-thrift.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-thrift U: hbase-thrift | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3257: HBASE-25817 Use GuavaCache in thrift server hashmap
Apache-HBase commented on pull request #3257: URL: https://github.com/apache/hbase/pull/3257#issuecomment-839409056 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 43s | master passed | | +1 :green_heart: | compile | 0m 38s | master passed | | +1 :green_heart: | shadedjars | 8m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 22s | the patch passed | | +1 :green_heart: | compile | 0m 38s | the patch passed | | +1 :green_heart: | javac | 0m 38s | the patch passed | | +1 :green_heart: | shadedjars | 8m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 4m 1s | hbase-thrift in the patch passed. | | | | 35m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3257 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d5ed99b55838 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/2/testReport/ | | Max. process+thread count | 2061 (vs. ulimit of 3) | | modules | C: hbase-thrift U: hbase-thrift | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3257: HBASE-25817 Use GuavaCache in thrift server hashmap
Apache-HBase commented on pull request #3257: URL: https://github.com/apache/hbase/pull/3257#issuecomment-839408296 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | master passed | | +1 :green_heart: | compile | 0m 32s | master passed | | +1 :green_heart: | shadedjars | 8m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | compile | 0m 34s | the patch passed | | +1 :green_heart: | javac | 0m 34s | the patch passed | | +1 :green_heart: | shadedjars | 8m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 4m 6s | hbase-thrift in the patch passed. | | | | 32m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3257 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c0c784e1c755 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/2/testReport/ | | Max. process+thread count | 1928 (vs. ulimit of 3) | | modules | C: hbase-thrift U: hbase-thrift | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3244: HBASE-25869 WAL value compression
Apache-HBase commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-839407558 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 38s | master passed | | +1 :green_heart: | compile | 2m 11s | master passed | | +1 :green_heart: | shadedjars | 8m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 12s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 2m 12s | the patch passed | | +1 :green_heart: | javac | 2m 12s | the patch passed | | +1 :green_heart: | shadedjars | 8m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 49s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 47s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 147m 10s | hbase-server in the patch passed. | | | | 184m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3244 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1245e11a2275 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/9/testReport/ | | Max. process+thread count | 4160 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/9/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3244: HBASE-25869 WAL value compression
Apache-HBase commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-839407299 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 10s | master passed | | +1 :green_heart: | compile | 2m 39s | master passed | | +1 :green_heart: | shadedjars | 8m 6s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 19s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 17s | the patch passed | | +1 :green_heart: | compile | 2m 36s | the patch passed | | +1 :green_heart: | javac | 2m 36s | the patch passed | | +1 :green_heart: | shadedjars | 8m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 0s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 2m 4s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 143m 40s | hbase-server in the patch passed. | | | | 183m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3244 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ece61faa51ba 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/9/testReport/ | | Max. process+thread count | 3886 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/9/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3252: HBASE-25862 update reference guide for log4j2.xml
Apache-HBase commented on pull request #3252: URL: https://github.com/apache/hbase/pull/3252#issuecomment-839406274 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 2m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3252 | | Optional Tests | | | uname | Linux 99e4bc1143eb 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Max. process+thread count | 46 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3252: HBASE-25862 update reference guide for log4j2.xml
Apache-HBase commented on pull request #3252: URL: https://github.com/apache/hbase/pull/3252#issuecomment-839406139 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 1m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3252 | | Optional Tests | | | uname | Linux b157b07ada5a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Max. process+thread count | 52 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3252: HBASE-25862 update reference guide for log4j2.xml
Apache-HBase commented on pull request #3252: URL: https://github.com/apache/hbase/pull/3252#issuecomment-839405255 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 8m 47s | master passed | | +0 :ok: | refguide | 4m 6s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 3m 40s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 23m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3252 | | Optional Tests | dupname asflicense refguide | | uname | Linux b47f9a2068b6 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/2/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/2/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 65 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25867) Extra doc around ITBLL
[
https://issues.apache.org/jira/browse/HBASE-25867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342982#comment-17342982
]
Hudson commented on HBASE-25867:
Results for branch branch-2
[build #248 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/248/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/248/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/248/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/248/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/248/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Extra doc around ITBLL
> --
>
> Key: HBASE-25867
> URL: https://issues.apache.org/jira/browse/HBASE-25867
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> Added some doc around ITBLL to explain stuff I had difficultly with. Minor
> items such as log message & javadoc edits and explaining how to pass
> configuration to the ChaosMonkeyRunner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Affects Version/s: (was: 1.5.0) 1.7.0 > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 3.0.0-alpha-1, 2.0.0, 2.2.0, 1.7.0 >Reporter: yeonsuk Choi >Priority: Minor > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Attachment: (was: HBASE-25817.patch) > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 3.0.0-alpha-1, 1.5.0, 2.0.0, 2.2.0 >Reporter: yeonsuk Choi >Priority: Minor > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Attachment: HBASE-25817.patch > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 3.0.0-alpha-1, 1.5.0, 2.0.0, 2.2.0 >Reporter: yeonsuk Choi >Priority: Minor > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Fix Version/s: 3.0.0-alpha-1 > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 3.0.0-alpha-1, 1.5.0, 2.0.0, 2.2.0 >Reporter: yeonsuk Choi >Priority: Minor > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Affects Version/s: (was: 2.6.7) 3.0.0-alpha-1 1.5.0 2.0.0 2.2.0 > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 3.0.0-alpha-1, 1.5.0, 2.0.0, 2.2.0 >Reporter: yeonsuk Choi >Priority: Minor > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Affects Version/s: 2.6.7 > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 2.6.7 >Reporter: yeonsuk Choi >Priority: Minor > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] chexunn commented on a change in pull request #3252: HBASE-25862 update reference guide for log4j2.xml
chexunn commented on a change in pull request #3252: URL: https://github.com/apache/hbase/pull/3252#discussion_r630693983 ## File path: src/main/asciidoc/_chapters/configuration.adoc ## @@ -636,7 +636,9 @@ read by HBase daemons on startup. Changes here will require a cluster restart for HBase to notice the change. [[log4j]] -=== _log4j.properties_ +=== _log4j2.xml_ + +Since version 3.0.0, HBase has upgraded to Log4j2, so the configuration file name and format has changed.Read more in link:https://logging.apache.org/log4j/2.x/index.html[Apache Log4j2]. Review comment: Thanks for reminding, I have added a space after the end of the first sentence, please review the new merged commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3252: HBASE-25862 update reference guide for log4j2.xml
Apache-HBase commented on pull request #3252: URL: https://github.com/apache/hbase/pull/3252#issuecomment-839398335 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 3m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3252 | | Optional Tests | | | uname | Linux e86f3e6a3df0 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Max. process+thread count | 41 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3252: HBASE-25862 update reference guide for log4j2.xml
Apache-HBase commented on pull request #3252: URL: https://github.com/apache/hbase/pull/3252#issuecomment-839397600 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 1m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3252 | | Optional Tests | | | uname | Linux cf57439e5be7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Max. process+thread count | 39 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3252/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3257: HBASE-25817 Use GuavaCache in thrift server hashmap
Apache-HBase commented on pull request #3257: URL: https://github.com/apache/hbase/pull/3257#issuecomment-839397244 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 32s | master passed | | +1 :green_heart: | compile | 0m 39s | master passed | | +1 :green_heart: | shadedjars | 8m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | master passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 3m 54s | root in the patch failed. | | -1 :x: | compile | 0m 15s | hbase-thrift in the patch failed. | | -0 :warning: | javac | 0m 15s | hbase-thrift in the patch failed. | | +1 :green_heart: | shadedjars | 8m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 0m 15s | hbase-thrift in the patch failed. | | | | 29m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3257 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9da30299b098 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-11.0.10+9 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-thrift.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-thrift.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-thrift.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/testReport/ | | Max. process+thread count | 98 (vs. ulimit of 3) | | modules | C: hbase-thrift U: hbase-thrift | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3257: HBASE-25817 Use GuavaCache in thrift server hashmap
Apache-HBase commented on pull request #3257: URL: https://github.com/apache/hbase/pull/3257#issuecomment-839396971 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 0s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | master passed | | +1 :green_heart: | compile | 0m 33s | master passed | | +1 :green_heart: | shadedjars | 8m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 3m 27s | root in the patch failed. | | -1 :x: | compile | 0m 14s | hbase-thrift in the patch failed. | | -0 :warning: | javac | 0m 14s | hbase-thrift in the patch failed. | | +1 :green_heart: | shadedjars | 8m 16s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 0m 16s | hbase-thrift in the patch failed. | | | | 28m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3257 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a1cb2cd6ab33 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-thrift.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-thrift.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-thrift.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/testReport/ | | Max. process+thread count | 88 (vs. ulimit of 3) | | modules | C: hbase-thrift U: hbase-thrift | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3257: HBASE-25817 Use GuavaCache in thrift server hashmap
Apache-HBase commented on pull request #3257: URL: https://github.com/apache/hbase/pull/3257#issuecomment-839394827 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | compile | 1m 0s | master passed | | +1 :green_heart: | checkstyle | 0m 40s | master passed | | +1 :green_heart: | spotbugs | 1m 26s | master passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 3m 18s | root in the patch failed. | | -1 :x: | compile | 0m 14s | hbase-thrift in the patch failed. | | -0 :warning: | javac | 0m 14s | hbase-thrift in the patch failed. | | -0 :warning: | checkstyle | 0m 39s | hbase-thrift: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | hadoopcheck | 2m 29s | The patch causes 10 errors with Hadoop v3.1.2. | | -1 :x: | hadoopcheck | 5m 8s | The patch causes 10 errors with Hadoop v3.2.1. | | -1 :x: | hadoopcheck | 7m 44s | The patch causes 10 errors with Hadoop v3.3.0. | | -1 :x: | spotbugs | 0m 14s | hbase-thrift in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 21m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3257 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 3c5eaabe10f7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-general-check/output/patch-compile-hbase-thrift.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-general-check/output/patch-compile-hbase-thrift.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-thrift.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-general-check/output/patch-javac-3.1.2.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-general-check/output/patch-javac-3.2.1.txt | | hadoopcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-general-check/output/patch-javac-3.3.0.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/artifact/yetus-general-check/output/patch-spotbugs-hbase-thrift.txt | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-thrift U: hbase-thrift | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3257/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Affects Version/s: (was: 1.2.12) > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: yeonsuk Choi >Priority: Minor > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Attachment: (was: HBASE-25817.patch) > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 1.2.12 >Reporter: yeonsuk Choi >Priority: Minor > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Attachment: HBASE-25817.patch > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 1.2.12 >Reporter: yeonsuk Choi >Priority: Minor > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Attachment: HBASE-25817.patch > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 1.2.12 >Reporter: yeonsuk Choi >Priority: Minor > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Attachment: (was: HBASE-25817.patch) > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 1.2.12 >Reporter: yeonsuk Choi >Priority: Minor > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] sambong0113 opened a new pull request #3257: HBASE-25817 Use GuavaCache in thrift server hashmap
sambong0113 opened a new pull request #3257: URL: https://github.com/apache/hbase/pull/3257 When opened scanner is not properly closed, it remains scannerMap forever. To resolve this problem, make scannerMap to Guava Cache so that can be expired on set time. https://issues.apache.org/jira/browse/HBASE-25817 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sambong0113 closed pull request #3256: HBASE-25817 Use GuavaCache in thrift server hashmap
sambong0113 closed pull request #3256: URL: https://github.com/apache/hbase/pull/3256 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Fix Version/s: (was: 1.2.12) > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 1.2.12 >Reporter: yeonsuk Choi >Priority: Minor > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To solve this problem, I make the hashMap to Guava Cache with expire time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Description: I found when the scanner does not closed successfully, it remains forever in the hashMap that stores the state of the scanner. To resolve this problem, I make scannerMap to Guava Cache so that can be expired on set time. was: I found when the scanner does not closed successfully, it remains forever in the hashMap that stores the state of the scanner. To solve this problem, I make the hashMap to Guava Cache with expire time. > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 1.2.12 >Reporter: yeonsuk Choi >Priority: Minor > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To resolve this problem, I make scannerMap to Guava Cache so that can be > expired on set time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630673256
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,61 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ int bufferSize = 1024;
+ byte[] buffer = new byte[bufferSize];
+ // Deflater#deflate will return 0 only if more input is required. We
iterate until
+ // that condition is met, sending the content of 'buffer' to the output
stream at
+ // each step, until deflate returns 0. Then the compressor must be
flushed in order
+ // for all of the value's output to be written into the corresponding
edit. (Otherwise
+ // the compressor would carry over some of the output for this value
into the output
+ // of the next.) To flush the compressor we call deflate again using the
method option
+ // that allows us to specify the SYNC_FLUSH flag. The sync output will
be placed into
+ // the buffer. When flushing we iterate until there is no more output.
Then the flush
+ // is complete and the compressor is ready for more input.
+ int bytesOut;
Review comment:
@bharathv I can come back to try this again, but the first time around
code that would seem to work did not. Must have missed something, although it
was not obvious.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sambong0113 opened a new pull request #3256: HBASE-25817 Use GuavaCache in thrift server hashmap
sambong0113 opened a new pull request #3256: URL: https://github.com/apache/hbase/pull/3256 When opened scanner is not properly closed, it remains scannerMap forever. To resolve this problem, make scannerMap to Guava Cache so that can be expired on set time. https://issues.apache.org/jira/browse/HBASE-25817 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25817) Memory leak from thrift server hashMap
[ https://issues.apache.org/jira/browse/HBASE-25817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yeonsuk Choi updated HBASE-25817: - Attachment: HBASE-25817.patch Fix Version/s: 1.2.12 Status: Patch Available (was: Open) > Memory leak from thrift server hashMap > -- > > Key: HBASE-25817 > URL: https://issues.apache.org/jira/browse/HBASE-25817 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 1.2.12 >Reporter: yeonsuk Choi >Priority: Minor > Fix For: 1.2.12 > > Attachments: HBASE-25817.patch > > > I found when the scanner does not closed successfully, it remains forever in > the hashMap that stores the state of the scanner. > To solve this problem, I make the hashMap to Guava Cache with expire time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630669258
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,61 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ int bufferSize = 1024;
+ byte[] buffer = new byte[bufferSize];
+ // Deflater#deflate will return 0 only if more input is required. We
iterate until
+ // that condition is met, sending the content of 'buffer' to the output
stream at
+ // each step, until deflate returns 0. Then the compressor must be
flushed in order
+ // for all of the value's output to be written into the corresponding
edit. (Otherwise
+ // the compressor would carry over some of the output for this value
into the output
+ // of the next.) To flush the compressor we call deflate again using the
method option
+ // that allows us to specify the SYNC_FLUSH flag. The sync output will
be placed into
+ // the buffer. When flushing we iterate until there is no more output.
Then the flush
+ // is complete and the compressor is ready for more input.
+ int bytesOut;
Review comment:
I had to back out this change:
Exception in thread "AsyncFSWAL-0-hdfs://localhost:8020/hbase/MasterData"
java.lang.AssertionError: should not happen
at
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.append(AsyncProtobufLogWriter.java:148)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.doAppend(AsyncFSWAL.java:773)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.doAppend(AsyncFSWAL.java:130)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendEntry(AbstractFSWAL.java:1016)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync(AsyncFSWAL.java:468)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:556)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: write beyond end of stream
at
java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:201)
at
org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvEncoder.compressValue(WALCellCodec.java:282)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630657193
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,61 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ int bufferSize = 1024;
+ byte[] buffer = new byte[bufferSize];
+ // Deflater#deflate will return 0 only if more input is required. We
iterate until
+ // that condition is met, sending the content of 'buffer' to the output
stream at
+ // each step, until deflate returns 0. Then the compressor must be
flushed in order
+ // for all of the value's output to be written into the corresponding
edit. (Otherwise
+ // the compressor would carry over some of the output for this value
into the output
+ // of the next.) To flush the compressor we call deflate again using the
method option
+ // that allows us to specify the SYNC_FLUSH flag. The sync output will
be placed into
+ // the buffer. When flushing we iterate until there is no more output.
Then the flush
+ // is complete and the compressor is ready for more input.
+ int bytesOut;
Review comment:
With the latest code this won't let us reuse the buffer needed for
Deflator#deflate. Maybe not important. I can look at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630660880
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -241,10 +245,17 @@ public void write(Cell cell) throws IOException {
compression.getDictionary(CompressionContext.DictionaryIndex.FAMILY));
PrivateCellUtil.compressQualifier(out, cell,
compression.getDictionary(CompressionContext.DictionaryIndex.QUALIFIER));
- // Write timestamp, type and value as uncompressed.
+ // Write timestamp, type and value.
StreamUtils.writeLong(out, cell.getTimestamp());
- out.write(cell.getTypeByte());
- PrivateCellUtil.writeValue(out, cell, cell.getValueLength());
+ byte type = cell.getTypeByte();
+ out.write(type);
+ if (compression.getValueCompressor() != null) {
Review comment:
Yes, we can optimize this with a final boolean field.
Also applies to the check for if tag compression is enabled as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630662501
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -302,18 +367,28 @@ protected Cell parseCell() throws IOException {
compression.getDictionary(CompressionContext.DictionaryIndex.QUALIFIER));
pos += elemLen;
- // timestamp, type and value
- int tsTypeValLen = length - pos;
+ // timestamp
+ long ts = StreamUtils.readLong(in);
+ pos = Bytes.putLong(backingArray, pos, ts);
+ // type and value
+ int typeValLen = length - pos;
if (tagsLength > 0) {
-tsTypeValLen = tsTypeValLen - tagsLength - KeyValue.TAGS_LENGTH_SIZE;
+typeValLen = typeValLen - tagsLength - KeyValue.TAGS_LENGTH_SIZE;
+ }
+ byte type = (byte)in.read();
+ pos = Bytes.putByte(backingArray, pos, type);
+ int valLen = typeValLen - 1;
+ if (compression.hasValueCompression()) {
Review comment:
Yes.
Also applies to the check for if tag compression is enabled as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630662113
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -349,6 +424,42 @@ private static void checkLength(int len, int max) throws
IOException {
throw new IOException("Invalid length for compresesed portion of
keyvalue: " + len);
}
}
+
+private void readCompressedValue(InputStream in, byte[] outArray, int
outOffset,
+int expectedLength) throws IOException {
+ // Read the size of the compressed value. We serialized it as a vint32.
+ int compressedLength = StreamUtils.readRawVarint32(in);
+ // Read all of the compressed value into a buffer for the Inflater.
+ byte[] buffer = new byte[compressedLength];
+ IOUtils.readFully(in, buffer, 0, compressedLength);
+ // Inflate the compressed value. We know the uncompressed size.
Inflator#inflate will
+ // return nonzero for as long as some compressed input remains, and 0
when done.
+ Inflater inflater = compression.getValueCompressor().getInflater();
Review comment:
Here I think it is better to use the buffer based API instead of a
stream, because we are inflating into place into the backing array for cell
data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630660880
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -241,10 +245,17 @@ public void write(Cell cell) throws IOException {
compression.getDictionary(CompressionContext.DictionaryIndex.FAMILY));
PrivateCellUtil.compressQualifier(out, cell,
compression.getDictionary(CompressionContext.DictionaryIndex.QUALIFIER));
- // Write timestamp, type and value as uncompressed.
+ // Write timestamp, type and value.
StreamUtils.writeLong(out, cell.getTimestamp());
- out.write(cell.getTypeByte());
- PrivateCellUtil.writeValue(out, cell, cell.getValueLength());
+ byte type = cell.getTypeByte();
+ out.write(type);
+ if (compression.getValueCompressor() != null) {
Review comment:
Yes, we can optimize this with a final boolean field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630659919
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,61 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ int bufferSize = 1024;
+ byte[] buffer = new byte[bufferSize];
+ // Deflater#deflate will return 0 only if more input is required. We
iterate until
+ // that condition is met, sending the content of 'buffer' to the output
stream at
+ // each step, until deflate returns 0. Then the compressor must be
flushed in order
+ // for all of the value's output to be written into the corresponding
edit. (Otherwise
+ // the compressor would carry over some of the output for this value
into the output
+ // of the next.) To flush the compressor we call deflate again using the
method option
+ // that allows us to specify the SYNC_FLUSH flag. The sync output will
be placed into
+ // the buffer. When flushing we iterate until there is no more output.
Then the flush
+ // is complete and the compressor is ready for more input.
+ int bytesOut;
Review comment:
Yes, I think this can work. DeflaterOutputStream has a constructor that
will make flush() do SYNC_FLUSH like we need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3244: HBASE-25869 WAL value compression
Apache-HBase commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-839367282 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 1s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 35s | master passed | | +1 :green_heart: | compile | 5m 28s | master passed | | +1 :green_heart: | checkstyle | 1m 40s | master passed | | +1 :green_heart: | spotbugs | 6m 28s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | compile | 5m 20s | the patch passed | | +1 :green_heart: | cc | 5m 20s | the patch passed | | -0 :warning: | javac | 0m 47s | hbase-common generated 1 new + 158 unchanged - 1 fixed = 159 total (was 159) | | +1 :green_heart: | checkstyle | 1m 41s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 2m 3s | the patch passed | | +1 :green_heart: | spotbugs | 7m 8s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 65m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/9/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3244 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux e03cba43355d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/9/artifact/yetus-general-check/output/diff-compile-javac-hbase-common.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/9/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[ https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342956#comment-17342956 ] Francis Christopher Liu commented on HBASE-25761: - Been working on this the past couple of weeks. I'm hacking my existing Root Table branch and converting it to first hbase:meta region as root. I've been making changes to get the bare minimum working for us to better understand the pros/cons. > POC: hbase:meta,,1 as ROOT > -- > > Key: HBASE-25761 > URL: https://issues.apache.org/jira/browse/HBASE-25761 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Francis Christopher Liu >Priority: Major > > One of the proposals up in the split-meta design doc suggests a > sleight-of-hand where the current hard-coded hbase:meta,,1 Region is > leveraged to serve as first Region of a split hbase:meta but also does > double-duty as 'ROOT'. This suggestion was put aside as a complicating > recursion in chat but then Francis noticed on a re-read of the BigTable > paper, that this is how they describe they do 'ROOT': "The root tablet is > just the first tablet in the METADATA table, but is treated specially -- it > is never split..." > This issue is for playing around with this notion to see what the problems > are so can do a better description of this approach here, in the design: > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630657741
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,61 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
Review comment:
This didn't work out actually. Yeah, we can just use getValueArray to
simplify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630657193
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,61 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ int bufferSize = 1024;
+ byte[] buffer = new byte[bufferSize];
+ // Deflater#deflate will return 0 only if more input is required. We
iterate until
+ // that condition is met, sending the content of 'buffer' to the output
stream at
+ // each step, until deflate returns 0. Then the compressor must be
flushed in order
+ // for all of the value's output to be written into the corresponding
edit. (Otherwise
+ // the compressor would carry over some of the output for this value
into the output
+ // of the next.) To flush the compressor we call deflate again using the
method option
+ // that allows us to specify the SYNC_FLUSH flag. The sync output will
be placed into
+ // the buffer. When flushing we iterate until there is no more output.
Then the flush
+ // is complete and the compressor is ready for more input.
+ int bytesOut;
Review comment:
With the latest code this won't let us reuse the buffer needed for
Deflator#deflate. Maybe not important. I can look at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (HBASE-25761) POC: hbase:meta,,1 as ROOT
[ https://issues.apache.org/jira/browse/HBASE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Christopher Liu reassigned HBASE-25761: --- Assignee: Francis Christopher Liu > POC: hbase:meta,,1 as ROOT > -- > > Key: HBASE-25761 > URL: https://issues.apache.org/jira/browse/HBASE-25761 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Francis Christopher Liu >Priority: Major > > One of the proposals up in the split-meta design doc suggests a > sleight-of-hand where the current hard-coded hbase:meta,,1 Region is > leveraged to serve as first Region of a split hbase:meta but also does > double-duty as 'ROOT'. This suggestion was put aside as a complicating > recursion in chat but then Francis noticed on a re-read of the BigTable > paper, that this is how they describe they do 'ROOT': "The root tablet is > just the first tablet in the METADATA table, but is treated specially -- it > is never split..." > This issue is for playing around with this notion to see what the problems > are so can do a better description of this approach here, in the design: > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit?ts=606c120f#heading=h.ikbhxlcthjle -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630656411
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -241,10 +245,17 @@ public void write(Cell cell) throws IOException {
compression.getDictionary(CompressionContext.DictionaryIndex.FAMILY));
PrivateCellUtil.compressQualifier(out, cell,
compression.getDictionary(CompressionContext.DictionaryIndex.QUALIFIER));
- // Write timestamp, type and value as uncompressed.
+ // Write timestamp, type and value.
StreamUtils.writeLong(out, cell.getTimestamp());
- out.write(cell.getTypeByte());
- PrivateCellUtil.writeValue(out, cell, cell.getValueLength());
+ byte type = cell.getTypeByte();
Review comment:
Sure. This was left over from changes to make single step debugging
easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3244: HBASE-25869 WAL value compression
bharathv commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630410074
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -241,10 +245,17 @@ public void write(Cell cell) throws IOException {
compression.getDictionary(CompressionContext.DictionaryIndex.FAMILY));
PrivateCellUtil.compressQualifier(out, cell,
compression.getDictionary(CompressionContext.DictionaryIndex.QUALIFIER));
- // Write timestamp, type and value as uncompressed.
+ // Write timestamp, type and value.
StreamUtils.writeLong(out, cell.getTimestamp());
- out.write(cell.getTypeByte());
- PrivateCellUtil.writeValue(out, cell, cell.getValueLength());
+ byte type = cell.getTypeByte();
Review comment:
nit: remove local variable assignment
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -349,6 +424,42 @@ private static void checkLength(int len, int max) throws
IOException {
throw new IOException("Invalid length for compresesed portion of
keyvalue: " + len);
}
}
+
+private void readCompressedValue(InputStream in, byte[] outArray, int
outOffset,
+int expectedLength) throws IOException {
+ // Read the size of the compressed value. We serialized it as a vint32.
+ int compressedLength = StreamUtils.readRawVarint32(in);
+ // Read all of the compressed value into a buffer for the Inflater.
+ byte[] buffer = new byte[compressedLength];
+ IOUtils.readFully(in, buffer, 0, compressedLength);
+ // Inflate the compressed value. We know the uncompressed size.
Inflator#inflate will
+ // return nonzero for as long as some compressed input remains, and 0
when done.
+ Inflater inflater = compression.getValueCompressor().getInflater();
Review comment:
Same comment as above, use InflaterInputStream?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -302,18 +367,28 @@ protected Cell parseCell() throws IOException {
compression.getDictionary(CompressionContext.DictionaryIndex.QUALIFIER));
pos += elemLen;
- // timestamp, type and value
- int tsTypeValLen = length - pos;
+ // timestamp
+ long ts = StreamUtils.readLong(in);
+ pos = Bytes.putLong(backingArray, pos, ts);
+ // type and value
+ int typeValLen = length - pos;
if (tagsLength > 0) {
-tsTypeValLen = tsTypeValLen - tagsLength - KeyValue.TAGS_LENGTH_SIZE;
+typeValLen = typeValLen - tagsLength - KeyValue.TAGS_LENGTH_SIZE;
+ }
+ byte type = (byte)in.read();
+ pos = Bytes.putByte(backingArray, pos, type);
+ int valLen = typeValLen - 1;
+ if (compression.hasValueCompression()) {
Review comment:
Same here on avoiding a branch, should we do that?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,61 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
Review comment:
q: Is this special handling to avoid an array copy from the bytebuffer?
(rather than using Cell interface getValueArray() methods). I see this kinda
special handling in multiple places but don't have the context..
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,61 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ int bufferSize = 1024;
+ byte[] buffer = new byte[bufferSize];
+ // Deflater#deflate will return 0 only if more input is required. We
iterate until
+ // that condition is met, sending the content of 'buffer' to the output
stream at
+ // each step, until deflate returns 0. Then the compressor must be
flushed in order
+ // for all of the value's output to be written into the corresponding
edit. (Otherwise
+ // the
[GitHub] [hbase] ndimiduk commented on a change in pull request #3244: HBASE-25869 WAL value compression
ndimiduk commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630593612
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
##
@@ -34,21 +36,49 @@
@InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.COPROC,
HBaseInterfaceAudience.PHOENIX})
public class CompressionContext {
- static final String ENABLE_WAL_TAGS_COMPRESSION =
+ public static final String ENABLE_WAL_TAGS_COMPRESSION =
Review comment:
Oh, it is related to your change. Should this new test class,
`TestWALSplitValueCompression`, be in the package
`org.apache.hadoop.hbase.regionserver.wal` ? I see our WAL implementation and
test classes are spread across these two packages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #3244: HBASE-25869 WAL value compression
ndimiduk commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630591194
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
##
@@ -34,21 +36,49 @@
@InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.COPROC,
HBaseInterfaceAudience.PHOENIX})
public class CompressionContext {
- static final String ENABLE_WAL_TAGS_COMPRESSION =
+ public static final String ENABLE_WAL_TAGS_COMPRESSION =
Review comment:
Oh -- it seems that `TestWALSplitValueCompression` is in the wrong
package :)
Not related to your change, and I found several similar cases to this while
closing down symbol permissions in other areas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25032) Wait for region server to become online before adding it to online servers in Master
[ https://issues.apache.org/jira/browse/HBASE-25032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342905#comment-17342905 ] Nick Dimiduk commented on HBASE-25032: -- bq. Oops, this patch was committed on 25th March and 2.3.5RC1 came out on 26th March, so I got confused and ended up adding the wrong release version. Is there is a better way to avoid this race rather than keeping track of emails in dev@? Tracking the content of the CHANGES.md file is one way. Another way is to check the git history of the branch for a commit's place in relation to the release tags. Either way, a release manager usually cleans up these kinds of inconsistencies as they prepare the next patch release, so it would have been caught as part of preparation for 2.3.6RC0. > Wait for region server to become online before adding it to online servers in > Master > > > Key: HBASE-25032 > URL: https://issues.apache.org/jira/browse/HBASE-25032 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Guggilam >Assignee: Caroline Zhou >Priority: Major > Labels: master, regionserver > Fix For: 3.0.0-alpha-1, 2.5.0 > > > As part of RS start up, RS reports for duty to Master . Master acknowledges > the request and adds it to the onlineServers list for further assigning any > regions to the RS > Once Master acknowledges the reportForDuty and sends back the response, RS > does a bunch of stuff like initializing replication sources etc before > becoming online. However, sometimes there could be an issue with initializing > replication sources when it is unable to connect to peer clusters because of > some kerberos configuration and there would be a delay of around 20 mins in > becoming online. > > Since master considers it online, it tries to assign regions and which fails > with ServerNotRunningYet exception, then the master tries to unassign which > again fails with the same exception leading the region to FAILED_CLOSE state. > > It would be good to have a check to see if the RS is ready to accept the > assignment requests before adding it to online servers list which would > account for any such delays as described above -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on pull request #3063: HBASE-25651: NORMALIZER_TARGET_REGION_SIZE needs a unit in its name
ndimiduk commented on pull request #3063: URL: https://github.com/apache/hbase/pull/3063#issuecomment-839246587 Sounds good. You might be interested in concurrent working that's happening now, to improve and correct how HBase handles deprecated configurations. See #3249 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #3249: HBASE-25861 Correct the usage of Configuration#addDeprecation
ndimiduk commented on pull request #3249: URL: https://github.com/apache/hbase/pull/3249#issuecomment-839245398 > Thanks @ndimiduk and @saintstack for review. It's cleaner after using the configuration string values directly in the HBaseConfiguration#addDeprecatedKeys(). Yes, this looks much better. It seems the test failure is related though, mind taking a look? If the jenkins test result screen doesn't include enough information for you, you can download all the [test logs](https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3249/2/artifact/yetus-jdk11-hadoop3-check/output/test_logs.zip). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3244: HBASE-25869 WAL value compression
Apache-HBase commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-839244870 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 54s | master passed | | +1 :green_heart: | compile | 2m 47s | master passed | | +1 :green_heart: | shadedjars | 8m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 18s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 48s | the patch passed | | +1 :green_heart: | compile | 2m 47s | the patch passed | | +1 :green_heart: | javac | 2m 47s | the patch passed | | +1 :green_heart: | shadedjars | 9m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 4s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 2m 13s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 205m 32s | hbase-server in the patch passed. | | | | 248m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3244 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3efdf6c6cbee 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/8/testReport/ | | Max. process+thread count | 3061 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/8/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3250: HBASE-25875 RegionServer failed to start due to IllegalThreadStateException in AuthenticationTokenSecretManager.start
Apache-HBase commented on pull request #3250: URL: https://github.com/apache/hbase/pull/3250#issuecomment-839208074 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 1m 1s | master passed | | +1 :green_heart: | shadedjars | 8m 52s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 9m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 210m 48s | hbase-server in the patch passed. | | | | 243m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3250/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3250 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4827d9c026c9 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 29bd3dd586 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3250/3/testReport/ | | Max. process+thread count | 2944 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3250/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3244: HBASE-25869 WAL value compression
Apache-HBase commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-839193926 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 37s | master passed | | +1 :green_heart: | compile | 2m 13s | master passed | | +1 :green_heart: | shadedjars | 8m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 14s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | +1 :green_heart: | compile | 2m 13s | the patch passed | | +1 :green_heart: | javac | 2m 13s | the patch passed | | +1 :green_heart: | shadedjars | 8m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 45s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 48s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 147m 37s | hbase-server in the patch passed. | | | | 184m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3244 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux df8366263ce9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/8/testReport/ | | Max. process+thread count | 4337 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/8/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25810) The nightly build for master branch is failing when running UTs in hbase-procedure after HBASE-22120 merged
[ https://issues.apache.org/jira/browse/HBASE-25810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342881#comment-17342881 ] Sean Busbey commented on HBASE-25810: - okay if I delete the feature branch for this issue? or disable the nightly builds if the branch is still needed? > The nightly build for master branch is failing when running UTs in > hbase-procedure after HBASE-22120 merged > --- > > Key: HBASE-25810 > URL: https://issues.apache.org/jira/browse/HBASE-25810 > Project: HBase > Issue Type: Bug > Components: proc-v2, test >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3255: HBASE-25880 remove files in CompactionContext from filesCompacting when clear com…
Apache-HBase commented on pull request #3255: URL: https://github.com/apache/hbase/pull/3255#issuecomment-839167894 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 3s | master passed | | +1 :green_heart: | compile | 1m 19s | master passed | | +1 :green_heart: | shadedjars | 9m 6s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 45s | the patch passed | | +1 :green_heart: | compile | 1m 21s | the patch passed | | +1 :green_heart: | javac | 1m 21s | the patch passed | | +1 :green_heart: | shadedjars | 9m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 208m 32s | hbase-server in the patch failed. | | | | 243m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3255/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3255 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cb9e35e01a2b 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 29bd3dd586 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3255/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3255/1/testReport/ | | Max. process+thread count | 3505 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3255/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3250: HBASE-25875 RegionServer failed to start due to IllegalThreadStateException in AuthenticationTokenSecretManager.start
Apache-HBase commented on pull request #3250: URL: https://github.com/apache/hbase/pull/3250#issuecomment-839153376 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 36s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 8m 20s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 30s | the patch passed | | +1 :green_heart: | compile | 1m 16s | the patch passed | | +1 :green_heart: | javac | 1m 16s | the patch passed | | +1 :green_heart: | shadedjars | 8m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 142m 3s | hbase-server in the patch passed. | | | | 175m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3250/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3250 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5c5c7948aa0e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 29bd3dd586 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3250/3/testReport/ | | Max. process+thread count | 4336 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3250/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #3243: HBASE-25864 Use shaded javax.ws.rs package classes
ndimiduk commented on pull request #3243: URL: https://github.com/apache/hbase/pull/3243#issuecomment-839150225 We might also introduce a new shaded jar that is jackson-core, jackson-databind, and jackson-jaxb-annotations. We would then make use of those symbols from the shaded-jackson-jaxrs-json-provider jar, as well as any mentions within hbase-core. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #3243: HBASE-25864 Use shaded javax.ws.rs package classes
ndimiduk commented on pull request #3243: URL: https://github.com/apache/hbase/pull/3243#issuecomment-839085013 @apurtell how would you like this change to proceed? Would you prefer that we also shade all of jackson within the `hbase-shaded-jackson-jaxrs-json-provider` module? For your reference, `mvn dependency:tree` of the new module: ``` [INFO] +- org.apache.hbase.thirdparty:hbase-shaded-jackson-jaxrs-json-provider:jar:3.5.1-SNAPSHOT:compile [INFO] | +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.1:compile [INFO] | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile [INFO] | \- jakarta.activation:jakarta.activation-api:jar:1.2.1:compile ``` Dependency list from the dependency-reduced-pom.xml ``` com.fasterxml.jackson.core jackson-core 2.10.1 compile com.fasterxml.jackson.core jackson-databind 2.10.1 compile com.fasterxml.jackson.module jackson-module-jaxb-annotations 2.10.1 compile com.fasterxml.jackson.core jackson-annotations 2.10.1 compile jakarta.xml.bind jakarta.xml.bind-api 2.3.2 compile jakarta.activation jakarta.activation-api 1.2.1 compile ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3255: HBASE-25880 remove files in CompactionContext from filesCompacting when clear com…
Apache-HBase commented on pull request #3255: URL: https://github.com/apache/hbase/pull/3255#issuecomment-839082168 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 0m 59s | master passed | | +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 8m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 147m 9s | hbase-server in the patch failed. | | | | 177m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3255/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3255 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5b484d08be72 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 29bd3dd586 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3255/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3255/1/testReport/ | | Max. process+thread count | 4574 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3255/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25859) Reference class incorrectly parses the protobuf magic marker
[
https://issues.apache.org/jira/browse/HBASE-25859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342831#comment-17342831
]
Hudson commented on HBASE-25859:
Results for branch branch-2.4
[build #114 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/114/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/114/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/114/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/114/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/114/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Reference class incorrectly parses the protobuf magic marker
>
>
> Key: HBASE-25859
> URL: https://issues.apache.org/jira/browse/HBASE-25859
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.4.1
>Reporter: Constantin-Catalin Luca
>Assignee: Constantin-Catalin Luca
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> The Reference class incorrectly parses the protobuf magic marker.
> It uses:
> {code:java}
> // DataInputStream.read(byte[lengthOfPNMagic]){code}
> but this call does not guarantee to read all the bytes of the marker.
> The fix is the same as the one for
> https://issues.apache.org/jira/browse/HBASE-25674
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3244: HBASE-25869 WAL value compression
Apache-HBase commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-839070312 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 1s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | compile | 5m 35s | master passed | | +1 :green_heart: | checkstyle | 1m 44s | master passed | | +1 :green_heart: | spotbugs | 6m 45s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 5m 31s | the patch passed | | +1 :green_heart: | cc | 5m 31s | the patch passed | | -0 :warning: | javac | 0m 46s | hbase-common generated 1 new + 158 unchanged - 1 fixed = 159 total (was 159) | | +1 :green_heart: | checkstyle | 1m 43s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 22m 13s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 2m 13s | the patch passed | | +1 :green_heart: | spotbugs | 8m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 74m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/8/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3244 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux c1adcec0a2e0 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 630c73fda4 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/8/artifact/yetus-general-check/output/diff-compile-javac-hbase-common.txt | | Max. process+thread count | 87 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3244/8/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25881) Create a chore to update age related metrics.
Rushabh Shah created HBASE-25881: Summary: Create a chore to update age related metrics. Key: HBASE-25881 URL: https://issues.apache.org/jira/browse/HBASE-25881 Project: HBase Issue Type: Improvement Reporter: Rushabh Shah We had a case where logRoller and ReplicationShipper thread were stuck for a day since some other thread was holding the lock. We were not rolling the wal for 1 day and we were not shipping any edits for 1 day. Still the oldestWalAge and age of last ship metric were not spiking as they should. The way we calculate any age related metric is we calculate the diff between current time and the time at which any event happens and we add that to metrics Framework. We lose the event timestamp at that point. If the thread populating the metric is stuck then we will always carry forward the same value forever. This will make it look like there is no problem in the system. In this case the oldestWalAge metric was stuck at 809 value and age of last ship metric was 0 the whole time and no PD alert was fired. >From Andrew Purtell, We have the Chore/ScheduledChore framework. We could be making more use of it. Much of this is legacy, before Chore was formalized as it is today. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3250: HBASE-25875 RegionServer failed to start due to IllegalThreadStateException in AuthenticationTokenSecretManager.start
Apache-HBase commented on pull request #3250: URL: https://github.com/apache/hbase/pull/3250#issuecomment-838982912 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | compile | 3m 25s | master passed | | +1 :green_heart: | checkstyle | 1m 7s | master passed | | +1 :green_heart: | spotbugs | 2m 10s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 3m 18s | the patch passed | | +1 :green_heart: | javac | 3m 18s | the patch passed | | +1 :green_heart: | checkstyle | 1m 7s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 50s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | -1 :x: | spotbugs | 2m 26s | hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 51m 5s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | org.apache.hadoop.hbase.ipc.NettyRpcServer.start() synchronizes on updated field NettyRpcServer.authTokenSecretMgr At NettyRpcServer.java:NettyRpcServer.authTokenSecretMgr At NettyRpcServer.java:[line 135] | | | org.apache.hadoop.hbase.ipc.SimpleRpcServer.start() synchronizes on updated field SimpleRpcServer.authTokenSecretMgr At SimpleRpcServer.java:SimpleRpcServer.authTokenSecretMgr At SimpleRpcServer.java:[line 422] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3250/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3250 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 5b93d7ba4749 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 29bd3dd586 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3250/3/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3250/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell edited a comment on pull request #3244: HBASE-25869 WAL value compression
apurtell edited a comment on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-838973994 > Speaking out loud. During RU, the wal written by RS that has ENABLE_WAL_VALUE_COMPRESSION set to false died and another RS which was recently upgraded has the above property set to true. Will the new RS be able to decode the cell ? @shahrs87 Yes. Each WAL file carries metadata about compression particulars for the file, so we can set up the CompressionContext correctly no matter the site file defaults. This logic is in ProtobufLogReader.java. The inverse is not true, though. There is no downgrade scenario for when after an upgrade now new WALs have value compression enabled, and you want to downgrade to a version that does not support value compression. However this is the obvious expectation that comes with enabling a new optional feature, and will be documented in release notes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3244: HBASE-25869 WAL value compression
apurtell commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-838973994 > Speaking out loud. During RU, the wal written by RS that has ENABLE_WAL_VALUE_COMPRESSION set to false died and another RS which was recently upgraded has the above property set to true. Will the new RS be able to decode the cell ? @shahrs87 Yes. Each WAL file carries metadata about compression particulars for the file, so we can set up the CompressionContext correctly no matter the site file defaults. This logic is in ProtobufLogReader.java. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25859) Reference class incorrectly parses the protobuf magic marker
[
https://issues.apache.org/jira/browse/HBASE-25859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342795#comment-17342795
]
Hudson commented on HBASE-25859:
Results for branch branch-2
[build #247 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/247/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/247/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/247/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/247/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/247/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Reference class incorrectly parses the protobuf magic marker
>
>
> Key: HBASE-25859
> URL: https://issues.apache.org/jira/browse/HBASE-25859
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.4.1
>Reporter: Constantin-Catalin Luca
>Assignee: Constantin-Catalin Luca
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> The Reference class incorrectly parses the protobuf magic marker.
> It uses:
> {code:java}
> // DataInputStream.read(byte[lengthOfPNMagic]){code}
> but this call does not guarantee to read all the bytes of the marker.
> The fix is the same as the one for
> https://issues.apache.org/jira/browse/HBASE-25674
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #3252: HBASE-25862 update reference guide for log4j2.xml
ndimiduk commented on a change in pull request #3252: URL: https://github.com/apache/hbase/pull/3252#discussion_r630427184 ## File path: src/main/asciidoc/_chapters/configuration.adoc ## @@ -636,7 +636,9 @@ read by HBase daemons on startup. Changes here will require a cluster restart for HBase to notice the change. [[log4j]] -=== _log4j.properties_ +=== _log4j2.xml_ + +Since version 3.0.0, HBase has upgraded to Log4j2, so the configuration file name and format has changed.Read more in link:https://logging.apache.org/log4j/2.x/index.html[Apache Log4j2]. Review comment: Needs a space after the end of the first sentence. `s/changed.Read/changed. Read/`. Otherwise, looks good to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] shahrs87 edited a comment on pull request #3244: HBASE-25869 WAL value compression
shahrs87 edited a comment on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-838959572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25867) Extra doc around ITBLL
[ https://issues.apache.org/jira/browse/HBASE-25867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25867. --- Hadoop Flags: Reviewed Assignee: Michael Stack Resolution: Fixed Pushed to branch-2.4+. Thanks for review [~busbey] > Extra doc around ITBLL > -- > > Key: HBASE-25867 > URL: https://issues.apache.org/jira/browse/HBASE-25867 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4 > > > Added some doc around ITBLL to explain stuff I had difficultly with. Minor > items such as log message & javadoc edits and explaining how to pass > configuration to the ChaosMonkeyRunner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] shahrs87 commented on pull request #3244: HBASE-25869 WAL value compression
shahrs87 commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-838959572 Speaking out loud. During RU, the wal written by RS that has `ENABLE_WAL_VALUE_COMPRESSION` set to `false` died and another RS which was recently upgraded has the above property set to true. Will the new RS be able to decode the cell ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #3242: HBASE-25867 Extra doc around ITBLL
saintstack merged pull request #3242: URL: https://github.com/apache/hbase/pull/3242 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3253: HBASE-25877:add access check for switchCompaction
Apache-HBase commented on pull request #3253: URL: https://github.com/apache/hbase/pull/3253#issuecomment-838938680 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 48s | master passed | | +1 :green_heart: | compile | 1m 17s | master passed | | +1 :green_heart: | shadedjars | 9m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 13s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | shadedjars | 10m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 208m 28s | hbase-server in the patch passed. | | | | 245m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3253/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3253 | | JIRA Issue | HBASE-25877 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 53be86a0d69a 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 29bd3dd586 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3253/2/testReport/ | | Max. process+thread count | 3422 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3253/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3255: HBASE-25880 remove files in CompactionContext from filesCompacting when clear com…
Apache-HBase commented on pull request #3255: URL: https://github.com/apache/hbase/pull/3255#issuecomment-838914364 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | master passed | | +1 :green_heart: | compile | 3m 22s | master passed | | +1 :green_heart: | checkstyle | 1m 10s | master passed | | +1 :green_heart: | spotbugs | 2m 13s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | compile | 3m 20s | the patch passed | | +1 :green_heart: | javac | 3m 20s | the patch passed | | +1 :green_heart: | checkstyle | 1m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 57s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 56m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3255/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3255 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 2c7f7f25f216 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 29bd3dd586 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3255/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pankaj72981 commented on pull request #3250: HBASE-25875 RegionServer failed to start due to IllegalThreadStateException in AuthenticationTokenSecretManager.start
pankaj72981 commented on pull request #3250: URL: https://github.com/apache/hbase/pull/3250#issuecomment-838897019 Missed to handle in SimpleRpcServer, handled in second commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3253: HBASE-25877:add access check for switchCompaction
Apache-HBase commented on pull request #3253: URL: https://github.com/apache/hbase/pull/3253#issuecomment-838844668 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 45s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 8m 20s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | shadedjars | 8m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 150m 12s | hbase-server in the patch failed. | | | | 180m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3253/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3253 | | JIRA Issue | HBASE-25877 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 121f1c46296c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 29bd3dd586 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3253/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3253/2/testReport/ | | Max. process+thread count | 4264 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3253/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] shahrs87 commented on a change in pull request #3244: HBASE-25869 WAL value compression
shahrs87 commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r63036
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,43 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
Review comment:
@apurtell I increased the buffer to 100, changed the input string to
500 and everything works fine. The decompressed string was the same as original
raw string.
On a side note, `Deflater#deflateBytes(long addr, byte[] b, int off, int
len, int flush)` which is a native method doesn't have much documentation which
help newbies to understand whats going on but not related to this PR.
Sorry for the noise if I have created. :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] shahrs87 commented on a change in pull request #3244: HBASE-25869 WAL value compression
shahrs87 commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r63036
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,43 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
Review comment:
@apurtell I increased the buffer to 100, changed the input string to
500 and everything works fine. The decompressed string was the same as original
raw string.
On a side note, `Deflater#deflateBytes(long addr, byte[] b, int off, int
len, int flush)` which is a native method doesn't have much documentation which
help newbies to understand whats going on but not related to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] frostruan opened a new pull request #3255: HBASE-25880 remove files in CompactionContext from filesCompacting when clear com…
frostruan opened a new pull request #3255: URL: https://github.com/apache/hbase/pull/3255 When clear compaction queues, we just clear the workQueue of ThreadPoolExecutor, but files in compaction request are still in filesCompacting list. maybe we should clear it also. https://issues.apache.org/jira/browse/HBASE-25880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25880) remove files from filesCompacting when clear compaction queues
ruanhui created HBASE-25880: --- Summary: remove files from filesCompacting when clear compaction queues Key: HBASE-25880 URL: https://issues.apache.org/jira/browse/HBASE-25880 Project: HBase Issue Type: Bug Components: Compaction Reporter: ruanhui Assignee: ruanhui When clear compaction queues, we just clear the workQueue of ThreadPoolExecutor, but files in compaction request are still in filesCompacting list. maybe we should clear it also. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell edited a comment on pull request #3244: HBASE-25869 WAL value compression
apurtell edited a comment on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-838765223 There was a period of time where there was a bad patch. I pushed a replacement. The latest precommits look better. There is one test to fix, back soon. The reason it fails now is after the latest changes it is now an invalid test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630330429
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,43 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
Review comment:
Ah, I see what you mean. Never mind. Let me fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630326851
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,43 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
Review comment:
You are sizing your buffer too small, but I see what you mean. Let me
address this in the next update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] pankaj72981 edited a comment on pull request #3250: HBASE-25875 RegionServer failed to start due to IllegalThreadStateException in AuthenticationTokenSecretManager.start
pankaj72981 edited a comment on pull request #3250:
URL: https://github.com/apache/hbase/pull/3250#issuecomment-838765723
> LGTM. After fixing the spot bugs!
Spotbugs warn looks like a false positive case which will never happen
technically, as multiple thread can't call NettyRpcServer#start concurrently
(start() is synchronized) and we have a check whether NettyRpcServer started
before creating a new AuthenticationTokenSecretManager instance.
```
@Override
public synchronized void start() {
if (started) {
return;
}
authTokenSecretMgr = createSecretManager();
if (authTokenSecretMgr != null) {
// Start AuthenticationTokenSecretManager in synchronized way to avoid
race conditions in
// LeaderElector start. See HBASE-25875
synchronized (authTokenSecretMgr) {
setSecretManager(authTokenSecretMgr);
authTokenSecretMgr.start();
}
}
this.authManager = new ServiceAuthorizationManager();
HBasePolicyProvider.init(conf, authManager);
scheduler.start();
started = true;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630330429
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,43 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
Review comment:
Ah, I see what you mean. Never mind. Let me fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell edited a comment on pull request #3244: HBASE-25869 WAL value compression
apurtell edited a comment on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-838765223 There was a period of time where there was a bad patch. I pushed a replacement. The latest precommits look better. There is one test to fix, back soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pankaj72981 commented on pull request #3250: HBASE-25875 RegionServer failed to start due to IllegalThreadStateException in AuthenticationTokenSecretManager.start
pankaj72981 commented on pull request #3250:
URL: https://github.com/apache/hbase/pull/3250#issuecomment-838765723
> LGTM. After fixing the spot bugs!
Spotbugs warn looks like a false positive case which will never happen
technically, as multiple thread can't call NettyRpcServer#start concurrently
(start() is synchronized) and we have a check whether NettyRpcServer started
before creating a new AuthenticationTokenSecretManager instance.
{code}
@Override
public synchronized void start() {
if (started) {
return;
}
authTokenSecretMgr = createSecretManager();
if (authTokenSecretMgr != null) {
// Start AuthenticationTokenSecretManager in synchronized way to avoid
race conditions in
// LeaderElector start. See HBASE-25875
synchronized (authTokenSecretMgr) {
setSecretManager(authTokenSecretMgr);
authTokenSecretMgr.start();
}
}
this.authManager = new ServiceAuthorizationManager();
HBasePolicyProvider.init(conf, authManager);
scheduler.start();
started = true;
}
{code}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3244: HBASE-25869 WAL value compression
apurtell commented on pull request #3244: URL: https://github.com/apache/hbase/pull/3244#issuecomment-838765223 There was a period of time where there was a bad patch. I pushed a replacement. The latest precommits look good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on a change in pull request #3244: HBASE-25869 WAL value compression
apurtell commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630326851
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,43 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
Review comment:
You are sizing your buffer too small. Read the Deflater/Inflater javadoc
and code to learn more, please. Do let me know if I am doing something wrong.
As far as I am aware, my API usage is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #3244: HBASE-25869 WAL value compression
bharathv commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630293441
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,43 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
Review comment:
@shahrs87
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8133170 , seems like a
known issue with small buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3254: HBASE-25879 [branch-1] Update CHANGES.txt and tag 1.7.0RC0 to the most recent commit
Apache-HBase commented on pull request #3254: URL: https://github.com/apache/hbase/pull/3254#issuecomment-838679738 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 7m 2s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-1 Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 28s | The patch does not generate ASF License warnings. | | | | 11m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3254/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3254 | | Optional Tests | dupname asflicense | | uname | Linux fe842090cd37 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-3254/out/precommit/personality/provided.sh | | git revision | branch-1 / f312cfe | | Max. process+thread count | 41 (vs. ulimit of 1) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3254/1/console | | versions | git=1.9.1 maven=3.0.5 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd opened a new pull request #3254: HBASE-25879 [branch-1] Update CHANGES.txt and tag 1.7.0RC0 to the most recent commit
Reidd opened a new pull request #3254: URL: https://github.com/apache/hbase/pull/3254 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25879) [branch-1] Update CHANGES.txt and tag 1.7.0RC0 to the most recent commit
[ https://issues.apache.org/jira/browse/HBASE-25879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-25879: -- Summary: [branch-1] Update CHANGES.txt and tag 1.7.0RC0 to the most recent commit (was: [branch-1] Update CHANGES.txt and the 1.7.0RC0 the most recent commit) > [branch-1] Update CHANGES.txt and tag 1.7.0RC0 to the most recent commit > > > Key: HBASE-25879 > URL: https://issues.apache.org/jira/browse/HBASE-25879 > Project: HBase > Issue Type: Task >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] shahrs87 commented on a change in pull request #3244: HBASE-25869 WAL value compression
shahrs87 commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630264621
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##
@@ -256,6 +267,43 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
Review comment:
@apurtell I was trying to understand how this buffer works while we
compress the data. So I wrote a small test case to understand it better.
```
@Test
public void testCodec() throws Exception {
String inputRawString = "helloworld";
int rawStringLength = inputRawString.length();
byte[] compressedBytesArr = compressValue(inputRawString);
int compressedBytesLength = compressedBytesArr.length;
}
private byte[] compressValue(String inputRawString) throws IOException {
Deflater deflater = new Deflater();
deflater.setLevel(Deflater.BEST_SPEED);
byte[] input = inputRawString.getBytes("UTF-8");
deflater.setInput(input, 0, input.length);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[5];
int bytesOut;
do {
bytesOut = deflater.deflate(buffer);
if (bytesOut > 0) {
baos.write(buffer, 0, bytesOut);
}
} while (bytesOut > 0);
// Done compressing value, now flush until deflater buffers are empty
do {
bytesOut = deflater.deflate(buffer, 0, buffer.length,
Deflater.SYNC_FLUSH);
baos.write(buffer, 0, bytesOut);
} while (bytesOut == buffer.length); // See javadoc for Deflater#deflate
return baos.toByteArray();
}
```
Basically the above snippet is copied pasted from your PR where you compress
the cell. I replcaed the cell with some random String.
Also I made sure that the size of buffer (5) is less than input string
length (which is 10) since in real world the size of Cell value will be more
than 1024 which is the buffer size in PR.
The above test goes into indefinite while loop in the second do..while loop.
If I change the size of buffer to 10 then it succeeds.
Please educate me if I am doing something wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25032) Wait for region server to become online before adding it to online servers in Master
[ https://issues.apache.org/jira/browse/HBASE-25032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342633#comment-17342633 ] Reid Chan commented on HBASE-25032: --- Ooh, I saw already reverted. > Wait for region server to become online before adding it to online servers in > Master > > > Key: HBASE-25032 > URL: https://issues.apache.org/jira/browse/HBASE-25032 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Guggilam >Assignee: Caroline Zhou >Priority: Major > Labels: master, regionserver > Fix For: 3.0.0-alpha-1, 2.5.0 > > > As part of RS start up, RS reports for duty to Master . Master acknowledges > the request and adds it to the onlineServers list for further assigning any > regions to the RS > Once Master acknowledges the reportForDuty and sends back the response, RS > does a bunch of stuff like initializing replication sources etc before > becoming online. However, sometimes there could be an issue with initializing > replication sources when it is unable to connect to peer clusters because of > some kerberos configuration and there would be a delay of around 20 mins in > becoming online. > > Since master considers it online, it tries to assign regions and which fails > with ServerNotRunningYet exception, then the master tries to unassign which > again fails with the same exception leading the region to FAILED_CLOSE state. > > It would be good to have a check to see if the RS is ready to accept the > assignment requests before adding it to online servers list which would > account for any such delays as described above -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25032) Wait for region server to become online before adding it to online servers in Master
[ https://issues.apache.org/jira/browse/HBASE-25032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342630#comment-17342630 ] Reid Chan commented on HBASE-25032: --- Catch up late, do I still need to revert it from branch-1? > Wait for region server to become online before adding it to online servers in > Master > > > Key: HBASE-25032 > URL: https://issues.apache.org/jira/browse/HBASE-25032 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Guggilam >Assignee: Caroline Zhou >Priority: Major > Labels: master, regionserver > Fix For: 3.0.0-alpha-1, 2.5.0 > > > As part of RS start up, RS reports for duty to Master . Master acknowledges > the request and adds it to the onlineServers list for further assigning any > regions to the RS > Once Master acknowledges the reportForDuty and sends back the response, RS > does a bunch of stuff like initializing replication sources etc before > becoming online. However, sometimes there could be an issue with initializing > replication sources when it is unable to connect to peer clusters because of > some kerberos configuration and there would be a delay of around 20 mins in > becoming online. > > Since master considers it online, it tries to assign regions and which fails > with ServerNotRunningYet exception, then the master tries to unassign which > again fails with the same exception leading the region to FAILED_CLOSE state. > > It would be good to have a check to see if the RS is ready to accept the > assignment requests before adding it to online servers list which would > account for any such delays as described above -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25878) Fix various formatting issues in the refrence guide book
[ https://issues.apache.org/jira/browse/HBASE-25878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342621#comment-17342621 ] Che Xun commented on HBASE-25878: - Hi [~ndimiduk],I adopt your suggestion, using a separate JIRA to fix the formatting issues in the refrence guide. > Fix various formatting issues in the refrence guide book > > > Key: HBASE-25878 > URL: https://issues.apache.org/jira/browse/HBASE-25878 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha-1 >Reporter: Che Xun >Assignee: Che Xun >Priority: Minor > Fix For: 3.0.0-alpha-1 > > Attachments: image-2021-05-11-22-55-13-810.png > > > This jira is used to collect and fix various formatting issues in the > refrence guide book. > For example,in the first paragraph of the chapter 8.1，the bulleted list is > not displayed correctly，as show below： > !image-2021-05-11-22-55-13-810.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
