[jira] [Resolved] (HBASE-27003) Optimize log format for PerformanceEvaluation
[ https://issues.apache.org/jira/browse/HBASE-27003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-27003. -- Fix Version/s: 2.5.0 3.0.0-alpha-3 2.4.13 Hadoop Flags: Reviewed Resolution: Fixed > Optimize log format for PerformanceEvaluation > - > > Key: HBASE-27003 > URL: https://issues.apache.org/jira/browse/HBASE-27003 > Project: HBase > Issue Type: Improvement >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.13 > > Attachments: image-2022-05-06-18-13-50-763.png, > image-2022-05-06-18-14-28-578.png, image-2022-05-06-18-15-13-913.png > > > The logs in PerformanceEvaluation look a little confusing to new users, we > should optimize the format. > Before: > !image-2022-05-06-18-13-50-763.png|width=787,height=156! > After: > !image-2022-05-06-18-15-13-913.png|width=674,height=147! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] virajjasani commented on pull request #4411: HBASE-27003 Optimize log format for PerformanceEvaluation
virajjasani commented on PR #4411: URL: https://github.com/apache/hbase/pull/4411#issuecomment-1123204339 Thanks for the contribution @tomscut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani merged pull request #4411: HBASE-27003 Optimize log format for PerformanceEvaluation
virajjasani merged PR #4411: URL: https://github.com/apache/hbase/pull/4411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] xiaowangzhixiao commented on pull request #4335: HBASE-26942 cache region locations when getAllRegionLocations
xiaowangzhixiao commented on PR #4335: URL: https://github.com/apache/hbase/pull/4335#issuecomment-1123189429 @frostruan Thanks for your honest. I will try to submit a new issue to add a function with a cacheAll argument on top of your implementation. Thanks for your suggestion. @bbeaudreault -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4410: HBASE-27002 Config BucketCache as victim handler of LRUCache
Apache-HBase commented on PR #4410: URL: https://github.com/apache/hbase/pull/4410#issuecomment-1123182190 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 23s | master passed | | +1 :green_heart: | compile | 2m 16s | master passed | | +1 :green_heart: | checkstyle | 0m 32s | master passed | | -1 :x: | spotless | 0m 38s | branch has 68 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 9s | the patch passed | | +1 :green_heart: | compile | 2m 13s | the patch passed | | +1 :green_heart: | javac | 2m 13s | the patch passed | | +1 :green_heart: | checkstyle | 0m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 37s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | -1 :x: | spotless | 0m 36s | patch has 69 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 31m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4410/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4410 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux fa67db06ce07 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4410/2/artifact/yetus-general-check/output/branch-spotless.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4410/2/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4410/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache-HBase commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1123182035 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 18s | master passed | | +1 :green_heart: | compile | 2m 8s | master passed | | +1 :green_heart: | checkstyle | 0m 27s | master passed | | -1 :x: | spotless | 0m 36s | branch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 3s | the patch passed | | +1 :green_heart: | compile | 2m 13s | the patch passed | | +1 :green_heart: | javac | 2m 13s | the patch passed | | -0 :warning: | checkstyle | 0m 28s | hbase-server: The patch generated 1 new + 4 unchanged - 1 fixed = 5 total (was 5) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 49s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotless | 0m 38s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 8s | The patch does not generate ASF License warnings. | | | | 31m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4417 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 044b85019a33 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/4/artifact/yetus-general-check/output/branch-spotless.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] tomscut commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
tomscut commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1123164939 Thanks @Apache9 for your review and suggestion. I ran `mvn spotless:apply` and `mvn spotless:check`, build success locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
Apache-HBase commented on PR #4414: URL: https://github.com/apache/hbase/pull/4414#issuecomment-1123137144 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 5s | master passed | | +1 :green_heart: | compile | 2m 39s | master passed | | +1 :green_heart: | checkstyle | 0m 38s | master passed | | -1 :x: | spotless | 0m 32s | branch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 39s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 14s | the patch passed | | +1 :green_heart: | compile | 2m 40s | the patch passed | | +1 :green_heart: | javac | 2m 40s | the patch passed | | -0 :warning: | checkstyle | 0m 26s | hbase-server: The patch generated 1 new + 4 unchanged - 1 fixed = 5 total (was 5) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 42s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 34m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4414 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux a600dfdd98b0 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/6/artifact/yetus-general-check/output/branch-spotless.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 69 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/6/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4419: HBASE-27021 StoreFileInfo should set its initialPath in a consistent way
Apache9 commented on PR #4419: URL: https://github.com/apache/hbase/pull/4419#issuecomment-1123127903 OK, there is a FileSystem instance in the StoreFileInfo, so I think we just need to make a makeQualified to normalize the initial path? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4419: HBASE-27021 StoreFileInfo should set its initialPath in a consistent way
Apache9 commented on PR #4419: URL: https://github.com/apache/hbase/pull/4419#issuecomment-1123126805 I think the key problem here is we use StoreFileInfo as the key of a hash map. In general, within a typical store, we just need to use the storefile's name to decide whether they are the same StoreFileInfo. Or at least, when comparing, we should normalize the Path object to the same format... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a diff in pull request #4419: HBASE-27021 StoreFileInfo should set its initialPath in a consistent way
Apache9 commented on code in PR #4419:
URL: https://github.com/apache/hbase/pull/4419#discussion_r869823745
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java:
##
@@ -119,18 +119,18 @@ public class StoreFileInfo implements Configurable {
*/
public StoreFileInfo(final Configuration conf, final FileSystem fs, final
Path initialPath,
final boolean primaryReplica) throws IOException {
-this(conf, fs, null, initialPath, primaryReplica);
+this(conf, fs, fs.getFileStatus(initialPath), primaryReplica);
Review Comment:
I think this is intentional, we want to reduce the request to namenode...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache9 commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1123118328 Please just run spotless:apply to fix the spotless errors? Will check the checkstyle configurations later to align them. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] 2005hithlj closed pull request #4412: HBASE-26977 Fix branch-2 spotless check error
2005hithlj closed pull request #4412: HBASE-26977 Fix branch-2 spotless check error URL: https://github.com/apache/hbase/pull/4412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
Apache-HBase commented on PR #4414: URL: https://github.com/apache/hbase/pull/4414#issuecomment-1123116750 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 54s | master passed | | +1 :green_heart: | compile | 1m 29s | master passed | | +1 :green_heart: | shadedjars | 5m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 12s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | shadedjars | 5m 16s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-common generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 24s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 232m 16s | hbase-server in the patch passed. | | | | 260m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4414 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 961db863f0cb 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | javadoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-common.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/testReport/ | | Max. process+thread count | 2328 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #4407: HBASE-26999 HStore should try write WAL compaction marker before repl…
Apache9 commented on PR #4407: URL: https://github.com/apache/hbase/pull/4407#issuecomment-1123114603 Please do not ignore the spotless error before merging? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4420: HBASE-27019 Minor compression performance improvements
Apache-HBase commented on PR #4420: URL: https://github.com/apache/hbase/pull/4420#issuecomment-1123098880 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 22s | master passed | | +1 :green_heart: | compile | 1m 42s | master passed | | +1 :green_heart: | checkstyle | 0m 48s | master passed | | -1 :x: | spotless | 0m 37s | branch has 68 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 48s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 11s | the patch passed | | +1 :green_heart: | compile | 1m 40s | the patch passed | | +1 :green_heart: | javac | 1m 40s | the patch passed | | +1 :green_heart: | checkstyle | 0m 47s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 37s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | -1 :x: | spotless | 0m 36s | patch has 68 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 2m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 54s | The patch does not generate ASF License warnings. | | | | 34m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4420/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4420 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux c341815e694c 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4420/1/artifact/yetus-general-check/output/branch-spotless.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4420/1/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 69 (vs. ulimit of 3) | | modules | C: hbase-compression/hbase-compression-aircompressor hbase-compression/hbase-compression-brotli hbase-compression/hbase-compression-lz4 hbase-compression/hbase-compression-snappy hbase-compression/hbase-compression-xz hbase-compression/hbase-compression-zstd U: hbase-compression | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4420/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] tomscut commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
tomscut commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1123096530 Hi @Apache9 @apurtell I found some spotless warn on another PR at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/1/artifact/yetus-general-check/output/branch-spotless.txt. So I repaired them by running "mvn spotless:apply". Then I executed "mvn spotless:check" locally, and the build succeeded. But there are some checkstyle errors https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt. Then I fix the checkstyle errors. The spotless warn came again. There seems to be some conflict between the default spotless format and hbase checkstyle in indentation checking. I don't know if there have been similar cases before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4420: HBASE-27019 Minor compression performance improvements
Apache-HBase commented on PR #4420: URL: https://github.com/apache/hbase/pull/4420#issuecomment-1123094567 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 46s | master passed | | +1 :green_heart: | compile | 1m 26s | master passed | | +1 :green_heart: | shadedjars | 3m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 15s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 34s | the patch passed | | +1 :green_heart: | compile | 1m 26s | the patch passed | | +1 :green_heart: | javac | 1m 26s | the patch passed | | +1 :green_heart: | shadedjars | 3m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 50s | hbase-compression-aircompressor in the patch passed. | | +1 :green_heart: | unit | 0m 35s | hbase-compression-brotli in the patch passed. | | +1 :green_heart: | unit | 0m 30s | hbase-compression-lz4 in the patch passed. | | +1 :green_heart: | unit | 0m 29s | hbase-compression-snappy in the patch passed. | | +1 :green_heart: | unit | 0m 57s | hbase-compression-xz in the patch passed. | | +1 :green_heart: | unit | 0m 31s | hbase-compression-zstd in the patch passed. | | | | 25m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4420/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4420 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 519488f20e92 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4420/1/testReport/ | | Max. process+thread count | 603 (vs. ulimit of 3) | | modules | C: hbase-compression/hbase-compression-aircompressor hbase-compression/hbase-compression-brotli hbase-compression/hbase-compression-lz4 hbase-compression/hbase-compression-snappy hbase-compression/hbase-compression-xz hbase-compression/hbase-compression-zstd U: hbase-compression | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4420/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a diff in pull request #4419: HBASE-27021 StoreFileInfo should set its initialPath in a consistent way
joshelser commented on code in PR #4419:
URL: https://github.com/apache/hbase/pull/4419#discussion_r869802361
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java:
##
@@ -119,18 +119,18 @@ public class StoreFileInfo implements Configurable {
*/
public StoreFileInfo(final Configuration conf, final FileSystem fs, final
Path initialPath,
final boolean primaryReplica) throws IOException {
-this(conf, fs, null, initialPath, primaryReplica);
+this(conf, fs, fs.getFileStatus(initialPath), primaryReplica);
Review Comment:
Should make sure constructor javadoc is updated as this is a Namenode call.
If we call this often, we're incurring remote reads which could be a bad perf
hit. Need to make sure current callers are behaving nicely (we _should_ be
caching StoreFileInfo's already)
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStore.java:
##
@@ -940,13 +940,15 @@ private void archiveStoreFile(int index) throws
IOException {
private void closeCompactedFile(int index) throws IOException {
Collection files =
this.store.getStoreEngine().getStoreFileManager().getCompactedfiles();
-HStoreFile sf = null;
-Iterator it = files.iterator();
-for (int i = 0; i <= index; i++) {
- sf = it.next();
+if (files.size() > 0) {
+ HStoreFile sf = null;
+ Iterator it = files.iterator();
+ for (int i = 0; i <= index; i++) {
+sf = it.next();
+ }
+ sf.closeStoreFile(true);
+
store.getStoreEngine().getStoreFileManager().removeCompactedFiles(Lists.newArrayList(sf));
Review Comment:
nit: `Collections.singletonList(sf)` might be a little less heavy than a new
ArrayList (since you're changing this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4420: HBASE-27019 Minor compression performance improvements
Apache-HBase commented on PR #4420: URL: https://github.com/apache/hbase/pull/4420#issuecomment-1123093620 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 20s | master passed | | +1 :green_heart: | compile | 1m 20s | master passed | | +1 :green_heart: | shadedjars | 3m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 7s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 13s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 3m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 43s | hbase-compression-aircompressor in the patch passed. | | +1 :green_heart: | unit | 0m 35s | hbase-compression-brotli in the patch passed. | | +1 :green_heart: | unit | 0m 29s | hbase-compression-lz4 in the patch passed. | | +1 :green_heart: | unit | 0m 31s | hbase-compression-snappy in the patch passed. | | +1 :green_heart: | unit | 0m 54s | hbase-compression-xz in the patch passed. | | +1 :green_heart: | unit | 0m 31s | hbase-compression-zstd in the patch passed. | | | | 23m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4420/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4420 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0fa4c07e72d6 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4420/1/testReport/ | | Max. process+thread count | 798 (vs. ulimit of 3) | | modules | C: hbase-compression/hbase-compression-aircompressor hbase-compression/hbase-compression-brotli hbase-compression/hbase-compression-lz4 hbase-compression/hbase-compression-snappy hbase-compression/hbase-compression-xz hbase-compression/hbase-compression-zstd U: hbase-compression | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4420/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27013) Introduce read all bytes when using pread for prefetch
[
https://issues.apache.org/jira/browse/HBASE-27013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534638#comment-17534638
]
Josh Elser commented on HBASE-27013:
{quote}In the case of the input stream read short and when the input stream
read passed the length of the necessary data block with few more bytes within
the size of next block header, the
[BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
returns to the caller without a cached the next block header. As a result,
before HBase tries to read the next block,
[HFileBlock#readBlockDataInternal|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1648-L1664]
in hbase tries to re-read the next block header from the input stream.
{quote}
If we read the comment on the code that Stephen called out in
readBlockDataInternal, you can find:
{code:java}
If header was not cached (see getCachedHeader above), need to seek to pull it
in. This is costly and should happen very rarely {code}
And then you had also said:
{quote}The root cause of above issue was due to
[BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
is reading an input stream that does not guarrentee to return the data block
and the next block header as an option data to be cached.
{quote}
I think what you're saying is the following.
# Read header for block1
# Read block1 and try to read block2's header
# Read block2 and try to read block3's header
# Repeat
This would align with the comment, too. Where, the last time we read, we tried
to get the header cached, such that the _next_ time we come back to read, we
have that header already cached and can avoid another {{seek()}} (through the
pread).
The very high-level reading of the HBase code would indicate to me that we
_expect_ to read the n+1th block header when reading the nth block. I would
assume that we also want this for HDFS base cluster, but HDFS just does a good
enough job that we haven't noticed this being a problem (short-circuit reads
making our live happy?).
I think attempting to read off the end of a file is not a big concern since
we're just pulling those extra bytes off in the current read. I am thinking
about a different drawback where, if the InputStream isn't giving us the bytes
we asked for back, why was that? Did it take over some threshold of time? If we
go back and ask HDFS (or S3) again "give me those extra bytes", would we
increase the overall latency? Genuinely not sure.
I think, long-term, it makes sense for this configuration to be on by default,
but I am motivated by the expose this configuration property for additional
testing on HDFS while committing this change to try to help the S3-based
prefetching workload. I'm leaning towards putting this in since the risk is low
(given my understanding).
WDYT, Duo?
> Introduce read all bytes when using pread for prefetch
> --
>
> Key: HBASE-27013
> URL: https://issues.apache.org/jira/browse/HBASE-27013
> Project: HBase
> Issue Type: Improvement
> Components: HFile, Performance
>Affects Versions: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
>
> h2. Problem statement
> When prefetching HFiles from blob storage like S3 and use it with the storage
> implementation like S3A, we found there is a logical issue in HBase pread
> that causes the reading of the remote HFile aborts the input stream multiple
> times. This aborted stream and reopen slow down the reads and trigger many
> aborted bytes and waste time in recreating the connection especially when SSL
> is enabled.
> h2. ROOT CAUSE
> The root cause of above issue was due to
> [BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
> is reading an input stream that does not guarrentee to return the data block
> and the next block header as an option data to be cached.
> In the case of the input stream read short and when the input stream read
> passed the length of the necessary data block with few more bytes within the
> size of next block header, the
> [BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
> returns to the caller
[jira] [Comment Edited] (HBASE-27013) Introduce read all bytes when using pread for prefetch
[
https://issues.apache.org/jira/browse/HBASE-27013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534638#comment-17534638
]
Josh Elser edited comment on HBASE-27013 at 5/11/22 1:46 AM:
-
{quote}In the case of the input stream read short and when the input stream
read passed the length of the necessary data block with few more bytes within
the size of next block header, the
[BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
returns to the caller without a cached the next block header. As a result,
before HBase tries to read the next block,
[HFileBlock#readBlockDataInternal|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1648-L1664]
in hbase tries to re-read the next block header from the input stream.
{quote}
If we read the comment on the code that Stephen called out in
readBlockDataInternal, you can find:
{code:java}
If header was not cached (see getCachedHeader above), need to seek to pull it
in. This is costly and should happen very rarely {code}
And then you had also said:
{quote}The root cause of above issue was due to
[BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
is reading an input stream that does not guarrentee to return the data block
and the next block header as an option data to be cached.
{quote}
I think what you're saying is the following.
# Read header for block1
# Read block1 and try to read block2's header
# Read block2 and try to read block3's header
# Repeat
This would align with the comment, too. Where, the last time we read, we tried
to get the header cached, such that the _next_ time we come back to read, we
have that header already cached and can avoid another {{seek()}} (through the
pread).
The very high-level reading of the HBase code would indicate to me that we
_expect_ to read the n+1th block header when reading the nth block. I would
assume that we also want this for HDFS base cluster, but HDFS just does a good
enough job that we haven't noticed this being a problem (short-circuit reads
making our live happy?).
I think attempting to read off the end of a file is not a big concern since
we're just pulling those extra bytes off in the current read. I am thinking
about a different drawback where, if the InputStream isn't giving us the bytes
we asked for back, why was that? Did it take over some threshold of time? If we
go back and ask HDFS (or S3) again "give me those extra bytes", would we
increase the overall latency? Genuinely not sure.
I think, long-term, it makes sense for this configuration to be on by default,
but I am motivated by the expose this configuration property for additional
testing on HDFS while committing this change to try to help the S3-based
prefetching workload. I'm leaning towards putting this in since the risk is low
(given my understanding).
WDYT, Duo? Stephen, did I get this all correct? (please correct me if I'm wrong)
was (Author: elserj):
{quote}In the case of the input stream read short and when the input stream
read passed the length of the necessary data block with few more bytes within
the size of next block header, the
[BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
returns to the caller without a cached the next block header. As a result,
before HBase tries to read the next block,
[HFileBlock#readBlockDataInternal|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1648-L1664]
in hbase tries to re-read the next block header from the input stream.
{quote}
If we read the comment on the code that Stephen called out in
readBlockDataInternal, you can find:
{code:java}
If header was not cached (see getCachedHeader above), need to seek to pull it
in. This is costly and should happen very rarely {code}
And then you had also said:
{quote}The root cause of above issue was due to
[BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
is reading an input stream that does not guarrentee to return the data block
and the next block header as an option data to be cached.
{quote}
I think what you're saying is the following.
# Read header for block1
# Read block1 and try to read block2's header
# Read block2 and try to read block3's header
[GitHub] [hbase] Apache-HBase commented on pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
Apache-HBase commented on PR #4414: URL: https://github.com/apache/hbase/pull/4414#issuecomment-1123087304 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 4s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | shadedjars | 3m 49s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 5s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 3m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 36s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 182m 59s | hbase-server in the patch passed. | | | | 201m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4414 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 123c61a315e4 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/testReport/ | | Max. process+thread count | 2265 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26905) ReplicationPeerManager#checkPeerExists should throw ReplicationPeerNotFoundException if peer doesn't exists
[
https://issues.apache.org/jira/browse/HBASE-26905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534635#comment-17534635
]
Andrew Kyle Purtell commented on HBASE-26905:
-
Apologies for the long delayed response. This seems like a good idea
[~shahrs87]. Did you or will you open a PR for it? This can go as far back as
branch-2.5.
> ReplicationPeerManager#checkPeerExists should throw
> ReplicationPeerNotFoundException if peer doesn't exists
> ---
>
> Key: HBASE-26905
> URL: https://issues.apache.org/jira/browse/HBASE-26905
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Rushabh Shah
>Priority: Major
>
> ReplicationPeerManager#checkPeerExists should throw
> ReplicationPeerNotFoundException if peer doesn't exists. Currently it throws
> generic DoNotRetryIOException.
> {code:java}
> private ReplicationPeerDescription checkPeerExists(String peerId) throws
> DoNotRetryIOException {
> ReplicationPeerDescription desc = peers.get(peerId);
> if (desc == null) {
> throw new DoNotRetryIOException("Replication peer " + peerId + " does not
> exist");
> }
> return desc;
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] joshelser commented on a diff in pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
joshelser commented on code in PR #4414:
URL: https://github.com/apache/hbase/pull/4414#discussion_r869794629
##
hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java:
##
@@ -284,6 +311,10 @@ private static boolean preadWithExtraDirectly(ByteBuff
buff, FSDataInputStream d
throw e;
}
if (ret < 0) {
+if (remain <= extraLen) {
+ // break for the "extra data" when hitting end of stream and
remaining is necessary
+ break;
Review Comment:
That's a fair point -- I would have expected that we were very explicit and
did exactly what the caller said. I think you did the right thing. May be good
to capture that in a comment :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4382: HBASE-26925 Create WAL event tracker table to track all the WAL events.
Apache-HBase commented on PR #4382: URL: https://github.com/apache/hbase/pull/4382#issuecomment-1123077071 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 22s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-26913-replication-observability-framework Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 22s | HBASE-26913-replication-observability-framework passed | | +1 :green_heart: | compile | 1m 26s | HBASE-26913-replication-observability-framework passed | | +1 :green_heart: | shadedjars | 3m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | HBASE-26913-replication-observability-framework passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 15s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 3m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 27s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 0m 34s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. | | -1 :x: | unit | 266m 47s | hbase-server in the patch failed. | | | | 289m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4382/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4382 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b9c5649ab1ab 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-26913-replication-observability-framework / 64a6ba3647 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4382/7/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4382/7/testReport/ | | Max. process+thread count | 2533 (vs. ulimit of 3) | | modules | C: hbase-common hbase-hadoop-compat hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4382/7/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell opened a new pull request, #4420: HBASE-27019 Minor compression performance improvements
apurtell opened a new pull request, #4420: URL: https://github.com/apache/hbase/pull/4420 TRACE level logging is expensive enough to warrant removal. They were useful during development but are now just overhead. Also we unnecessarily create new compressor and decompressor instances in the reset() methods for the Aircompressor and Lz4 codecs. Remove. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-27019) Minor compression performance improvements
[
https://issues.apache.org/jira/browse/HBASE-27019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-27019:
Description:
TRACE level logging is expensive enough to warrant removal. They were useful
during development but now are just overhead.
{noformat}
12700390224.07% 127 jbyte_disjoint_arraycopy
{noformat}
e.g.
{noformat}
[ 0] jbyte_disjoint_arraycopy
[ 1] org.slf4j.impl.Reload4jLoggerAdapter.isTraceEnabled
[ 2] org.slf4j.impl.Reload4jLoggerAdapter.trace
[ 3]
org.apache.hadoop.hbase.io.compress.aircompressor.HadoopCompressor.setInput
[ 4] org.apache.hadoop.io.compress.BlockCompressorStream.write
[ 5] java.io.OutputStream.write
[ 6] com.salesforce.hbase.util.TestUtils.outputStreamTest
[ 7] com.salesforce.hbase.util.TestUtils.outputStreamTest
[ 8] com.salesforce.hbase.BenchmarkAircompressorLz4.test
[ 9]
com.salesforce.hbase.jmh_generated.BenchmarkAircompressorLz4_test_jmhTest.test_avgt_jmhStub
[10]
com.salesforce.hbase.jmh_generated.BenchmarkAircompressorLz4_test_jmhTest.test_AverageTime
{noformat}
Also we unnecessarily create new LZ4 compressor and decompressor instances in
the reset() methods.
was:
TRACE level logging is expensive enough to warrant removal. They were useful
during development but now are just overhead.
{noformat}
12700390224.07% 127 jbyte_disjoint_arraycopy
{noformat}
e.g.
{noformat}
[ 0] jbyte_disjoint_arraycopy
[ 1] org.slf4j.impl.Reload4jLoggerAdapter.isTraceEnabled
[ 2] org.slf4j.impl.Reload4jLoggerAdapter.trace
[ 3]
org.apache.hadoop.hbase.io.compress.aircompressor.HadoopCompressor.setInput
[ 4] org.apache.hadoop.io.compress.BlockCompressorStream.write
[ 5] java.io.OutputStream.write
[ 6] com.salesforce.hbase.util.TestUtils.outputStreamTest
[ 7] com.salesforce.hbase.util.TestUtils.outputStreamTest
[ 8] com.salesforce.hbase.BenchmarkAircompressorLz4.test
[ 9]
com.salesforce.hbase.jmh_generated.BenchmarkAircompressorLz4_test_jmhTest.test_avgt_jmhStub
[10]
com.salesforce.hbase.jmh_generated.BenchmarkAircompressorLz4_test_jmhTest.test_AverageTime
{noformat}
> Minor compression performance improvements
> --
>
> Key: HBASE-27019
> URL: https://issues.apache.org/jira/browse/HBASE-27019
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Trivial
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> TRACE level logging is expensive enough to warrant removal. They were useful
> during development but now are just overhead.
> {noformat}
> 12700390224.07% 127 jbyte_disjoint_arraycopy
> {noformat}
> e.g.
> {noformat}
> [ 0] jbyte_disjoint_arraycopy
> [ 1] org.slf4j.impl.Reload4jLoggerAdapter.isTraceEnabled
> [ 2] org.slf4j.impl.Reload4jLoggerAdapter.trace
> [ 3]
> org.apache.hadoop.hbase.io.compress.aircompressor.HadoopCompressor.setInput
> [ 4] org.apache.hadoop.io.compress.BlockCompressorStream.write
> [ 5] java.io.OutputStream.write
> [ 6] com.salesforce.hbase.util.TestUtils.outputStreamTest
> [ 7] com.salesforce.hbase.util.TestUtils.outputStreamTest
> [ 8] com.salesforce.hbase.BenchmarkAircompressorLz4.test
> [ 9]
> com.salesforce.hbase.jmh_generated.BenchmarkAircompressorLz4_test_jmhTest.test_avgt_jmhStub
> [10]
> com.salesforce.hbase.jmh_generated.BenchmarkAircompressorLz4_test_jmhTest.test_AverageTime
> {noformat}
> Also we unnecessarily create new LZ4 compressor and decompressor instances in
> the reset() methods.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HBASE-27019) Minor compression performance improvements
[
https://issues.apache.org/jira/browse/HBASE-27019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-27019:
Summary: Minor compression performance improvements (was: Remove TRACE
level logging in hbase-compression modules)
> Minor compression performance improvements
> --
>
> Key: HBASE-27019
> URL: https://issues.apache.org/jira/browse/HBASE-27019
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Trivial
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> TRACE level logging is expensive enough to warrant removal. They were useful
> during development but now are just overhead.
> {noformat}
> 12700390224.07% 127 jbyte_disjoint_arraycopy
> {noformat}
> e.g.
> {noformat}
> [ 0] jbyte_disjoint_arraycopy
> [ 1] org.slf4j.impl.Reload4jLoggerAdapter.isTraceEnabled
> [ 2] org.slf4j.impl.Reload4jLoggerAdapter.trace
> [ 3]
> org.apache.hadoop.hbase.io.compress.aircompressor.HadoopCompressor.setInput
> [ 4] org.apache.hadoop.io.compress.BlockCompressorStream.write
> [ 5] java.io.OutputStream.write
> [ 6] com.salesforce.hbase.util.TestUtils.outputStreamTest
> [ 7] com.salesforce.hbase.util.TestUtils.outputStreamTest
> [ 8] com.salesforce.hbase.BenchmarkAircompressorLz4.test
> [ 9]
> com.salesforce.hbase.jmh_generated.BenchmarkAircompressorLz4_test_jmhTest.test_avgt_jmhStub
> [10]
> com.salesforce.hbase.jmh_generated.BenchmarkAircompressorLz4_test_jmhTest.test_AverageTime
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4382: HBASE-26925 Create WAL event tracker table to track all the WAL events.
Apache-HBase commented on PR #4382: URL: https://github.com/apache/hbase/pull/4382#issuecomment-1123036372 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-26913-replication-observability-framework Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 32s | HBASE-26913-replication-observability-framework passed | | +1 :green_heart: | compile | 1m 27s | HBASE-26913-replication-observability-framework passed | | +1 :green_heart: | shadedjars | 3m 54s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 2s | HBASE-26913-replication-observability-framework passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 30s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 3m 52s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 45s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 0m 33s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 1m 20s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 200m 11s | hbase-server in the patch passed. | | | | 225m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4382/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4382 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fad2a9c8357e 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-26913-replication-observability-framework / 64a6ba3647 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4382/7/testReport/ | | Max. process+thread count | 2520 (vs. ulimit of 3) | | modules | C: hbase-common hbase-hadoop-compat hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4382/7/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] tomscut commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
tomscut commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1123024227 Thanks @shahrs87 for your review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-20430) Improve store file management for non-HDFS filesystems
[ https://issues.apache.org/jira/browse/HBASE-20430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-20430: Parent: (was: HBASE-20429) Issue Type: New Feature (was: Sub-task) > Improve store file management for non-HDFS filesystems > -- > > Key: HBASE-20430 > URL: https://issues.apache.org/jira/browse/HBASE-20430 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Kyle Purtell >Priority: Major > Labels: s3 > > HBase keeps a file open for every active store file so no additional round > trips to the NameNode are needed after the initial open. HDFS internally > multiplexes open files, but the Hadoop S3 filesystem implementations do not, > or, at least, not as well. As the bulk of data under management increases we > observe the required number of concurrently open connections will rise, and > expect it will eventually exhaust a limit somewhere (the client, the OS file > descriptor table or open file limits, or the S3 service). > Initially we can simply introduce an option to close every store file after > the reader has finished, and determine the performance impact. Use cases > backed by non-HDFS filesystems will already have to cope with a different > read performance profile. Based on experiments with the S3 backed Hadoop > filesystems, notably S3A, even with aggressively tuned options simple reads > can be very slow when there are blockcache misses, 15-20 seconds observed for > Get of a single small row, for example. We expect extensive use of the > BucketCache to mitigate in this application already. Could be backed by > offheap storage, but more likely a large number of cache files managed by the > file engine on local SSD storage. If misses are already going to be super > expensive, then the motivation to do more than simply open store files on > demand is largely absent. > Still, we could employ a predictive cache. Where frequent access to a given > store file (or, at least, its store) is predicted, keep a reference to the > store file open. Can keep statistics about read frequency, write it out to > HFiles during compaction, and note these stats when opening the region, > perhaps by reading all meta blocks of region HFiles when opening. Otherwise, > close the file after reading and open again on demand. Need to be careful not > to use ARC or equivalent as cache replacement strategy as it is encumbered. > The size of the cache can be determined at startup after detecting the > underlying filesystem. Eg. setCacheSize(VERY_LARGE_CONSTANT) if (fs > instanceof DistributedFileSystem), so we don't lose much when on HDFS still. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HBASE-20430) Improve store file management for non-HDFS filesystems
[ https://issues.apache.org/jira/browse/HBASE-20430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-20430: --- Assignee: (was: Mingliang Liu) > Improve store file management for non-HDFS filesystems > -- > > Key: HBASE-20430 > URL: https://issues.apache.org/jira/browse/HBASE-20430 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Priority: Major > Labels: s3 > > HBase keeps a file open for every active store file so no additional round > trips to the NameNode are needed after the initial open. HDFS internally > multiplexes open files, but the Hadoop S3 filesystem implementations do not, > or, at least, not as well. As the bulk of data under management increases we > observe the required number of concurrently open connections will rise, and > expect it will eventually exhaust a limit somewhere (the client, the OS file > descriptor table or open file limits, or the S3 service). > Initially we can simply introduce an option to close every store file after > the reader has finished, and determine the performance impact. Use cases > backed by non-HDFS filesystems will already have to cope with a different > read performance profile. Based on experiments with the S3 backed Hadoop > filesystems, notably S3A, even with aggressively tuned options simple reads > can be very slow when there are blockcache misses, 15-20 seconds observed for > Get of a single small row, for example. We expect extensive use of the > BucketCache to mitigate in this application already. Could be backed by > offheap storage, but more likely a large number of cache files managed by the > file engine on local SSD storage. If misses are already going to be super > expensive, then the motivation to do more than simply open store files on > demand is largely absent. > Still, we could employ a predictive cache. Where frequent access to a given > store file (or, at least, its store) is predicted, keep a reference to the > store file open. Can keep statistics about read frequency, write it out to > HFiles during compaction, and note these stats when opening the region, > perhaps by reading all meta blocks of region HFiles when opening. Otherwise, > close the file after reading and open again on demand. Need to be careful not > to use ARC or equivalent as cache replacement strategy as it is encumbered. > The size of the cache can be determined at startup after detecting the > underlying filesystem. Eg. setCacheSize(VERY_LARGE_CONSTANT) if (fs > instanceof DistributedFileSystem), so we don't lose much when on HDFS still. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
Apache-HBase commented on PR #4414: URL: https://github.com/apache/hbase/pull/4414#issuecomment-1122988690 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 28s | master passed | | +1 :green_heart: | compile | 2m 43s | master passed | | +1 :green_heart: | checkstyle | 0m 38s | master passed | | -1 :x: | spotless | 0m 36s | branch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 39s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 9s | the patch passed | | +1 :green_heart: | compile | 2m 42s | the patch passed | | +1 :green_heart: | javac | 2m 42s | the patch passed | | -0 :warning: | checkstyle | 0m 27s | hbase-server: The patch generated 1 new + 4 unchanged - 1 fixed = 5 total (was 5) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 58s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotless | 0m 39s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 34m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4414 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 5dbb22e0cce8 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/artifact/yetus-general-check/output/branch-spotless.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/5/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-27022) SFT seems to apparently tracking invalid/malformed store files
[
https://issues.apache.org/jira/browse/HBASE-27022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-27022:
Fix Version/s: 2.5.0
3.0.0-alpha-3
> SFT seems to apparently tracking invalid/malformed store files
> --
>
> Key: HBASE-27022
> URL: https://issues.apache.org/jira/browse/HBASE-27022
> Project: HBase
> Issue Type: Sub-task
>Reporter: Wellington Chevreuil
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> Opening this on behalf of [~apurtell] , who first reported this issue on
> HBASE-26999: When running scale tests using ITLCC, the following errors were
> observed:
> {noformat}
> [00]2022-05-05 15:59:52,280 WARN [region-location-0]
> regionserver.StoreFileInfo:
> Skipping
> hdfs://ip-172-31-58-47.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestLoadCommonCrawl/9eafc10e1b5a25532a4f0adf550828fc/c/9d07757144a7404fac02e161b5bd035e
> because it is empty. HBASE-646 DATA LOSS?
> ...
> [00]2022-05-05 15:59:52,320 WARN [region-location-2]
> regionserver.StoreFileInfo:
> Skipping
> hdfs://ip-172-31-58-47.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestLoadCommonCrawl/5322c54b9a899eae03cb16e956a836d5/c/184b4f55ab1a4dbc813e77aeae1343ae
>
> because it is empty. HBASE-646 DATA LOSS? {noformat}
>
> From some discussions in HBASE-26999, it seems that SFT has wrongly tracked
> an incomplete/unfinished store file.
> For further context, follow the [comments thread on
> HBASE-26999|https://issues.apache.org/jira/browse/HBASE-26999?focusedCommentId=17533508=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17533508].
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4382: HBASE-26925 Create WAL event tracker table to track all the WAL events.
Apache-HBase commented on PR #4382: URL: https://github.com/apache/hbase/pull/4382#issuecomment-1122872126 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 20s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-26913-replication-observability-framework Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 28s | HBASE-26913-replication-observability-framework passed | | +1 :green_heart: | compile | 3m 43s | HBASE-26913-replication-observability-framework passed | | +1 :green_heart: | checkstyle | 1m 10s | HBASE-26913-replication-observability-framework passed | | +0 :ok: | refguide | 2m 23s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 46s | HBASE-26913-replication-observability-framework passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 15s | the patch passed | | +1 :green_heart: | compile | 3m 40s | the patch passed | | +1 :green_heart: | javac | 3m 40s | the patch passed | | +1 :green_heart: | checkstyle | 1m 2s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 1m 53s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 11m 59s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | -1 :x: | spotless | 0m 36s | patch has 34 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 3m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 45m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4382/7/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4382 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile refguide xml | | uname | Linux 3e38e67511a4 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-26913-replication-observability-framework / 64a6ba3647 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4382/7/yetus-general-check/output/branch-site/book.html | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4382/7/yetus-general-check/output/patch-site/book.html | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4382/7/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-common hbase-hadoop-compat hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4382/7/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26999) HStore should try write WAL compaction marker before replacing compacted files in StoreEngine
[
https://issues.apache.org/jira/browse/HBASE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534566#comment-17534566
]
Hudson commented on HBASE-26999:
Results for branch branch-2.5
[build #115 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/115/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/115/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/115/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/115/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/115/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HStore should try write WAL compaction marker before replacing compacted
> files in StoreEngine
> -
>
> Key: HBASE-26999
> URL: https://issues.apache.org/jira/browse/HBASE-26999
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> On HBASE-26064, it seems we altered the order we update different places with
> the results of a compaction:
> {noformat}
> @@ -1510,14 +1149,13 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> List newFiles) throws IOException {
> // Do the steps necessary to complete the compaction.
> setStoragePolicyFromFileName(newFiles);
> - List sfs = commitStoreFiles(newFiles, true);
> + List sfs = storeEngine.commitStoreFiles(newFiles, true);
> if (this.getCoprocessorHost() != null) {
> for (HStoreFile sf : sfs) {
> getCoprocessorHost().postCompact(this, sf, cr.getTracker(), cr,
> user);
> }
> }
> - writeCompactionWalRecord(filesToCompact, sfs);
> - replaceStoreFiles(filesToCompact, sfs);
> + replaceStoreFiles(filesToCompact, sfs, true);
> ...
> @@ -1581,25 +1219,24 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> this.region.getRegionInfo(), compactionDescriptor,
> this.region.getMVCC());
> }
>
> - void replaceStoreFiles(Collection compactedFiles,
> Collection result)
> - throws IOException {
> - this.lock.writeLock().lock();
> - try {
> -
> this.storeEngine.getStoreFileManager().addCompactionResults(compactedFiles,
> result);
> - synchronized (filesCompacting) {
> - filesCompacting.removeAll(compactedFiles);
> - }
> -
> - // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> - // sizes later so it's not super-critical if we miss these.
> - RegionServerServices rsServices = region.getRegionServerServices();
> - if (rsServices != null &&
> rsServices.getRegionServerSpaceQuotaManager() != null) {
> - updateSpaceQuotaAfterFileReplacement(
> -
> rsServices.getRegionServerSpaceQuotaManager().getRegionSizeStore(),
> getRegionInfo(),
> - compactedFiles, result);
> - }
> - } finally {
> - this.lock.writeLock().unlock();
> + @RestrictedApi(explanation = "Should only be called in TestHStore", link =
> "",
> + allowedOnPath = ".*/(HStore|TestHStore).java")
> + void replaceStoreFiles(Collection compactedFiles,
> Collection result,
> + boolean writeCompactionMarker) throws IOException {
> + storeEngine.replaceStoreFiles(compactedFiles, result);
> + if (writeCompactionMarker) {
> + writeCompactionWalRecord(compactedFiles, result);
> + }
> + synchronized (filesCompacting) {
> + filesCompacting.removeAll(compactedFiles);
> + }
> + // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> + // sizes later so it's not super-critical if we miss these.
> + RegionServerServices rsServices = region.getRegionServerServices();
> + if (rsServices != null && rsServices.getRegionServerSpaceQuotaManager()
> != null) {
> + updateSpaceQuotaAfterFileReplacement(
> +
[jira] [Updated] (HBASE-27022) SFT seems to apparently tracking invalid/malformed store files
[
https://issues.apache.org/jira/browse/HBASE-27022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-27022:
-
Summary: SFT seems to apparently tracking invalid/malformed store files
(was: SFT seems apparently tracking invalid/malformed store files)
> SFT seems to apparently tracking invalid/malformed store files
> --
>
> Key: HBASE-27022
> URL: https://issues.apache.org/jira/browse/HBASE-27022
> Project: HBase
> Issue Type: Sub-task
>Reporter: Wellington Chevreuil
>Priority: Major
>
> Opening this on behalf of [~apurtell] , who first reported this issue on
> HBASE-26999: When running scale tests using ITLCC, the following errors were
> observed:
> {noformat}
> [00]2022-05-05 15:59:52,280 WARN [region-location-0]
> regionserver.StoreFileInfo:
> Skipping
> hdfs://ip-172-31-58-47.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestLoadCommonCrawl/9eafc10e1b5a25532a4f0adf550828fc/c/9d07757144a7404fac02e161b5bd035e
> because it is empty. HBASE-646 DATA LOSS?
> ...
> [00]2022-05-05 15:59:52,320 WARN [region-location-2]
> regionserver.StoreFileInfo:
> Skipping
> hdfs://ip-172-31-58-47.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestLoadCommonCrawl/5322c54b9a899eae03cb16e956a836d5/c/184b4f55ab1a4dbc813e77aeae1343ae
>
> because it is empty. HBASE-646 DATA LOSS? {noformat}
>
> From some discussions in HBASE-26999, it seems that SFT has wrongly tracked
> an incomplete/unfinished store file.
> For further context, follow the [comments thread on
> HBASE-26999|https://issues.apache.org/jira/browse/HBASE-26999?focusedCommentId=17533508=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17533508].
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (HBASE-27022) SFT seems apparently tracking invalid/malformed store files
Wellington Chevreuil created HBASE-27022:
Summary: SFT seems apparently tracking invalid/malformed store
files
Key: HBASE-27022
URL: https://issues.apache.org/jira/browse/HBASE-27022
Project: HBase
Issue Type: Sub-task
Reporter: Wellington Chevreuil
Opening this on behalf of [~apurtell] , who first reported this issue on
HBASE-26999: When running scale tests using ITLCC, the following errors were
observed:
{noformat}
[00]2022-05-05 15:59:52,280 WARN [region-location-0] regionserver.StoreFileInfo:
Skipping
hdfs://ip-172-31-58-47.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestLoadCommonCrawl/9eafc10e1b5a25532a4f0adf550828fc/c/9d07757144a7404fac02e161b5bd035e
because it is empty. HBASE-646 DATA LOSS?
...
[00]2022-05-05 15:59:52,320 WARN [region-location-2]
regionserver.StoreFileInfo:
Skipping
hdfs://ip-172-31-58-47.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestLoadCommonCrawl/5322c54b9a899eae03cb16e956a836d5/c/184b4f55ab1a4dbc813e77aeae1343ae
because it is empty. HBASE-646 DATA LOSS? {noformat}
>From some discussions in HBASE-26999, it seems that SFT has wrongly tracked an
>incomplete/unfinished store file.
For further context, follow the [comments thread on
HBASE-26999|https://issues.apache.org/jira/browse/HBASE-26999?focusedCommentId=17533508=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17533508].
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (HBASE-26999) HStore should try write WAL compaction marker before replacing compacted files in StoreEngine
[
https://issues.apache.org/jira/browse/HBASE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-26999:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Original issue fix is merged to master, branch-2 and branch-2.5. Thanks for the
reviews, [~apurtell] and [~zhangduo] . Will open another Jira for the
additional issue reported by [~apurtell] .
> HStore should try write WAL compaction marker before replacing compacted
> files in StoreEngine
> -
>
> Key: HBASE-26999
> URL: https://issues.apache.org/jira/browse/HBASE-26999
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> On HBASE-26064, it seems we altered the order we update different places with
> the results of a compaction:
> {noformat}
> @@ -1510,14 +1149,13 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> List newFiles) throws IOException {
> // Do the steps necessary to complete the compaction.
> setStoragePolicyFromFileName(newFiles);
> - List sfs = commitStoreFiles(newFiles, true);
> + List sfs = storeEngine.commitStoreFiles(newFiles, true);
> if (this.getCoprocessorHost() != null) {
> for (HStoreFile sf : sfs) {
> getCoprocessorHost().postCompact(this, sf, cr.getTracker(), cr,
> user);
> }
> }
> - writeCompactionWalRecord(filesToCompact, sfs);
> - replaceStoreFiles(filesToCompact, sfs);
> + replaceStoreFiles(filesToCompact, sfs, true);
> ...
> @@ -1581,25 +1219,24 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> this.region.getRegionInfo(), compactionDescriptor,
> this.region.getMVCC());
> }
>
> - void replaceStoreFiles(Collection compactedFiles,
> Collection result)
> - throws IOException {
> - this.lock.writeLock().lock();
> - try {
> -
> this.storeEngine.getStoreFileManager().addCompactionResults(compactedFiles,
> result);
> - synchronized (filesCompacting) {
> - filesCompacting.removeAll(compactedFiles);
> - }
> -
> - // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> - // sizes later so it's not super-critical if we miss these.
> - RegionServerServices rsServices = region.getRegionServerServices();
> - if (rsServices != null &&
> rsServices.getRegionServerSpaceQuotaManager() != null) {
> - updateSpaceQuotaAfterFileReplacement(
> -
> rsServices.getRegionServerSpaceQuotaManager().getRegionSizeStore(),
> getRegionInfo(),
> - compactedFiles, result);
> - }
> - } finally {
> - this.lock.writeLock().unlock();
> + @RestrictedApi(explanation = "Should only be called in TestHStore", link =
> "",
> + allowedOnPath = ".*/(HStore|TestHStore).java")
> + void replaceStoreFiles(Collection compactedFiles,
> Collection result,
> + boolean writeCompactionMarker) throws IOException {
> + storeEngine.replaceStoreFiles(compactedFiles, result);
> + if (writeCompactionMarker) {
> + writeCompactionWalRecord(compactedFiles, result);
> + }
> + synchronized (filesCompacting) {
> + filesCompacting.removeAll(compactedFiles);
> + }
> + // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> + // sizes later so it's not super-critical if we miss these.
> + RegionServerServices rsServices = region.getRegionServerServices();
> + if (rsServices != null && rsServices.getRegionServerSpaceQuotaManager()
> != null) {
> + updateSpaceQuotaAfterFileReplacement(
> + rsServices.getRegionServerSpaceQuotaManager().getRegionSizeStore(),
> getRegionInfo(),
> + compactedFiles, result); {noformat}
> While running some large scale load test, we run into File SFT metafiles
> inconsistency that we believe could have been avoided if the original order
> was in place. Here the scenario we had:
> 1) Region R with one CF f was open on RS1. At this time, the given store had
> some files, let's say these were file1, file2 and file3;
> 2) Compaction started on RS1;
> 3) RS1 entered a long GC pause, lost ZK lock. Compaction is still running,
> though.
> 4) RS2 opens R. The related File SFT instance for this store then creates a
> new meta file with file1, file2 and file3.
> 5) Compaction on RS1 successfully completes the
> *storeEngine.replaceStoreFiles* call. This updates the in memory cache of
> valid files (StoreFileManager) and the SFT metafile for the store engine on
> RS1 with the compaction resulting
[GitHub] [hbase] elliottneilclark commented on pull request #4406: HBASE-27000 Block cache stats (Misses Caching) display error in RS web UI
elliottneilclark commented on PR #4406: URL: https://github.com/apache/hbase/pull/4406#issuecomment-1122764517 Nice catch. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26999) HStore should try write WAL compaction marker before replacing compacted files in StoreEngine
[
https://issues.apache.org/jira/browse/HBASE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534536#comment-17534536
]
Hudson commented on HBASE-26999:
Results for branch master
[build #583 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HStore should try write WAL compaction marker before replacing compacted
> files in StoreEngine
> -
>
> Key: HBASE-26999
> URL: https://issues.apache.org/jira/browse/HBASE-26999
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> On HBASE-26064, it seems we altered the order we update different places with
> the results of a compaction:
> {noformat}
> @@ -1510,14 +1149,13 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> List newFiles) throws IOException {
> // Do the steps necessary to complete the compaction.
> setStoragePolicyFromFileName(newFiles);
> - List sfs = commitStoreFiles(newFiles, true);
> + List sfs = storeEngine.commitStoreFiles(newFiles, true);
> if (this.getCoprocessorHost() != null) {
> for (HStoreFile sf : sfs) {
> getCoprocessorHost().postCompact(this, sf, cr.getTracker(), cr,
> user);
> }
> }
> - writeCompactionWalRecord(filesToCompact, sfs);
> - replaceStoreFiles(filesToCompact, sfs);
> + replaceStoreFiles(filesToCompact, sfs, true);
> ...
> @@ -1581,25 +1219,24 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> this.region.getRegionInfo(), compactionDescriptor,
> this.region.getMVCC());
> }
>
> - void replaceStoreFiles(Collection compactedFiles,
> Collection result)
> - throws IOException {
> - this.lock.writeLock().lock();
> - try {
> -
> this.storeEngine.getStoreFileManager().addCompactionResults(compactedFiles,
> result);
> - synchronized (filesCompacting) {
> - filesCompacting.removeAll(compactedFiles);
> - }
> -
> - // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> - // sizes later so it's not super-critical if we miss these.
> - RegionServerServices rsServices = region.getRegionServerServices();
> - if (rsServices != null &&
> rsServices.getRegionServerSpaceQuotaManager() != null) {
> - updateSpaceQuotaAfterFileReplacement(
> -
> rsServices.getRegionServerSpaceQuotaManager().getRegionSizeStore(),
> getRegionInfo(),
> - compactedFiles, result);
> - }
> - } finally {
> - this.lock.writeLock().unlock();
> + @RestrictedApi(explanation = "Should only be called in TestHStore", link =
> "",
> + allowedOnPath = ".*/(HStore|TestHStore).java")
> + void replaceStoreFiles(Collection compactedFiles,
> Collection result,
> + boolean writeCompactionMarker) throws IOException {
> + storeEngine.replaceStoreFiles(compactedFiles, result);
> + if (writeCompactionMarker) {
> + writeCompactionWalRecord(compactedFiles, result);
> + }
> + synchronized (filesCompacting) {
> + filesCompacting.removeAll(compactedFiles);
> + }
> + // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> + // sizes later so it's not super-critical if we miss these.
> + RegionServerServices rsServices = region.getRegionServerServices();
> + if (rsServices != null && rsServices.getRegionServerSpaceQuotaManager()
> != null) {
> + updateSpaceQuotaAfterFileReplacement(
> + rsServices.getRegionServerSpaceQuotaManager().getRegionSizeStore(),
> getRegionInfo(),
> + compactedFiles, result); {noformat}
> While running some large scale load test, we run into File SFT metafiles
> inconsistency that we believe could have been avoided if
[jira] [Commented] (HBASE-27015) Fix log format for ServerManager
[
https://issues.apache.org/jira/browse/HBASE-27015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534535#comment-17534535
]
Hudson commented on HBASE-27015:
Results for branch master
[build #583 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix log format for ServerManager
>
>
> Key: HBASE-27015
> URL: https://issues.apache.org/jira/browse/HBASE-27015
> Project: HBase
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Minor
> Fix For: 3.0.0-alpha-3
>
> Attachments: ServerManagerLog.jpg
>
>
> A space is missing from the ServerManager log.
> !ServerManagerLog.jpg|width=919,height=128!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26994) MasterFileSystem create directory without permission check
[
https://issues.apache.org/jira/browse/HBASE-26994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534534#comment-17534534
]
Hudson commented on HBASE-26994:
Results for branch master
[build #583 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/583/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MasterFileSystem create directory without permission check
> --
>
> Key: HBASE-26994
> URL: https://issues.apache.org/jira/browse/HBASE-26994
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 2.4.12
>Reporter: Zhang Dongsheng
>Assignee: Zhang Dongsheng
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.13
>
> Attachments: HBASE-26994.patch
>
>
> Method checkStagingDir and checkSubDir first check if directory is exist ,if
> not , create it with special permission. If exist then setPermission for this
> directory. BUT if not exist ,we still need set special permission for this
> directory
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache-HBase commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1122736170 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 52s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 4m 25s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 0s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 4m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 193m 27s | hbase-server in the patch passed. | | | | 214m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4417 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 62e470a09c2b 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/3/testReport/ | | Max. process+thread count | 2511 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache-HBase commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1122719519 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 37s | master passed | | +1 :green_heart: | compile | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 3m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 37s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 3m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 180m 6s | hbase-server in the patch passed. | | | | 196m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4417 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 43933c798f42 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/3/testReport/ | | Max. process+thread count | 2739 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] frostruan commented on pull request #4335: HBASE-26942 cache region locations when getAllRegionLocations
frostruan commented on PR #4335: URL: https://github.com/apache/hbase/pull/4335#issuecomment-1122686601 Thanks for the reply. @bbeaudreault hi @xiaowangzhixiao would you mind opening a new jira to provide your solution ? If there is any conflict, please just overwrite mine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26960) Another case for unnecessary replication suspending in RegionReplicationSink
[
https://issues.apache.org/jira/browse/HBASE-26960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534488#comment-17534488
]
Huaxiang Sun commented on HBASE-26960:
--
Does this issue exist in branch-2? If so, can we backport to branch-2/2.5/2.4?
Thanks.
> Another case for unnecessary replication suspending in RegionReplicationSink
>
>
> Key: HBASE-26960
> URL: https://issues.apache.org/jira/browse/HBASE-26960
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 3.0.0-alpha-2
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Fix For: 3.0.0-alpha-3
>
>
> Besides HBASE-26768, there is another case replication in
> {{RegionReplicationSink}} would be suspend:
> For {{RegionReplicationSink}}, when there is a replication error ,
> {{RegionReplicationSink}} invokes {{MemStoreFlusher#requestFlush}} to request
> a flush, and after receiving the {{FlushAction#START_FLUSH}} or
> {{FlushAction#CANNOT_FLUSH}} flush marker, it would resume the replication.
> But when {{MemStoreFlusher}} flushing, it invokes following method
> {{HRegion.flushcache}} with the {{writeFlushRequestWalMarker}} set to false:
> {code:java}
> public FlushResultImpl flushcache(List families,
> boolean writeFlushRequestWalMarker, FlushLifeCycleTracker tracker)
> throws IOException {
> }
> {code}
> When {{writeFlushRequestWalMarker}} is set to false, {{HRegion.flushcache}}
> does not write the {{FlushAction#CANNOT_FLUSH}} flush marker to {{WAL}} when
> the memstore is empty, just as following
> {{HRegion.writeFlushRequestMarkerToWAL}} illustrated:
> {code:java}
> private boolean writeFlushRequestMarkerToWAL(WAL wal, boolean
> writeFlushWalMarker) {
> if (writeFlushWalMarker && wal != null && !writestate.readOnly) {
> FlushDescriptor desc =
> ProtobufUtil.toFlushDescriptor(FlushAction.CANNOT_FLUSH,
> getRegionInfo(), -1, new TreeMap<>(Bytes.BYTES_COMPARATOR));
> try {
> WALUtil.writeFlushMarker(wal, this.getReplicationScope(),
> getRegionInfo(), desc, true, mvcc,
> regionReplicationSink.orElse(null));
> return true;
> } catch (IOException e) {
> LOG.warn(getRegionInfo().getEncodedName() + " : " +
> "Received exception while trying to write the flush request to
> wal", e);
> }
> }
> return false;
> }
> {code}
> so when there is a replication error when the memstore is empty(eg.
> replicating the {{FlushAction#START_FLUSH}} or {{FlushAction#COMMIT_FLUSH}}
> ), the replication may suspend until next memstore flush,even though later
> there are user writes and it could replicate normally.
> I simulate this problem in the PR , and for {{writeFlushRequestWalMarker}}
> paramter, it is introduced by HBASE-11580 and just only determines whether or
> not writing the {{FlushAction#CANNOT_FLUSH}} flush marker to WAL when the
> memstore is empty, so I think for simplicity, we could set it to true always
> for {{MemStoreFlusher}}.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (HBASE-18678) Add more metrics to the Procedure MBean; currently one metric only for a loci of macro churn
[
https://issues.apache.org/jira/browse/HBASE-18678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani reassigned HBASE-18678:
Assignee: Anwesha Banerjee
> Add more metrics to the Procedure MBean; currently one metric only for a loci
> of macro churn
>
>
> Key: HBASE-18678
> URL: https://issues.apache.org/jira/browse/HBASE-18678
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Michael Stack
>Assignee: Anwesha Banerjee
>Priority: Major
> Labels: beginner
>
> I just noticed the procedurev2 MBean published by the master and how little
> it exposes:
> {code}{
> name: "Hadoop:service=HBase,name=Master,sub=Procedure",
> modelerType: "Master,sub=Procedure",
> tag.Context: "master",
> tag.Hostname: "ve0524.halxg.cloudera.com",
> numMasterWALs: 1
> },
> {code]
> There is a bunch of other stuff we could add in here... to offer insight on
> the Master's reactor
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] bbeaudreault commented on pull request #4335: HBASE-26942 cache region locations when getAllRegionLocations
bbeaudreault commented on PR #4335: URL: https://github.com/apache/hbase/pull/4335#issuecomment-1122661351 I think it's an honest mistake. I did not see any other jira or PR related to this. Since this one has already been merged to all branches, and the jira resolved, I think the best course of action is to leave this as is. If someone wants to submit a new issue to add a cacheAll argument, we can add it on top of this implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4419: HBASE-27021 StoreFileInfo should set its initialPath in a consistent way
Apache-HBase commented on PR #4419: URL: https://github.com/apache/hbase/pull/4419#issuecomment-1122660067 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 56s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 1m 3s | root in master failed. | | -1 :x: | compile | 0m 27s | hbase-server in master failed. | | -0 :warning: | checkstyle | 0m 4s | The patch fails to run checkstyle in hbase-server | | -1 :x: | spotless | 0m 48s | branch has 66 errors when running spotless:check, run spotless:apply to fix. | | -1 :x: | spotbugs | 0m 6s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 30s | the patch passed | | +1 :green_heart: | compile | 2m 38s | the patch passed | | -0 :warning: | javac | 2m 38s | hbase-server generated 193 new + 0 unchanged - 0 fixed = 193 total (was 0) | | +1 :green_heart: | checkstyle | 0m 34s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 13m 34s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | -1 :x: | spotless | 0m 40s | patch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 31m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4419 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 2520e49a53ad 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-general-check/output/branch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-general-check/output/branch-compile-hbase-server.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-general-check/output/buildtool-branch-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-general-check/output/branch-spotless.txt | | spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-general-check/output/branch-spotbugs-hbase-server.txt | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] frostruan commented on pull request #4335: HBASE-26942 cache region locations when getAllRegionLocations
frostruan commented on PR #4335: URL: https://github.com/apache/hbase/pull/4335#issuecomment-1122656925 I am sorry for that. Didn't know you were also interested in contributing to the community. Anyway, I think finding the problem is more important than the solution. So Let's revert this patch and please submit your own solution to this problem. Thanks. @xiaowangzhixiao Would you mind helping do that ? @bbeaudreault -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4419: HBASE-27021 StoreFileInfo should set its initialPath in a consistent way
Apache-HBase commented on PR #4419: URL: https://github.com/apache/hbase/pull/4419#issuecomment-1122657764 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 0s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 0m 58s | root in master failed. | | -1 :x: | compile | 0m 26s | hbase-server in master failed. | | -1 :x: | shadedjars | 2m 33s | branch has 10 errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 8s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 18s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 4m 42s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 31s | hbase-server generated 96 new + 0 unchanged - 0 fixed = 96 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 13m 14s | hbase-server in the patch failed. | | | | 28m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4419 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 90c6ef53f8a7 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-jdk11-hadoop3-check/output/branch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-jdk11-hadoop3-check/output/branch-compile-hbase-server.txt | | shadedjars | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-jdk11-hadoop3-check/output/branch-shadedjars.txt | | javadoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/testReport/ | | Max. process+thread count | 610 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4419: HBASE-27021 StoreFileInfo should set its initialPath in a consistent way
Apache-HBase commented on PR #4419: URL: https://github.com/apache/hbase/pull/4419#issuecomment-1122654753 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 3m 48s | root in master failed. | | +1 :green_heart: | compile | 0m 34s | master passed | | +1 :green_heart: | shadedjars | 3m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 13s | the patch passed | | +1 :green_heart: | compile | 0m 34s | the patch passed | | +1 :green_heart: | javac | 0m 34s | the patch passed | | +1 :green_heart: | shadedjars | 3m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 7m 56s | hbase-server in the patch failed. | | | | 25m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4419 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e8eaf504c86b 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-jdk8-hadoop3-check/output/branch-mvninstall-root.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/testReport/ | | Max. process+thread count | 567 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4419/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4418: HBASE-26969:Eliminate MOB renames when SFT is enabled
Apache-HBase commented on PR #4418: URL: https://github.com/apache/hbase/pull/4418#issuecomment-1122647466 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 58s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 33s | master passed | | +1 :green_heart: | compile | 0m 49s | master passed | | +1 :green_heart: | shadedjars | 4m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 7s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 4m 20s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 215m 38s | hbase-server in the patch failed. | | | | 238m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4418 | | JIRA Issue | HBASE-26969 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2bac7fac6d6c 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/testReport/ | | Max. process+thread count | 2869 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4418: HBASE-26969:Eliminate MOB renames when SFT is enabled
Apache-HBase commented on PR #4418: URL: https://github.com/apache/hbase/pull/4418#issuecomment-1122617643 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | master passed | | +1 :green_heart: | compile | 0m 34s | master passed | | +1 :green_heart: | shadedjars | 3m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 15s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | shadedjars | 3m 52s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 189m 57s | hbase-server in the patch failed. | | | | 207m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4418 | | JIRA Issue | HBASE-26969 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6129249e5b3a 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/testReport/ | | Max. process+thread count | 2438 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-27021) StoreFileInfo should set its initialPath in a consistent way
Wellington Chevreuil created HBASE-27021:
Summary: StoreFileInfo should set its initialPath in a consistent
way
Key: HBASE-27021
URL: https://issues.apache.org/jira/browse/HBASE-27021
Project: HBase
Issue Type: Bug
Reporter: Wellington Chevreuil
Assignee: Wellington Chevreuil
Currently, StoreFileInfo provides overloaded public constructors where the
related file path can be passed as either a Path or FileStatus instance. This
can lead to the StoreFileInfo instances related to the same file entry to have
different representations of the file path, which could create problems for
functions relying on equality for comparing store files. One example I could
find is the StoreEngine.refreshStoreFiles method, which list some files from
the SFT, then compares against a list of files from the SFM to decide how it
should update SFM internal cache. Here's a sample output from the
TestHStore.testRefreshStoreFiles:
---
2022-05-10T15:06:42,831 INFO [Time-limited test] regionserver.StoreEngine(399):
Refreshing store files for
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine@69d58ac1 files to add:
[file:/hbase/hbase-server/target/test-data/e3eac5ce-9bdf-8624-bcec-09c89790d682/TestStoretestRefreshStoreFiles/data/default/table/da6a3cf38941b37cd16438d554b13bbc/family/6e92c2f5cf1f40f7b8c6b6b34a176fa5,
file:/hbase/hbase-server/target/test-data/e3eac5ce-9bdf-8624-bcec-09c89790d682/TestStoretestRefreshStoreFiles/data/default/table/da6a3cf38941b37cd16438d554b13bbc/family/{*}fa4d5909da644d94873cbfdc6b5a07da{*}]
files to remove:
[/hbase/hbase-server/target/test-data/e3eac5ce-9bdf-8624-bcec-09c89790d682/TestStoretestRefreshStoreFiles/data/default/table/da6a3cf38941b37cd16438d554b13bbc/family/{*}fa4d5909da644d94873cbfdc6b5a07da{*}]
---
The above will wrongly add it to SFM's list of compacted files, making a valid
file potentially eligible for deletion and data loss.
I think we can avoid that by always converting Path instances passed in
StoreFileInfo constructors to a FileStatus, for consistently build the internal
StoreFileInfo path.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] xiaowangzhixiao commented on pull request #4335: HBASE-26942 cache region locations when getAllRegionLocations
xiaowangzhixiao commented on PR #4335: URL: https://github.com/apache/hbase/pull/4335#issuecomment-1122586497 The question you came up 20 days ago is the question I point out in the internal HBase repository of our company 30 days ago, which I had offered a solution at the same time. You commented on My Pull Request why I added a function getAllRegionLocation() with a parameter cacheAll to solve this problem, and I answered your comment. Although the details of your implementation are different, I discovered this problem first and resolved it in our company 's internal code repository. I feel very pity that you get ahead of me to point out this problem in the community which was come up by me firstly. Your implementation looks good too, whatever, congratulations for your contribute to HBase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26999) HStore should try write WAL compaction marker before replacing compacted files in StoreEngine
[
https://issues.apache.org/jira/browse/HBASE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534427#comment-17534427
]
Andrew Kyle Purtell commented on HBASE-26999:
-
Yes, of course.
> HStore should try write WAL compaction marker before replacing compacted
> files in StoreEngine
> -
>
> Key: HBASE-26999
> URL: https://issues.apache.org/jira/browse/HBASE-26999
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> On HBASE-26064, it seems we altered the order we update different places with
> the results of a compaction:
> {noformat}
> @@ -1510,14 +1149,13 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> List newFiles) throws IOException {
> // Do the steps necessary to complete the compaction.
> setStoragePolicyFromFileName(newFiles);
> - List sfs = commitStoreFiles(newFiles, true);
> + List sfs = storeEngine.commitStoreFiles(newFiles, true);
> if (this.getCoprocessorHost() != null) {
> for (HStoreFile sf : sfs) {
> getCoprocessorHost().postCompact(this, sf, cr.getTracker(), cr,
> user);
> }
> }
> - writeCompactionWalRecord(filesToCompact, sfs);
> - replaceStoreFiles(filesToCompact, sfs);
> + replaceStoreFiles(filesToCompact, sfs, true);
> ...
> @@ -1581,25 +1219,24 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> this.region.getRegionInfo(), compactionDescriptor,
> this.region.getMVCC());
> }
>
> - void replaceStoreFiles(Collection compactedFiles,
> Collection result)
> - throws IOException {
> - this.lock.writeLock().lock();
> - try {
> -
> this.storeEngine.getStoreFileManager().addCompactionResults(compactedFiles,
> result);
> - synchronized (filesCompacting) {
> - filesCompacting.removeAll(compactedFiles);
> - }
> -
> - // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> - // sizes later so it's not super-critical if we miss these.
> - RegionServerServices rsServices = region.getRegionServerServices();
> - if (rsServices != null &&
> rsServices.getRegionServerSpaceQuotaManager() != null) {
> - updateSpaceQuotaAfterFileReplacement(
> -
> rsServices.getRegionServerSpaceQuotaManager().getRegionSizeStore(),
> getRegionInfo(),
> - compactedFiles, result);
> - }
> - } finally {
> - this.lock.writeLock().unlock();
> + @RestrictedApi(explanation = "Should only be called in TestHStore", link =
> "",
> + allowedOnPath = ".*/(HStore|TestHStore).java")
> + void replaceStoreFiles(Collection compactedFiles,
> Collection result,
> + boolean writeCompactionMarker) throws IOException {
> + storeEngine.replaceStoreFiles(compactedFiles, result);
> + if (writeCompactionMarker) {
> + writeCompactionWalRecord(compactedFiles, result);
> + }
> + synchronized (filesCompacting) {
> + filesCompacting.removeAll(compactedFiles);
> + }
> + // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> + // sizes later so it's not super-critical if we miss these.
> + RegionServerServices rsServices = region.getRegionServerServices();
> + if (rsServices != null && rsServices.getRegionServerSpaceQuotaManager()
> != null) {
> + updateSpaceQuotaAfterFileReplacement(
> + rsServices.getRegionServerSpaceQuotaManager().getRegionSizeStore(),
> getRegionInfo(),
> + compactedFiles, result); {noformat}
> While running some large scale load test, we run into File SFT metafiles
> inconsistency that we believe could have been avoided if the original order
> was in place. Here the scenario we had:
> 1) Region R with one CF f was open on RS1. At this time, the given store had
> some files, let's say these were file1, file2 and file3;
> 2) Compaction started on RS1;
> 3) RS1 entered a long GC pause, lost ZK lock. Compaction is still running,
> though.
> 4) RS2 opens R. The related File SFT instance for this store then creates a
> new meta file with file1, file2 and file3.
> 5) Compaction on RS1 successfully completes the
> *storeEngine.replaceStoreFiles* call. This updates the in memory cache of
> valid files (StoreFileManager) and the SFT metafile for the store engine on
> RS1 with the compaction resulting file, say file4, removing file1, file2 and
> file3. Note that the SFT meta file used by RS1 here is different (older) than
> the one used by RS2.
> 6) Compaction on RS1 tries to update WAL marker, but fails to
[GitHub] [hbase] Apache-HBase commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache-HBase commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1122562919 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 57s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 9s | master passed | | +1 :green_heart: | compile | 2m 49s | master passed | | +1 :green_heart: | checkstyle | 0m 34s | master passed | | -1 :x: | spotless | 0m 41s | branch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 34s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 5s | the patch passed | | +1 :green_heart: | compile | 2m 46s | the patch passed | | +1 :green_heart: | javac | 2m 46s | the patch passed | | +1 :green_heart: | checkstyle | 0m 32s | hbase-server: The patch generated 0 new + 0 unchanged - 5 fixed = 0 total (was 5) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 16m 45s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | -1 :x: | spotless | 0m 46s | patch has 38 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 43m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4417 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 9ce9c995d5bc 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/3/artifact/yetus-general-check/output/branch-spotless.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/3/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26999) HStore should try write WAL compaction marker before replacing compacted files in StoreEngine
[
https://issues.apache.org/jira/browse/HBASE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534364#comment-17534364
]
Wellington Chevreuil commented on HBASE-26999:
--
Hi [~apurtell] , since your reported issue differs from the original reported
here (for which we have a fix already committed), can we resolve this one and
continue the investigations on the additional error on a separate jira?
> HStore should try write WAL compaction marker before replacing compacted
> files in StoreEngine
> -
>
> Key: HBASE-26999
> URL: https://issues.apache.org/jira/browse/HBASE-26999
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> On HBASE-26064, it seems we altered the order we update different places with
> the results of a compaction:
> {noformat}
> @@ -1510,14 +1149,13 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> List newFiles) throws IOException {
> // Do the steps necessary to complete the compaction.
> setStoragePolicyFromFileName(newFiles);
> - List sfs = commitStoreFiles(newFiles, true);
> + List sfs = storeEngine.commitStoreFiles(newFiles, true);
> if (this.getCoprocessorHost() != null) {
> for (HStoreFile sf : sfs) {
> getCoprocessorHost().postCompact(this, sf, cr.getTracker(), cr,
> user);
> }
> }
> - writeCompactionWalRecord(filesToCompact, sfs);
> - replaceStoreFiles(filesToCompact, sfs);
> + replaceStoreFiles(filesToCompact, sfs, true);
> ...
> @@ -1581,25 +1219,24 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> this.region.getRegionInfo(), compactionDescriptor,
> this.region.getMVCC());
> }
>
> - void replaceStoreFiles(Collection compactedFiles,
> Collection result)
> - throws IOException {
> - this.lock.writeLock().lock();
> - try {
> -
> this.storeEngine.getStoreFileManager().addCompactionResults(compactedFiles,
> result);
> - synchronized (filesCompacting) {
> - filesCompacting.removeAll(compactedFiles);
> - }
> -
> - // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> - // sizes later so it's not super-critical if we miss these.
> - RegionServerServices rsServices = region.getRegionServerServices();
> - if (rsServices != null &&
> rsServices.getRegionServerSpaceQuotaManager() != null) {
> - updateSpaceQuotaAfterFileReplacement(
> -
> rsServices.getRegionServerSpaceQuotaManager().getRegionSizeStore(),
> getRegionInfo(),
> - compactedFiles, result);
> - }
> - } finally {
> - this.lock.writeLock().unlock();
> + @RestrictedApi(explanation = "Should only be called in TestHStore", link =
> "",
> + allowedOnPath = ".*/(HStore|TestHStore).java")
> + void replaceStoreFiles(Collection compactedFiles,
> Collection result,
> + boolean writeCompactionMarker) throws IOException {
> + storeEngine.replaceStoreFiles(compactedFiles, result);
> + if (writeCompactionMarker) {
> + writeCompactionWalRecord(compactedFiles, result);
> + }
> + synchronized (filesCompacting) {
> + filesCompacting.removeAll(compactedFiles);
> + }
> + // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> + // sizes later so it's not super-critical if we miss these.
> + RegionServerServices rsServices = region.getRegionServerServices();
> + if (rsServices != null && rsServices.getRegionServerSpaceQuotaManager()
> != null) {
> + updateSpaceQuotaAfterFileReplacement(
> + rsServices.getRegionServerSpaceQuotaManager().getRegionSizeStore(),
> getRegionInfo(),
> + compactedFiles, result); {noformat}
> While running some large scale load test, we run into File SFT metafiles
> inconsistency that we believe could have been avoided if the original order
> was in place. Here the scenario we had:
> 1) Region R with one CF f was open on RS1. At this time, the given store had
> some files, let's say these were file1, file2 and file3;
> 2) Compaction started on RS1;
> 3) RS1 entered a long GC pause, lost ZK lock. Compaction is still running,
> though.
> 4) RS2 opens R. The related File SFT instance for this store then creates a
> new meta file with file1, file2 and file3.
> 5) Compaction on RS1 successfully completes the
> *storeEngine.replaceStoreFiles* call. This updates the in memory cache of
> valid files (StoreFileManager) and the SFT metafile for the store engine on
> RS1 with the compaction
[GitHub] [hbase] Apache-HBase commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache-HBase commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1122470490 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 7s | master passed | | +1 :green_heart: | compile | 0m 38s | master passed | | +1 :green_heart: | shadedjars | 3m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 12s | the patch passed | | +1 :green_heart: | compile | 0m 39s | the patch passed | | +1 :green_heart: | javac | 0m 39s | the patch passed | | +1 :green_heart: | shadedjars | 3m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 190m 28s | hbase-server in the patch passed. | | | | 206m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4417 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6e60011df830 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/2/testReport/ | | Max. process+thread count | 2465 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534360#comment-17534360
]
Szabolcs Bukros commented on HBASE-26969:
-
I would like to start by stating that this issue grow bigger than just removing
the renames and exposed multiple issues in the MOB-SFT interaction.
I have uploaded a draft PR containing my changes. I intend to use it as a
reference to show the issues when it comes to using MOB on FileBased SFT.
My main problem was that while MOB files were already tracked in the hfile
metadata, the "single source of truth" is widely distributed and not easily
available.
Both the WriterCreationTracker and the StoreFileTracker are RS based data and
the MOB cleaner needs it to work reliably when FileBased SFT is used. Exposing
this data and allowing the Master to request this from RSes, collect it and run
the cleaner based on this, while technically possible, looked less than
optimal. It would result in a single cluster wide spike that we should try to
avoid and considering the delay that certain RSes could have (uneven load, GC
pauses, etc) the data can be already outdated by the time the collection is
done. So instead I tried to move the cleaner to the RSes. This solution also
had it's drawbacks.
MOB file names contain the encoded name of the region that created them so the
RS hosting that specific region can check it's hfiles for references and can
clean it up if it does not find anything. The problem comes with merge/split
parent regions. When the parent region is archived the new region's hfiles will
still hold references to the old MOB files but now the only way to make sure if
the old MOB file is referenced or not is to check every single hfile in every
store belonging to the same columnfamily, because we can not tell based on it's
name where it could be referenced from. Like the old cleaner did. So while I
moved the MOB cleaner to the RS level and reduced it's scope to only clean up
MOB files belonging to regions hosted by that RS I had to leave a "global" MOB
cleaner running on Master to deal with MOB files created by archived regions
but potentially still being referenced. And I think this is very ugly.
This whole process could have been significantly simpler if we would have
tracker files in MOB stores but then we would have TWO competing sources of
truth. The tracker files and the hfile metadata.
HBASE-27017 is a related issue where the snapshot code tries to get the active
MOB files based on the configured SFT, but since MOB stores do not have tracker
files it returns an empty list. If the store had tracker files it would work.
Without a tracker file we either include every MOB files in the dir (garbage
included) or scan every single hfile metadata for MOB references.
What I'm trying to say is that while I think my solution would work and solve
the immediate issues I would much prefer if there would be a centralized,
easily available active MOB list and create a solution based on that.
[~apurtell] ,[~zhangduo],[~elserj] ,[~wchevreuil] What do you think?
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache-HBase commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1122460217 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 52s | master passed | | +1 :green_heart: | compile | 0m 46s | master passed | | +1 :green_heart: | shadedjars | 3m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 29s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | shadedjars | 3m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 180m 3s | hbase-server in the patch passed. | | | | 198m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4417 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3feacd434f12 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/2/testReport/ | | Max. process+thread count | 2775 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4418: HBASE-26969:Eliminate MOB renames when SFT is enabled
Apache-HBase commented on PR #4418: URL: https://github.com/apache/hbase/pull/4418#issuecomment-1122400802 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 25s | master passed | | +1 :green_heart: | compile | 2m 16s | master passed | | +1 :green_heart: | checkstyle | 0m 32s | master passed | | -1 :x: | spotless | 0m 37s | branch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 10s | the patch passed | | +1 :green_heart: | compile | 2m 16s | the patch passed | | +1 :green_heart: | javac | 2m 16s | the patch passed | | -0 :warning: | checkstyle | 0m 32s | hbase-server: The patch generated 13 new + 19 unchanged - 0 fixed = 32 total (was 19) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 11m 34s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | -1 :x: | spotless | 0m 35s | patch has 66 errors when running spotless:check, run spotless:apply to fix. | | -1 :x: | spotbugs | 1m 26s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 32m 10s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | Possible null pointer dereference of regionDirs in org.apache.hadoop.hbase.mob.MobFileCleanerChore.cleanupObsoleteMobFiles(Configuration, TableName) Dereferenced at MobFileCleanerChore.java:regionDirs in org.apache.hadoop.hbase.mob.MobFileCleanerChore.cleanupObsoleteMobFiles(Configuration, TableName) Dereferenced at MobFileCleanerChore.java:[line 174] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4418 | | JIRA Issue | HBASE-26969 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 5a3c10599332 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/artifact/yetus-general-check/output/branch-spotless.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/artifact/yetus-general-check/output/whitespace-eol.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/artifact/yetus-general-check/output/patch-spotless.txt | | spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4418/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] BukrosSzabolcs opened a new pull request, #4418: HBASE-26969:Eliminate MOB renames when SFT is enabled
BukrosSzabolcs opened a new pull request, #4418: URL: https://github.com/apache/hbase/pull/4418 Get rid of renames in the MOB compaction and flush. Also refactor the MOB cleaner to be compatible with FileBased StoreFileTracker -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
Apache-HBase commented on PR #4414: URL: https://github.com/apache/hbase/pull/4414#issuecomment-1122314449 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 39s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 3m 43s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 34s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 4s | the patch passed | | +1 :green_heart: | shadedjars | 3m 42s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | hbase-common generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 45s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 212m 51s | hbase-server in the patch passed. | | | | 233m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4414 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2ece5342128a 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | javadoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-common.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/testReport/ | | Max. process+thread count | 2551 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
Apache-HBase commented on PR #4414: URL: https://github.com/apache/hbase/pull/4414#issuecomment-1122286080 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 1s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 3m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 2s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | shadedjars | 3m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 36s | hbase-common in the patch passed. | | -1 :x: | unit | 187m 39s | hbase-server in the patch failed. | | | | 205m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4414 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 27b6880ecae9 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/testReport/ | | Max. process+thread count | 2400 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache-HBase commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1122267825 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 7s | master passed | | +1 :green_heart: | compile | 2m 10s | master passed | | +1 :green_heart: | checkstyle | 0m 27s | master passed | | -1 :x: | spotless | 0m 32s | branch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 11s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 4s | the patch passed | | +1 :green_heart: | compile | 2m 11s | the patch passed | | +1 :green_heart: | javac | 2m 11s | the patch passed | | -0 :warning: | checkstyle | 0m 27s | hbase-server: The patch generated 1 new + 4 unchanged - 1 fixed = 5 total (was 5) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 38s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotless | 0m 38s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 7s | The patch does not generate ASF License warnings. | | | | 30m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4417 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 246fe2bb533e 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/2/artifact/yetus-general-check/output/branch-spotless.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache-HBase commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1122238376 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 25s | master passed | | +1 :green_heart: | compile | 0m 39s | master passed | | +1 :green_heart: | shadedjars | 3m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 13s | the patch passed | | +1 :green_heart: | compile | 0m 38s | the patch passed | | +1 :green_heart: | javac | 0m 38s | the patch passed | | +1 :green_heart: | shadedjars | 3m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 191m 23s | hbase-server in the patch passed. | | | | 208m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4417 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0593d77d1d3d 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/testReport/ | | Max. process+thread count | 2727 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26953) CallDroppedException message says "server 0.0.0.0 is overloaded"
[ https://issues.apache.org/jira/browse/HBASE-26953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534283#comment-17534283 ] Bryan Beaudreault commented on HBASE-26953: --- [~ayeshasilvia] sorry for the confusion. Yes, the exception includes the port for us as well. In terms of hours to reproduce, you could potentially deploy hbase with your own implementation of the pluggable RPC queue which always calls call.drop(). Otherwise I think it would only happen if using the codel queue, and only under high saturation. I’m out of office not, so can’t give better suggestion until next week > CallDroppedException message says "server 0.0.0.0 is overloaded" > > > Key: HBASE-26953 > URL: https://issues.apache.org/jira/browse/HBASE-26953 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Assignee: Ayesha Mosaddeque >Priority: Trivial > Labels: newbie > > This is not a super useful error when seen on the client side. Maybe we can > have it pull in the ServerName instead, or at the very least resolve to the > hostname. > > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java#L213 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache-HBase commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-118200 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 51s | master passed | | +1 :green_heart: | compile | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 3m 43s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 33s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 3m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 180m 41s | hbase-server in the patch passed. | | | | 198m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4417 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6f638ceee80e 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/testReport/ | | Max. process+thread count | 2668 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-27020) Fix spotless warn for master branch
[ https://issues.apache.org/jira/browse/HBASE-27020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HBASE-27020: --- Description: Fix spotless warn for master branch. We can see the spotless warn at [https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/1/artifact/yetus-general-check/output/branch-spotless.txt]. There are some compile tasks that fail because of this. was: Fix spotless warn for master branch. We can see the spotless warn at [https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/1/artifact/yetus-general-check/output/branch-spotless.txt]. There are many compile tasks that fail because of this. > Fix spotless warn for master branch > --- > > Key: HBASE-27020 > URL: https://issues.apache.org/jira/browse/HBASE-27020 > Project: HBase > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > > Fix spotless warn for master branch. We can see the spotless warn at > [https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/1/artifact/yetus-general-check/output/branch-spotless.txt]. > > There are some compile tasks that fail because of this. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
Apache-HBase commented on PR #4414: URL: https://github.com/apache/hbase/pull/4414#issuecomment-1122118753 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 10s | master passed | | +1 :green_heart: | compile | 2m 36s | master passed | | +1 :green_heart: | checkstyle | 0m 37s | master passed | | -1 :x: | spotless | 0m 32s | branch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 7s | the patch passed | | +1 :green_heart: | compile | 2m 40s | the patch passed | | +1 :green_heart: | javac | 2m 40s | the patch passed | | -0 :warning: | checkstyle | 0m 27s | hbase-server: The patch generated 1 new + 4 unchanged - 1 fixed = 5 total (was 5) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 50s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 33m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4414 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 2853ddde71ca 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/artifact/yetus-general-check/output/branch-spotless.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 65 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] tomscut commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
tomscut commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1122086831 After `mvn spotless:apply`, I run `mvn spotless:check` locally, and `build success`. But failed here. I feel a little confused. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4416: HBASE-27018 Add a tool command list_liveservers
Apache-HBase commented on PR #4416: URL: https://github.com/apache/hbase/pull/4416#issuecomment-1122061904 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | master passed | | +1 :green_heart: | javadoc | 0m 13s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 37s | the patch passed | | +1 :green_heart: | javadoc | 0m 10s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 14s | hbase-shell in the patch failed. | | | | 18m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4416 | | Optional Tests | javac javadoc unit | | uname | Linux 22b7c623d026 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-shell.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/testReport/ | | Max. process+thread count | 1405 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
Apache-HBase commented on PR #4414: URL: https://github.com/apache/hbase/pull/4414#issuecomment-1122061206 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 38s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 3m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 37s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 4s | the patch passed | | +1 :green_heart: | shadedjars | 3m 37s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | hbase-common generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 45s | hbase-common in the patch passed. | | -1 :x: | unit | 192m 33s | hbase-server in the patch failed. | | | | 212m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4414 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ea6066255f46 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | javadoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/2/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-common.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/2/testReport/ | | Max. process+thread count | 2536 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4416: HBASE-27018 Add a tool command list_liveservers
Apache-HBase commented on PR #4416: URL: https://github.com/apache/hbase/pull/4416#issuecomment-1122058718 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 50s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 42s | master passed | | +1 :green_heart: | javadoc | 0m 9s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 7s | the patch passed | | +1 :green_heart: | javadoc | 0m 8s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 7m 35s | hbase-shell in the patch failed. | | | | 15m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4416 | | Optional Tests | javac javadoc unit | | uname | Linux 45ab26c434c1 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-shell.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/testReport/ | | Max. process+thread count | 1493 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
Apache-HBase commented on PR #4414: URL: https://github.com/apache/hbase/pull/4414#issuecomment-1122054142 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 4s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | shadedjars | 3m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 6s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 3m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 36s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 184m 21s | hbase-server in the patch passed. | | | | 206m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4414 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 37471aaa7ef0 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/2/testReport/ | | Max. process+thread count | 2447 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4414/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4416: HBASE-27018 Add a tool command list_liveservers
Apache-HBase commented on PR #4416: URL: https://github.com/apache/hbase/pull/4416#issuecomment-1122050915 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 10s | master passed | | -1 :x: | spotless | 0m 36s | branch has 68 errors when running spotless:check, run spotless:apply to fix. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 11s | the patch passed | | -0 :warning: | rubocop | 0m 3s | The patch generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | spotless | 0m 36s | patch has 68 errors when running spotless:check, run spotless:apply to fix. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 8s | The patch does not generate ASF License warnings. | | | | 7m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4416 | | Optional Tests | dupname asflicense javac spotless rubocop | | uname | Linux 794f71fc514c 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/artifact/yetus-general-check/output/branch-spotless.txt | | rubocop | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 46 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/2/console | | versions | git=2.17.1 maven=3.6.3 rubocop=0.80.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4417: HBASE-27020 Fix spotless warn for master branch
Apache-HBase commented on PR #4417: URL: https://github.com/apache/hbase/pull/4417#issuecomment-1122049555 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | -1 :x: | mvninstall | 0m 12s | root in master failed. | | -1 :x: | compile | 0m 59s | hbase-server in master failed. | | -0 :warning: | checkstyle | 0m 3s | The patch fails to run checkstyle in hbase-server | | -1 :x: | spotless | 0m 35s | branch has 66 errors when running spotless:check, run spotless:apply to fix. | | -1 :x: | spotbugs | 0m 5s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 0s | the patch passed | | +1 :green_heart: | compile | 2m 12s | the patch passed | | -0 :warning: | javac | 2m 12s | hbase-server generated 193 new + 0 unchanged - 0 fixed = 193 total (was 0) | | -0 :warning: | checkstyle | 0m 32s | hbase-server: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | hadoopcheck | 9m 27s | The patch causes 23 errors with Hadoop v3.3.1. | | +1 :green_heart: | spotless | 0m 38s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 29s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 21m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4417 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 585a54ef664e 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-general-check/output/branch-mvninstall-root.txt | | compile | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-general-check/output/branch-compile-hbase-server.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-general-check/output/buildtool-branch-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-general-check/output/branch-spotless.txt | | spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-general-check/output/branch-spotbugs-hbase-server.txt | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | hadoopcheck | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/artifact/yetus-general-check/output/patch-javac-3.3.1.txt | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4417/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-27020) Fix spotless warn for master branch
[ https://issues.apache.org/jira/browse/HBASE-27020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HBASE-27020: --- Description: Fix spotless warn for master branch. We can see the spotless warn at [https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/1/artifact/yetus-general-check/output/branch-spotless.txt]. There are many compile tasks that fail because of this. was:Fix spotless warn for master branch. We can see the spotless warn at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/1/artifact/yetus-general-check/output/branch-spotless.txt. > Fix spotless warn for master branch > --- > > Key: HBASE-27020 > URL: https://issues.apache.org/jira/browse/HBASE-27020 > Project: HBase > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > > Fix spotless warn for master branch. We can see the spotless warn at > [https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/1/artifact/yetus-general-check/output/branch-spotless.txt]. > > There are many compile tasks that fail because of this. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4411: HBASE-27003 Optimize log format for PerformanceEvaluation
Apache-HBase commented on PR #4411: URL: https://github.com/apache/hbase/pull/4411#issuecomment-1122043990 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 19s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 2s | master passed | | +1 :green_heart: | compile | 0m 21s | master passed | | +1 :green_heart: | shadedjars | 3m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | shadedjars | 3m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 13m 46s | hbase-mapreduce in the patch passed. | | | | 30m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4411/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4411 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3270deb944f0 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4411/3/testReport/ | | Max. process+thread count | 2610 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4411/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4411: HBASE-27003 Optimize log format for PerformanceEvaluation
Apache-HBase commented on PR #4411: URL: https://github.com/apache/hbase/pull/4411#issuecomment-1122043540 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | master passed | | +1 :green_heart: | compile | 0m 16s | master passed | | +1 :green_heart: | shadedjars | 4m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 13s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 13s | the patch passed | | +1 :green_heart: | compile | 0m 16s | the patch passed | | +1 :green_heart: | javac | 0m 16s | the patch passed | | +1 :green_heart: | shadedjars | 3m 58s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 12m 53s | hbase-mapreduce in the patch passed. | | | | 29m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4411/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4411 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7a39a340d51d 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4411/3/testReport/ | | Max. process+thread count | 2896 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4411/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4411: HBASE-27003 Optimize log format for PerformanceEvaluation
Apache-HBase commented on PR #4411: URL: https://github.com/apache/hbase/pull/4411#issuecomment-1122039734 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 13s | master passed | | +1 :green_heart: | compile | 0m 30s | master passed | | +1 :green_heart: | checkstyle | 0m 11s | master passed | | -1 :x: | spotless | 0m 34s | branch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 0m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 7s | the patch passed | | +1 :green_heart: | compile | 0m 30s | the patch passed | | +1 :green_heart: | javac | 0m 30s | the patch passed | | +1 :green_heart: | checkstyle | 0m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 19s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | -1 :x: | spotless | 0m 36s | patch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 0m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | | | | 25m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4411/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4411 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux c4a21e61ac93 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f6e9d3e1dd | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4411/3/artifact/yetus-general-check/output/branch-spotless.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4411/3/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 65 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4411/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26953) CallDroppedException message says "server 0.0.0.0 is overloaded"
[
https://issues.apache.org/jira/browse/HBASE-26953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534196#comment-17534196
]
Ayesha Mosaddeque commented on HBASE-26953:
---
Hi [~bbeaudreault],
Can you please let me know the steps to reproduce this exception? From what I
see:
{code:java}
call.setResponse(null, null, CALL_DROPPED_EXCEPTION, "Call dropped, server " +
(address != null ? address : "(channel closed)") + " is overloaded, please
retry."); {code}
This part is simply using the toString() method of InetSocketAddress class.
Which is:
{code:java}
@Override
public String toString() {
if (isUnresolved()) {
return hostname + ":" + port;
} else {
return addr.toString() + ":" + port;
}
} {code}
So, there should be a ":" in the message that you get. But this ticket does
not mention that, which is why I am a little confused, where this "0.0.0.0" is
generated from.
It will be great if you can post further logs and way to reproduce it.
Thanks.
> CallDroppedException message says "server 0.0.0.0 is overloaded"
>
>
> Key: HBASE-26953
> URL: https://issues.apache.org/jira/browse/HBASE-26953
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Ayesha Mosaddeque
>Priority: Trivial
> Labels: newbie
>
> This is not a super useful error when seen on the client side. Maybe we can
> have it pull in the ServerName instead, or at the very least resolve to the
> hostname.
>
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java#L213
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] tomscut opened a new pull request, #4417: HBASE-27020 Fix spotless warn for master branch
tomscut opened a new pull request, #4417: URL: https://github.com/apache/hbase/pull/4417 JIRA: HBASE-27020. Fix spotless warn for master branch. We can see the spotless warn at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/1/artifact/yetus-general-check/output/branch-spotless.txt. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-27020) Fix spotless warn for master branch
Tao Li created HBASE-27020: -- Summary: Fix spotless warn for master branch Key: HBASE-27020 URL: https://issues.apache.org/jira/browse/HBASE-27020 Project: HBase Issue Type: Bug Reporter: Tao Li Assignee: Tao Li Fix spotless warn for master branch. We can see the spotless warn at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4416/1/artifact/yetus-general-check/output/branch-spotless.txt. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26994) MasterFileSystem create directory without permission check
[
https://issues.apache.org/jira/browse/HBASE-26994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534188#comment-17534188
]
Hudson commented on HBASE-26994:
Results for branch branch-2
[build #537 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> MasterFileSystem create directory without permission check
> --
>
> Key: HBASE-26994
> URL: https://issues.apache.org/jira/browse/HBASE-26994
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 2.4.12
>Reporter: Zhang Dongsheng
>Assignee: Zhang Dongsheng
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.13
>
> Attachments: HBASE-26994.patch
>
>
> Method checkStagingDir and checkSubDir first check if directory is exist ,if
> not , create it with special permission. If exist then setPermission for this
> directory. BUT if not exist ,we still need set special permission for this
> directory
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26999) HStore should try write WAL compaction marker before replacing compacted files in StoreEngine
[
https://issues.apache.org/jira/browse/HBASE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534189#comment-17534189
]
Hudson commented on HBASE-26999:
Results for branch branch-2
[build #537 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/537//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> HStore should try write WAL compaction marker before replacing compacted
> files in StoreEngine
> -
>
> Key: HBASE-26999
> URL: https://issues.apache.org/jira/browse/HBASE-26999
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.5.0, 3.0.0-alpha-2, 2.6.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> On HBASE-26064, it seems we altered the order we update different places with
> the results of a compaction:
> {noformat}
> @@ -1510,14 +1149,13 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> List newFiles) throws IOException {
> // Do the steps necessary to complete the compaction.
> setStoragePolicyFromFileName(newFiles);
> - List sfs = commitStoreFiles(newFiles, true);
> + List sfs = storeEngine.commitStoreFiles(newFiles, true);
> if (this.getCoprocessorHost() != null) {
> for (HStoreFile sf : sfs) {
> getCoprocessorHost().postCompact(this, sf, cr.getTracker(), cr,
> user);
> }
> }
> - writeCompactionWalRecord(filesToCompact, sfs);
> - replaceStoreFiles(filesToCompact, sfs);
> + replaceStoreFiles(filesToCompact, sfs, true);
> ...
> @@ -1581,25 +1219,24 @@ public class HStore implements Store, HeapSize,
> StoreConfigInformation,
> this.region.getRegionInfo(), compactionDescriptor,
> this.region.getMVCC());
> }
>
> - void replaceStoreFiles(Collection compactedFiles,
> Collection result)
> - throws IOException {
> - this.lock.writeLock().lock();
> - try {
> -
> this.storeEngine.getStoreFileManager().addCompactionResults(compactedFiles,
> result);
> - synchronized (filesCompacting) {
> - filesCompacting.removeAll(compactedFiles);
> - }
> -
> - // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> - // sizes later so it's not super-critical if we miss these.
> - RegionServerServices rsServices = region.getRegionServerServices();
> - if (rsServices != null &&
> rsServices.getRegionServerSpaceQuotaManager() != null) {
> - updateSpaceQuotaAfterFileReplacement(
> -
> rsServices.getRegionServerSpaceQuotaManager().getRegionSizeStore(),
> getRegionInfo(),
> - compactedFiles, result);
> - }
> - } finally {
> - this.lock.writeLock().unlock();
> + @RestrictedApi(explanation = "Should only be called in TestHStore", link =
> "",
> + allowedOnPath = ".*/(HStore|TestHStore).java")
> + void replaceStoreFiles(Collection compactedFiles,
> Collection result,
> + boolean writeCompactionMarker) throws IOException {
> + storeEngine.replaceStoreFiles(compactedFiles, result);
> + if (writeCompactionMarker) {
> + writeCompactionWalRecord(compactedFiles, result);
> + }
> + synchronized (filesCompacting) {
> + filesCompacting.removeAll(compactedFiles);
> + }
> + // These may be null when the RS is shutting down. The space quota
> Chores will fix the Region
> + // sizes later so it's not super-critical if we miss these.
> + RegionServerServices rsServices
[jira] [Work started] (HBASE-26953) CallDroppedException message says "server 0.0.0.0 is overloaded"
[ https://issues.apache.org/jira/browse/HBASE-26953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-26953 started by Ayesha Mosaddeque. - > CallDroppedException message says "server 0.0.0.0 is overloaded" > > > Key: HBASE-26953 > URL: https://issues.apache.org/jira/browse/HBASE-26953 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Assignee: Ayesha Mosaddeque >Priority: Trivial > Labels: newbie > > This is not a super useful error when seen on the client side. Maybe we can > have it pull in the ServerName instead, or at the very least resolve to the > hostname. > > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java#L213 -- This message was sent by Atlassian Jira (v8.20.7#820007)
