[GitHub] [hbase] Apache-HBase commented on pull request #4439: Backport HBASE-26523 + HBASE-25465 + HBASE-26855 to branch-2.4
Apache-HBase commented on PR #4439: URL: https://github.com/apache/hbase/pull/4439#issuecomment-1128439289 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 55s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 56s | branch-2.4 passed | | +1 :green_heart: | compile | 1m 26s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 4m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 17s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 9s | the patch passed | | +1 :green_heart: | compile | 1m 29s | the patch passed | | +1 :green_heart: | javac | 1m 29s | root generated 0 new + 1 unchanged - 36 fixed = 1 total (was 37) | | +1 :green_heart: | shadedjars | 4m 13s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 13s | hbase-common generated 35 new + 16 unchanged - 0 fixed = 51 total (was 16) | | -0 :warning: | javadoc | 1m 42s | root generated 35 new + 201 unchanged - 0 fixed = 236 total (was 201) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 190m 50s | root in the patch passed. | | | | 218m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4439 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7894a0da7357 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 5b319752de | | Default Java | AdoptOpenJDK-11.0.10+9 | | javadoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-common.txt | | javadoc | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/testReport/ | | Max. process+thread count | 6119 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-http hbase-server hbase-it hbase-rest . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26856) BufferedDataBlockEncoder.OnheapDecodedCell value can get corrupted
[
https://issues.apache.org/jira/browse/HBASE-26856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar resolved HBASE-26856.
--
Fix Version/s: 2.5.0
2.6.0
3.0.0-alpha-3
2.4.13
Resolution: Fixed
Pushed to branch-2.4+.
Thanks [~arshad.mohammad] for the contribution & [~apurtell] for reviewing.
> BufferedDataBlockEncoder.OnheapDecodedCell value can get corrupted
> --
>
> Key: HBASE-26856
> URL: https://issues.apache.org/jira/browse/HBASE-26856
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13
>
>
> In our production cluster we observed the cell value is modified after
> successful scanner read. After analyzing we observed OnheapDecodedCell is not
> created properly.
> We create OnheapDecodedCell with complete valAndTagsBuffer underlying array.
> {code:java}
> return new OnheapDecodedCell(Bytes.copy(keyBuffer, 0, this.keyLength),
> currentKey.getRowLength(), currentKey.getFamilyOffset(),
> currentKey.getFamilyLength(),
> currentKey.getQualifierOffset(), currentKey.getQualifierLength(),
> currentKey.getTimestamp(), currentKey.getTypeByte(),
> valAndTagsBuffer.array(),
> valAndTagsBuffer.arrayOffset() + vOffset, this.valueLength,
> memstoreTS, tagsArray,
> tOffset, this.tagsLength);
> {code}
> Here we are passing valAndTagsBuffer.array() for value extraction.
> The underlying array will be modified if it is altered anywhere.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] pankaj72981 merged pull request #4394: HBASE-26856:BufferedDataBlockEncoder.OnheapDecodedCell value can get corrupted
pankaj72981 merged PR #4394: URL: https://github.com/apache/hbase/pull/4394 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27032) The draining region servers metric description is incorrect
[ https://issues.apache.org/jira/browse/HBASE-27032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537949#comment-17537949 ] Pankaj Kumar commented on HBASE-27032: -- [~tomscut] You can raise another JIRA for correcting the typo mistake in metric name. > The draining region servers metric description is incorrect > --- > > Key: HBASE-27032 > URL: https://issues.apache.org/jira/browse/HBASE-27032 > Project: HBase > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13 > > > The draining region servers metric description is incorrect. > Fixed the description and added unit test. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-27032) The draining region servers metric description is incorrect
[ https://issues.apache.org/jira/browse/HBASE-27032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Kumar resolved HBASE-27032. -- Fix Version/s: 2.5.0 2.6.0 3.0.0-alpha-3 2.4.13 Resolution: Fixed Pushed to branch-2.4+. Thanks [~tomscut] for the contribution and [~Xiaolin Ha] for reviewing. > The draining region servers metric description is incorrect > --- > > Key: HBASE-27032 > URL: https://issues.apache.org/jira/browse/HBASE-27032 > Project: HBase > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13 > > > The draining region servers metric description is incorrect. > Fixed the description and added unit test. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] pankaj72981 merged pull request #4428: HBASE-27032 The draining region servers metric description is incorrect
pankaj72981 merged PR #4428: URL: https://github.com/apache/hbase/pull/4428 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-27046) The filenum in AbstractFSWAL should be monotone increasing
Duo Zhang created HBASE-27046:
-
Summary: The filenum in AbstractFSWAL should be monotone increasing
Key: HBASE-27046
URL: https://issues.apache.org/jira/browse/HBASE-27046
Project: HBase
Issue Type: Improvement
Reporter: Duo Zhang
This is the current code
{code}
/**
* retrieve the next path to use for writing. Increments the internal filenum.
*/
private Path getNewPath() throws IOException {
this.filenum.set(EnvironmentEdgeManager.currentTime());
Path newPath = getCurrentFileName();
while (fs.exists(newPath)) {
this.filenum.incrementAndGet();
newPath = getCurrentFileName();
}
return newPath;
}
{code}
In some tests, we inject our own EnvironmentEdge, it may return the same ts
always or even go backwards, the logic here is not rnough to keep the filenum
monotone increasing, as we may have already archive the old file.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-27013) Introduce read all bytes when using pread for prefetch
[
https://issues.apache.org/jira/browse/HBASE-27013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537946#comment-17537946
]
Hudson commented on HBASE-27013:
Results for branch branch-2.4
[build #351 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Introduce read all bytes when using pread for prefetch
> --
>
> Key: HBASE-27013
> URL: https://issues.apache.org/jira/browse/HBASE-27013
> Project: HBase
> Issue Type: Improvement
> Components: HFile, Performance
>Affects Versions: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> h2. Problem statement
> When prefetching HFiles from blob storage like S3 and use it with the storage
> implementation like S3A, we found there is a logical issue in HBase pread
> that causes the reading of the remote HFile aborts the input stream multiple
> times. This aborted stream and reopen slow down the reads and trigger many
> aborted bytes and waste time in recreating the connection especially when SSL
> is enabled.
> h2. ROOT CAUSE
> The root cause of above issue was due to
> [BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
> is reading an input stream that does not guarrentee to return the data block
> and the next block header as an option data to be cached.
> In the case of the input stream read short and when the input stream read
> passed the length of the necessary data block with few more bytes within the
> size of next block header, the
> [BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
> returns to the caller without a cached the next block header. As a result,
> before HBase tries to read the next block,
> [HFileBlock#readBlockDataInternal|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1648-L1664]
> in hbase tries to re-read the next block header from the input stream. Here,
> the reusable input stream has move the current position pointer ahead from
> the offset of the last read data block, when using with the [S3A
> implementation|https://github.com/apache/hadoop/blob/29401c820377d02a992eecde51083cf87f8e57af/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L339-L361],
> the input stream is then closed, aborted all the remaining bytes and reopen
> a new input stream at the offset of the last read data block .
> h2. How do we fix it?
> S3A is doing the right job that HBase is telling to move the offset from
> position A back to A - N, so there is not much thing we can do on how S3A
> handle the inputstream. meanwhile in the case of HDFS, this operation is fast.
> Such that, we should fix in HBase level, and try always to read datablock +
> next block header when we're using blob storage to avoid expensive draining
> the bytes in a stream and reopen the socket with the remote storage.
> h2. Draw back and discussion
> * A known drawback is, when we're at the last block, we will read extra
> length that should not be a header, and we still read that into the byte
> buffer array.
[jira] [Commented] (HBASE-27033) Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch" to branch-2.4
[
https://issues.apache.org/jira/browse/HBASE-27033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537945#comment-17537945
]
Hudson commented on HBASE-27033:
Results for branch branch-2.4
[build #351 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/351//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch"
> to branch-2.4
> ---
>
> Key: HBASE-27033
> URL: https://issues.apache.org/jira/browse/HBASE-27033
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.4.13
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.4.13
>
>
> Backport HBASE-27013 to branch-2.4, it's required because it's not a clean
> backport.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4440: HBASE-27045 Disable TestClusterScopeQuotaThrottle
Apache-HBase commented on PR #4440: URL: https://github.com/apache/hbase/pull/4440#issuecomment-1128365751 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 2m 13s | master passed | | +1 :green_heart: | checkstyle | 0m 29s | master passed | | +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 13s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 1s | the patch passed | | +1 :green_heart: | compile | 2m 11s | the patch passed | | +1 :green_heart: | javac | 2m 11s | the patch passed | | +1 :green_heart: | checkstyle | 0m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 48s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotless | 0m 38s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | | | | 32m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4440/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4440 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 28da4a2abef8 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ad64a9baae | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4440/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4439: [DO_NOT_MERGE] Backport HBASE-26523 + HBASE-25465 + HBASE-26855 to branch-2.4
Apache-HBase commented on PR #4439: URL: https://github.com/apache/hbase/pull/4439#issuecomment-1128357734 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.4 Compile Tests _ | | +0 :ok: | mvndep | 0m 59s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 7s | branch-2.4 passed | | +1 :green_heart: | compile | 5m 45s | branch-2.4 passed | | +1 :green_heart: | checkstyle | 1m 1s | branch-2.4 passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 9m 8s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 1m 50s | root in the patch failed. | | +1 :green_heart: | compile | 5m 34s | the patch passed | | -0 :warning: | javac | 5m 34s | root generated 1 new + 1632 unchanged - 1 fixed = 1633 total (was 1633) | | -0 :warning: | checkstyle | 0m 56s | root: The patch generated 5 new + 107 unchanged - 2 fixed = 112 total (was 109) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 10m 36s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | -1 :x: | spotless | 0m 11s | patch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 10m 0s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 49s | The patch does not generate ASF License warnings. | | | | 57m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4439 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile xml | | uname | Linux fddf170dca65 5.4.0-1068-aws #72~18.04.1-Ubuntu SMP Thu Mar 3 08:49:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / 5b319752de | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | mvninstall | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/artifact/yetus-general-check/output/diff-compile-javac-root.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 139 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-http hbase-server hbase-it hbase-rest . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4439/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-27045) Disable TestClusterScopeQuotaThrottle
Duo Zhang created HBASE-27045: - Summary: Disable TestClusterScopeQuotaThrottle Key: HBASE-27045 URL: https://issues.apache.org/jira/browse/HBASE-27045 Project: HBase Issue Type: Sub-task Components: test Reporter: Duo Zhang Assignee: Duo Zhang We will inject a special EnvironmentEdge to control the return value of current time, which will mess up the wal filename and then fail the wal archiving. Let's disable it first before we find a suitable solution. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng reassigned HBASE-26981: Assignee: zhengsicheng > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] tomscut commented on pull request #4428: HBASE-27032 The draining region servers metric description is incorrect
tomscut commented on PR #4428: URL: https://github.com/apache/hbase/pull/4428#issuecomment-1128320927 Thanks @sunhelly for the review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-27043) Let lock wait timeout to improve performance of SnapshotHFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-27043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaolin Ha updated HBASE-27043: --- Fix Version/s: 2.5.0 3.0.0-alpha-3 > Let lock wait timeout to improve performance of SnapshotHFileCleaner > > > Key: HBASE-27043 > URL: https://issues.apache.org/jira/browse/HBASE-27043 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 2.5.0, 3.0.0-alpha-3 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > Attachments: clearner-before-and-after.png, namenode-callqueue.png > > > Currently, hfile cleaner uses the dir scanning threads to get deletable > files, by checking all the files under the scanned directories through the > cleaner chain. And before scanning a directory, cleaner sorted the > subdirectories by consumed spaces, but we all know getContentSummary is a > time consuming operation for HDFS. > SnapshotHFileCleaner filters all the unreferenced files of snapshots to > delete, and it tries to get write lock of SnapshotManager#takingSnapshotLock > before determining the deletable files. > [https://github.com/apache/hbase/blob/ad64a9baae2ef8ee56aa3ed6b96cb3d51f5daf0a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java#L195] > But when there is any snapshot taking and the cleaner gets the lock failed, > all the scanned files will be determined to be none-deletable, and all the > dir scanning threads will scan and getContentSummary of other dirs(no files > are deletable too) one by one until the snapshot taking lock is released. > This is a low efficiency behavior, we should let the clear wait the lock to > determine if the files it currently hold are deletable, instead of the > meaningless scanning and getContentSummary other/new directories while the > lock is acquired by taking snapshot operations. > I deployed this optimization in our production environment, and the effect is > very obvious. > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-27043) Let lock wait timeout to improve performance of SnapshotHFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-27043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaolin Ha updated HBASE-27043: --- Affects Version/s: 2.4.12 3.0.0-alpha-2 (was: 2.5.0) (was: 3.0.0-alpha-3) > Let lock wait timeout to improve performance of SnapshotHFileCleaner > > > Key: HBASE-27043 > URL: https://issues.apache.org/jira/browse/HBASE-27043 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0-alpha-2, 2.4.12 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > Attachments: clearner-before-and-after.png, namenode-callqueue.png > > > Currently, hfile cleaner uses the dir scanning threads to get deletable > files, by checking all the files under the scanned directories through the > cleaner chain. And before scanning a directory, cleaner sorted the > subdirectories by consumed spaces, but we all know getContentSummary is a > time consuming operation for HDFS. > SnapshotHFileCleaner filters all the unreferenced files of snapshots to > delete, and it tries to get write lock of SnapshotManager#takingSnapshotLock > before determining the deletable files. > [https://github.com/apache/hbase/blob/ad64a9baae2ef8ee56aa3ed6b96cb3d51f5daf0a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java#L195] > But when there is any snapshot taking and the cleaner gets the lock failed, > all the scanned files will be determined to be none-deletable, and all the > dir scanning threads will scan and getContentSummary of other dirs(no files > are deletable too) one by one until the snapshot taking lock is released. > This is a low efficiency behavior, we should let the clear wait the lock to > determine if the files it currently hold are deletable, instead of the > meaningless scanning and getContentSummary other/new directories while the > lock is acquired by taking snapshot operations. > I deployed this optimization in our production environment, and the effect is > very obvious. > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] tomscut commented on pull request #4428: HBASE-27032 The draining region servers metric description is incorrect
tomscut commented on PR #4428: URL: https://github.com/apache/hbase/pull/4428#issuecomment-1128256746 Hi @pankaj72981 , please take a look. Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27044) Serialized procedures which point to users from other Kerberos domains can prevent master startup
[
https://issues.apache.org/jira/browse/HBASE-27044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537812#comment-17537812
]
Josh Elser commented on HBASE-27044:
We could do a pretty naive "change" here where we just return back a {{User}}
which is "unknown" when we fail to parse the serialized protobuf which would be
enough to fix this problem on the surface.
However, I think this change is missing the root of the problem (the
expectation that HBase should just be able to "reattach" itself to an
hbase.rootdir).
I can't think of any way in which the above exception would be thrown other
than the cloud storage reattachment case I described. I'm happy to put up a
patch to gracefully handle a the failure to create the UGI if folks think there
is merit in that.
> Serialized procedures which point to users from other Kerberos domains can
> prevent master startup
> -
>
> Key: HBASE-27044
> URL: https://issues.apache.org/jira/browse/HBASE-27044
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Reporter: Josh Elser
>Priority: Major
>
> We ran into an interesting bug when test teams were running HBase against
> cloud storage without ensuring that the previous location was cleaned. This
> resulted in an hbase.rootdir that had:
> * A valid HBase MasterData Region
> * A valid hbase:meta
> * A valid collection of HBase tables
> * An empty ZooKeeper
> Through the changes that we've worked on prior, those described in
> HBASE-24286 were effective in getting every _except_ the Procedures back
> online without issue. Parsing the existing procedures produced an interesting
> error:
> {noformat}
> java.lang.IllegalArgumentException: Illegal principal name
> hbase/wrong-hostname.domain@WRONG_REALM:
> org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule:
> No rules applied to hbase/wrong-hostname.domain@WRONG_REALM
> at org.apache.hadoop.security.User.(User.java:51)
> at org.apache.hadoop.security.User.(User.java:43)
> at
> org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1418)
> at
> org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1402)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.toUserInfo(MasterProcedureUtil.java:60)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.deserializeStateData(ModifyTableProcedure.java:262)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureUtil.convertToProcedure(ProcedureUtil.java:294)
> at
> org.apache.hadoop.hbase.procedure2.store.ProtoAndProcedure.getProcedure(ProtoAndProcedure.java:43)
> at
> org.apache.hadoop.hbase.procedure2.store.InMemoryProcedureIterator.next(InMemoryProcedureIterator.java:90)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.loadProcedures(ProcedureExecutor.java:411)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.load(ProcedureExecutor.java:339)
> at
> org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.load(RegionProcedureStore.java:285)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:330)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:600)
> at
> org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1581)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:835)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2205)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:514)
> at java.lang.Thread.run(Thread.java:750) {noformat}
> What's actually happening is that we are storing the {{User}} into the
> procedure and then relying on UserGroupInformation to parse the {{User}}
> protobuf into a UGI to get the "short" username.
> When the serialized procedure (whether in the MasterData region over via PV2
> WAL files, I think) gets loaded, we end up needing Hadoop auth_to_local
> configuration to be able to parse that kerberos principal back to a name.
> However, Hadoop's KerberosName will only unwrap Kerberos principals which
> match the local Kerberos realm (defined by the krb5.conf's default_realm,
> [ref|https://github.com/frohoff/jdk8u-jdk/blob/master/src/share/classes/sun/security/krb5/Config.java#L978-L983])
> The interesting part is that we don't seem to ever use the user _other_ than
> to display the {{owner}} attribute for procedures on the HBase UI. There is a
>
[jira] [Created] (HBASE-27044) Serialized procedures which point to users from other Kerberos domains can prevent master startup
Josh Elser created HBASE-27044:
--
Summary: Serialized procedures which point to users from other
Kerberos domains can prevent master startup
Key: HBASE-27044
URL: https://issues.apache.org/jira/browse/HBASE-27044
Project: HBase
Issue Type: Bug
Components: proc-v2
Reporter: Josh Elser
We ran into an interesting bug when test teams were running HBase against cloud
storage without ensuring that the previous location was cleaned. This resulted
in an hbase.rootdir that had:
* A valid HBase MasterData Region
* A valid hbase:meta
* A valid collection of HBase tables
* An empty ZooKeeper
Through the changes that we've worked on prior, those described in HBASE-24286
were effective in getting every _except_ the Procedures back online without
issue. Parsing the existing procedures produced an interesting error:
{noformat}
java.lang.IllegalArgumentException: Illegal principal name
hbase/wrong-hostname.domain@WRONG_REALM:
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No
rules applied to hbase/wrong-hostname.domain@WRONG_REALM
at org.apache.hadoop.security.User.(User.java:51)
at org.apache.hadoop.security.User.(User.java:43)
at
org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1418)
at
org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1402)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.toUserInfo(MasterProcedureUtil.java:60)
at
org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.deserializeStateData(ModifyTableProcedure.java:262)
at
org.apache.hadoop.hbase.procedure2.ProcedureUtil.convertToProcedure(ProcedureUtil.java:294)
at
org.apache.hadoop.hbase.procedure2.store.ProtoAndProcedure.getProcedure(ProtoAndProcedure.java:43)
at
org.apache.hadoop.hbase.procedure2.store.InMemoryProcedureIterator.next(InMemoryProcedureIterator.java:90)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.loadProcedures(ProcedureExecutor.java:411)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:78)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.load(ProcedureExecutor.java:339)
at
org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.load(RegionProcedureStore.java:285)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:330)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:600)
at
org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1581)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:835)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2205)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:514)
at java.lang.Thread.run(Thread.java:750) {noformat}
What's actually happening is that we are storing the {{User}} into the
procedure and then relying on UserGroupInformation to parse the {{User}}
protobuf into a UGI to get the "short" username.
When the serialized procedure (whether in the MasterData region over via PV2
WAL files, I think) gets loaded, we end up needing Hadoop auth_to_local
configuration to be able to parse that kerberos principal back to a name.
However, Hadoop's KerberosName will only unwrap Kerberos principals which match
the local Kerberos realm (defined by the krb5.conf's default_realm,
[ref|https://github.com/frohoff/jdk8u-jdk/blob/master/src/share/classes/sun/security/krb5/Config.java#L978-L983])
The interesting part is that we don't seem to ever use the user _other_ than to
display the {{owner}} attribute for procedures on the HBase UI. There is a
method in hbase-procedure which can filter procedures based on Owner, but I
didn't see any usages of that method.
Given the pushback against HBASE-24286, I assume that, for the same reasons, we
would see pushback against fixing this issue. However, I wanted to call it out
for posterity. The expectation of users is that HBase _should_ implicitly
handle this case.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Comment Edited] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537787#comment-17537787
]
Szabolcs Bukros edited comment on HBASE-26969 at 5/16/22 8:45 PM:
--
{quote}I guess if we want to compact the mob files, we always need to compact
the normal files which references the mob files so we can update the references
in the metadata?
{quote}
Yes, that is how it works.
{quote}The mob files should have a different name prefix or under a different
directory?
{quote}
They have a different directory structure.
"/mobdir/data/default/table_name/a0209c070c85d1e4d500af8ba33c3c02/cf"
They are stored fully separated. Please note these regions only contain mob
files and are fully independent from the referencing regions. A single mob
region could theoretically contain every MOB file in hbase regardless of where
it is referenced from. Also their naming convention is different. For us the
only important thing is that it ends with "_", so something like this:
"0cc175b9c0f1b6a831c399e2697726612022050314ecf20b51674cd6bd647bfb2d88b1ff_b593e96e821ba6211d8a4b101a88"
{quote}So at least for loading, there will be no problem
{quote}
That's true. Reads are very straightforward.
{quote}I think the only problem here is how do we clean up the half written mob
files, I think the logic is mainly the same with what we have now, get all the
mob refs from all the normal storefiles, to construct the base list, and then
get all the mob files which are currently being written, all MOB files besides
them are the ones should be deleted.
{quote}
That is part of the problme, yes. To have access to the half written mob file
list the cleaner have to run on the RS. But each RS only has access to it's own
half written mob file list so each can only clean a subset of the existing mob
files. To be precise if a mob file name ends with a region's name that is
hosted on the current RS then the cleaner can decide if it can be archived or
not. Unfortunately with merges and splits regions get archived so after a point
there will be mob files containing names of regions not hosted on any RS and
none of the cleaners running on RSes could clean these up. So we need one more
cleaner specifically for these (I put it on master to replace the original
cleaner), that have to read every available hfile to make sure we have every
active mob reference and are able to decide if a mob file created by a since
archived region can be archived or not.
was (Author: bszabolcs):
{quote}I guess if we want to compact the mob files, we always need to compact
the normal files which references the mob files so we can update the references
in the metadata?
{quote}
Yes, that is how it works.
{quote}The mob files should have a different name prefix or under a different
directory?
{quote}
They have a different directory structure.
"/mobdir/data/default/table_name/a0209c070c85d1e4d500af8ba33c3c02/cf"
They are stored fully separated. Please note these regions only contain mob
files and are fully independent from the referencing regions. A single mob
region could theoretically contain every MOB file in hbase regardless of where
it is referenced from. Also their naming convention is different. For us the
only important thing is that it ends with "_", so something like this:
"0cc175b9c0f1b6a831c399e2697726612022050314ecf20b51674cd6bd647bfb2d88b1ff_b593e96e821ba6211d8a4b101a88"
{quote}So at least for loading, there will be no problem
{quote}
That's true. Reads are very straightforward.
{quote}I think the only problem here is how do we clean up the half written mob
files, I think the logic is mainly the same with what we have now, get all the
mob refs from all the normal storefiles, to construct the base list, and then
get all the mob files which are currently being written, all MOB files besides
them are the ones should be deleted.
{quote}
That is part of the problme, yes. To have access to the half written mob file
list the cleaner have to run on the RS. But each RS only has access to it's own
}}half written mob file list so each can only clean a subset of the existing
mob files. To be precise if a mob file name ends with a region's name that is
hosted on the current RS then the cleaner can decide if it can be archived or
not. Unfortunately with merges and splits regions get archived so after a point
there will be mob files containing names of regions not hosted on any RS and
none of the cleaners running on RSes could clean these up. So we need one more
cleaner specifically for these (I put it on master to replace the original
cleaner), that have to read every available hfile to make sure we have every
active mob reference and are able to decide if a mob file created by a since
archived region can be archived or not.
> Eliminate MOB renames when SFT is enabled
>
[jira] [Comment Edited] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537787#comment-17537787
]
Szabolcs Bukros edited comment on HBASE-26969 at 5/16/22 8:45 PM:
--
{quote}I guess if we want to compact the mob files, we always need to compact
the normal files which references the mob files so we can update the references
in the metadata?
{quote}
Yes, that is how it works.
{quote}The mob files should have a different name prefix or under a different
directory?
{quote}
They have a different directory structure.
"/mobdir/data/default/table_name/a0209c070c85d1e4d500af8ba33c3c02/cf"
They are stored fully separated. Please note these regions only contain mob
files and are fully independent from the referencing regions. A single mob
region could theoretically contain every MOB file in hbase regardless of where
it is referenced from. Also their naming convention is different. For us the
only important thing is that it ends with "_", so something like this:
"0cc175b9c0f1b6a831c399e2697726612022050314ecf20b51674cd6bd647bfb2d88b1ff_b593e96e821ba6211d8a4b101a88"
{quote}So at least for loading, there will be no problem
{quote}
That's true. Reads are very straightforward.
{quote}I think the only problem here is how do we clean up the half written mob
files, I think the logic is mainly the same with what we have now, get all the
mob refs from all the normal storefiles, to construct the base list, and then
get all the mob files which are currently being written, all MOB files besides
them are the ones should be deleted.
{quote}
That is part of the problme, yes. To have access to the half written mob file
list the cleaner have to run on the RS. But each RS only has access to it's own
}}half written mob file list so each can only clean a subset of the existing
mob files. To be precise if a mob file name ends with a region's name that is
hosted on the current RS then the cleaner can decide if it can be archived or
not. Unfortunately with merges and splits regions get archived so after a point
there will be mob files containing names of regions not hosted on any RS and
none of the cleaners running on RSes could clean these up. So we need one more
cleaner specifically for these (I put it on master to replace the original
cleaner), that have to read every available hfile to make sure we have every
active mob reference and are able to decide if a mob file created by a since
archived region can be archived or not.
was (Author: bszabolcs):
{{{quote}}}
I guess if we want to compact the mob files, we always need to compact the
normal files which references the mob files so we can update the references in
the metadata?
{quote}
Yes, that is how it works.
{{{quote}}}
The mob files should have a different name prefix or under a different
directory?
{{{}{quote}{}}}{{{}{}}}
They have a different directory structure.
"/mobdir/data/default/table_name/a0209c070c85d1e4d500af8ba33c3c02/cf"
They are stored fully separated. Please note these regions only contain mob
files and are fully independent from the referencing regions. A single mob
region could theoretically contain every MOB file in hbase regardless of where
it is referenced from. Also their naming convention is different. For us the
only important thing is that it ends with "_", so something like this:
"0cc175b9c0f1b6a831c399e2697726612022050314ecf20b51674cd6bd647bfb2d88b1ff_b593e96e821ba6211d8a4b101a88"
{quote}
So at least for loading, there will be no problem
{quote}
That's true. Reads are very straightforward.
{{{quote}}}
I think the only problem here is how do we clean up the half written mob files,
I think the logic is mainly the same with what we have now, get all the mob
refs from all the normal storefiles, to construct the base list, and then get
all the mob files which are currently being written, all MOB files besides them
are the ones should be deleted.
{quote}{{{}{}}}
{{That is part of the problme, yes. To have access to the half written mob file
list the cleaner have to run on the RS. But each RS only has access to it's own
}}half written mob file list so each can only clean a subset of the existing
mob files. To be precise if a mob file name ends with a region's name that is
hosted on the current RS then the cleaner can decide if it can be archived or
not. Unfortunately with merges and splits regions get archived so after a point
there will be mob files containing names of regions not hosted on any RS and
none of the cleaners running on RSes could clean these up. So we need one more
cleaner specifically for these (I put it on master to replace the original
cleaner), that have to read every available hfile to make sure we have every
active mob reference and are able to decide if a mob file created by a since
archived region can be archived or not.
> Eliminate MOB renames
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537787#comment-17537787
]
Szabolcs Bukros commented on HBASE-26969:
-
{{{quote}}}
I guess if we want to compact the mob files, we always need to compact the
normal files which references the mob files so we can update the references in
the metadata?
{quote}
Yes, that is how it works.
{{{quote}}}
The mob files should have a different name prefix or under a different
directory?
{{{}{quote}{}}}{{{}{}}}
They have a different directory structure.
"/mobdir/data/default/table_name/a0209c070c85d1e4d500af8ba33c3c02/cf"
They are stored fully separated. Please note these regions only contain mob
files and are fully independent from the referencing regions. A single mob
region could theoretically contain every MOB file in hbase regardless of where
it is referenced from. Also their naming convention is different. For us the
only important thing is that it ends with "_", so something like this:
"0cc175b9c0f1b6a831c399e2697726612022050314ecf20b51674cd6bd647bfb2d88b1ff_b593e96e821ba6211d8a4b101a88"
{quote}
So at least for loading, there will be no problem
{quote}
That's true. Reads are very straightforward.
{{{quote}}}
I think the only problem here is how do we clean up the half written mob files,
I think the logic is mainly the same with what we have now, get all the mob
refs from all the normal storefiles, to construct the base list, and then get
all the mob files which are currently being written, all MOB files besides them
are the ones should be deleted.
{quote}{{{}{}}}
{{That is part of the problme, yes. To have access to the half written mob file
list the cleaner have to run on the RS. But each RS only has access to it's own
}}half written mob file list so each can only clean a subset of the existing
mob files. To be precise if a mob file name ends with a region's name that is
hosted on the current RS then the cleaner can decide if it can be archived or
not. Unfortunately with merges and splits regions get archived so after a point
there will be mob files containing names of regions not hosted on any RS and
none of the cleaners running on RSes could clean these up. So we need one more
cleaner specifically for these (I put it on master to replace the original
cleaner), that have to read every available hfile to make sure we have every
active mob reference and are able to decide if a mob file created by a since
archived region can be archived or not.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4438: HBASE-27006 Move nightly integration testing to new larger test node class.
Apache-HBase commented on PR #4438: URL: https://github.com/apache/hbase/pull/4438#issuecomment-1128078583 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 39s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 3s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 5m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4438/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4438 | | Optional Tests | | | uname | Linux ab8b66dbe059 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ad64a9baae | | Max. process+thread count | 33 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4438/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4438: HBASE-27006 Move nightly integration testing to new larger test node class.
Apache-HBase commented on PR #4438: URL: https://github.com/apache/hbase/pull/4438#issuecomment-1128077150 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 4s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | spotless | 0m 37s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 3m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4438/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4438 | | Optional Tests | dupname asflicense spotless shellcheck shelldocs | | uname | Linux a8fe569230bd 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ad64a9baae | | Max. process+thread count | 33 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4438/1/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4438: HBASE-27006 Move nightly integration testing to new larger test node class.
Apache-HBase commented on PR #4438: URL: https://github.com/apache/hbase/pull/4438#issuecomment-1128076136 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 4s | Maven dependency ordering for patch | ||| _ Other Tests _ | | | | 2m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4438/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4438 | | Optional Tests | | | uname | Linux 2eb46e4e326a 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ad64a9baae | | Max. process+thread count | 42 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4438/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] BukrosSzabolcs commented on a diff in pull request #4418: HBASE-26969:Eliminate MOB renames when SFT is enabled
BukrosSzabolcs commented on code in PR #4418:
URL: https://github.com/apache/hbase/pull/4418#discussion_r874096575
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:
##
@@ -1867,6 +1870,10 @@ executorService.new
ExecutorConfig().setExecutorType(ExecutorType.RS_SNAPSHOT_OP
choreService.scheduleChore(brokenStoreFileCleaner);
}
+if (this.rsMobFileCleanerChore != null) {
+ choreService.scheduleChore(rsMobFileCleanerChore);
Review Comment:
Let me clarify the differences between the 2 cleaners.
RSMobFileCleanerChore:
- runs on RS to have access to currently written files and active storefile
list
- can only archive mob files created by regions hosted on the current RS
- only reads hfiles belonging to regions hosted by the current RS when
looking for references
This allow it to do the majority of the cleanup necessary as efficiently as
possible
MobFileCleanerChore:
- runs on Master
- can only archive mob files created by archived regions (regions no longer
existing in the /data folder). Thanks to this these mob files can no longer be
"currently written" so we do not need the data only available on the RS
- reads every single hfile in /data with a mob enabled CF. This is
necessary, because this is the only way if any of these mobs has active
references
So yes, there is an overlap, the same hfiles could be read by both cleansers
but the a mob file could be archived with either one of them. The cleaner on
the master is wasteful and not especially elegant, but thanks to the lack of
centralized mob tracking we do not have a better way of collecting the
references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] BukrosSzabolcs commented on a diff in pull request #4418: HBASE-26969:Eliminate MOB renames when SFT is enabled
BukrosSzabolcs commented on code in PR #4418:
URL: https://github.com/apache/hbase/pull/4418#discussion_r874089324
##
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java:
##
@@ -256,9 +274,11 @@ public void cleanupObsoleteMobFiles(Configuration conf,
TableName table) throws
while (rit.hasNext()) {
LocatedFileStatus lfs = rit.next();
Path p = lfs.getPath();
- if (!allActiveMobFileName.contains(p.getName())) {
-// MOB is not in a list of active references, but it can be too
-// fresh, skip it in this case
+ String[] mobParts = p.getName().split("_");
+ String regionName = mobParts[mobParts.length - 1];
+
+ if (!regionNames.contains(regionName)) {
Review Comment:
I think these are valid concerns and should be investigated, but maybe not
as part of this issue. For now I'm following the original logic when looking
for mob references and I hope that would be acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] busbey opened a new pull request, #4438: HBASE-27006 Move nightly integration testing to new larger test node class.
busbey opened a new pull request, #4438: URL: https://github.com/apache/hbase/pull/4438 Since the integration tests will eventually run on the large nodes w/k8s, have them start running their now to validate that we can schedule on the nodes. Did a scan through our other jenkinsfiles and I believe everything except for `dev-support/adhoc_run_tests/Jenkinsfile` is strongly tied to the `hbase` label and thus will stay off of `hbase-large`. I also don't think we have that job configured to run at all, so maybe a follow on should delete it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-27033) Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch" to branch-2.4
[ https://issues.apache.org/jira/browse/HBASE-27033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27033. -- Hadoop Flags: Reviewed Release Note: see detail in HBASE-27013 Resolution: Fixed > Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch" > to branch-2.4 > --- > > Key: HBASE-27033 > URL: https://issues.apache.org/jira/browse/HBASE-27033 > Project: HBase > Issue Type: Task >Affects Versions: 2.4.13 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.4.13 > > > Backport HBASE-27013 to branch-2.4, it's required because it's not a clean > backport. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-27033) Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch" to branch-2.4
[ https://issues.apache.org/jira/browse/HBASE-27033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537708#comment-17537708 ] Tak-Lon (Stephen) Wu commented on HBASE-27033: -- https://github.com/apache/hbase/commit/5b319752dea879071ee3a23ed8faba4891977f62 merged > Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch" > to branch-2.4 > --- > > Key: HBASE-27033 > URL: https://issues.apache.org/jira/browse/HBASE-27033 > Project: HBase > Issue Type: Task >Affects Versions: 2.4.13 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.4.13 > > > Backport HBASE-27013 to branch-2.4, it's required because it's not a clean > backport. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] taklwu commented on a diff in pull request #4429: HBASE-27033 Backport "HBASE-27013 Introduce read all bytes when using…
taklwu commented on code in PR #4429:
URL: https://github.com/apache/hbase/pull/4429#discussion_r874030288
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:
##
@@ -1449,7 +1452,9 @@ protected boolean readAtOffset(FSDataInputStream istream,
ByteBuff dest, int siz
} else {
// Positional read. Better for random reads; or when the streamLock is
already locked.
int extraSize = peekIntoNextBlock ? hdrSize : 0;
-if (!BlockIOUtils.preadWithExtra(dest, istream, fileOffset, size,
extraSize)) {
+if (
+ !BlockIOUtils.preadWithExtra(dest, istream, fileOffset, size,
extraSize, isPreadAllBytes)
+) {
Review Comment:
that was modified by `spotless:apply`, and yet the checkstyle was happy
about it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] taklwu merged pull request #4429: HBASE-27033 Backport "HBASE-27013 Introduce read all bytes when using…
taklwu merged PR #4429: URL: https://github.com/apache/hbase/pull/4429 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4437: HBASE-27043 Let lock wait timeout to improve performance of SnapshotHFileCleaner
Apache-HBase commented on PR #4437: URL: https://github.com/apache/hbase/pull/4437#issuecomment-1127970804 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 28s | master passed | | +1 :green_heart: | compile | 0m 38s | master passed | | +1 :green_heart: | shadedjars | 3m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 11s | the patch passed | | +1 :green_heart: | compile | 0m 39s | the patch passed | | +1 :green_heart: | javac | 0m 39s | the patch passed | | +1 :green_heart: | shadedjars | 3m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 206m 51s | hbase-server in the patch passed. | | | | 229m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4437/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4437 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a4e2026ee622 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ad64a9baae | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4437/1/testReport/ | | Max. process+thread count | 2536 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4437/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4437: HBASE-27043 Let lock wait timeout to improve performance of SnapshotHFileCleaner
Apache-HBase commented on PR #4437: URL: https://github.com/apache/hbase/pull/4437#issuecomment-1127967128 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 5m 58s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 53s | master passed | | +1 :green_heart: | compile | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 3m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 35s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 3m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 201m 41s | hbase-server in the patch passed. | | | | 224m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4437/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4437 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3578bc607b31 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ad64a9baae | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4437/1/testReport/ | | Max. process+thread count | 2813 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4437/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] steveloughran commented on a diff in pull request #4414: HBASE-27013 Introduce read all bytes when using pread for prefetch
steveloughran commented on code in PR #4414:
URL: https://github.com/apache/hbase/pull/4414#discussion_r873936144
##
hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java:
##
@@ -228,21 +228,43 @@ public static boolean readWithExtra(ByteBuff buf,
FSDataInputStream dis, int nec
*/
public static boolean preadWithExtra(ByteBuff buff, FSDataInputStream dis,
long position,
int necessaryLen, int extraLen) throws IOException {
+return preadWithExtra(buff, dis, position, necessaryLen, extraLen, false);
+ }
+
+ /**
+ * Read from an input stream at least necessaryLen and if
possible,
+ * extraLen also if available. Analogous to
+ * {@link IOUtils#readFully(InputStream, byte[], int, int)}, but uses
positional read and
+ * specifies a number of "extra" bytes that would be desirable but not
absolutely necessary to
+ * read. If the input stream supports ByteBufferPositionedReadable, it reads
to the byte buffer
+ * directly, and does not allocate a temporary byte array.
+ * @param buff ByteBuff to read into.
+ * @param dis the input stream to read from
+ * @param position the position within the stream from which to start
reading
+ * @param necessaryLen the number of bytes that are absolutely necessary to
read
+ * @param extraLen the number of extra bytes that would be nice to read
+ * @param readAllBytes whether we must read the necessaryLen and extraLen
+ * @return true if and only if extraLen is > 0 and reading those extra bytes
was successful
+ * @throws IOException if failed to read the necessary bytes
+ */
+ public static boolean preadWithExtra(ByteBuff buff, FSDataInputStream dis,
long position,
+int necessaryLen, int extraLen, boolean readAllBytes) throws IOException {
boolean preadbytebuffer = dis.hasCapability("in:preadbytebuffer");
if (preadbytebuffer) {
- return preadWithExtraDirectly(buff, dis, position, necessaryLen,
extraLen);
+ return preadWithExtraDirectly(buff, dis, position, necessaryLen,
extraLen, readAllBytes);
} else {
- return preadWithExtraOnHeap(buff, dis, position, necessaryLen, extraLen);
+ return preadWithExtraOnHeap(buff, dis, position, necessaryLen, extraLen,
readAllBytes);
}
}
private static boolean preadWithExtraOnHeap(ByteBuff buff, FSDataInputStream
dis, long position,
-int necessaryLen, int extraLen) throws IOException {
+int necessaryLen, int extraLen, boolean readAllBytes) throws IOException {
int remain = necessaryLen + extraLen;
byte[] buf = new byte[remain];
int bytesRead = 0;
-while (bytesRead < necessaryLen) {
+int lengthMustRead = readAllBytes ? remain : necessaryLen;
+while (bytesRead < lengthMustRead) {
int ret = dis.read(position + bytesRead, buf, bytesRead, remain);
Review Comment:
any reason for not using dis.readFully() into the byte array?
##
hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java:
##
@@ -257,11 +279,12 @@ private static boolean preadWithExtraOnHeap(ByteBuff
buff, FSDataInputStream dis
}
private static boolean preadWithExtraDirectly(ByteBuff buff,
FSDataInputStream dis, long position,
Review Comment:
in line 305 on this method, the NPE raised should be included as the cause
of the IOE. that way if the input stream code raises NPEs, debugging it becomes
possible
##
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockIOUtils.java:
##
@@ -92,6 +109,103 @@ public void testReadFully() throws IOException {
assertArrayEquals(Bytes.toBytes(s), heapBuf);
}
+ @Test
+ public void testPreadWithReadFullBytes() throws IOException {
+testPreadReadFullBytesInternal(true, EnvironmentEdgeManager.currentTime());
+ }
+
+ @Test
+ public void testPreadWithoutReadFullBytes() throws IOException {
+testPreadReadFullBytesInternal(false,
EnvironmentEdgeManager.currentTime());
+ }
+
+ private void testPreadReadFullBytesInternal(boolean readAllBytes, long
randomSeed)
+throws IOException {
+Configuration conf = TEST_UTIL.getConfiguration();
+conf.setBoolean(HConstants.HFILE_PREAD_ALL_BYTES_ENABLED_KEY,
readAllBytes);
+FileSystem fs = TEST_UTIL.getTestFileSystem();
+Path path = new Path(TEST_UTIL.getDataTestDirOnTestFS(),
testName.getMethodName());
+// give a fixed seed such we can see failure easily.
+Random rand = new Random(randomSeed);
+long totalDataBlockBytes =
+ writeBlocks(TEST_UTIL.getConfiguration(), rand, COMPRESSION_ALGO, path);
+readDataBlocksAndVerify(fs, path, COMPRESSION_ALGO, totalDataBlockBytes);
+ }
+
+ private long writeBlocks(Configuration conf, Random rand,
Compression.Algorithm compressAlgo,
+Path path) throws IOException {
+FileSystem fs = HFileSystem.get(conf);
+FSDataOutputStream os = fs.create(path);
Review Comment:
safer within try/finally or try with
[GitHub] [hbase] Apache-HBase commented on pull request #4428: HBASE-27032 The draining region servers metric description is incorrect
Apache-HBase commented on PR #4428: URL: https://github.com/apache/hbase/pull/4428#issuecomment-1127869393 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 5s | master passed | | +1 :green_heart: | compile | 0m 44s | master passed | | +1 :green_heart: | shadedjars | 3m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 2s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed | | +1 :green_heart: | javac | 0m 44s | the patch passed | | +1 :green_heart: | shadedjars | 3m 48s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 32s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 196m 36s | hbase-server in the patch passed. | | | | 213m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4428/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4428 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1d7d0b7136d7 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ad64a9baae | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4428/3/testReport/ | | Max. process+thread count | 2384 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4428/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4428: HBASE-27032 The draining region servers metric description is incorrect
Apache-HBase commented on PR #4428: URL: https://github.com/apache/hbase/pull/4428#issuecomment-1127863748 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 36s | master passed | | +1 :green_heart: | compile | 0m 53s | master passed | | +1 :green_heart: | shadedjars | 3m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 25s | the patch passed | | +1 :green_heart: | compile | 0m 53s | the patch passed | | +1 :green_heart: | javac | 0m 53s | the patch passed | | +1 :green_heart: | shadedjars | 3m 48s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 35s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 189m 12s | hbase-server in the patch passed. | | | | 208m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4428/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4428 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fb127d306e36 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ad64a9baae | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4428/3/testReport/ | | Max. process+thread count | 2634 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4428/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-27041) Flakey test detector generated garbage, breaks build automation
[
https://issues.apache.org/jira/browse/HBASE-27041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-27041:
--
Attachment: jstack
> Flakey test detector generated garbage, breaks build automation
> ---
>
> Key: HBASE-27041
> URL: https://issues.apache.org/jira/browse/HBASE-27041
> Project: HBase
> Issue Type: Task
> Components: flakies, test
>Reporter: Nick Dimiduk
>Priority: Major
> Attachments: jstack
>
>
> Flakey test automation is injecting nonsense into the mvn invocation used by
> builds. The skip list includes: {{**/htWtApR.NTIeNsGt:H tAtnp
> Sielrlveegra.java}}, which causes the build to fail with
> {noformat}
> [ERROR] Unknown lifecycle phase "tAtnp". You must specify a valid lifecycle
> phase or a goal in the format : or
> :[:]:. Available
> lifecycle phases are: validate, initialize, generate-sources,
> process-sources, generate-resources, process-resources, compile,
> process-classes, generate-test-sources, process-test-sources,
> generate-test-resources, process-test-resources, test-compile,
> process-test-classes, test, prepare-package, package, pre-integration-test,
> integration-test, post-integration-test, verify, install, deploy, pre-clean,
> clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-27041) Flakey test detector generated garbage, breaks build automation
[
https://issues.apache.org/jira/browse/HBASE-27041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537619#comment-17537619
]
Duo Zhang commented on HBASE-27041:
---
I've already deleted the broken output for two find flaky tests, so it should
be OK for now.
The state of the flaky tests job for branch-2 is not good. The problem is
TestClusterScopeQuotaThrottle, it will hang there forever and cause the build
to timeout.
Let me upload the jstack result of the hanging process. Need to dig more.
> Flakey test detector generated garbage, breaks build automation
> ---
>
> Key: HBASE-27041
> URL: https://issues.apache.org/jira/browse/HBASE-27041
> Project: HBase
> Issue Type: Task
> Components: flakies, test
>Reporter: Nick Dimiduk
>Priority: Major
>
> Flakey test automation is injecting nonsense into the mvn invocation used by
> builds. The skip list includes: {{**/htWtApR.NTIeNsGt:H tAtnp
> Sielrlveegra.java}}, which causes the build to fail with
> {noformat}
> [ERROR] Unknown lifecycle phase "tAtnp". You must specify a valid lifecycle
> phase or a goal in the format : or
> :[:]:. Available
> lifecycle phases are: validate, initialize, generate-sources,
> process-sources, generate-resources, process-resources, compile,
> process-classes, generate-test-sources, process-test-sources,
> generate-test-resources, process-test-resources, test-compile,
> process-test-classes, test, prepare-package, package, pre-integration-test,
> integration-test, post-integration-test, verify, install, deploy, pre-clean,
> clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase-filesystem] steveloughran commented on pull request #33: HBASE-27042. Remove S3Guard awareness from HBoss
steveloughran commented on PR #33: URL: https://github.com/apache/hbase-filesystem/pull/33#issuecomment-1127796373 s3 london; directory markers=keep Tests run: 225, Failures: 0, Errors: 0, Skipped: 55 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a diff in pull request #4429: HBASE-27033 Backport "HBASE-27013 Introduce read all bytes when using…
joshelser commented on code in PR #4429:
URL: https://github.com/apache/hbase/pull/4429#discussion_r873834582
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:
##
@@ -1449,7 +1452,9 @@ protected boolean readAtOffset(FSDataInputStream istream,
ByteBuff dest, int siz
} else {
// Positional read. Better for random reads; or when the streamLock is
already locked.
int extraSize = peekIntoNextBlock ? hdrSize : 0;
-if (!BlockIOUtils.preadWithExtra(dest, istream, fileOffset, size,
extraSize)) {
+if (
+ !BlockIOUtils.preadWithExtra(dest, istream, fileOffset, size,
extraSize, isPreadAllBytes)
+) {
Review Comment:
```suggestion
if (!BlockIOUtils.preadWithExtra(
dest, istream, fileOffset, size, extraSize,
isPreadAllBytes)) {
```
Typically, we break long lines with method invocations around the arguments,
rather than break up the `if` condition itself.
However, if checkstyle is happy with this how it is, then it's fine :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4437: HBASE-27043 Let lock wait timeout to improve performance of SnapshotHFileCleaner
Apache-HBase commented on PR #4437: URL: https://github.com/apache/hbase/pull/4437#issuecomment-1127765579 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 29s | master passed | | +1 :green_heart: | compile | 2m 14s | master passed | | +1 :green_heart: | checkstyle | 0m 32s | master passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 13s | the patch passed | | +1 :green_heart: | compile | 2m 13s | the patch passed | | +1 :green_heart: | javac | 2m 13s | the patch passed | | +1 :green_heart: | checkstyle | 0m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 3s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 33m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4437/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4437 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 5c626156e268 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ad64a9baae | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 69 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4437/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4435: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2
Apache-HBase commented on PR #4435: URL: https://github.com/apache/hbase/pull/4435#issuecomment-1127722851 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 19s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 15s | branch-2 passed | | +1 :green_heart: | compile | 3m 19s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 0s | branch-2 passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 22s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 10s | the patch passed | | +1 :green_heart: | compile | 3m 19s | the patch passed | | +1 :green_heart: | javac | 3m 19s | the patch passed | | +1 :green_heart: | checkstyle | 0m 57s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 7m 56s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotless | 0m 39s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 35m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4435 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 335be8c1742a 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2efaa7d221 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 68 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-procedure hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sunhelly opened a new pull request, #4437: HBASE-27043 Let lock wait timeout to improve performance of SnapshotHFileCleaner
sunhelly opened a new pull request, #4437: URL: https://github.com/apache/hbase/pull/4437 More details and improve result is at https://issues.apache.org/jira/browse/HBASE-27043 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4435: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2
Apache-HBase commented on PR #4435: URL: https://github.com/apache/hbase/pull/4435#issuecomment-1127706710 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 53s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 47s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 35s | branch-2 passed | | +1 :green_heart: | compile | 1m 10s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 50s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 24s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed | | +1 :green_heart: | javac | 1m 11s | the patch passed | | +1 :green_heart: | shadedjars | 3m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 0m 6s | hbase-client in the patch failed. | | -1 :x: | unit | 0m 6s | hbase-procedure in the patch failed. | | -1 :x: | unit | 0m 6s | hbase-server in the patch failed. | | | | 21m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4435 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5fe3e6eb3a5b 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2efaa7d221 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-client.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-procedure.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/testReport/ | | Max. process+thread count | 78 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-procedure hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4435: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2
Apache-HBase commented on PR #4435: URL: https://github.com/apache/hbase/pull/4435#issuecomment-1127705285 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 54s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 51s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 16s | branch-2 passed | | +1 :green_heart: | compile | 1m 4s | branch-2 passed | | +1 :green_heart: | shadedjars | 3m 56s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 47s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 7s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 3m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 0m 5s | hbase-client in the patch failed. | | -1 :x: | unit | 0m 6s | hbase-procedure in the patch failed. | | -1 :x: | unit | 0m 6s | hbase-server in the patch failed. | | | | 20m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4435 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 397643b751c8 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2efaa7d221 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-client.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-procedure.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/testReport/ | | Max. process+thread count | 64 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-procedure hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-27043) Let lock wait timeout to improve performance of SnapshotHFileCleaner
Xiaolin Ha created HBASE-27043: -- Summary: Let lock wait timeout to improve performance of SnapshotHFileCleaner Key: HBASE-27043 URL: https://issues.apache.org/jira/browse/HBASE-27043 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 2.5.0, 3.0.0-alpha-3 Reporter: Xiaolin Ha Assignee: Xiaolin Ha Attachments: clearner-before-and-after.png, namenode-callqueue.png Currently, hfile cleaner uses the dir scanning threads to get deletable files, by checking all the files under the scanned directories through the cleaner chain. And before scanning a directory, cleaner sorted the subdirectories by consumed spaces, but we all know getContentSummary is a time consuming operation for HDFS. SnapshotHFileCleaner filters all the unreferenced files of snapshots to delete, and it tries to get write lock of SnapshotManager#takingSnapshotLock before determining the deletable files. [https://github.com/apache/hbase/blob/ad64a9baae2ef8ee56aa3ed6b96cb3d51f5daf0a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java#L195] But when there is any snapshot taking and the cleaner gets the lock failed, all the scanned files will be determined to be none-deletable, and all the dir scanning threads will scan and getContentSummary of other dirs(no files are deletable too) one by one until the snapshot taking lock is released. This is a low efficiency behavior, we should let the clear wait the lock to determine if the files it currently hold are deletable, instead of the meaningless scanning and getContentSummary other/new directories while the lock is acquired by taking snapshot operations. I deployed this optimization in our production environment, and the effect is very obvious. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase-filesystem] steveloughran commented on pull request #33: HBASE-27042. Remove S3Guard awareness from HBoss
steveloughran commented on PR #33: URL: https://github.com/apache/hbase-filesystem/pull/33#issuecomment-1127690408 (yes, it was being picked up, adding an invalid bucket triggered test run). bulild options were ``` -Phadoop3.3 -Dhadoop33.version=3.3.3 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-filesystem] steveloughran commented on pull request #33: HBASE-27042. Remove S3Guard awareness from HBoss
steveloughran commented on PR #33: URL: https://github.com/apache/hbase-filesystem/pull/33#issuecomment-1127687278 tested in the mock client, and in progress is a test with fs.hboss.data.uri pointing at a real s3a bucket. not convinced that this setting is being picked up though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-filesystem] steveloughran opened a new pull request, #33: HBASE-27042. Remove S3Guard awareness from HBoss
steveloughran opened a new pull request, #33: URL: https://github.com/apache/hbase-filesystem/pull/33 * Cut HBoss test code which would explicitly disable S3Guard -obsolete and breaks the build on forthcoming hadoop releases. * Also cut all references in docs and XML. * EmbeddedS3 to implement deleteObject() and so support AWS SDKs 1.11.1026+, which expects this (hadoop 3.3.2+). * Maven test runs to provide full stack traces on failures, so runtime failures can be debugged and fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4436: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2.5
Apache-HBase commented on PR #4436: URL: https://github.com/apache/hbase/pull/4436#issuecomment-1127682279 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.5 Compile Tests _ | | +0 :ok: | mvndep | 0m 50s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 40s | branch-2.5 passed | | +1 :green_heart: | compile | 1m 15s | branch-2.5 passed | | +1 :green_heart: | shadedjars | 4m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | branch-2.5 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 31s | the patch passed | | +1 :green_heart: | compile | 1m 15s | the patch passed | | +1 :green_heart: | javac | 1m 15s | the patch passed | | +1 :green_heart: | shadedjars | 4m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 50s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 56s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 1m 21s | hbase-procedure in the patch passed. | | +1 :green_heart: | unit | 175m 43s | hbase-server in the patch passed. | | | | 203m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4436/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4436 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ad8218017507 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.5 / 2e4d9a2adf | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4436/1/testReport/ | | Max. process+thread count | 2534 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-procedure hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4436/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4436: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2.5
Apache-HBase commented on PR #4436: URL: https://github.com/apache/hbase/pull/4436#issuecomment-1127677824 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.5 Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 11s | branch-2.5 passed | | +1 :green_heart: | compile | 1m 11s | branch-2.5 passed | | +1 :green_heart: | shadedjars | 3m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | branch-2.5 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 6s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed | | +1 :green_heart: | javac | 1m 11s | the patch passed | | +1 :green_heart: | shadedjars | 3m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 32s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 1m 20s | hbase-procedure in the patch passed. | | +1 :green_heart: | unit | 176m 38s | hbase-server in the patch passed. | | | | 199m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4436/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4436 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1eafdbde142d 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.5 / 2e4d9a2adf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4436/1/testReport/ | | Max. process+thread count | 2643 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-procedure hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4436/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4428: HBASE-27032 The draining region servers metric description is incorrect
Apache-HBase commented on PR #4428: URL: https://github.com/apache/hbase/pull/4428#issuecomment-1127663075 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 18s | master passed | | +1 :green_heart: | compile | 2m 34s | master passed | | +1 :green_heart: | checkstyle | 0m 40s | master passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 37s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 14s | the patch passed | | +1 :green_heart: | compile | 2m 32s | the patch passed | | +1 :green_heart: | javac | 2m 32s | the patch passed | | +1 :green_heart: | checkstyle | 0m 38s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 40s | Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1. | | +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 50s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 20s | The patch does not generate ASF License warnings. | | | | 35m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4428/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4428 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 07cae5340274 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / ad64a9baae | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 64 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4428/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27042) hboss doesn't compile against hadoop branch-3.3 now that s3guard is cut
[
https://issues.apache.org/jira/browse/HBASE-27042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537531#comment-17537531
]
Steve Loughran commented on HBASE-27042:
an update of the AWS SDK also breaks the tests, because the AWS client now
requires another method to be implemented
{code}
[ERROR]
testListFilesEmptyDirectoryNonrecursive(org.apache.hadoop.hbase.oss.contract.TestHBOSSContractGetFileStatus)
Time elapsed: 4.248 s <<< ERROR!
java.lang.UnsupportedOperationException: Extend AbstractAmazonS3 to provide an
implementation
{code}
once you tell maven to give you useful stack traces, you can track this down
{code}
[ERROR]
testListLocatedStatusEmptyDirectory(org.apache.hadoop.hbase.oss.contract.TestHBOSSContractGetFileStatus)
Time elapsed: 1.365 s <<< ERROR!
java.lang.UnsupportedOperationException: Extend AbstractAmazonS3 to provide an
implementation
at
com.amazonaws.services.s3.AbstractAmazonS3.deleteObject(AbstractAmazonS3.java:642)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$null$13(S3AFileSystem.java:2696)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:464)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObject$14(S3AFileSystem.java:2694)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:377)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObject(S3AFileSystem.java:2690)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjectAtPath(S3AFileSystem.java:2725)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.lambda$deleteObjectAtPath$0(S3AFileSystem.java:2055)
at org.apache.hadoop.fs.s3a.Invoker.lambda$once$0(Invoker.java:135)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:133)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.deleteObjectAtPath(S3AFileSystem.java:2054)
{code}
> hboss doesn't compile against hadoop branch-3.3 now that s3guard is cut
> ---
>
> Key: HBASE-27042
> URL: https://issues.apache.org/jira/browse/HBASE-27042
> Project: HBase
> Issue Type: Bug
> Components: hboss
>Reporter: Steve Loughran
>Priority: Minor
>
> HBoss doesn't compile against hadoop builds containing HADOOP-17409, "remove
> s3guard", as test setup tries to turn it off.
> there's no need for s3guard any more, so hboss can just avoid all settings
> and expect it to be disabled (hadoop 3.3.3. or earlier) or removed (3.4+)
> (hboss version is 1.0.0-alpha2-SNAPSHOT)
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Work started] (HBASE-27034) NegativeArraySizeException was encountered during compaction
[
https://issues.apache.org/jira/browse/HBASE-27034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-27034 started by fengxianjing.
> NegativeArraySizeException was encountered during compaction
>
>
> Key: HBASE-27034
> URL: https://issues.apache.org/jira/browse/HBASE-27034
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.3.4
>Reporter: fengxianjing
>Assignee: fengxianjing
>Priority: Critical
>
>
> {code:java}
> 2022-04-13 12:45:37,122 ERROR [regionserver/xx:26020-shortCompactions-0]
> regionserver.CompactSplit: Compaction failed
> region=XXX:XXX,002CX21205070934507532021052320210523174923,162
> 8091302516.7d2e05ad63b91843d438d2464a908d49.,
> storeName=7d2e05ad63b91843d438d2464a908d49/info, priority=90,
> startTime=1649825135950
> java.lang.NegativeArraySizeException
> at org.apache.hadoop.hbase.CellUtil.cloneQualifier(CellUtil.java:120)
> at
> org.apache.hadoop.hbase.ByteBufferKeyValue.getQualifierArray(ByteBufferKeyValue.java:112)
> at
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(CellUtil.java:1335)
> at
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(CellUtil.java:1318)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.getMidpoint(HFileWriterImpl.java:384)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishBlock(HFileWriterImpl.java:349)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.checkBlockBoundary(HFileWriterImpl.java:328)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.append(HFileWriterImpl.java:739)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter.append(StoreFileWriter.java:299)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:410)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:333)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1544)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2288)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:619)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:661)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
>
> We encounter the exeception above many times. Usually, it will retry and
> compact success next time. But sometime it make _getMidpoint return_ a wrong
> result and then make an abnormal index block as follow
>
> {code:java}
> 068c892d122//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> 1//LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> 068c896a6fc6f155beaddab036a4225ef79_12022011420220114205945/info:q/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> {code}
>
> And this index block will lead to an endless loop in
> _org.apache.hadoop.hbase.regionserver.KeyValueHeap#generalizedSeek_
> The cause of this problem is _lastCellOfPreviousBlock_ reference to the cells
> in read path([HBASE-16372|https://issues.apache.org/jira/browse/HBASE-16372])
> I have fixed it and will create a PR for it
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (HBASE-27042) hboss doesn't compile against hadoop branch-3.3 now that s3guard is cut
Steve Loughran created HBASE-27042: -- Summary: hboss doesn't compile against hadoop branch-3.3 now that s3guard is cut Key: HBASE-27042 URL: https://issues.apache.org/jira/browse/HBASE-27042 Project: HBase Issue Type: Bug Components: hboss Reporter: Steve Loughran HBoss doesn't compile against hadoop builds containing HADOOP-17409, "remove s3guard", as test setup tries to turn it off. there's no need for s3guard any more, so hboss can just avoid all settings and expect it to be disabled (hadoop 3.3.3. or earlier) or removed (3.4+) (hboss version is 1.0.0-alpha2-SNAPSHOT) -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4436: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2.5
Apache-HBase commented on PR #4436: URL: https://github.com/apache/hbase/pull/4436#issuecomment-1127515933 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 52s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.5 Compile Tests _ | | +0 :ok: | mvndep | 0m 46s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 17s | branch-2.5 passed | | +1 :green_heart: | compile | 3m 10s | branch-2.5 passed | | +1 :green_heart: | checkstyle | 0m 49s | branch-2.5 passed | | +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 7s | branch-2.5 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 1s | the patch passed | | +1 :green_heart: | compile | 3m 9s | the patch passed | | +1 :green_heart: | javac | 3m 9s | the patch passed | | +1 :green_heart: | checkstyle | 0m 48s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 23s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotless | 0m 36s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 38m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4436/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4436 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 96e8d37c8457 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.5 / 2e4d9a2adf | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 65 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-procedure hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4436/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #4435: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2
ndimiduk commented on PR #4435: URL: https://github.com/apache/hbase/pull/4435#issuecomment-1127512896 Broken by [HBASE-27041](https://issues.apache.org/jira/browse/HBASE-27041). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-27041) Flakey test detector generated garbage, breaks build automation
Nick Dimiduk created HBASE-27041:
Summary: Flakey test detector generated garbage, breaks build
automation
Key: HBASE-27041
URL: https://issues.apache.org/jira/browse/HBASE-27041
Project: HBase
Issue Type: Task
Components: flakies, test
Reporter: Nick Dimiduk
Flakey test automation is injecting nonsense into the mvn invocation used by
builds. The skip list includes: {{**/htWtApR.NTIeNsGt:H tAtnp
Sielrlveegra.java}}, which causes the build to fail with
{noformat}
[ERROR] Unknown lifecycle phase "tAtnp". You must specify a valid lifecycle
phase or a goal in the format : or
:[:]:. Available
lifecycle phases are: validate, initialize, generate-sources, process-sources,
generate-resources, process-resources, compile, process-classes,
generate-test-sources, process-test-sources, generate-test-resources,
process-test-resources, test-compile, process-test-classes, test,
prepare-package, package, pre-integration-test, integration-test,
post-integration-test, verify, install, deploy, pre-clean, clean, post-clean,
pre-site, site, post-site, site-deploy. -> [Help 1]
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [hbase] Apache-HBase commented on pull request #4435: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2
Apache-HBase commented on PR #4435: URL: https://github.com/apache/hbase/pull/4435#issuecomment-1127501346 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 18s | branch-2 passed | | +1 :green_heart: | compile | 3m 21s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 0s | branch-2 passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 27s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 10s | the patch passed | | +1 :green_heart: | compile | 3m 23s | the patch passed | | +1 :green_heart: | javac | 3m 23s | the patch passed | | +1 :green_heart: | checkstyle | 0m 58s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 7m 55s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 41m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4435 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux e5b0579b1a7d 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2efaa7d221 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 65 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-procedure hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4435: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2
Apache-HBase commented on PR #4435: URL: https://github.com/apache/hbase/pull/4435#issuecomment-1127483621 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 24s | branch-2 passed | | +1 :green_heart: | compile | 1m 3s | branch-2 passed | | +1 :green_heart: | shadedjars | 3m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 3s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 4m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 47s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 0m 6s | hbase-client in the patch failed. | | -1 :x: | unit | 0m 6s | hbase-procedure in the patch failed. | | -1 :x: | unit | 0m 6s | hbase-server in the patch failed. | | | | 23m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4435 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3808d90bd802 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2efaa7d221 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-client.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-procedure.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/testReport/ | | Max. process+thread count | 64 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-procedure hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4435: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2
Apache-HBase commented on PR #4435: URL: https://github.com/apache/hbase/pull/4435#issuecomment-1127480966 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 52s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 50s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 36s | branch-2 passed | | +1 :green_heart: | compile | 1m 11s | branch-2 passed | | +1 :green_heart: | shadedjars | 3m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 30s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 3m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 0m 7s | hbase-client in the patch failed. | | -1 :x: | unit | 0m 6s | hbase-procedure in the patch failed. | | -1 :x: | unit | 0m 6s | hbase-server in the patch failed. | | | | 20m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4435 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4b7361d0053b 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 2efaa7d221 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-client.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-procedure.txt | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/testReport/ | | Max. process+thread count | 73 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-procedure hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4435/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request, #4436: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2.5
ndimiduk opened a new pull request, #4436: URL: https://github.com/apache/hbase/pull/4436 This one adds a tracing context to the Master's ProcedureV2 execution. This ties all the spans that result from a procedure execution back to a procedure by name. Maybe in a follow-on issue, we can serialize span context along with procedures and display procedure start/stop/retry events and parent-child hierarchy in tracing visualization. Signed-off-by: Andrew Purtell Signed-off-by: Duo Zhang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26986) Trace a one-shot execution of a Master procedure
[ https://issues.apache.org/jira/browse/HBASE-26986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-26986: - Fix Version/s: 3.0.0-alpha-3 > Trace a one-shot execution of a Master procedure > > > Key: HBASE-26986 > URL: https://issues.apache.org/jira/browse/HBASE-26986 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.5.0, 3.0.0-alpha-2 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0-alpha-3 > > > A self-contained effort towards wrangling the stray traces identified in > HBASE-26366. This one adds a tracing context to the Master's ProcedureV2 > execution. This ties all the spans that result from a procedure execution > back to a procedure by name. > Maybe in a follow-on issue, we can serialize span context along with > procedures and display procedure start/stop/retry events and parent-child > hierarchy in tracing visualization. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [hbase] ndimiduk opened a new pull request, #4435: Backport "HBASE-26986 Trace a one-shot execution of a Master procedure" to branch-2
ndimiduk opened a new pull request, #4435: URL: https://github.com/apache/hbase/pull/4435 This one adds a tracing context to the Master's ProcedureV2 execution. This ties all the spans that result from a procedure execution back to a procedure by name. Maybe in a follow-on issue, we can serialize span context along with procedures and display procedure start/stop/retry events and parent-child hierarchy in tracing visualization. Signed-off-by: Andrew Purtell Signed-off-by: Duo Zhang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #4381: HBASE-26986 Trace a one-shot execution of a Master procedure
ndimiduk merged PR #4381: URL: https://github.com/apache/hbase/pull/4381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4434: HBASE-27040 Optimize the log display of the ZKProcedureUtil.java
Apache-HBase commented on PR #4434: URL: https://github.com/apache/hbase/pull/4434#issuecomment-1127333182 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 16s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 25s | master passed | | +1 :green_heart: | compile | 0m 38s | master passed | | +1 :green_heart: | shadedjars | 3m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 13s | the patch passed | | +1 :green_heart: | compile | 0m 39s | the patch passed | | +1 :green_heart: | javac | 0m 39s | the patch passed | | +1 :green_heart: | shadedjars | 3m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 197m 41s | hbase-server in the patch passed. | | | | 214m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4434/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4434 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c405f0d1a68d 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 369adcdf35 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4434/1/testReport/ | | Max. process+thread count | 2414 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4434/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #4434: HBASE-27040 Optimize the log display of the ZKProcedureUtil.java
Apache-HBase commented on PR #4434: URL: https://github.com/apache/hbase/pull/4434#issuecomment-1127327600 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | master passed | | +1 :green_heart: | compile | 0m 43s | master passed | | +1 :green_heart: | shadedjars | 3m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 26s | the patch passed | | +1 :green_heart: | compile | 0m 41s | the patch passed | | +1 :green_heart: | javac | 0m 41s | the patch passed | | +1 :green_heart: | shadedjars | 3m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 188m 33s | hbase-server in the patch passed. | | | | 207m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4434/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4434 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 106e08657bc6 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 369adcdf35 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4434/1/testReport/ | | Max. process+thread count | 2707 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4434/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] tomscut commented on pull request #4428: HBASE-27032 The draining region servers metric description is incorrect
tomscut commented on PR #4428: URL: https://github.com/apache/hbase/pull/4428#issuecomment-1127264426 > > Hi @pankaj72981 , I updated it. Please take a look again. Thank you very much. > > Adding UT is always good, I meant to say correcting metric name will be incompatible change draininigRegionServers -> drainingRegionServers Thanks @pankaj72981 for your comments. Do you mean that if the metric name is changed, some tasks that depend on the metric name may be affected if the user upgrades the hbase version? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
