[jira] [Commented] (HBASE-28222) Leak in ExportSnapshot during verifySnapshot on S3A
[
https://issues.apache.org/jira/browse/HBASE-28222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792334#comment-17792334
]
Hudson commented on HBASE-28222:
Results for branch branch-2
[build #935 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Leak in ExportSnapshot during verifySnapshot on S3A
> ---
>
> Key: HBASE-28222
> URL: https://issues.apache.org/jira/browse/HBASE-28222
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Fix For: 2.6.0, 3.0.0-beta-1
>
>
> Each S3AFileSystem creates an S3AInstrumentation and various metrics sources,
> with no real way to disable that. In HADOOP-18526, a bug was fixed so that
> these are not leaked. But in order to use that, you must call
> S3AFileSystem.close() when done.
> In ExportSnapshot, ever since HBASE-12819 we set fs.impl.disable.cache to
> true. It looks like that was added in order to prevent conflicting calls to
> close() between mapper and main thread when running in a single JVM.
> When verifySnapshot is enabled, SnapshotReferenceUtil.verifySnapshot iterates
> all storefiles (could be many thousands) and calls
> SnapshotReferenceUtil.verifyStoreFile on them. verifyStoreFile makes a number
> of static calls which end up in CommonFSUtils.getRootDir, which does
> Path.getFileSystem().
> Since the FS cache is disabled, every single call to Path.getFileSystem()
> creates a new FileSystem instance. That FS is short lived, and gets GC'd. But
> in the case of S3AFileSystem, this leaks all of the metrics stuff.
> We have two easy possible fixes:
> # Only set fs.impl.disable.cache when running hadoop in local mode, since
> that was the original problem.
> # When calling verifySnapshot, create a new Configuration which does not
> include the fs.impl.disable.cache setting.
> I tested out #2 in my environment and it fixed the leak.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28212) Do not need to maintain rollback step when root procedure does not support rollback
[
https://issues.apache.org/jira/browse/HBASE-28212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792335#comment-17792335
]
Hudson commented on HBASE-28212:
Results for branch branch-2
[build #935 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Do not need to maintain rollback step when root procedure does not support
> rollback
> ---
>
> Key: HBASE-28212
> URL: https://issues.apache.org/jira/browse/HBASE-28212
> Project: HBase
> Issue Type: Improvement
> Components: master, proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> In HBASE-28210, we need to synchronized on RootProcedureState when updating
> procedure store, which will decrease the performance a lot when there are
> lots of sub procedures for a procedure, such as SCP, so it will impact MTTR.
> But actually, SCP does not support rollback, so I do not think we need to
> maintain this information. If so, I think we do not need the synchronization
> added in HBASE-28210, at least for SCP then. This could get the performance
> back.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[
https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792333#comment-17792333
]
Hudson commented on HBASE-28204:
Results for branch branch-2
[build #935 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Canary can take lot more time If any region (except the first region) starts
> with delete markers
>
>
> Key: HBASE-28204
> URL: https://issues.apache.org/jira/browse/HBASE-28204
> Project: HBase
> Issue Type: Bug
> Components: canary
>Reporter: Mihir Monani
>Assignee: Mihir Monani
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7, 2.7.0
>
>
> In CanaryTool.java, Canary reads only the first row of the region using
> [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33]
> for any region of the table. Canary uses [Scan with FirstRowKeyFilter for
> table
> scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530]
> If the said region has empty start key (This will only happen when region is
> the first region for a table)
> With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]-
> RawScan was
> [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534]
> to improve performance for regions which can have high number of delete
> markers. Based on currently implementation, [RawScan is only
> enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519]
> if region has empty start-key (or region is first region for the table).
> RawScan doesn't work for rest of the regions in the table except first
> region. Also If the region has all the rows or majority of the rows with
> delete markers, Get Operation can take a lot of time. This is can cause
> timeouts for CanaryTool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-25549) Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms.
[
https://issues.apache.org/jira/browse/HBASE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792336#comment-17792336
]
Hudson commented on HBASE-25549:
Results for branch branch-2
[build #935 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Provide a switch that allows avoiding reopening all regions when modifying a
> table to prevent RIT storms.
> -
>
> Key: HBASE-25549
> URL: https://issues.apache.org/jira/browse/HBASE-25549
> Project: HBase
> Issue Type: Improvement
> Components: master, shell
>Affects Versions: 3.0.0-alpha-1
>Reporter: Zhuoyue Huang
>Assignee: Zhuoyue Huang
>Priority: Major
> Fix For: 2.6.0, 3.0.0-beta-1, 2.5.7
>
>
> Under normal circumstances, modifying a table will cause all regions
> belonging to the table to enter RIT. Imagine the following two scenarios：
> # Someone entered the wrong configuration （e.g. negative
> 'hbase.busy.wait.multiplier.max' value） when altering the table, causing
> thousands of online regions to fail to open, leading to online accidents.
> # Modify the configuration of a table, but this modification is not urgent,
> the regions are not expected to enter RIT immediately.
> -'alter_lazy' is a new command to modify a table without reopening any online
> regions except those regions were assigned by other threads or split etc.-
>
> Provide an optional lazy_mode for the alter command to modify the
> TableDescriptor without the region entering the RIT. The modification will
> take effect when the region is reopened.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28230) Update branch-2 version to 2.7-SNAPSHOT
[
https://issues.apache.org/jira/browse/HBASE-28230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792332#comment-17792332
]
Hudson commented on HBASE-28230:
Results for branch branch-2
[build #935 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/935/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Update branch-2 version to 2.7-SNAPSHOT
> ---
>
> Key: HBASE-28230
> URL: https://issues.apache.org/jira/browse/HBASE-28230
> Project: HBase
> Issue Type: Sub-task
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Fix For: 2.7.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
gourabtaparia commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1837052398 Thanks @bbeaudreault for merging this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[
https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792326#comment-17792326
]
Hudson commented on HBASE-28204:
Results for branch branch-2.4
[build #660 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/660/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/660/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/660/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/660/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/660/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Canary can take lot more time If any region (except the first region) starts
> with delete markers
>
>
> Key: HBASE-28204
> URL: https://issues.apache.org/jira/browse/HBASE-28204
> Project: HBase
> Issue Type: Bug
> Components: canary
>Reporter: Mihir Monani
>Assignee: Mihir Monani
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7, 2.7.0
>
>
> In CanaryTool.java, Canary reads only the first row of the region using
> [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33]
> for any region of the table. Canary uses [Scan with FirstRowKeyFilter for
> table
> scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530]
> If the said region has empty start key (This will only happen when region is
> the first region for a table)
> With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]-
> RawScan was
> [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534]
> to improve performance for regions which can have high number of delete
> markers. Based on currently implementation, [RawScan is only
> enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519]
> if region has empty start-key (or region is first region for the table).
> RawScan doesn't work for rest of the regions in the table except first
> region. Also If the region has all the rows or majority of the rows with
> delete markers, Get Operation can take a lot of time. This is can cause
> timeouts for CanaryTool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28212) Do not need to maintain rollback step when root procedure does not support rollback
[
https://issues.apache.org/jira/browse/HBASE-28212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792327#comment-17792327
]
Hudson commented on HBASE-28212:
Results for branch branch-2.4
[build #660 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/660/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/660/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/660/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/660/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/660/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Do not need to maintain rollback step when root procedure does not support
> rollback
> ---
>
> Key: HBASE-28212
> URL: https://issues.apache.org/jira/browse/HBASE-28212
> Project: HBase
> Issue Type: Improvement
> Components: master, proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> In HBASE-28210, we need to synchronized on RootProcedureState when updating
> procedure store, which will decrease the performance a lot when there are
> lots of sub procedures for a procedure, such as SCP, so it will impact MTTR.
> But actually, SCP does not support rollback, so I do not think we need to
> maintain this information. If so, I think we do not need the synchronization
> added in HBASE-28210, at least for SCP then. This could get the performance
> back.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28215: region reopen procedure batching/throttling [hbase]
Apache-HBase commented on PR #5534: URL: https://github.com/apache/hbase/pull/5534#issuecomment-1837042766 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | master passed | | +1 :green_heart: | compile | 0m 49s | master passed | | +1 :green_heart: | shadedjars | 4m 54s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 42s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 4m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 230m 22s | hbase-server in the patch passed. | | | | 252m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5534 | | JIRA Issue | HBASE-28215 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 19671a15633b 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7dd4d0c532 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/9/testReport/ | | Max. process+thread count | 4841 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/9/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28215: region reopen procedure batching/throttling [hbase]
Apache-HBase commented on PR #5534: URL: https://github.com/apache/hbase/pull/5534#issuecomment-1837040443 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 19s | master passed | | +1 :green_heart: | compile | 0m 34s | master passed | | +1 :green_heart: | shadedjars | 5m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 13s | the patch passed | | +1 :green_heart: | compile | 0m 36s | the patch passed | | +1 :green_heart: | javac | 0m 36s | the patch passed | | +1 :green_heart: | shadedjars | 5m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 219m 36s | hbase-server in the patch passed. | | | | 240m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5534 | | JIRA Issue | HBASE-28215 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 39c77c676370 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7dd4d0c532 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/9/testReport/ | | Max. process+thread count | 4488 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/9/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28222) Leak in ExportSnapshot during verifySnapshot on S3A
[
https://issues.apache.org/jira/browse/HBASE-28222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792319#comment-17792319
]
Hudson commented on HBASE-28222:
Results for branch branch-2.6
[build #4 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/4/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/4/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/4/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/4/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/4/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Leak in ExportSnapshot during verifySnapshot on S3A
> ---
>
> Key: HBASE-28222
> URL: https://issues.apache.org/jira/browse/HBASE-28222
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Fix For: 2.6.0, 3.0.0-beta-1
>
>
> Each S3AFileSystem creates an S3AInstrumentation and various metrics sources,
> with no real way to disable that. In HADOOP-18526, a bug was fixed so that
> these are not leaked. But in order to use that, you must call
> S3AFileSystem.close() when done.
> In ExportSnapshot, ever since HBASE-12819 we set fs.impl.disable.cache to
> true. It looks like that was added in order to prevent conflicting calls to
> close() between mapper and main thread when running in a single JVM.
> When verifySnapshot is enabled, SnapshotReferenceUtil.verifySnapshot iterates
> all storefiles (could be many thousands) and calls
> SnapshotReferenceUtil.verifyStoreFile on them. verifyStoreFile makes a number
> of static calls which end up in CommonFSUtils.getRootDir, which does
> Path.getFileSystem().
> Since the FS cache is disabled, every single call to Path.getFileSystem()
> creates a new FileSystem instance. That FS is short lived, and gets GC'd. But
> in the case of S3AFileSystem, this leaks all of the metrics stuff.
> We have two easy possible fixes:
> # Only set fs.impl.disable.cache when running hadoop in local mode, since
> that was the original problem.
> # When calling verifySnapshot, create a new Configuration which does not
> include the fs.impl.disable.cache setting.
> I tested out #2 in my environment and it fixed the leak.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[
https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792318#comment-17792318
]
Hudson commented on HBASE-28204:
Results for branch branch-2.6
[build #4 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/4/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/4/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/4/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/4/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/4/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Canary can take lot more time If any region (except the first region) starts
> with delete markers
>
>
> Key: HBASE-28204
> URL: https://issues.apache.org/jira/browse/HBASE-28204
> Project: HBase
> Issue Type: Bug
> Components: canary
>Reporter: Mihir Monani
>Assignee: Mihir Monani
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7, 2.7.0
>
>
> In CanaryTool.java, Canary reads only the first row of the region using
> [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33]
> for any region of the table. Canary uses [Scan with FirstRowKeyFilter for
> table
> scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530]
> If the said region has empty start key (This will only happen when region is
> the first region for a table)
> With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]-
> RawScan was
> [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534]
> to improve performance for regions which can have high number of delete
> markers. Based on currently implementation, [RawScan is only
> enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519]
> if region has empty start-key (or region is first region for the table).
> RawScan doesn't work for rest of the regions in the table except first
> region. Also If the region has all the rows or majority of the rows with
> delete markers, Get Operation can take a lot of time. This is can cause
> timeouts for CanaryTool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28215: region reopen procedure batching/throttling [hbase]
rmdmattingly commented on PR #5534: URL: https://github.com/apache/hbase/pull/5534#issuecomment-1836998471 The javac warning is legit, will get to that asap. The test failure was for the previous commit I believe -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28215: region reopen procedure batching/throttling [hbase]
bbeaudreault commented on PR #5534: URL: https://github.com/apache/hbase/pull/5534#issuecomment-1836997744 No rush, but looks like test failures are related and new compile/javac warnings -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28215: region reopen procedure batching/throttling [hbase]
Apache-HBase commented on PR #5534: URL: https://github.com/apache/hbase/pull/5534#issuecomment-1836980804 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 15s | master passed | | +1 :green_heart: | compile | 2m 49s | master passed | | +1 :green_heart: | checkstyle | 0m 35s | master passed | | +1 :green_heart: | spotless | 0m 39s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 32s | the patch passed | | -1 :x: | compile | 1m 3s | hbase-server in the patch failed. | | -0 :warning: | javac | 1m 3s | hbase-server in the patch failed. | | +1 :green_heart: | checkstyle | 0m 33s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 47s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 39s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 8s | The patch does not generate ASF License warnings. | | | | 31m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/9/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5534 | | JIRA Issue | HBASE-28215 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 02c6bf96adc1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7dd4d0c532 | | Default Java | Eclipse Adoptium-11.0.17+8 | | compile | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/9/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/9/artifact/yetus-general-check/output/patch-compile-hbase-server.txt | | Max. process+thread count | 76 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/9/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28215: region reopen procedure batching/throttling [hbase]
Apache-HBase commented on PR #5534: URL: https://github.com/apache/hbase/pull/5534#issuecomment-1836967457 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 59s | master passed | | +1 :green_heart: | compile | 0m 49s | master passed | | +1 :green_heart: | shadedjars | 4m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 52s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 4m 54s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 230m 26s | hbase-server in the patch failed. | | | | 253m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5534 | | JIRA Issue | HBASE-28215 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b27d3643db05 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7dd4d0c532 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/8/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/8/testReport/ | | Max. process+thread count | 4860 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/8/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28215: region reopen procedure batching/throttling [hbase]
Apache-HBase commented on PR #5534: URL: https://github.com/apache/hbase/pull/5534#issuecomment-1836963555 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 38s | master passed | | +1 :green_heart: | compile | 0m 36s | master passed | | +1 :green_heart: | shadedjars | 5m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 17s | the patch passed | | +1 :green_heart: | compile | 0m 36s | the patch passed | | +1 :green_heart: | javac | 0m 36s | the patch passed | | +1 :green_heart: | shadedjars | 5m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 222m 9s | hbase-server in the patch failed. | | | | 243m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5534 | | JIRA Issue | HBASE-28215 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ae8b71ee0239 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7dd4d0c532 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/8/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/8/testReport/ | | Max. process+thread count | 4648 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/8/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
Apache-HBase commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1836930404 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 40s | branch-2 passed | | +1 :green_heart: | compile | 3m 9s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 49s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 23s | the patch passed | | +1 :green_heart: | compile | 2m 59s | the patch passed | | +1 :green_heart: | javac | 2m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 48s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 9m 1s | hbase-client in the patch passed. | | -1 :x: | unit | 253m 54s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 8m 10s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 1s | hbase-shell in the patch passed. | | | | 313m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5549 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8aa5dbf633a5 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / dbc165a3cf | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/testReport/ | | Max. process+thread count | 4460 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-27795: Define RPC API for cache cleaning [hbase]
Apache-HBase commented on PR #5492: URL: https://github.com/apache/hbase/pull/5492#issuecomment-1836917586 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 53s | master passed | | +1 :green_heart: | compile | 2m 12s | master passed | | +1 :green_heart: | shadedjars | 4m 54s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 20s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 38s | the patch passed | | +1 :green_heart: | compile | 2m 11s | the patch passed | | +1 :green_heart: | javac | 2m 11s | the patch passed | | +1 :green_heart: | shadedjars | 4m 52s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 38s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 28s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 248m 12s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 6m 54s | hbase-thrift in the patch passed. | | | | 286m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5492/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5492 | | JIRA Issue | HBASE-27795 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1369a012cd31 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7dd4d0c532 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5492/7/testReport/ | | Max. process+thread count | 4860 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5492/7/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-27795: Define RPC API for cache cleaning [hbase]
Apache-HBase commented on PR #5492: URL: https://github.com/apache/hbase/pull/5492#issuecomment-1836890515 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 18s | master passed | | +1 :green_heart: | compile | 1m 50s | master passed | | +1 :green_heart: | shadedjars | 4m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 11s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 20s | the patch passed | | +1 :green_heart: | compile | 1m 50s | the patch passed | | +1 :green_heart: | javac | 1m 50s | the patch passed | | +1 :green_heart: | shadedjars | 4m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 30s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 18s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 226m 12s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 7m 3s | hbase-thrift in the patch passed. | | | | 260m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5492/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5492 | | JIRA Issue | HBASE-27795 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a1083a2726ca 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7dd4d0c532 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5492/7/testReport/ | | Max. process+thread count | 4502 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5492/7/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
Apache-HBase commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1836888615 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 49s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 30s | branch-2 passed | | +1 :green_heart: | compile | 2m 5s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 29s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 25s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 48s | the patch passed | | +1 :green_heart: | compile | 2m 28s | the patch passed | | +1 :green_heart: | javac | 2m 28s | the patch passed | | +1 :green_heart: | shadedjars | 5m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 29s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 8m 5s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 213m 47s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 6m 47s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 37s | hbase-shell in the patch passed. | | | | 266m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5549 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d596ac0a538a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / dbc165a3cf | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/testReport/ | | Max. process+thread count | 4672 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-28120) Provide the switch to avoid reopening regions in the alter sync command
[ https://issues.apache.org/jira/browse/HBASE-28120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault resolved HBASE-28120. --- Fix Version/s: (was: 2.6.0) Resolution: Invalid > Provide the switch to avoid reopening regions in the alter sync command > --- > > Key: HBASE-28120 > URL: https://issues.apache.org/jira/browse/HBASE-28120 > Project: HBase > Issue Type: Sub-task > Components: master, shell >Affects Versions: 2.0.0-alpha-1 >Reporter: Gourab Taparia >Assignee: Gourab Taparia >Priority: Major > > As part of the sub-task, as HBase 2 supports both Async and Sync API, this > task is to add this support/feature to HBase 2's Sync API. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28121) Port the switch to avoid reopening regions in the alter async in HBase 2
[ https://issues.apache.org/jira/browse/HBASE-28121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault resolved HBASE-28121. --- Fix Version/s: (was: 2.6.0) Resolution: Invalid > Port the switch to avoid reopening regions in the alter async in HBase 2 > > > Key: HBASE-28121 > URL: https://issues.apache.org/jira/browse/HBASE-28121 > Project: HBase > Issue Type: Sub-task > Components: master, shell >Affects Versions: 2.0.0-alpha-1 >Reporter: Gourab Taparia >Assignee: Gourab Taparia >Priority: Major > > As part of the sub-task, as HBase 2 supports both Async and Sync API, this > task is to port the feature added in HBase 3 alter(async default) layer to > HBase 2's async side. > There is a separate sub-task for adding it to HBase 2's sync side. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28215: region reopen procedure batching/throttling [hbase]
rmdmattingly commented on code in PR #5534:
URL: https://github.com/apache/hbase/pull/5534#discussion_r1412616954
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ReopenTableRegionsProcedure.java:
##
@@ -61,20 +73,43 @@ public class ReopenTableRegionsProcedure
private List regions = Collections.emptyList();
+ private List currentRegionBatch = Collections.emptyList();
+
private RetryCounter retryCounter;
+ private long reopenBatchBackoffMillis;
+ private int reopenBatchSize;
+ private int reopenBatchSizeMax;
+ private long regionsReopened = 0;
+ private long batchesProcessed = 0;
+
public ReopenTableRegionsProcedure() {
-regionNames = Collections.emptyList();
+this(null);
}
public ReopenTableRegionsProcedure(TableName tableName) {
-this.tableName = tableName;
-this.regionNames = Collections.emptyList();
+this(tableName, Collections.emptyList());
}
public ReopenTableRegionsProcedure(final TableName tableName, final
List regionNames) {
+this(tableName, regionNames, PROGRESSIVE_BATCH_BACKOFF_MILLIS_DEFAULT,
+ PROGRESSIVE_BATCH_SIZE_MAX_DISABLED);
+ }
+
+ public ReopenTableRegionsProcedure(final TableName tableName, long
reopenBatchBackoffMillis,
+int reopenBatchSizeMax) {
+this(tableName, Collections.emptyList(), reopenBatchBackoffMillis,
reopenBatchSizeMax);
+ }
+
+ public ReopenTableRegionsProcedure(final TableName tableName, final
List regionNames,
+long reopenBatchBackoffMillis, int reopenBatchSizeMax) {
this.tableName = tableName;
this.regionNames = regionNames;
+this.reopenBatchBackoffMillis = reopenBatchBackoffMillis;
+this.reopenBatchSize = reopenBatchSizeMax !=
PROGRESSIVE_BATCH_SIZE_MAX_DISABLED
+ ? 1
+ : PROGRESSIVE_BATCH_SIZE_MAX_DEFAULT_VALUE;
+this.reopenBatchSizeMax = Math.max(reopenBatchSizeMax,
MINIMUM_BATCH_SIZE_MAX);
Review Comment:
very good catch re: overflow. I've added a test which fails without our
overflow protection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
kadirozde commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1412493239 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java: ## @@ -135,11 +136,17 @@ public static List getScannersForStoreFiles(Collection
Re: [PR] HBASE-28215: region reopen procedure batching/throttling [hbase]
bbeaudreault commented on code in PR #5534:
URL: https://github.com/apache/hbase/pull/5534#discussion_r1412589480
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ReopenTableRegionsProcedure.java:
##
@@ -61,20 +73,43 @@ public class ReopenTableRegionsProcedure
private List regions = Collections.emptyList();
+ private List currentRegionBatch = Collections.emptyList();
+
private RetryCounter retryCounter;
+ private long reopenBatchBackoffMillis;
+ private int reopenBatchSize;
+ private int reopenBatchSizeMax;
+ private long regionsReopened = 0;
+ private long batchesProcessed = 0;
+
public ReopenTableRegionsProcedure() {
-regionNames = Collections.emptyList();
+this(null);
}
public ReopenTableRegionsProcedure(TableName tableName) {
-this.tableName = tableName;
-this.regionNames = Collections.emptyList();
+this(tableName, Collections.emptyList());
}
public ReopenTableRegionsProcedure(final TableName tableName, final
List regionNames) {
+this(tableName, regionNames, PROGRESSIVE_BATCH_BACKOFF_MILLIS_DEFAULT,
+ PROGRESSIVE_BATCH_SIZE_MAX_DISABLED);
+ }
+
+ public ReopenTableRegionsProcedure(final TableName tableName, long
reopenBatchBackoffMillis,
+int reopenBatchSizeMax) {
+this(tableName, Collections.emptyList(), reopenBatchBackoffMillis,
reopenBatchSizeMax);
+ }
+
+ public ReopenTableRegionsProcedure(final TableName tableName, final
List regionNames,
+long reopenBatchBackoffMillis, int reopenBatchSizeMax) {
this.tableName = tableName;
this.regionNames = regionNames;
+this.reopenBatchBackoffMillis = reopenBatchBackoffMillis;
+this.reopenBatchSize = reopenBatchSizeMax !=
PROGRESSIVE_BATCH_SIZE_MAX_DISABLED
+ ? 1
+ : PROGRESSIVE_BATCH_SIZE_MAX_DEFAULT_VALUE;
+this.reopenBatchSizeMax = Math.max(reopenBatchSizeMax,
MINIMUM_BATCH_SIZE_MAX);
Review Comment:
sorry, this is now a little confusing. at first i thought this was a bug. if
reopenBatchSizeMax was -1, we'd set the batchSize to int.max, and
reopenBatchSizeMax to 1. so then below in our batch handling in CONFIRM_REGIONS
we'd reset reopenBatchSize to 1 because of the Math.min but then I
realized, since we use Int.Max it wouldn't go into the code. anyway, it might
just be a bit cleaner to do:
```
if (reopenBatchSizeMax == PROGRESSIVE_BATCH_SIZE_MAX_DISABLED) {
this.reopenBatchSize = Integer.MAX_VALUE;
this.reopenBatchSizeMax = Integer.MAX_VALUE;
} else {
this.reopenBatchSize = 1;
this.reopenBatchSizeMax = Math.max(reopenBatchSizeMax,
MINIMUM_BATCH_SIZE_MAX);
}
```
And now that I look again below, I actually _do_ think this is a bug. Let's
say batch is disabled, and REOPEN_REGIONS tries to reopen all regions at once.
But then in CONFIRM_REGIONS some of them did not open. We'd fall into the
canSchedule block and `shouldBatchBackoff` would be true. So then we'd do
`Math.min(reopenBatchSizeMax, 2 * Integer.Max)`. I actually think the 2 * max
would overflow negative and who knows what would happen.
So maybe we should do the clarification above, but we also need to make sure
we don't overflow below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28215: region reopen procedure batching/throttling [hbase]
Apache-HBase commented on PR #5534: URL: https://github.com/apache/hbase/pull/5534#issuecomment-1836787833 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 12s | master passed | | +1 :green_heart: | compile | 2m 57s | master passed | | +1 :green_heart: | checkstyle | 0m 37s | master passed | | +1 :green_heart: | spotless | 0m 48s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 30s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 2m 38s | the patch passed | | +1 :green_heart: | javac | 2m 38s | the patch passed | | +1 :green_heart: | checkstyle | 0m 33s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 50s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 45s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | | | | 35m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/8/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5534 | | JIRA Issue | HBASE-28215 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 4ca742952943 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7dd4d0c532 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 77 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5534/8/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[ https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792193#comment-17792193 ] Rushabh Shah commented on HBASE-28204: -- > FYI Bryan Beaudreault, we have seen a bit of perf regression so need to > revert the commit. Just wanted to keep you in loop in case you have started > preparing RC0 already. [~vjasani] Can you share more details where you saw this regression and/or if there is any thread I can follow? > Canary can take lot more time If any region (except the first region) starts > with delete markers > > > Key: HBASE-28204 > URL: https://issues.apache.org/jira/browse/HBASE-28204 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: Mihir Monani >Assignee: Mihir Monani >Priority: Major > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7, 2.7.0 > > > In CanaryTool.java, Canary reads only the first row of the region using > [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33] > for any region of the table. Canary uses [Scan with FirstRowKeyFilter for > table > scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530] > If the said region has empty start key (This will only happen when region is > the first region for a table) > With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]- > RawScan was > [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534] > to improve performance for regions which can have high number of delete > markers. Based on currently implementation, [RawScan is only > enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519] > if region has empty start-key (or region is first region for the table). > RawScan doesn't work for rest of the regions in the table except first > region. Also If the region has all the rows or majority of the rows with > delete markers, Get Operation can take a lot of time. This is can cause > timeouts for CanaryTool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[
https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792190#comment-17792190
]
Hudson commented on HBASE-28204:
Results for branch branch-3
[build #94 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/94/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/94/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/94/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/94/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Canary can take lot more time If any region (except the first region) starts
> with delete markers
>
>
> Key: HBASE-28204
> URL: https://issues.apache.org/jira/browse/HBASE-28204
> Project: HBase
> Issue Type: Bug
> Components: canary
>Reporter: Mihir Monani
>Assignee: Mihir Monani
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7, 2.7.0
>
>
> In CanaryTool.java, Canary reads only the first row of the region using
> [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33]
> for any region of the table. Canary uses [Scan with FirstRowKeyFilter for
> table
> scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530]
> If the said region has empty start key (This will only happen when region is
> the first region for a table)
> With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]-
> RawScan was
> [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534]
> to improve performance for regions which can have high number of delete
> markers. Based on currently implementation, [RawScan is only
> enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519]
> if region has empty start-key (or region is first region for the table).
> RawScan doesn't work for rest of the regions in the table except first
> region. Also If the region has all the rows or majority of the rows with
> delete markers, Get Operation can take a lot of time. This is can cause
> timeouts for CanaryTool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[
https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792186#comment-17792186
]
Hudson commented on HBASE-28204:
Results for branch master
[build #955 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/955/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/955/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/955/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/955/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Canary can take lot more time If any region (except the first region) starts
> with delete markers
>
>
> Key: HBASE-28204
> URL: https://issues.apache.org/jira/browse/HBASE-28204
> Project: HBase
> Issue Type: Bug
> Components: canary
>Reporter: Mihir Monani
>Assignee: Mihir Monani
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7, 2.7.0
>
>
> In CanaryTool.java, Canary reads only the first row of the region using
> [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33]
> for any region of the table. Canary uses [Scan with FirstRowKeyFilter for
> table
> scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530]
> If the said region has empty start key (This will only happen when region is
> the first region for a table)
> With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]-
> RawScan was
> [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534]
> to improve performance for regions which can have high number of delete
> markers. Based on currently implementation, [RawScan is only
> enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519]
> if region has empty start-key (or region is first region for the table).
> RawScan doesn't work for rest of the regions in the table except first
> region. Also If the region has all the rows or majority of the rows with
> delete markers, Get Operation can take a lot of time. This is can cause
> timeouts for CanaryTool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-25549) Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms.
[ https://issues.apache.org/jira/browse/HBASE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-25549: -- Release Note: New APIs are added to Admin, AsyncAdmin, and hbase shell to allow modifying a table without reopening all regions. Care should be used in using this API, as regions will be in an inconsistent state until they are all reopened. Whether this matters depends on the change, and some changes are disallowed (such as enabling region replication or adding/removing a column family). Resolution: Fixed Status: Resolved (was: Patch Available) > Provide a switch that allows avoiding reopening all regions when modifying a > table to prevent RIT storms. > - > > Key: HBASE-25549 > URL: https://issues.apache.org/jira/browse/HBASE-25549 > Project: HBase > Issue Type: Improvement > Components: master, shell >Affects Versions: 3.0.0-alpha-1 >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 2.6.0, 3.0.0-beta-1, 2.5.7 > > > Under normal circumstances, modifying a table will cause all regions > belonging to the table to enter RIT. Imagine the following two scenarios： > # Someone entered the wrong configuration （e.g. negative > 'hbase.busy.wait.multiplier.max' value） when altering the table, causing > thousands of online regions to fail to open, leading to online accidents. > # Modify the configuration of a table, but this modification is not urgent, > the regions are not expected to enter RIT immediately. > -'alter_lazy' is a new command to modify a table without reopening any online > regions except those regions were assigned by other threads or split etc.- > > Provide an optional lazy_mode for the alter command to modify the > TableDescriptor without the region entering the RIT. The modification will > take effect when the region is reopened. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-25549) Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms.
[ https://issues.apache.org/jira/browse/HBASE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792184#comment-17792184 ] Bryan Beaudreault commented on HBASE-25549: --- Backport has been pushed to branch-2, branch-2.6, and branch-2.5. Thanks [~gourab.taparia] for the contribution! FYI the original master commit had been missed backporting to branch-3, so I took care of that. Thanks to [~GeorryHuang] for the work here, and everyone else for the interest, input, and reviews! It was nice to see such an active thread of discussion and contribution on this one, sort of reminiscent of the old days :) > Provide a switch that allows avoiding reopening all regions when modifying a > table to prevent RIT storms. > - > > Key: HBASE-25549 > URL: https://issues.apache.org/jira/browse/HBASE-25549 > Project: HBase > Issue Type: Improvement > Components: master, shell >Affects Versions: 3.0.0-alpha-1 >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 2.6.0, 3.0.0-beta-1, 2.5.7 > > > Under normal circumstances, modifying a table will cause all regions > belonging to the table to enter RIT. Imagine the following two scenarios： > # Someone entered the wrong configuration （e.g. negative > 'hbase.busy.wait.multiplier.max' value） when altering the table, causing > thousands of online regions to fail to open, leading to online accidents. > # Modify the configuration of a table, but this modification is not urgent, > the regions are not expected to enter RIT immediately. > -'alter_lazy' is a new command to modify a table without reopening any online > regions except those regions were assigned by other threads or split etc.- > > Provide an optional lazy_mode for the alter command to modify the > TableDescriptor without the region entering the RIT. The modification will > take effect when the region is reopened. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
bbeaudreault merged PR #5549: URL: https://github.com/apache/hbase/pull/5549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-27795: Define RPC API for cache cleaning [hbase]
Apache-HBase commented on PR #5492: URL: https://github.com/apache/hbase/pull/5492#issuecomment-1836661223 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 7s | master passed | | +1 :green_heart: | compile | 4m 18s | master passed | | +1 :green_heart: | checkstyle | 1m 21s | master passed | | +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 5m 25s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 48s | the patch passed | | +1 :green_heart: | compile | 4m 16s | the patch passed | | +1 :green_heart: | cc | 4m 16s | the patch passed | | +1 :green_heart: | javac | 4m 16s | the patch passed | | +1 :green_heart: | checkstyle | 1m 22s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 10m 46s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | hbaseprotoc | 1m 34s | the patch passed | | +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 5m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 51m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5492/7/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5492 | | JIRA Issue | HBASE-27795 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool | | uname | Linux 899bbf7af033 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7dd4d0c532 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 83 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5492/7/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[ https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-28204: - Fix Version/s: 2.7.0 > Canary can take lot more time If any region (except the first region) starts > with delete markers > > > Key: HBASE-28204 > URL: https://issues.apache.org/jira/browse/HBASE-28204 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: Mihir Monani >Assignee: Mihir Monani >Priority: Major > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7, 2.7.0 > > > In CanaryTool.java, Canary reads only the first row of the region using > [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33] > for any region of the table. Canary uses [Scan with FirstRowKeyFilter for > table > scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530] > If the said region has empty start key (This will only happen when region is > the first region for a table) > With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]- > RawScan was > [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534] > to improve performance for regions which can have high number of delete > markers. Based on currently implementation, [RawScan is only > enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519] > if region has empty start-key (or region is first region for the table). > RawScan doesn't work for rest of the regions in the table except first > region. Also If the region has all the rows or majority of the rows with > delete markers, Get Operation can take a lot of time. This is can cause > timeouts for CanaryTool. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
kadirozde commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1412493239 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java: ## @@ -135,11 +136,17 @@ public static List getScannersForStoreFiles(Collection
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
Apache-HBase commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1836650922 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 55s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 49s | branch-2 passed | | +1 :green_heart: | compile | 4m 42s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 39s | branch-2 passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 5m 15s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 4m 44s | the patch passed | | +1 :green_heart: | cc | 4m 44s | the patch passed | | +1 :green_heart: | javac | 4m 44s | the patch passed | | -0 :warning: | checkstyle | 0m 36s | hbase-server: The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) | | -0 :warning: | rubocop | 0m 10s | The patch generated 10 new + 422 unchanged - 5 fixed = 432 total (was 427) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 13m 59s | Patch does not cause any errors with Hadoop 2.10.2 or 3.2.4 3.3.6. | | +1 :green_heart: | hbaseprotoc | 1m 54s | the patch passed | | +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 5m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 43s | The patch does not generate ASF License warnings. | | | | 50m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5549 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool rubocop | | uname | Linux 6e85577e2eb2 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / dbc165a3cf | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | rubocop | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 81 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/6/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 rubocop=1.37.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
kadirozde commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1412480023 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
kadirozde commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1412480023 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine
Re: [PR] HBASE-27795: Define RPC API for cache cleaning [hbase]
Kota-SH commented on PR #5492: URL: https://github.com/apache/hbase/pull/5492#issuecomment-1836635277 @wchevreuil - Could you please review this updated patch and provide feedback? Thanks, Shanmukha K -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28204) Canary can take lot more time If any region (except the first region) starts with delete markers
[
https://issues.apache.org/jira/browse/HBASE-28204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792174#comment-17792174
]
Hudson commented on HBASE-28204:
Results for branch branch-2.5
[build #443 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/443/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/443/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/443/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/443/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/443/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Canary can take lot more time If any region (except the first region) starts
> with delete markers
>
>
> Key: HBASE-28204
> URL: https://issues.apache.org/jira/browse/HBASE-28204
> Project: HBase
> Issue Type: Bug
> Components: canary
>Reporter: Mihir Monani
>Assignee: Mihir Monani
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> In CanaryTool.java, Canary reads only the first row of the region using
> [Get|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L520C33-L520C33]
> for any region of the table. Canary uses [Scan with FirstRowKeyFilter for
> table
> scan|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L530]
> If the said region has empty start key (This will only happen when region is
> the first region for a table)
> With -[HBASE-16091|https://issues.apache.org/jira/browse/HBASE-16091]-
> RawScan was
> [implemented|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519-L534]
> to improve performance for regions which can have high number of delete
> markers. Based on currently implementation, [RawScan is only
> enabled|https://github.com/apache/hbase/blob/23c41560d58cc1353b8a466deacd02dfee9e6743/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L519]
> if region has empty start-key (or region is first region for the table).
> RawScan doesn't work for rest of the regions in the table except first
> region. Also If the region has all the rows or majority of the rows with
> delete markers, Get Operation can take a lot of time. This is can cause
> timeouts for CanaryTool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-27795: Define RPC API for cache cleaning [hbase]
Kota-SH commented on PR #5525: URL: https://github.com/apache/hbase/pull/5525#issuecomment-1836612742 @Apache9 - Thanks for the review. > What will happen if the region is moved back while we are cleaning its blocks in block cache? As of now, we check the region state before calling evictBlocksByHfileName() method in uncacheStaleBlocks(). So we only evict blocks if the region state isClosed/isClosing. If the region is moved back, it will be in a different state, and we do not evict its blocks. > What about other types of BlockCache? Currently, we only support BucketCache and CombinedBlockCache. We can extend this feature to other types of block cache if required. I can create a separate jira to handle those scenarios. > I prefer here we pass an interface / If we do not clean any blocks, just return an empty map? Yes, we pass an empty map if we don't evict any blocks. I made the change to pass an Interface in a separate PR to the master branch (https://github.com/apache/hbase/pull/5492). Could you please review it there? Thanks, Shanmukha K -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
Apache-HBase commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1836586869 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 54s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 39s | branch-2 passed | | +1 :green_heart: | compile | 2m 20s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 33s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 42s | the patch passed | | +1 :green_heart: | compile | 2m 12s | the patch passed | | +1 :green_heart: | javac | 2m 12s | the patch passed | | +1 :green_heart: | shadedjars | 5m 48s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 27s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 29s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 8m 9s | hbase-client in the patch passed. | | -1 :x: | unit | 234m 35s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 6m 57s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 58s | hbase-shell in the patch passed. | | | | 289m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5549 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 65eaf4bf2fc1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 0f63885e7e | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/testReport/ | | Max. process+thread count | 4552 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
Apache-HBase commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1836567752 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 6s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 5s | branch-2 passed | | +1 :green_heart: | compile | 2m 49s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 23s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 41s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 52s | the patch passed | | +1 :green_heart: | compile | 2m 26s | the patch passed | | +1 :green_heart: | javac | 2m 26s | the patch passed | | +1 :green_heart: | shadedjars | 4m 57s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 35s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 8m 18s | hbase-client in the patch passed. | | -1 :x: | unit | 219m 25s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 7m 57s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 5s | hbase-shell in the patch passed. | | | | 274m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5549 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux aec47f0eda8e 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 0f63885e7e | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/testReport/ | | Max. process+thread count | 4414 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28209: Create a jmx metrics to expose the oldWALs directory size [hbase]
Apache-HBase commented on PR #5528: URL: https://github.com/apache/hbase/pull/5528#issuecomment-1836451385 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 32s | master passed | | +1 :green_heart: | compile | 1m 34s | master passed | | +1 :green_heart: | shadedjars | 5m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 3s | master passed | | -0 :warning: | patch | 7m 39s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 19s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed | | +1 :green_heart: | javac | 1m 36s | the patch passed | | +1 :green_heart: | shadedjars | 5m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 4s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 350m 47s | root in the patch failed. | | | | 379m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5528/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5528 | | JIRA Issue | HBASE-28209 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d2e71cbcd399 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7f3e400268 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5528/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5528/5/testReport/ | | Max. process+thread count | 5126 (vs. ulimit of 3) | | modules | C: hbase-common hbase-hadoop-compat hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5528/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28209: Create a jmx metrics to expose the oldWALs directory size [hbase]
Apache-HBase commented on PR #5528: URL: https://github.com/apache/hbase/pull/5528#issuecomment-1836378941 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 50s | master passed | | +1 :green_heart: | compile | 2m 0s | master passed | | +1 :green_heart: | shadedjars | 4m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 47s | master passed | | -0 :warning: | patch | 8m 8s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 1m 57s | the patch passed | | +1 :green_heart: | javac | 1m 57s | the patch passed | | +1 :green_heart: | shadedjars | 4m 52s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 310m 10s | root in the patch passed. | | | | 343m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5528/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5528 | | JIRA Issue | HBASE-28209 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d43c094e0126 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7f3e400268 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5528/5/testReport/ | | Max. process+thread count | 8322 (vs. ulimit of 3) | | modules | C: hbase-common hbase-hadoop-compat hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5528/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28031) TestClusterScopeQuotaThrottle is still failing with broken WAL writer
[
https://issues.apache.org/jira/browse/HBASE-28031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792113#comment-17792113
]
Bryan Beaudreault commented on HBASE-28031:
---
Ok so that one, it looks like the test causing all that spam
(testUserTableClusterScopeQuota) actually succeeds. The job times out during
testUserNamespaceClusterScopeQuota. According to the thread dump at the end,
it's while trying to refresh quota cache:
{code:java}
"Listener at localhost/42451" daemon prio=5 tid=18 runnable
java.lang.Thread.State: RUNNABLE
at javax.security.auth.Subject.getSubject(Subject.java:297)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
at
org.apache.hadoop.hbase.security.User$SecureHadoopUser.(User.java:280)
at org.apache.hadoop.hbase.security.User.getCurrent(User.java:160)
at
org.apache.hadoop.hbase.quotas.ThrottleQuotaTestUtil.triggerCacheRefresh(ThrottleQuotaTestUtil.java:156)
at
org.apache.hadoop.hbase.quotas.ThrottleQuotaTestUtil.triggerUserCacheRefresh(ThrottleQuotaTestUtil.java:109)
at
org.apache.hadoop.hbase.quotas.TestClusterScopeQuotaThrottle.testUserNamespaceClusterScopeQuota(TestClusterScopeQuotaThrottle.java:197)
{code}
Looks like there's a [while loop in
there|https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/ThrottleQuotaTestUtil.java#L150-L184]
which doesn't have a timeout. We should probably update that to use Waiter
with a timeout, so at least the test will fail more usefully.
The while loop triggers the QuotaRefresherChore. According to the thread dump,
that chore is waiting on a future:
{code:java}
"regionserver/jenkins-hbase19:0.Chore.1" daemon prio=5 tid=366 in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
at
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.hadoop.hbase.util.FutureUtils.get(FutureUtils.java:182)
at
org.apache.hadoop.hbase.client.TableOverAsyncTable.get(TableOverAsyncTable.java:193)
at
org.apache.hadoop.hbase.quotas.QuotaTableUtil.doGet(QuotaTableUtil.java:910)
at
org.apache.hadoop.hbase.quotas.QuotaUtil.fetchGlobalQuotas(QuotaUtil.java:375)
at
org.apache.hadoop.hbase.quotas.QuotaUtil.fetchNamespaceQuotas(QuotaUtil.java:342)
at
org.apache.hadoop.hbase.quotas.QuotaCache$QuotaRefresherChore$1.fetchEntries(QuotaCache.java:274)
at
org.apache.hadoop.hbase.quotas.QuotaCache$QuotaRefresherChore.fetch(QuotaCache.java:365)
at
org.apache.hadoop.hbase.quotas.QuotaCache$QuotaRefresherChore.fetchNamespaceQuotaState(QuotaCache.java:266)
at
org.apache.hadoop.hbase.quotas.QuotaCache$QuotaRefresherChore.chore(QuotaCache.java:257)
{code}
This is just a point-in-time, and I don't see any exceptions. I don't know if
that Future is actually blocked, or maybe it returns quickly but not what the
triggerCacheRefresh is looking for. We might need more logging. There's a
LOG.debug("QuotaCache") dump _after_ the while loop. Maybe we need to move it
to _inside_ the while loop.
> TestClusterScopeQuotaThrottle is still failing with broken WAL writer
> -
>
> Key: HBASE-28031
> URL: https://issues.apache.org/jira/browse/HBASE-28031
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> {noformat}
> 2023-08-17T10:47:31,026 WARN [regionserver/jenkins-hbase19:0.logRoller {}]
> asyncfs.FanOutOneBlockAsyncDFSOutputHelper(515): create fan-out dfs output
> /user/jenkins/test-data/bb8017fa-92f5-92c9-2f1d-aa9b90cf4b80/WALs/jenkins-hbase19.apache.org,43363,1692269230784/jenkins-hbase19.apache.org%2C43363%2C1692269230784.meta.1692433272886.meta
> failed, retry = 0
> org.apache.hadoop.ipc.RemoteException: File
> /user/jenkins/test-data/bb8017fa-92f5-92c9-2f1d-aa9b90cf4b80/WALs/jenkins-hbase19.apache.org,43363,1692269230784/jenkins-hbase19.apache.org%2C43363%2C1692269230784.meta.1692433272886.meta
> could only be written to 0 of the 1 minReplication nodes. There are 2
> datanode(s) running and 2 node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2276)
> at
> org.apache.hadoop.h
Re: [PR] HBASE-28212 Addendum fix TestShell [hbase]
Apache9 merged PR #: URL: https://github.com/apache/hbase/pull/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28212 Addendum fix TestShell [hbase]
Apache-HBase commented on PR #: URL: https://github.com/apache/hbase/pull/#issuecomment-1836304743 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 15s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 50s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 52s | hbase-shell in the patch passed. | | | | 15m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/ | | Optional Tests | unit | | uname | Linux c1969bb5f84d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 02217e1abc | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-/1/testReport/ | | Max. process+thread count | 1495 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28212 Addendum fix TestShell [hbase]
Apache-HBase commented on PR #: URL: https://github.com/apache/hbase/pull/#issuecomment-1836302779 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 4s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 27s | hbase-shell in the patch passed. | | | | 14m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/ | | Optional Tests | unit | | uname | Linux f429852f3e3b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 02217e1abc | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-/1/testReport/ | | Max. process+thread count | 1540 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28212 Addendum fix TestShell [hbase]
Apache-HBase commented on PR #: URL: https://github.com/apache/hbase/pull/#issuecomment-1836285948 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | spotless | 0m 48s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | -0 :warning: | rubocop | 0m 6s | The patch generated 6 new + 51 unchanged - 1 fixed = 57 total (was 52) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | spotless | 0m 37s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | | | | 3m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/ | | Optional Tests | dupname asflicense spotless rubocop | | uname | Linux ec26980dd2a2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 02217e1abc | | rubocop | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-/1/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 45 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-/1/console | | versions | git=2.34.1 maven=3.8.6 rubocop=1.37.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
Apache-HBase commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1836266082 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 24s | branch-2 passed | | +1 :green_heart: | compile | 7m 58s | branch-2 passed | | +1 :green_heart: | checkstyle | 2m 7s | branch-2 passed | | +1 :green_heart: | spotless | 1m 13s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 8m 51s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 28s | the patch passed | | +1 :green_heart: | compile | 7m 1s | the patch passed | | +1 :green_heart: | cc | 7m 1s | the patch passed | | +1 :green_heart: | javac | 7m 1s | the patch passed | | -0 :warning: | checkstyle | 1m 2s | hbase-server: The patch generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) | | -0 :warning: | rubocop | 0m 17s | The patch generated 10 new + 422 unchanged - 5 fixed = 432 total (was 427) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 21m 54s | Patch does not cause any errors with Hadoop 2.10.2 or 3.2.4 3.3.6. | | +1 :green_heart: | hbaseprotoc | 2m 38s | the patch passed | | -1 :x: | spotless | 0m 27s | patch has 49 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 9m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 1s | The patch does not generate ASF License warnings. | | | | 80m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5549 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool rubocop | | uname | Linux 6e8385aeb028 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 0f63885e7e | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | rubocop | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 81 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/5/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 rubocop=1.37.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] HBASE-28212 Addendum fix TestShell [hbase]
Apache9 opened a new pull request, #: URL: https://github.com/apache/hbase/pull/ We added a new field in Procedure so the json output is also changed thus we need to change the assertion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28031) TestClusterScopeQuotaThrottle is still failing with broken WAL writer
[
https://issues.apache.org/jira/browse/HBASE-28031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792086#comment-17792086
]
Duo Zhang commented on HBASE-28031:
---
https://nightlies.apache.org/hbase/HBase-Flaky-Tests/master/14210/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.quotas.TestClusterScopeQuotaThrottle-output.txt
> TestClusterScopeQuotaThrottle is still failing with broken WAL writer
> -
>
> Key: HBASE-28031
> URL: https://issues.apache.org/jira/browse/HBASE-28031
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> {noformat}
> 2023-08-17T10:47:31,026 WARN [regionserver/jenkins-hbase19:0.logRoller {}]
> asyncfs.FanOutOneBlockAsyncDFSOutputHelper(515): create fan-out dfs output
> /user/jenkins/test-data/bb8017fa-92f5-92c9-2f1d-aa9b90cf4b80/WALs/jenkins-hbase19.apache.org,43363,1692269230784/jenkins-hbase19.apache.org%2C43363%2C1692269230784.meta.1692433272886.meta
> failed, retry = 0
> org.apache.hadoop.ipc.RemoteException: File
> /user/jenkins/test-data/bb8017fa-92f5-92c9-2f1d-aa9b90cf4b80/WALs/jenkins-hbase19.apache.org,43363,1692269230784/jenkins-hbase19.apache.org%2C43363%2C1692269230784.meta.1692433272886.meta
> could only be written to 0 of the 1 minReplication nodes. There are 2
> datanode(s) running and 2 node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2276)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2820)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:910)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:577)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:549)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:518)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1035)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:963)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2960)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612)
> ~[hadoop-common-3.2.4.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1558)
> ~[hadoop-common-3.2.4.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1455)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:231)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> ~[hadoop-common-3.2.4.jar:?]
> at com.sun.proxy.$Proxy41.addBlock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:520)
> ~[hadoop-hdfs-client-3.2.4.jar:?]
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_362]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:433)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
> ~[hadoop-common-3.2.4.jar:?]
> at com.sun.proxy.$Proxy42.addBlock(Unknown Source) ~[?:?]
> at sun.reflect.Genera
[jira] [Commented] (HBASE-28031) TestClusterScopeQuotaThrottle is still failing with broken WAL writer
[
https://issues.apache.org/jira/browse/HBASE-28031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792083#comment-17792083
]
Bryan Beaudreault commented on HBASE-28031:
---
Is the nightlies browser really slow? I can load the main index at
[https://nightlies.apache.org/hbase/HBase-Flaky-Tests/] but can't click into
master, it just hangs forever. Can you link me a more recent failure?
> TestClusterScopeQuotaThrottle is still failing with broken WAL writer
> -
>
> Key: HBASE-28031
> URL: https://issues.apache.org/jira/browse/HBASE-28031
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> {noformat}
> 2023-08-17T10:47:31,026 WARN [regionserver/jenkins-hbase19:0.logRoller {}]
> asyncfs.FanOutOneBlockAsyncDFSOutputHelper(515): create fan-out dfs output
> /user/jenkins/test-data/bb8017fa-92f5-92c9-2f1d-aa9b90cf4b80/WALs/jenkins-hbase19.apache.org,43363,1692269230784/jenkins-hbase19.apache.org%2C43363%2C1692269230784.meta.1692433272886.meta
> failed, retry = 0
> org.apache.hadoop.ipc.RemoteException: File
> /user/jenkins/test-data/bb8017fa-92f5-92c9-2f1d-aa9b90cf4b80/WALs/jenkins-hbase19.apache.org,43363,1692269230784/jenkins-hbase19.apache.org%2C43363%2C1692269230784.meta.1692433272886.meta
> could only be written to 0 of the 1 minReplication nodes. There are 2
> datanode(s) running and 2 node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2276)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2820)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:910)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:577)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:549)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:518)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1035)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:963)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2960)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612)
> ~[hadoop-common-3.2.4.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1558)
> ~[hadoop-common-3.2.4.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1455)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:231)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> ~[hadoop-common-3.2.4.jar:?]
> at com.sun.proxy.$Proxy41.addBlock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:520)
> ~[hadoop-hdfs-client-3.2.4.jar:?]
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_362]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:433)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
> ~[hadoop-common-3.2.4.jar:?]
> at com.sun.proxy.$Proxy42.addB
[jira] [Reopened] (HBASE-28212) Do not need to maintain rollback step when root procedure does not support rollback
[ https://issues.apache.org/jira/browse/HBASE-28212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-28212: --- This seems breaks TestShell. Let me confirm. > Do not need to maintain rollback step when root procedure does not support > rollback > --- > > Key: HBASE-28212 > URL: https://issues.apache.org/jira/browse/HBASE-28212 > Project: HBase > Issue Type: Improvement > Components: master, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7 > > > In HBASE-28210, we need to synchronized on RootProcedureState when updating > procedure store, which will decrease the performance a lot when there are > lots of sub procedures for a procedure, such as SCP, so it will impact MTTR. > But actually, SCP does not support rollback, so I do not think we need to > maintain this information. If so, I think we do not need the synchronization > added in HBASE-28210, at least for SCP then. This could get the performance > back. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28031) TestClusterScopeQuotaThrottle is still failing with broken WAL writer
[
https://issues.apache.org/jira/browse/HBASE-28031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17792074#comment-17792074
]
Duo Zhang commented on HBASE-28031:
---
Because we use a manual EnvironmentEdge for getting timestamp. I have fixed
this problem in the last merge PR for this issue, where I limited the scope of
the EnvironmentEdge to only take effect in quotas package.
You can see newer failure logs, which should not have the JVM pauses then.
Thanks.
> TestClusterScopeQuotaThrottle is still failing with broken WAL writer
> -
>
> Key: HBASE-28031
> URL: https://issues.apache.org/jira/browse/HBASE-28031
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> {noformat}
> 2023-08-17T10:47:31,026 WARN [regionserver/jenkins-hbase19:0.logRoller {}]
> asyncfs.FanOutOneBlockAsyncDFSOutputHelper(515): create fan-out dfs output
> /user/jenkins/test-data/bb8017fa-92f5-92c9-2f1d-aa9b90cf4b80/WALs/jenkins-hbase19.apache.org,43363,1692269230784/jenkins-hbase19.apache.org%2C43363%2C1692269230784.meta.1692433272886.meta
> failed, retry = 0
> org.apache.hadoop.ipc.RemoteException: File
> /user/jenkins/test-data/bb8017fa-92f5-92c9-2f1d-aa9b90cf4b80/WALs/jenkins-hbase19.apache.org,43363,1692269230784/jenkins-hbase19.apache.org%2C43363%2C1692269230784.meta.1692433272886.meta
> could only be written to 0 of the 1 minReplication nodes. There are 2
> datanode(s) running and 2 node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2276)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2820)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:910)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:577)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:549)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:518)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1035)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:963)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2960)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612)
> ~[hadoop-common-3.2.4.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1558)
> ~[hadoop-common-3.2.4.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1455)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:231)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> ~[hadoop-common-3.2.4.jar:?]
> at com.sun.proxy.$Proxy41.addBlock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:520)
> ~[hadoop-hdfs-client-3.2.4.jar:?]
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_362]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:433)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
>
[jira] [Resolved] (HBASE-20433) HBase Export Snapshot utility does not close FileSystem instances
[ https://issues.apache.org/jira/browse/HBASE-20433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault resolved HBASE-20433. --- Resolution: Duplicate Resolving this as a duplicate of HBASE-28222 where I fixed this as best I could, by re-enabling the cache (by reverting HBASE-12819). ExportSnapshot is designed to be run as a standalone job. If someone plans to run ExportSnapshot many times in a single process, they should run FileSystem.closeAll() between each run. This is not safe for ExportSnapshot itself to do, since it could inadvertently close FileSystem objects referenced elsewhere in the user code. See HBASE-28222 for more details. > HBase Export Snapshot utility does not close FileSystem instances > - > > Key: HBASE-20433 > URL: https://issues.apache.org/jira/browse/HBASE-20433 > Project: HBase > Issue Type: Bug > Components: Client, Filesystem Integration, snapshots >Affects Versions: 1.2.6, 1.4.3 >Reporter: Voyta >Priority: Major > > It seems org.apache.hadoop.hbase.snapshot.ExportSnapshot disallows FileSystem > instance caching. > When verifySnapshot method is being run it calls often methods like > org.apache.hadoop.hbase.util.FSUtils#getRootDir that instantiate FileSystem > but never calls org.apache.hadoop.fs.FileSystem#close method. This behaviour > allows allocation of unwanted objects potentially causing memory leaks. > Related issue: https://issues.apache.org/jira/browse/HADOOP-15392 > > Expectation: > * HBase should properly release/close all objects, especially FileSystem > instances. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28222) Leak in ExportSnapshot during verifySnapshot on S3A
[ https://issues.apache.org/jira/browse/HBASE-28222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault resolved HBASE-28222. --- Fix Version/s: 2.6.0 3.0.0-beta-1 Release Note: ExportSnapshot now uses FileSystems from the global FileSystem cache, and as such does not close those FileSystems when it finishes. If users plan to run ExportSnapshot over and over in a single process for different FileSystem urls, they should run FileSystem.closeAll() between runs. See JIRA for details. Assignee: Bryan Beaudreault Resolution: Fixed Pushed to master, branch-3, branch-2, branch-2.6. Thanks for the review [~wchevreuil]! I did not push to older branches, even though this is a bug. It might be an unexpected change, but we can if there is a desire. > Leak in ExportSnapshot during verifySnapshot on S3A > --- > > Key: HBASE-28222 > URL: https://issues.apache.org/jira/browse/HBASE-28222 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Fix For: 2.6.0, 3.0.0-beta-1 > > > Each S3AFileSystem creates an S3AInstrumentation and various metrics sources, > with no real way to disable that. In HADOOP-18526, a bug was fixed so that > these are not leaked. But in order to use that, you must call > S3AFileSystem.close() when done. > In ExportSnapshot, ever since HBASE-12819 we set fs.impl.disable.cache to > true. It looks like that was added in order to prevent conflicting calls to > close() between mapper and main thread when running in a single JVM. > When verifySnapshot is enabled, SnapshotReferenceUtil.verifySnapshot iterates > all storefiles (could be many thousands) and calls > SnapshotReferenceUtil.verifyStoreFile on them. verifyStoreFile makes a number > of static calls which end up in CommonFSUtils.getRootDir, which does > Path.getFileSystem(). > Since the FS cache is disabled, every single call to Path.getFileSystem() > creates a new FileSystem instance. That FS is short lived, and gets GC'd. But > in the case of S3AFileSystem, this leaks all of the metrics stuff. > We have two easy possible fixes: > # Only set fs.impl.disable.cache when running hadoop in local mode, since > that was the original problem. > # When calling verifySnapshot, create a new Configuration which does not > include the fs.impl.disable.cache setting. > I tested out #2 in my environment and it fixed the leak. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
gourabtaparia commented on code in PR #5549:
URL: https://github.com/apache/hbase/pull/5549#discussion_r1412113806
##
hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java:
##
@@ -1535,7 +1535,16 @@ default void modifyTable(TableName tableName,
TableDescriptor td) throws IOExcep
throw new IllegalArgumentException("the specified table name '" +
tableName
+ "' doesn't match with the HTD one: " + td.getTableName());
}
-modifyTable(td);
+modifyTable(td, true);
+ }
+
+ /**
+ * Modify an existing table, more IRB friendly version.
+ * @param td modified description of the table
Review Comment:
@bbeaudreault Done. Thanks for pointing out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
gourabtaparia commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1836134087 @bbeaudreault Thanks for pointing this out, i have pushed a commit adding the related javadoc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28222 Leak in ExportSnapshot during verifySnapshot on S3A [hbase]
bbeaudreault merged PR #5554: URL: https://github.com/apache/hbase/pull/5554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
bbeaudreault commented on code in PR #5549:
URL: https://github.com/apache/hbase/pull/5549#discussion_r1412097064
##
hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java:
##
@@ -1535,7 +1535,16 @@ default void modifyTable(TableName tableName,
TableDescriptor td) throws IOExcep
throw new IllegalArgumentException("the specified table name '" +
tableName
+ "' doesn't match with the HTD one: " + td.getTableName());
}
-modifyTable(td);
+modifyTable(td, true);
+ }
+
+ /**
+ * Modify an existing table, more IRB friendly version.
+ * @param td modified description of the table
Review Comment:
can you please add javadoc `@param reopenRegions` tag for this and below
overload?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
gourabtaparia commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1836093101 Hi @bbeaudreault @GeorryHuang Retriggered the build again, and all the previous failed test succeeded, and this time a new test TestClientScannerTimeouts.testNormalScanTimeoutOnNextAsync is failing. This looks not related to the changes made. Can you guys please help to get this merged ? Or should i continue to re-trigger it until all the tests succeed ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28209: Create a jmx metrics to expose the oldWALs directory size [hbase]
Apache-HBase commented on PR #5528: URL: https://github.com/apache/hbase/pull/5528#issuecomment-1835940180 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 41s | master passed | | +1 :green_heart: | compile | 4m 50s | master passed | | +1 :green_heart: | checkstyle | 1m 6s | master passed | | +0 :ok: | refguide | 2m 48s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotless | 0m 41s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 9m 39s | master passed | | -0 :warning: | patch | 7m 45s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 41s | the patch passed | | +1 :green_heart: | compile | 4m 44s | the patch passed | | +1 :green_heart: | javac | 4m 44s | the patch passed | | +1 :green_heart: | checkstyle | 1m 6s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 2m 13s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 9m 22s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 10m 8s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 41s | The patch does not generate ASF License warnings. | | | | 61m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5528/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5528 | | JIRA Issue | HBASE-28209 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile refguide xml | | uname | Linux 968724d4c239 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7f3e400268 | | Default Java | Eclipse Adoptium-11.0.17+8 | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-5528/5/yetus-general-check/output/branch-site/book.html | | refguide | https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-5528/5/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 176 (vs. ulimit of 3) | | modules | C: hbase-common hbase-hadoop-compat hbase-server . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5528/5/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
Apache-HBase commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1835913691 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 35s | branch-2 passed | | +1 :green_heart: | compile | 1m 49s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 13s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 14s | the patch passed | | +1 :green_heart: | compile | 1m 48s | the patch passed | | +1 :green_heart: | javac | 1m 48s | the patch passed | | +1 :green_heart: | shadedjars | 5m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 24s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 7m 53s | hbase-client in the patch passed. | | -1 :x: | unit | 225m 31s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 7m 25s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 55s | hbase-shell in the patch passed. | | | | 276m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5549 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 992dca863b6c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 0f63885e7e | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/4/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/4/testReport/ | | Max. process+thread count | 4671 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25549 Provide a switch that allows avoiding reopening all regions when modifying a table to prevent RIT storms [hbase]
Apache-HBase commented on PR #5549: URL: https://github.com/apache/hbase/pull/5549#issuecomment-1835909767 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 8s | branch-2 passed | | +1 :green_heart: | compile | 2m 59s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 34s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 7s | the patch passed | | +1 :green_heart: | compile | 2m 26s | the patch passed | | +1 :green_heart: | javac | 2m 26s | the patch passed | | +1 :green_heart: | shadedjars | 4m 57s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 35s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 8m 17s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 217m 40s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 7m 50s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 4s | hbase-shell in the patch passed. | | | | 273m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5549 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d1d1dd376599 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 0f63885e7e | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/4/testReport/ | | Max. process+thread count | 4697 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5549/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-28203) Graceful shutdown of active hmaster
[ https://issues.apache.org/jira/browse/HBASE-28203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Kumar reassigned HBASE-28203: --- Assignee: Rahul Kumar > Graceful shutdown of active hmaster > --- > > Key: HBASE-28203 > URL: https://issues.apache.org/jira/browse/HBASE-28203 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Rahul Kumar >Priority: Major > > We recently had an operational incident due to bad interplay between an > ongoing maintenance of regionservers, and a new maintenance of hmasters. We'd > been running a rolling restart of regionservers. Someone unknowingly started > a rolling restart of hmasters. This caused long RITs on a few clusters, what > happened was: > * regionserver restart executed a region move > * hmaster saw it and started TSRP, sending request to RS to close region > * immediately after, the hmaster stopped due to that rolling restart > * the regionserver saw the close request, closed the region, and tried to > report state back to the hmaster. > * This spammed tons of failures for the next 30+ secs while the new hmaster > became active. > * Finally the new hmaster started up and recovered the state and the RIT > finished. > Long RITs are really painful and effectively downtime for hbase. I think we > should have a shutdown hook on the hmaster which: > * Sets state so new move requests are rejected > * Waits for any existing move TSRP to finish > * Shuts down -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-25972 Dual File Compactor [hbase]
wchevreuil commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1411894932 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java: ## @@ -135,11 +136,17 @@ public static List getScannersForStoreFiles(Collection sortedFiles? Won't you get a NPE on lines #153/155? ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DualFileStoreEngine.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import java.io.IOException; +import java.util.List; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.CellComparator; +import org.apache.hadoop.hbase.regionserver.compactions.CompactionContext; +import org.apache.hadoop.hbase.regionserver.compactions.DualFileCompactor; +import org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy; +import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController; +import org.apache.hadoop.hbase.security.User; +import org.apache.hadoop.hbase.util.ReflectionUtils; +import org.apache.yetus.audience.InterfaceAudience; + +/** + * HBASE-25972 This store engine allows us to store data in two files, + * one for the latest put cells and the other for the rest of the cells (i.e., + * older put cells and delete markers). + */ +@InterfaceAudience.Private +public class DualFileStoreEngine extends StoreEngine Or perhaps its fine as is because it would be great to get this ported till 2.5 release line? We may allow that on the brackport for 2.5 only, but I don't think this would justify adding such technical debit in master branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-28221) Introduce regionserver metric for delayed flushes
[
https://issues.apache.org/jira/browse/HBASE-28221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Kumar reassigned HBASE-28221:
---
Assignee: Rahul Kumar
> Introduce regionserver metric for delayed flushes
> -
>
> Key: HBASE-28221
> URL: https://issues.apache.org/jira/browse/HBASE-28221
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.4.17, 2.5.6
>Reporter: Viraj Jasani
>Assignee: Rahul Kumar
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> If compaction is disabled temporarily to allow stabilizing hdfs load, we can
> forget re-enabling the compaction. This can result into flushes getting
> delayed for "hbase.hstore.blockingWaitTime" time (90s). While flushes do
> happen eventually after waiting for max blocking time, it is important to
> realize that any cluster cannot function well with compaction disabled for
> significant amount of time.
>
> We would also block any write requests until region is flushed (90+ sec, by
> default):
> {code:java}
> 2023-11-27 20:40:52,124 WARN [,queue=18,port=60020] regionserver.HRegion -
> Region is too busy due to exceeding memstore size limit.
> org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit,
> regionName=table1,1699923733811.4fd5e52e2133df1e347f32c646f23ab4.,
> server=server-1,60020,1699421714454, memstoreSize=1073820928,
> blockingMemStoreSize=1073741824
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4200)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3264)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3215)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:895)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2524)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36812)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2432)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291)
> {code}
>
> Delayed flush logs:
> {code:java}
> LOG.warn("{} has too many store files({}); delaying flush up to {} ms",
> region.getRegionInfo().getEncodedName(), getStoreFileCount(region),
> this.blockingWaitTime); {code}
> Suggestion: Introduce regionserver metric (MetricsRegionServerSource) for the
> num of flushes getting delayed due to too many store files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
