[jira] [Commented] (HBASE-28326) All nightly jobs are failing
[
https://issues.apache.org/jira/browse/HBASE-28326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810244#comment-17810244
]
Duo Zhang commented on HBASE-28326:
---
{code}
"${WORKSPACE}/component/dev-support/jenkins-scripts/cache-apache-project-artifact.sh"
\
--working-dir "${WORKSPACE}/downloads-yetus" \
--keys 'https://www.apache.org/dist/yetus/KEYS' \
--verify-tar-gz \
"${WORKSPACE}/yetus-${YETUS_RELEASE}-bin.tar.gz" \
"yetus/${YETUS_RELEASE}/apache-yetus-${YETUS_RELEASE}-bin.tar.gz"
{code}
Seems we should use download.a.o instead of dist here. I tried to access
'https://www.apache.org/dist/yetus/KEYS' through browser and it just redirected
to the main page of download.a.o.
> All nightly jobs are failing
>
>
> Key: HBASE-28326
> URL: https://issues.apache.org/jira/browse/HBASE-28326
> Project: HBase
> Issue Type: Bug
> Components: jenkins
>Reporter: Duo Zhang
>Priority: Major
>
> https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/991/console
> Failed in yetus install stage, seems to be related to gpg.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28326) All nightly jobs are failing
Duo Zhang created HBASE-28326: - Summary: All nightly jobs are failing Key: HBASE-28326 URL: https://issues.apache.org/jira/browse/HBASE-28326 Project: HBase Issue Type: Bug Components: jenkins Reporter: Duo Zhang https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/991/console Failed in yetus install stage, seems to be related to gpg. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28271) Infinite waiting on lock acquisition by snapshot can result in unresponsive master
[
https://issues.apache.org/jira/browse/HBASE-28271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810213#comment-17810213
]
Hudson commented on HBASE-28271:
Results for branch branch-2
[build #971 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/971/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/971/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/971/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/971/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/971/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Infinite waiting on lock acquisition by snapshot can result in unresponsive
> master
> --
>
> Key: HBASE-28271
> URL: https://issues.apache.org/jira/browse/HBASE-28271
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.7
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 2.6.0, 2.5.8, 3.0.0-beta-2
>
> Attachments: image.png
>
>
> When a region is stuck in transition for significant time, any attempt to
> take snapshot on the table would keep master handler thread in forever
> waiting state. As part of the creating snapshot on enabled or disabled table,
> in order to get the table level lock, LockProcedure is executed but if any
> region of the table is in transition, LockProcedure could not be executed by
> the snapshot handler, resulting in forever waiting until the region
> transition is completed, allowing the table level lock to be acquired by the
> snapshot handler.
> In cases where a region stays in RIT for considerable time, if enough
> attempts are made by the client to create snapshots on the table, it can
> easily exhaust all handler threads, leading to potentially unresponsive
> master. Attached a sample thread dump.
> Proposal: The snapshot handler should not stay stuck forever if it cannot
> take table level lock, it should fail-fast.
> !image.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28302) Add tracking of fs read times in ScanMetrics and slow logs
[
https://issues.apache.org/jira/browse/HBASE-28302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810214#comment-17810214
]
Hudson commented on HBASE-28302:
Results for branch branch-2
[build #971 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/971/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/971/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/971/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/971/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/971/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add tracking of fs read times in ScanMetrics and slow logs
> --
>
> Key: HBASE-28302
> URL: https://issues.apache.org/jira/browse/HBASE-28302
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Fix For: 2.6.0, 3.0.0-beta-2
>
>
> We've had this in our production for a while, and it's useful info to have.
> We already track FS read times in
> [HFileBlock|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1828-L1831C10].
> We can project that into the ScanMetrics instance and slow log pretty
> easily. It is also helpful to add a slow.fs.read.threshold, over which we log
> a warn
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28216) HDFS erasure coding support for table data dirs
[
https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810193#comment-17810193
]
ASF GitHub Bot commented on HBASE-28216:
Apache-HBase commented on PR #5591:
URL: https://github.com/apache/hbase/pull/5591#issuecomment-1907359918
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| +0 :ok: | reexec | 3m 42s | Docker mode activated. |
| -0 :warning: | yetus | 0m 5s | Unprocessed flag(s):
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list
--whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2 Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 58s | branch-2 passed |
| +1 :green_heart: | compile | 2m 3s | branch-2 passed |
| +1 :green_heart: | shadedjars | 5m 15s | branch has no errors when
building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 2s | branch-2 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 50s | the patch passed |
| +1 :green_heart: | compile | 2m 2s | the patch passed |
| +1 :green_heart: | javac | 2m 2s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 11s | patch has no errors when
building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | hbase-protocol-shaded in the patch
passed. |
| +1 :green_heart: | unit | 8m 6s | hbase-client in the patch passed.
|
| -1 :x: | unit | 228m 10s | hbase-server in the patch failed. |
| +1 :green_heart: | unit | 8m 3s | hbase-shell in the patch passed. |
| | | 276m 22s | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
|
| GITHUB PR | https://github.com/apache/hbase/pull/5591 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 3debd6bee7e9 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23
14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / 5488acfeaa |
| Default Java | Eclipse Adoptium-11.0.17+8 |
| unit |
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
|
| Test Results |
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/testReport/
|
| Max. process+thread count | 4418 (vs. ulimit of 3) |
| modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell
U: . |
| Console output |
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/console
|
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> HDFS erasure coding support for table data dirs
> ---
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy {}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the reque
Re: [PR] HBASE-28216 HDFS erasure coding support for table data dirs [hbase]
Apache-HBase commented on PR #5591: URL: https://github.com/apache/hbase/pull/5591#issuecomment-1907359918 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 58s | branch-2 passed | | +1 :green_heart: | compile | 2m 3s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 2s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 50s | the patch passed | | +1 :green_heart: | compile | 2m 2s | the patch passed | | +1 :green_heart: | javac | 2m 2s | the patch passed | | +1 :green_heart: | shadedjars | 5m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 35s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 8m 6s | hbase-client in the patch passed. | | -1 :x: | unit | 228m 10s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 8m 3s | hbase-shell in the patch passed. | | | | 276m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5591 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3debd6bee7e9 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 5488acfeaa | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/testReport/ | | Max. process+thread count | 4418 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28216) HDFS erasure coding support for table data dirs
[
https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810192#comment-17810192
]
ASF GitHub Bot commented on HBASE-28216:
Apache-HBase commented on PR #5591:
URL: https://github.com/apache/hbase/pull/5591#issuecomment-1907358582
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s):
--brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list
--whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2 Compile Tests _ |
| +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 2m 27s | branch-2 passed |
| +1 :green_heart: | compile | 1m 45s | branch-2 passed |
| +1 :green_heart: | shadedjars | 4m 37s | branch has no errors when
building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 3s | branch-2 passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 18s | the patch passed |
| +1 :green_heart: | compile | 1m 44s | the patch passed |
| +1 :green_heart: | javac | 1m 44s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 39s | patch has no errors when
building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 29s | hbase-protocol-shaded in the patch
passed. |
| +1 :green_heart: | unit | 8m 4s | hbase-client in the patch passed.
|
| -1 :x: | unit | 230m 23s | hbase-server in the patch failed. |
| +1 :green_heart: | unit | 9m 52s | hbase-shell in the patch passed. |
| | | 274m 19s | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
|
| GITHUB PR | https://github.com/apache/hbase/pull/5591 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux a6ab2979596c 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5
13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2 / 5488acfeaa |
| Default Java | Temurin-1.8.0_352-b08 |
| unit |
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt
|
| Test Results |
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/testReport/
|
| Max. process+thread count | 4452 (vs. ulimit of 3) |
| modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell
U: . |
| Console output |
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/console
|
| versions | git=2.34.1 maven=3.8.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> HDFS erasure coding support for table data dirs
> ---
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy {}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the requested pol
Re: [PR] HBASE-28216 HDFS erasure coding support for table data dirs [hbase]
Apache-HBase commented on PR #5591: URL: https://github.com/apache/hbase/pull/5591#issuecomment-1907358582 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 27s | branch-2 passed | | +1 :green_heart: | compile | 1m 45s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 18s | the patch passed | | +1 :green_heart: | compile | 1m 44s | the patch passed | | +1 :green_heart: | javac | 1m 44s | the patch passed | | +1 :green_heart: | shadedjars | 4m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 29s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 8m 4s | hbase-client in the patch passed. | | -1 :x: | unit | 230m 23s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 9m 52s | hbase-shell in the patch passed. | | | | 274m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5591 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a6ab2979596c 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 5488acfeaa | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/testReport/ | | Max. process+thread count | 4452 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28184) Tailing the WAL is very slow if there are multiple peers.
[
https://issues.apache.org/jira/browse/HBASE-28184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810158#comment-17810158
]
ASF GitHub Bot commented on HBASE-28184:
shahrs87 commented on code in PR #5521:
URL: https://github.com/apache/hbase/pull/5521#discussion_r1464207388
##
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java:
##
@@ -259,10 +259,11 @@ private boolean readNextEntryAndRecordReaderPosition()
throws IOException {
Entry readEntry = reader.next();
long readerPos = reader.getPosition();
OptionalLong fileLength;
-if (logQueue.getQueueSize(walGroupId) > 1) {
+if (logQueue.getQueueSize(walGroupId) > 2) {
Review Comment:
@Apache9 ^^
> Tailing the WAL is very slow if there are multiple peers.
> -
>
> Key: HBASE-28184
> URL: https://issues.apache.org/jira/browse/HBASE-28184
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.7
>
>
> Noticed in one of our production clusters which has 4 peers.
> Due to sudden ingestion of data, the size of log queue increased to a peak of
> 506. We have configured log roll size to 256 MB. Most of the edits in the WAL
> were from a table for which replication is disabled.
> So all ReplicationSourceWALReader thread had to do was to replay the WAL and
> NOT replicate them. Still it took 12 hours to drain the queue.
> Took few jstacks and found that ReplicationSourceWALReader was waiting to
> acquire rollWriterLock
> [here|https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java#L1231]
> {noformat}
> "regionserver/,1" #1036 daemon prio=5 os_prio=0 tid=0x7f44b374e800
> nid=0xbd7f waiting on condition [0x7f37b4d19000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f3897a3e150> (a
> java.util.concurrent.locks.ReentrantLock$FairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:872)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1202)
> at
> java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:228)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.getLogFileSizeIfBeingWritten(AbstractFSWAL.java:1102)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$null$0(WALProvider.java:128)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$177/1119730685.apply(Unknown
> Source)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)
> at
> java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
> at
> java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:536)
> at
> org.apache.hadoop.hbase.wal.WALProvider.lambda$getWALFileLengthProvider$2(WALProvider.java:129)
> at
> org.apache.hadoop.hbase.wal.WALProvider$$Lambda$140/1246380717.getLogFileSizeIfBeingWritten(Unknown
> Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:260)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
> at
> org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:222)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(Replicati
Re: [PR] HBASE-28184 Addendum PR [hbase]
shahrs87 commented on code in PR #5521:
URL: https://github.com/apache/hbase/pull/5521#discussion_r1464207388
##
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java:
##
@@ -259,10 +259,11 @@ private boolean readNextEntryAndRecordReaderPosition()
throws IOException {
Entry readEntry = reader.next();
long readerPos = reader.getPosition();
OptionalLong fileLength;
-if (logQueue.getQueueSize(walGroupId) > 1) {
+if (logQueue.getQueueSize(walGroupId) > 2) {
Review Comment:
@Apache9 ^^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-28216) HDFS erasure coding support for table data dirs
[ https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810145#comment-17810145 ] ASF GitHub Bot commented on HBASE-28216: Apache-HBase commented on PR #5591: URL: https://github.com/apache/hbase/pull/5591#issuecomment-1907152047 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 18s | branch-2 passed | | +1 :green_heart: | compile | 4m 11s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 4s | branch-2 passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 4m 16s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 52s | the patch passed | | +1 :green_heart: | compile | 4m 4s | the patch passed | | +1 :green_heart: | cc | 4m 4s | the patch passed | | +1 :green_heart: | javac | 4m 4s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-server: The patch generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) | | -0 :warning: | rubocop | 0m 8s | The patch generated 8 new + 701 unchanged - 3 fixed = 709 total (was 704) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 46s | Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6. | | +1 :green_heart: | hbaseprotoc | 1m 28s | the patch passed | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 4m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 41m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5591 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool rubocop | | uname | Linux f3b39c633c58 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 5488acfeaa | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | rubocop | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 rubocop=1.37.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. > HDFS erasure coding support for table data dirs > --- > > Key: HBASE-28216 > URL: https://issues.apache.org/jira/browse/HBASE-28216 > Project: HBase > Issue Type: New Feature >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Labels: patch-available > > [Erasure > coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html] > (EC) is a hadoop-3 feature which can drastically reduce storage > requirements, at the expense of locality. At my company we have a few hbase > clusters which are extremely data dense and take mostly write traffic, fewer > reads (cold data). We'd li
Re: [PR] HBASE-28216 HDFS erasure coding support for table data dirs [hbase]
Apache-HBase commented on PR #5591: URL: https://github.com/apache/hbase/pull/5591#issuecomment-1907152047 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 18s | branch-2 passed | | +1 :green_heart: | compile | 4m 11s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 4s | branch-2 passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 4m 16s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 52s | the patch passed | | +1 :green_heart: | compile | 4m 4s | the patch passed | | +1 :green_heart: | cc | 4m 4s | the patch passed | | +1 :green_heart: | javac | 4m 4s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-server: The patch generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) | | -0 :warning: | rubocop | 0m 8s | The patch generated 8 new + 701 unchanged - 3 fixed = 709 total (was 704) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 46s | Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6. | | +1 :green_heart: | hbaseprotoc | 1m 28s | the patch passed | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 4m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 41m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5591 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool rubocop | | uname | Linux f3b39c633c58 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 5488acfeaa | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | rubocop | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5591/3/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 rubocop=1.37.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28303) Interrupt cache prefetch thread when a heap usage threshold is reached
[ https://issues.apache.org/jira/browse/HBASE-28303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810144#comment-17810144 ] ASF GitHub Bot commented on HBASE-28303: Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1907139288 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 32s | master passed | | +1 :green_heart: | compile | 0m 42s | master passed | | +1 :green_heart: | shadedjars | 5m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 44s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 6m 22s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 242m 58s | hbase-server in the patch failed. | | | | 268m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2d6e2398e3fc 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3a91c5b698 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/testReport/ | | Max. process+thread count | 5331 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. > Interrupt cache prefetch thread when a heap usage threshold is reached > -- > > Key: HBASE-28303 > URL: https://issues.apache.org/jira/browse/HBASE-28303 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > Mostly critical when using non heap cache implementations, such as offheap or > file based. If the cache medium is too large and there are many blocks to be > cached, it may create a lot of cache index object in the RegionServer heap. > We should have guardrails to preventing caching from exhausting available > heap. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28303 Interrupt cache prefetch thread when a heap usage threshold is reached [hbase]
Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1907139288 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 32s | master passed | | +1 :green_heart: | compile | 0m 42s | master passed | | +1 :green_heart: | shadedjars | 5m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 44s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 6m 22s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 242m 58s | hbase-server in the patch failed. | | | | 268m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2d6e2398e3fc 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3a91c5b698 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/testReport/ | | Max. process+thread count | 5331 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28303) Interrupt cache prefetch thread when a heap usage threshold is reached
[ https://issues.apache.org/jira/browse/HBASE-28303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810140#comment-17810140 ] ASF GitHub Bot commented on HBASE-28303: Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1907113360 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 44s | master passed | | +1 :green_heart: | compile | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 5m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 218m 40s | hbase-server in the patch passed. | | | | 241m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bd6ebb4e87fe 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3a91c5b698 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/testReport/ | | Max. process+thread count | 4698 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. > Interrupt cache prefetch thread when a heap usage threshold is reached > -- > > Key: HBASE-28303 > URL: https://issues.apache.org/jira/browse/HBASE-28303 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > Mostly critical when using non heap cache implementations, such as offheap or > file based. If the cache medium is too large and there are many blocks to be > cached, it may create a lot of cache index object in the RegionServer heap. > We should have guardrails to preventing caching from exhausting available > heap. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28303 Interrupt cache prefetch thread when a heap usage threshold is reached [hbase]
Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1907113360 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 44s | master passed | | +1 :green_heart: | compile | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 5m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 218m 40s | hbase-server in the patch passed. | | | | 241m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bd6ebb4e87fe 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3a91c5b698 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/testReport/ | | Max. process+thread count | 4698 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28317) RpcCallContext should expose client's TLS certificate
[
https://issues.apache.org/jira/browse/HBASE-28317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810113#comment-17810113
]
ASF GitHub Bot commented on HBASE-28317:
bbeaudreault commented on code in PR #5644:
URL: https://github.com/apache/hbase/pull/5644#discussion_r1464030161
##
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java:
##
@@ -413,6 +415,19 @@ private void initSSL(ChannelPipeline p, boolean
supportPlaintext)
sslHandler.setWrapDataSize(
conf.getInt(HBASE_SERVER_NETTY_TLS_WRAP_SIZE,
DEFAULT_HBASE_SERVER_NETTY_TLS_WRAP_SIZE));
+ sslHandler.handshakeFuture().addListener(future -> {
+try {
+ Certificate[] certificates =
sslHandler.engine().getSession().getPeerCertificates();
+ if (certificates.length > 0) {
+conn.clientCertificate = (X509Certificate) certificates[0];
+ } else {
Review Comment:
We should inspect the `sslHandler.engine().getNeedClientAuth()` and throw an
exception if no certificate is found here and/or in the exception below. See
https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxnFactory.java#L418-L463
for how zookeeper handles hits.
Another thing I notice in zookeeper is that they set the entire
Certificate[] onto the connection, so that someone can inspect the whole
certificate chain if they want. I think we should probably do that in order to
avoid having to break this interface in the future if someone wants more than
the head of the chain.
> RpcCallContext should expose client's TLS certificate
> -
>
> Key: HBASE-28317
> URL: https://issues.apache.org/jira/browse/HBASE-28317
> Project: HBase
> Issue Type: Improvement
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Minor
>
> At my employer we plan on using a coprocessor to log information about some
> requests to HBase. For this to be useful to us, we need to know who each
> request is coming from. We use HBase's TLS support with mutual authentication
> to authenticate clients. I'd like a way to expose the client certificate used
> on a request to coprocessors. For setups using Kerberos authentication,
> RpcCall exposes the Kerberos principal shortname via {{getRequestUser()}}, so
> this would be the TLS equivalent to that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28317: Expose client TLS certificate on RpcCallContext [hbase]
bbeaudreault commented on code in PR #5644:
URL: https://github.com/apache/hbase/pull/5644#discussion_r1464030161
##
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java:
##
@@ -413,6 +415,19 @@ private void initSSL(ChannelPipeline p, boolean
supportPlaintext)
sslHandler.setWrapDataSize(
conf.getInt(HBASE_SERVER_NETTY_TLS_WRAP_SIZE,
DEFAULT_HBASE_SERVER_NETTY_TLS_WRAP_SIZE));
+ sslHandler.handshakeFuture().addListener(future -> {
+try {
+ Certificate[] certificates =
sslHandler.engine().getSession().getPeerCertificates();
+ if (certificates.length > 0) {
+conn.clientCertificate = (X509Certificate) certificates[0];
+ } else {
Review Comment:
We should inspect the `sslHandler.engine().getNeedClientAuth()` and throw an
exception if no certificate is found here and/or in the exception below. See
https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxnFactory.java#L418-L463
for how zookeeper handles hits.
Another thing I notice in zookeeper is that they set the entire
Certificate[] onto the connection, so that someone can inspect the whole
certificate chain if they want. I think we should probably do that in order to
avoid having to break this interface in the future if someone wants more than
the head of the chain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-28317) RpcCallContext should expose client's TLS certificate
[
https://issues.apache.org/jira/browse/HBASE-28317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810110#comment-17810110
]
ASF GitHub Bot commented on HBASE-28317:
bbeaudreault commented on code in PR #5644:
URL: https://github.com/apache/hbase/pull/5644#discussion_r1464019537
##
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java:
##
@@ -413,6 +415,19 @@ private void initSSL(ChannelPipeline p, boolean
supportPlaintext)
sslHandler.setWrapDataSize(
conf.getInt(HBASE_SERVER_NETTY_TLS_WRAP_SIZE,
DEFAULT_HBASE_SERVER_NETTY_TLS_WRAP_SIZE));
+ sslHandler.handshakeFuture().addListener(future -> {
Review Comment:
@Apache9 do you think a listener is the right approach here? I'm wondering
if we should instead pass the SSLHandler into NettyServerRpcConnection, and
directly call `handler.handshakeFuture().get()` in `setupHandlers()`. That way
it's more explicit that we have the certificate on the connection prior to
handling any requests. With the listener approach, I'm not 100% sure we can
guarantee that the listener is executed in the eventLoop prior to the
channelRead0 in NettyRpcServerPreambleHandler.
> RpcCallContext should expose client's TLS certificate
> -
>
> Key: HBASE-28317
> URL: https://issues.apache.org/jira/browse/HBASE-28317
> Project: HBase
> Issue Type: Improvement
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Minor
>
> At my employer we plan on using a coprocessor to log information about some
> requests to HBase. For this to be useful to us, we need to know who each
> request is coming from. We use HBase's TLS support with mutual authentication
> to authenticate clients. I'd like a way to expose the client certificate used
> on a request to coprocessors. For setups using Kerberos authentication,
> RpcCall exposes the Kerberos principal shortname via {{getRequestUser()}}, so
> this would be the TLS equivalent to that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28317: Expose client TLS certificate on RpcCallContext [hbase]
bbeaudreault commented on code in PR #5644:
URL: https://github.com/apache/hbase/pull/5644#discussion_r1464019537
##
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java:
##
@@ -413,6 +415,19 @@ private void initSSL(ChannelPipeline p, boolean
supportPlaintext)
sslHandler.setWrapDataSize(
conf.getInt(HBASE_SERVER_NETTY_TLS_WRAP_SIZE,
DEFAULT_HBASE_SERVER_NETTY_TLS_WRAP_SIZE));
+ sslHandler.handshakeFuture().addListener(future -> {
Review Comment:
@Apache9 do you think a listener is the right approach here? I'm wondering
if we should instead pass the SSLHandler into NettyServerRpcConnection, and
directly call `handler.handshakeFuture().get()` in `setupHandlers()`. That way
it's more explicit that we have the certificate on the connection prior to
handling any requests. With the listener approach, I'm not 100% sure we can
guarantee that the listener is executed in the eventLoop prior to the
channelRead0 in NettyRpcServerPreambleHandler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-28303) Interrupt cache prefetch thread when a heap usage threshold is reached
[ https://issues.apache.org/jira/browse/HBASE-28303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810094#comment-17810094 ] ASF GitHub Bot commented on HBASE-28303: Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1906862545 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 53s | master passed | | +1 :green_heart: | compile | 2m 33s | master passed | | +1 :green_heart: | checkstyle | 0m 35s | master passed | | +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 55s | the patch passed | | +1 :green_heart: | compile | 2m 36s | the patch passed | | +1 :green_heart: | javac | 2m 36s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-server: The patch generated 12 new + 24 unchanged - 0 fixed = 36 total (was 24) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 24s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 35s | patch has 21 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 7s | The patch does not generate ASF License warnings. | | | | 29m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 4c5e73ec4fa7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3a91c5b698 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. > Interrupt cache prefetch thread when a heap usage threshold is reached > -- > > Key: HBASE-28303 > URL: https://issues.apache.org/jira/browse/HBASE-28303 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > Mostly critical when using non heap cache implementations, such as offheap or > file based. If the cache medium is too large and there are many blocks to be > cached, it may create a lot of cache index object in the RegionServer heap. > We should have guardrails to preventing caching from exhausting available > heap. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28303 Interrupt cache prefetch thread when a heap usage threshold is reached [hbase]
Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1906862545 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 53s | master passed | | +1 :green_heart: | compile | 2m 33s | master passed | | +1 :green_heart: | checkstyle | 0m 35s | master passed | | +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 29s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 55s | the patch passed | | +1 :green_heart: | compile | 2m 36s | the patch passed | | +1 :green_heart: | javac | 2m 36s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-server: The patch generated 12 new + 24 unchanged - 0 fixed = 36 total (was 24) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 24s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 35s | patch has 21 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 7s | The patch does not generate ASF License warnings. | | | | 29m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 4c5e73ec4fa7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3a91c5b698 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/6/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28303) Interrupt cache prefetch thread when a heap usage threshold is reached
[ https://issues.apache.org/jira/browse/HBASE-28303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810089#comment-17810089 ] ASF GitHub Bot commented on HBASE-28303: Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1906820749 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 24s | master passed | | +1 :green_heart: | compile | 0m 41s | master passed | | +1 :green_heart: | shadedjars | 4m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 28s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 234m 24s | hbase-server in the patch failed. | | | | 256m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b985d921b098 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b8cea19d4a | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/testReport/ | | Max. process+thread count | 5711 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. > Interrupt cache prefetch thread when a heap usage threshold is reached > -- > > Key: HBASE-28303 > URL: https://issues.apache.org/jira/browse/HBASE-28303 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > Mostly critical when using non heap cache implementations, such as offheap or > file based. If the cache medium is too large and there are many blocks to be > cached, it may create a lot of cache index object in the RegionServer heap. > We should have guardrails to preventing caching from exhausting available > heap. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28303 Interrupt cache prefetch thread when a heap usage threshold is reached [hbase]
Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1906820749 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 24s | master passed | | +1 :green_heart: | compile | 0m 41s | master passed | | +1 :green_heart: | shadedjars | 4m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 28s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 234m 24s | hbase-server in the patch failed. | | | | 256m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b985d921b098 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b8cea19d4a | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/testReport/ | | Max. process+thread count | 5711 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28303) Interrupt cache prefetch thread when a heap usage threshold is reached
[ https://issues.apache.org/jira/browse/HBASE-28303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810088#comment-17810088 ] ASF GitHub Bot commented on HBASE-28303: Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1906803913 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 11s | master passed | | +1 :green_heart: | compile | 0m 44s | master passed | | +1 :green_heart: | shadedjars | 5m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 46s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 220m 45s | hbase-server in the patch failed. | | | | 244m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5436fd66b503 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b8cea19d4a | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/testReport/ | | Max. process+thread count | 4659 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. > Interrupt cache prefetch thread when a heap usage threshold is reached > -- > > Key: HBASE-28303 > URL: https://issues.apache.org/jira/browse/HBASE-28303 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > Mostly critical when using non heap cache implementations, such as offheap or > file based. If the cache medium is too large and there are many blocks to be > cached, it may create a lot of cache index object in the RegionServer heap. > We should have guardrails to preventing caching from exhausting available > heap. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28303 Interrupt cache prefetch thread when a heap usage threshold is reached [hbase]
Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1906803913 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 11s | master passed | | +1 :green_heart: | compile | 0m 44s | master passed | | +1 :green_heart: | shadedjars | 5m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 46s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 220m 45s | hbase-server in the patch failed. | | | | 244m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5436fd66b503 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b8cea19d4a | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/testReport/ | | Max. process+thread count | 4659 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-28302) Add tracking of fs read times in ScanMetrics and slow logs
[ https://issues.apache.org/jira/browse/HBASE-28302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault resolved HBASE-28302. --- Fix Version/s: 2.6.0 3.0.0-beta-2 Release Note: Adds a new getFsReadTime() to the slow log records, and fsReadTime counter to ScanMetrics. In both cases, this is the cumulative time spent reading blocks from hdfs for the given request. Additionally, a new fsSlowReadsCount jmx metric is added to the sub=IO bean. This is the count of HDFS reads which took longer than hbase.fs.reader.warn.time.ms. Assignee: Bryan Beaudreault Resolution: Fixed Thanks [~ndimiduk] for the review! Pushed to master, branch-3, branch-2, branch-2.6. > Add tracking of fs read times in ScanMetrics and slow logs > -- > > Key: HBASE-28302 > URL: https://issues.apache.org/jira/browse/HBASE-28302 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Major > Fix For: 2.6.0, 3.0.0-beta-2 > > > We've had this in our production for a while, and it's useful info to have. > We already track FS read times in > [HFileBlock|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1828-L1831C10]. > We can project that into the ScanMetrics instance and slow log pretty > easily. It is also helpful to add a slow.fs.read.threshold, over which we log > a warn -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27966) HBase Master/RS JVM metrics populated incorrectly
[
https://issues.apache.org/jira/browse/HBASE-27966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault resolved HBASE-27966.
---
Resolution: Fixed
Pushed to branch-3. Test looks good there. I quickly checked the other branches
and looks like they were properly backported.
> HBase Master/RS JVM metrics populated incorrectly
> -
>
> Key: HBASE-27966
> URL: https://issues.apache.org/jira/browse/HBASE-27966
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0-alpha-4
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 2.6.0, 3.0.0-beta-1, 2.5.6
>
> Attachments: test_patch.txt
>
>
> HBase Master/RS JVM metrics populated incorrectly due to regression causing
> ambari metrics system to not able to capture them.
> Based on my analysis the issue is relevant for all release post 2.0.0-alpha-4
> and seems to be caused due to HBASE-18846.
> Have been able to compare the JVM metrics across 3 versions of HBase and
> attaching results of same below:
> HBase: 1.1.2
> {code:java}
> {
> "name" : "Hadoop:service=HBase,name=JvmMetrics",
> "modelerType" : "JvmMetrics",
> "tag.Context" : "jvm",
> "tag.ProcessName" : "RegionServer",
> "tag.SessionId" : "",
> "tag.Hostname" : "HOSTNAME",
> "MemNonHeapUsedM" : 196.05664,
> "MemNonHeapCommittedM" : 347.60547,
> "MemNonHeapMaxM" : 4336.0,
> "MemHeapUsedM" : 7207.315,
> "MemHeapCommittedM" : 66080.0,
> "MemHeapMaxM" : 66080.0,
> "MemMaxM" : 66080.0,
> "GcCount" : 3953,
> "GcTimeMillis" : 662520,
> "ThreadsNew" : 0,
> "ThreadsRunnable" : 214,
> "ThreadsBlocked" : 0,
> "ThreadsWaiting" : 626,
> "ThreadsTimedWaiting" : 78,
> "ThreadsTerminated" : 0,
> "LogFatal" : 0,
> "LogError" : 0,
> "LogWarn" : 0,
> "LogInfo" : 0
> },
> {code}
> HBase 2.0.2
> {code:java}
> {
> "name" : "Hadoop:service=HBase,name=JvmMetrics",
> "modelerType" : "JvmMetrics",
> "tag.Context" : "jvm",
> "tag.ProcessName" : "IO",
> "tag.SessionId" : "",
> "tag.Hostname" : "HOSTNAME",
> "MemNonHeapUsedM" : 203.86688,
> "MemNonHeapCommittedM" : 740.6953,
> "MemNonHeapMaxM" : -1.0,
> "MemHeapUsedM" : 14879.477,
> "MemHeapCommittedM" : 31744.0,
> "MemHeapMaxM" : 31744.0,
> "MemMaxM" : 31744.0,
> "GcCount" : 75922,
> "GcTimeMillis" : 5134691,
> "ThreadsNew" : 0,
> "ThreadsRunnable" : 90,
> "ThreadsBlocked" : 3,
> "ThreadsWaiting" : 158,
> "ThreadsTimedWaiting" : 36,
> "ThreadsTerminated" : 0,
> "LogFatal" : 0,
> "LogError" : 0,
> "LogWarn" : 0,
> "LogInfo" : 0
> },
> {code}
> HBase: 2.5.2
> {code:java}
> {
> "name": "Hadoop:service=HBase,name=JvmMetrics",
> "modelerType": "JvmMetrics",
> "tag.Context": "jvm",
> "tag.ProcessName": "IO",
> "tag.SessionId": "",
> "tag.Hostname": "HOSTNAME",
> "MemNonHeapUsedM": 192.9798,
> "MemNonHeapCommittedM": 198.4375,
> "MemNonHeapMaxM": -1.0,
> "MemHeapUsedM": 773.23584,
> "MemHeapCommittedM": 1004.0,
> "MemHeapMaxM": 1024.0,
> "MemMaxM": 1024.0,
> "GcCount": 2048,
> "GcTimeMillis": 25440,
> "ThreadsNew": 0,
> "ThreadsRunnable": 22,
> "ThreadsBlocked": 0,
> "ThreadsWaiting": 121,
> "ThreadsTimedWaiting": 49,
> "ThreadsTerminated": 0,
> "LogFatal": 0,
> "LogError": 0,
> "LogWarn": 0,
> "LogInfo": 0
> },
> {code}
> It can be observed that 2.0.x onwards the field "tag.ProcessName" is
> populating as "IO" instead of expected "RegionServer" or "Master".
> Ambari relies on this field process name to create a metric
> 'jvm.RegionServer.JvmMetrics.GcTimeMillis' etc. See
> [code.|https://github.com/apache/ambari/blob/2ec4b055d99ec84c902da16dd57df91d571b48d6/ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/timeline/AMSPropertyProvider.java#L722]
> But post 2.0.x the field is getting populated as 'IO' and hence a metric with
> name 'jvm.JvmMetrics.GcTimeMillis' is created instead of expected
> 'jvm.RegionServer.JvmMetrics.GcTimeMillis', thus mixing up the metric with
> various other metrics coming from rs, master, spark executor etc. running on
> same host.
> *Expected*
> Field "tag.ProcessName" should be populated as "RegionServer" or "Master"
> instead of "IO".
> *Actual*
> Field "tag.ProcessName" is populating as "IO" instead of expected
> "RegionServer" or "Master" causing incorrect metric being published by ambari
> and thus mixing up all metrics and raising various alerts around JVM metrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-27966) HBase Master/RS JVM metrics populated incorrectly
[
https://issues.apache.org/jira/browse/HBASE-27966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault reopened HBASE-27966:
---
Re-opening because I just realized that this was not included in branch-3.
Perhaps it was committed around the time of our branching that. We need to
cherry-pick to branch-3, which I will do shortly.
> HBase Master/RS JVM metrics populated incorrectly
> -
>
> Key: HBASE-27966
> URL: https://issues.apache.org/jira/browse/HBASE-27966
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0-alpha-4
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 2.6.0, 2.5.6, 3.0.0-beta-1
>
> Attachments: test_patch.txt
>
>
> HBase Master/RS JVM metrics populated incorrectly due to regression causing
> ambari metrics system to not able to capture them.
> Based on my analysis the issue is relevant for all release post 2.0.0-alpha-4
> and seems to be caused due to HBASE-18846.
> Have been able to compare the JVM metrics across 3 versions of HBase and
> attaching results of same below:
> HBase: 1.1.2
> {code:java}
> {
> "name" : "Hadoop:service=HBase,name=JvmMetrics",
> "modelerType" : "JvmMetrics",
> "tag.Context" : "jvm",
> "tag.ProcessName" : "RegionServer",
> "tag.SessionId" : "",
> "tag.Hostname" : "HOSTNAME",
> "MemNonHeapUsedM" : 196.05664,
> "MemNonHeapCommittedM" : 347.60547,
> "MemNonHeapMaxM" : 4336.0,
> "MemHeapUsedM" : 7207.315,
> "MemHeapCommittedM" : 66080.0,
> "MemHeapMaxM" : 66080.0,
> "MemMaxM" : 66080.0,
> "GcCount" : 3953,
> "GcTimeMillis" : 662520,
> "ThreadsNew" : 0,
> "ThreadsRunnable" : 214,
> "ThreadsBlocked" : 0,
> "ThreadsWaiting" : 626,
> "ThreadsTimedWaiting" : 78,
> "ThreadsTerminated" : 0,
> "LogFatal" : 0,
> "LogError" : 0,
> "LogWarn" : 0,
> "LogInfo" : 0
> },
> {code}
> HBase 2.0.2
> {code:java}
> {
> "name" : "Hadoop:service=HBase,name=JvmMetrics",
> "modelerType" : "JvmMetrics",
> "tag.Context" : "jvm",
> "tag.ProcessName" : "IO",
> "tag.SessionId" : "",
> "tag.Hostname" : "HOSTNAME",
> "MemNonHeapUsedM" : 203.86688,
> "MemNonHeapCommittedM" : 740.6953,
> "MemNonHeapMaxM" : -1.0,
> "MemHeapUsedM" : 14879.477,
> "MemHeapCommittedM" : 31744.0,
> "MemHeapMaxM" : 31744.0,
> "MemMaxM" : 31744.0,
> "GcCount" : 75922,
> "GcTimeMillis" : 5134691,
> "ThreadsNew" : 0,
> "ThreadsRunnable" : 90,
> "ThreadsBlocked" : 3,
> "ThreadsWaiting" : 158,
> "ThreadsTimedWaiting" : 36,
> "ThreadsTerminated" : 0,
> "LogFatal" : 0,
> "LogError" : 0,
> "LogWarn" : 0,
> "LogInfo" : 0
> },
> {code}
> HBase: 2.5.2
> {code:java}
> {
> "name": "Hadoop:service=HBase,name=JvmMetrics",
> "modelerType": "JvmMetrics",
> "tag.Context": "jvm",
> "tag.ProcessName": "IO",
> "tag.SessionId": "",
> "tag.Hostname": "HOSTNAME",
> "MemNonHeapUsedM": 192.9798,
> "MemNonHeapCommittedM": 198.4375,
> "MemNonHeapMaxM": -1.0,
> "MemHeapUsedM": 773.23584,
> "MemHeapCommittedM": 1004.0,
> "MemHeapMaxM": 1024.0,
> "MemMaxM": 1024.0,
> "GcCount": 2048,
> "GcTimeMillis": 25440,
> "ThreadsNew": 0,
> "ThreadsRunnable": 22,
> "ThreadsBlocked": 0,
> "ThreadsWaiting": 121,
> "ThreadsTimedWaiting": 49,
> "ThreadsTerminated": 0,
> "LogFatal": 0,
> "LogError": 0,
> "LogWarn": 0,
> "LogInfo": 0
> },
> {code}
> It can be observed that 2.0.x onwards the field "tag.ProcessName" is
> populating as "IO" instead of expected "RegionServer" or "Master".
> Ambari relies on this field process name to create a metric
> 'jvm.RegionServer.JvmMetrics.GcTimeMillis' etc. See
> [code.|https://github.com/apache/ambari/blob/2ec4b055d99ec84c902da16dd57df91d571b48d6/ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/timeline/AMSPropertyProvider.java#L722]
> But post 2.0.x the field is getting populated as 'IO' and hence a metric with
> name 'jvm.JvmMetrics.GcTimeMillis' is created instead of expected
> 'jvm.RegionServer.JvmMetrics.GcTimeMillis', thus mixing up the metric with
> various other metrics coming from rs, master, spark executor etc. running on
> same host.
> *Expected*
> Field "tag.ProcessName" should be populated as "RegionServer" or "Master"
> instead of "IO".
> *Actual*
> Field "tag.ProcessName" is populating as "IO" instead of expected
> "RegionServer" or "Master" causing incorrect metric being published by ambari
> and thus mixing up all metrics and raising various alerts around JVM metrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28303) Interrupt cache prefetch thread when a heap usage threshold is reached
[ https://issues.apache.org/jira/browse/HBASE-28303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810026#comment-17810026 ] ASF GitHub Bot commented on HBASE-28303: Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1906394149 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 56s | master passed | | +1 :green_heart: | compile | 2m 32s | master passed | | +1 :green_heart: | checkstyle | 0m 32s | master passed | | +1 :green_heart: | spotless | 0m 41s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 30s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 52s | the patch passed | | +1 :green_heart: | compile | 2m 33s | the patch passed | | +1 :green_heart: | javac | 2m 33s | the patch passed | | -0 :warning: | checkstyle | 0m 34s | hbase-server: The patch generated 11 new + 24 unchanged - 0 fixed = 35 total (was 24) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 32s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 7s | The patch does not generate ASF License warnings. | | | | 29m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 342d5f4cc445 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b8cea19d4a | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 85 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. > Interrupt cache prefetch thread when a heap usage threshold is reached > -- > > Key: HBASE-28303 > URL: https://issues.apache.org/jira/browse/HBASE-28303 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > Mostly critical when using non heap cache implementations, such as offheap or > file based. If the cache medium is too large and there are many blocks to be > cached, it may create a lot of cache index object in the RegionServer heap. > We should have guardrails to preventing caching from exhausting available > heap. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28303 Interrupt cache prefetch thread when a heap usage threshold is reached [hbase]
Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1906394149 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 56s | master passed | | +1 :green_heart: | compile | 2m 32s | master passed | | +1 :green_heart: | checkstyle | 0m 32s | master passed | | +1 :green_heart: | spotless | 0m 41s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 30s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 52s | the patch passed | | +1 :green_heart: | compile | 2m 33s | the patch passed | | +1 :green_heart: | javac | 2m 33s | the patch passed | | -0 :warning: | checkstyle | 0m 34s | hbase-server: The patch generated 11 new + 24 unchanged - 0 fixed = 35 total (was 24) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 32s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 7s | The patch does not generate ASF License warnings. | | | | 29m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 342d5f4cc445 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b8cea19d4a | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 85 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/5/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28302) Add tracking of fs read times in ScanMetrics and slow logs
[ https://issues.apache.org/jira/browse/HBASE-28302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810024#comment-17810024 ] ASF GitHub Bot commented on HBASE-28302: bbeaudreault merged PR #5622: URL: https://github.com/apache/hbase/pull/5622 > Add tracking of fs read times in ScanMetrics and slow logs > -- > > Key: HBASE-28302 > URL: https://issues.apache.org/jira/browse/HBASE-28302 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Priority: Major > > We've had this in our production for a while, and it's useful info to have. > We already track FS read times in > [HFileBlock|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1828-L1831C10]. > We can project that into the ScanMetrics instance and slow log pretty > easily. It is also helpful to add a slow.fs.read.threshold, over which we log > a warn -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28302 Add tracking of fs read times in ScanMetrics and slow logs [hbase]
bbeaudreault merged PR #5622: URL: https://github.com/apache/hbase/pull/5622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28303) Interrupt cache prefetch thread when a heap usage threshold is reached
[ https://issues.apache.org/jira/browse/HBASE-28303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809981#comment-17809981 ] ASF GitHub Bot commented on HBASE-28303: Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1906242562 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | shadedjars | 5m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 5m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 218m 47s | hbase-server in the patch passed. | | | | 241m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bef757b94d2f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 877830e06c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/testReport/ | | Max. process+thread count | 4730 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. > Interrupt cache prefetch thread when a heap usage threshold is reached > -- > > Key: HBASE-28303 > URL: https://issues.apache.org/jira/browse/HBASE-28303 > Project: HBase > Issue Type: Improvement >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > Mostly critical when using non heap cache implementations, such as offheap or > file based. If the cache medium is too large and there are many blocks to be > cached, it may create a lot of cache index object in the RegionServer heap. > We should have guardrails to preventing caching from exhausting available > heap. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28303 Interrupt cache prefetch thread when a heap usage threshold is reached [hbase]
Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1906242562 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | shadedjars | 5m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 5m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 218m 47s | hbase-server in the patch passed. | | | | 241m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bef757b94d2f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 877830e06c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/testReport/ | | Max. process+thread count | 4730 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809980#comment-17809980 ] Bryan Beaudreault commented on HBASE-28325: --- Yea that would be really unfortunate since it's already really slow sometimes when you encounter flaky test timeouts. What i'd really love to do is get them on GH Actions, if that were possible. Or somehow get the old ones auto-hidden. > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 4.0.0-alpha-1 > > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28302) Add tracking of fs read times in ScanMetrics and slow logs
[ https://issues.apache.org/jira/browse/HBASE-28302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809979#comment-17809979 ] ASF GitHub Bot commented on HBASE-28302: bbeaudreault commented on PR #5622: URL: https://github.com/apache/hbase/pull/5622#issuecomment-1906229934 Thanks for the review! Yea it's pretty annoying to handle thrift, since I needed to go figure out how to generate them. But that seems to be how it works today. We could probably try to give it the protobuf-like treatment in a separate jira if anyone wants to improve the thrift build. > Add tracking of fs read times in ScanMetrics and slow logs > -- > > Key: HBASE-28302 > URL: https://issues.apache.org/jira/browse/HBASE-28302 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Priority: Major > > We've had this in our production for a while, and it's useful info to have. > We already track FS read times in > [HFileBlock|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1828-L1831C10]. > We can project that into the ScanMetrics instance and slow log pretty > easily. It is also helpful to add a slow.fs.read.threshold, over which we log > a warn -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28302 Add tracking of fs read times in ScanMetrics and slow logs [hbase]
bbeaudreault commented on PR #5622: URL: https://github.com/apache/hbase/pull/5622#issuecomment-1906229934 Thanks for the review! Yea it's pretty annoying to handle thrift, since I needed to go figure out how to generate them. But that seems to be how it works today. We could probably try to give it the protobuf-like treatment in a separate jira if anyone wants to improve the thrift build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809978#comment-17809978 ] Duo Zhang commented on HBASE-28325: --- Does multijdk build run different jdk builds in parallel? Otherwise it will double the pre commit time... > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 4.0.0-alpha-1 > > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809970#comment-17809970 ] Nick Dimiduk commented on HBASE-28325: -- The 3x posting on pre-commit is because Yetus's multi-jdk feature didn't work for us back when we added testing of jdk11. There have been a couple yetus releases since then and it's on my todo list to try out multi-jdk on a more recent version. > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 4.0.0-alpha-1 > > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809955#comment-17809955 ] Bryan Beaudreault commented on HBASE-28325: --- (Just to be clear, if we can get it to be just on PR creation I agree that'd be ideal for us right now. Just saying there are pros and cons to mirroring every comment, and most of the cons might be improved by reducing the noise of our pre-commits if possible) > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 4.0.0-alpha-1 > > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809952#comment-17809952 ] Bryan Beaudreault commented on HBASE-28325: --- HADOOP has that behavior... It's pretty noisy, but not totally bad and in some ways nice. That said, we have a more spammy set of pre-commit hooks, so might be really bad for us (3 large html comments for every commit) > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 4.0.0-alpha-1 > > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reopened HBASE-28325: -- I don't want a comment for every PR comment, only a comment when the PR is detected/linked. Asking for clarification on INFRA-25382. > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 4.0.0-alpha-1 > > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[
https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809945#comment-17809945
]
Nick Dimiduk commented on HBASE-28325:
--
{quote}
ndimiduk commented on PR #5645:
URL: https://github.com/apache/hbase/pull/5645#issuecomment-1906048012
Thanks folks.
{quote}
Oh no. That's exactly what I don't want.
> Enable infra automation to comment on a Jira when a new PR is posted
>
>
> Key: HBASE-28325
> URL: https://issues.apache.org/jira/browse/HBASE-28325
> Project: HBase
> Issue Type: Task
> Components: community
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 4.0.0-alpha-1
>
>
> https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions
> Currently we make use of the "link" feature. This does not send a
> notification to watchers, so I propose that we add the "comment" feature, so
> that a comment will also be sent, and watchers can find out about the
> availability of the PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28317) RpcCallContext should expose client's TLS certificate
[
https://issues.apache.org/jira/browse/HBASE-28317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809944#comment-17809944
]
Bryan Beaudreault commented on HBASE-28317:
---
Thanks for the thoughts [~ndimiduk]. Just to be clear, I do agree that adding
the cert to RpcCall seems in line with existing conventions. So no problems
with it as-is, more just thinking of the bigger picture. In hindsight, that
line of thinking might have been best explored in a different jira. If anyone
else has useful suggestions/context to provide in response to your comment,
I'll make sure to copy it into a new Jira for posterity and further discussion.
Your idea around doing some upfront general parsing is interesting, if there is
any opportunity there.
> RpcCallContext should expose client's TLS certificate
> -
>
> Key: HBASE-28317
> URL: https://issues.apache.org/jira/browse/HBASE-28317
> Project: HBase
> Issue Type: Improvement
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Minor
>
> At my employer we plan on using a coprocessor to log information about some
> requests to HBase. For this to be useful to us, we need to know who each
> request is coming from. We use HBase's TLS support with mutual authentication
> to authenticate clients. I'd like a way to expose the client certificate used
> on a request to coprocessors. For setups using Kerberos authentication,
> RpcCall exposes the Kerberos principal shortname via {{getRequestUser()}}, so
> this would be the TLS equivalent to that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809943#comment-17809943 ] ASF GitHub Bot commented on HBASE-28325: ndimiduk commented on PR #5645: URL: https://github.com/apache/hbase/pull/5645#issuecomment-1906048012 Thanks folks. > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-28325: - Fix Version/s: 4.0.0-alpha-1 Resolution: Fixed Status: Resolved (was: Patch Available) > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 4.0.0-alpha-1 > > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28325 Enable infra automation to comment on a Jira when a new PR is posted [hbase]
ndimiduk commented on PR #5645: URL: https://github.com/apache/hbase/pull/5645#issuecomment-1906048012 Thanks folks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28325 Enable infra automation to comment on a Jira when a new PR is posted [hbase]
ndimiduk merged PR #5645: URL: https://github.com/apache/hbase/pull/5645 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809938#comment-17809938 ] Bryan Beaudreault commented on HBASE-28325: --- +1 > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28292 Make Delay prefetch property to be dynamically configured [hbase]
kabhishek4 commented on code in PR #5605:
URL: https://github.com/apache/hbase/pull/5605#discussion_r1445999734
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java:
##
@@ -134,4 +139,25 @@ private PrefetchExecutor() {
static ScheduledExecutorService getExecutorPool() {
return prefetchExecutorPool;
}
+
+ public static int getPrefetchDelay() {
+return prefetchDelayMillis;
+ }
+
+ @Override
+ public void onConfigurationChange(Configuration conf) {
+PrefetchExecutor.loadConfiguration(conf);
+ }
+
+ @Override
+ public void registerChildren(ConfigurationManager manager) {
+ }
+
+ @Override
+ public void deregisterChildren(ConfigurationManager manager) {
+ }
+
+ public static void loadConfiguration(Configuration conf) {
+prefetchDelayMillis = conf.getInt(PREFETCH_DELAY, 1000);
Review Comment:
Yes, in the scenario you described, despite the delay is changed, prefetched
threads will wait. There are 2 ways to address this in my opinion.
1. Keep the behavior as it is and document above mentioned limitation
2. Interrupt threads which are doing prefetch, update the new value and
restart the prefetch threads
In approach 2, there are some complexities as HFilePreadReader class
'drives' the prefetch. Also, run() of HFilePreadReader class provides 'work
context' to the prefetch threads. Now, onConfigurationChange needs to cancel
the prefetch threads and restart them. For this can we store 'work context' in
a map
for ex,
private static final Map prefetchRunnable = new
ConcurrentSkipListMap<>();
and populate during request() method so that we can restart the prefetch
threads in following way,
For ex,
@Override
public void onConfigurationChange(Configuration conf) {
PrefetchExecutor.loadConfiguration(conf);
}
public static void loadConfiguration(Configuration conf) {
prefetchDelayMillis = conf.getInt(PREFETCH_DELAY, 1000);
prefetchFutures.forEach((k, v) -> {
cancel(k);
request(k,prefetchRunnable.get(k));
LOG.debug("Reset called on Prefetch of file {}", k);
});
}
Please comment. Also, let me know if there is another alternative.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java:
##
@@ -134,4 +139,25 @@ private PrefetchExecutor() {
static ScheduledExecutorService getExecutorPool() {
return prefetchExecutorPool;
}
+
+ public static int getPrefetchDelay() {
+return prefetchDelayMillis;
+ }
+
+ @Override
+ public void onConfigurationChange(Configuration conf) {
+PrefetchExecutor.loadConfiguration(conf);
+ }
+
+ @Override
+ public void registerChildren(ConfigurationManager manager) {
+ }
+
+ @Override
+ public void deregisterChildren(ConfigurationManager manager) {
+ }
+
+ public static void loadConfiguration(Configuration conf) {
+prefetchDelayMillis = conf.getInt(PREFETCH_DELAY, 1000);
Review Comment:
This document has been attached to the JIRA HBASE-28292.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-28292) Make Delay prefetch property to be dynamically configured
[ https://issues.apache.org/jira/browse/HBASE-28292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Kothalikar updated HBASE-28292: Description: Make the prefetch delay configurable. The prefetch delay is associated to hbase.hfile.prefetch.delay configuration. There are some cases where configuring hbase.hfile.prefetch.delay would help in achieving better throughput. (was: Rolling restart triggers region movement on the cluster while the RegionServers are getting restarted. During this time, the temporary RegionServers will start prefetching these files which are only hosted until the source RegionServer is restarted. Hence, in this timing window, fetches are executed on temporary region servers which takes few minutes. Currrently, this behaviour is governed by hbase.hfile.prefetch.delay Also, file-based block cache gives performance improvement on HBase clusters using object storage. COD uses a large ephemeral disk for this purpose. Possible solution is, The number of prefetches on the cluster caused by these temporary region movements during Rolling Upgrade (rolling restart) can be mininized by introducing delay before the prefetch starts. To achieve this, making the hbase.hfile.prefetch.delay dynamically configurable. The interval will let the rolling restart complete. Resultantly, prefetches on temporary region server can be minimized. ) > Make Delay prefetch property to be dynamically configured > - > > Key: HBASE-28292 > URL: https://issues.apache.org/jira/browse/HBASE-28292 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Kothalikar >Assignee: Abhishek Kothalikar >Priority: Major > Attachments: HBASE-28292.docx > > > Make the prefetch delay configurable. The prefetch delay is associated to > hbase.hfile.prefetch.delay configuration. There are some cases where > configuring hbase.hfile.prefetch.delay would help in achieving better > throughput. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28292 Make Delay prefetch property to be dynamically configured [hbase]
wchevreuil commented on code in PR #5605:
URL: https://github.com/apache/hbase/pull/5605#discussion_r1463134561
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java:
##
@@ -31,23 +31,28 @@
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.conf.ConfigurationManager;
+import org.apache.hadoop.hbase.conf.PropagatingConfigurationObserver;
import org.apache.hadoop.hbase.trace.TraceUtil;
import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@InterfaceAudience.Private
-public final class PrefetchExecutor {
+public final class PrefetchExecutor implements
PropagatingConfigurationObserver {
Review Comment:
Where are we registering this observer to get notified by
ConfigurationManager?
##
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java:
##
@@ -298,6 +299,44 @@ public void testPrefetchDoesntSkipRefs() throws Exception {
});
}
+ @Test
+ public void testOnConfigurationChange() {
+// change PREFETCH_DELAY_ENABLE_KEY from false to true
+conf.setInt(PREFETCH_DELAY, 4);
+PrefetchExecutor.loadConfiguration(conf);
+assertTrue(getPrefetchDelay() == 4);
+
+// restore
+conf.setInt(PREFETCH_DELAY, 3);
+PrefetchExecutor.loadConfiguration(conf);
+assertTrue(getPrefetchDelay() == 3);
+
+ }
+
+ @Test
+ public void testPrefetchWithDelay() throws Exception {
Review Comment:
Can we assert that 1) at first, no Prefetch thread is running, 2) After the
configured delay has elapsed, prefetch thread runs?
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/PrefetchExecutor.java:
##
@@ -134,4 +139,25 @@ private PrefetchExecutor() {
static ScheduledExecutorService getExecutorPool() {
return prefetchExecutorPool;
}
+
+ public static int getPrefetchDelay() {
+return prefetchDelayMillis;
+ }
+
+ @Override
+ public void onConfigurationChange(Configuration conf) {
Review Comment:
nit: please add info logging that config has changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28303 Interrupt cache prefetch thread when a heap usage threshold is reached [hbase]
Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1905852075 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 38s | master passed | | +1 :green_heart: | compile | 0m 40s | master passed | | +1 :green_heart: | shadedjars | 5m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 28s | the patch passed | | +1 :green_heart: | compile | 0m 41s | the patch passed | | +1 :green_heart: | javac | 0m 41s | the patch passed | | +1 :green_heart: | shadedjars | 5m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 11m 11s | hbase-server in the patch failed. | | | | 30m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4787bfda3c03 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 877830e06c | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/testReport/ | | Max. process+thread count | 1407 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28303 Interrupt cache prefetch thread when a heap usage threshold is reached [hbase]
Apache-HBase commented on PR #5615: URL: https://github.com/apache/hbase/pull/5615#issuecomment-1905850993 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 9s | master passed | | +1 :green_heart: | compile | 2m 34s | master passed | | +1 :green_heart: | checkstyle | 0m 34s | master passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 32s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 59s | the patch passed | | +1 :green_heart: | compile | 2m 33s | the patch passed | | +1 :green_heart: | javac | 2m 33s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-server: The patch generated 11 new + 24 unchanged - 0 fixed = 35 total (was 24) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 27s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 41s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 30m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5615 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux a97a873c8e23 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 877830e06c | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 81 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5615/4/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28317) RpcCallContext should expose client's TLS certificate
[
https://issues.apache.org/jira/browse/HBASE-28317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809893#comment-17809893
]
Nick Dimiduk commented on HBASE-28317:
--
For my part, I think that the approach [~charlesconnell] has proposed on the PR
is consistent with how HBase currently handles threading and exposing
connection details. I have no specific criticism of this approach. I wonder if
any of the specific parsing is useful broadly and can be done up-front and
placed on the Context object along with the client certificate.
[~bbeaudreault] has raised a good point about making all this machinery more
pluggable. I wonder how many of these interfaces would be different if we also
had the TLS implementation to consider when the Kerberos support was added.
Maybe [~apurtell] can shed some light here? It's been a while, but the proposal
of injecting AuthInfo via a pluggable mechanism reminds me of how a J2EE
application (or maybe this is [specific to
Jersey|https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest31x/security.html]?)
has its SecurityContext injected into resources. In J2EE, I don't know if the
implementation of the SecurityContextProvider is completely pluggable at the
spec level, or if the spec merely states that this interface must be injected
and it's up to the runtime implementation to intercept the connection/request
lifecycle as it sees fit. For our purposes, a connection coprocessor seems like
a nice "hbase native" means of implementing such a Provider, so long as it can
inject relevant interface implementations into the equivalent SecurityContext
object(s).
As a design choice, I'm fine with landing Charles's proposed approach in 2.x
(and probably 3.x) releases while we explore a more general approach. I do
wonder if we have enough examples of different implementations to justify the
refactor toward a pluggable architecture. Is there a reasonable expectation
that we'd want to add, for example, an OAuth auth provider as well?
Heya [~norman], I wonder if you can share any experience here about how this
kind of auth information is commonly plumbed through Netty applications? We
have a bit of friction here in that we also support a non-Netty RPC system, but
I think (hope) that the old implementation is on its way out.
> RpcCallContext should expose client's TLS certificate
> -
>
> Key: HBASE-28317
> URL: https://issues.apache.org/jira/browse/HBASE-28317
> Project: HBase
> Issue Type: Improvement
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Minor
>
> At my employer we plan on using a coprocessor to log information about some
> requests to HBase. For this to be useful to us, we need to know who each
> request is coming from. We use HBase's TLS support with mutual authentication
> to authenticate clients. I'd like a way to expose the client certificate used
> on a request to coprocessors. For setups using Kerberos authentication,
> RpcCall exposes the Kerberos principal shortname via {{getRequestUser()}}, so
> this would be the TLS equivalent to that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809891#comment-17809891 ] Duo Zhang commented on HBASE-28325: --- +1 > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28325 Enable infra automation to comment on a Jira when a new PR is posted [hbase]
Apache-HBase commented on PR #5645: URL: https://github.com/apache/hbase/pull/5645#issuecomment-1905725802 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | yamllint | 0m 0s | yamllint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | spotless | 0m 46s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | spotless | 0m 38s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 6m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5645/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5645 | | Optional Tests | dupname asflicense spotless yamllint | | uname | Linux 51f8ce2ef001 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 877830e06c | | Max. process+thread count | 44 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5645/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28325 Enable infra automation to comment on a Jira when a new PR is posted [hbase]
Apache-HBase commented on PR #5645: URL: https://github.com/apache/hbase/pull/5645#issuecomment-1905721495 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 2m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5645/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5645 | | Optional Tests | | | uname | Linux ef321e7a6faf 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 877830e06c | | Max. process+thread count | 29 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5645/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28325 Enable infra automation to comment on a Jira when a new PR is posted [hbase]
Apache-HBase commented on PR #5645: URL: https://github.com/apache/hbase/pull/5645#issuecomment-1905718304 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 1m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5645/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5645 | | Optional Tests | | | uname | Linux 895c717208c3 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 877830e06c | | Max. process+thread count | 40 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5645/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-28325: - Status: Patch Available (was: Open) > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted.
Nick Dimiduk created HBASE-28325: Summary: Enable infra automation to comment on a Jira when a new PR is posted. Key: HBASE-28325 URL: https://issues.apache.org/jira/browse/HBASE-28325 Project: HBase Issue Type: Task Components: community Reporter: Nick Dimiduk Assignee: Nick Dimiduk https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions Currently we make use of the "link" feature. This does not send a notification to watchers, so I propose that we add the "comment" feature, so that a comment will also be sent, and watchers can find out about the availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28325) Enable infra automation to comment on a Jira when a new PR is posted
[ https://issues.apache.org/jira/browse/HBASE-28325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-28325: - Summary: Enable infra automation to comment on a Jira when a new PR is posted (was: Enable infra automation to comment on a Jira when a new PR is posted.) > Enable infra automation to comment on a Jira when a new PR is posted > > > Key: HBASE-28325 > URL: https://issues.apache.org/jira/browse/HBASE-28325 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Jiranotificationoptions > Currently we make use of the "link" feature. This does not send a > notification to watchers, so I propose that we add the "comment" feature, so > that a comment will also be sent, and watchers can find out about the > availability of the PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
