[jira] [Commented] (HBASE-28254) Flaky test: TestTableShell
[
https://issues.apache.org/jira/browse/HBASE-28254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799280#comment-17799280
]
Hudson commented on HBASE-28254:
Results for branch branch-2
[build #948 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/948/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/948/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/948/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/948/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/948/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Flaky test: TestTableShell
> --
>
> Key: HBASE-28254
> URL: https://issues.apache.org/jira/browse/HBASE-28254
> Project: HBase
> Issue Type: Test
> Components: flakies, integration tests
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.8
>
>
> The test is running the following Ruby commands:
> {noformat}
> # Instert test data
> @test_table.put(1, "x:a", 1)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> {noformat}
> And validate the versions with:
> {noformat}
> args = { VERSIONS => 10, RAW => true } # Since 4 versions of row with rowkey
> 2 is been added, we can use any number >= 4 for VERSIONS to scan all 4
> versions.
> num_rows = 0
> @test_table._scan_internal(args) do # Raw Scan
> num_rows += 1
> end
> # 5 since , 1 from row key '1' and other 4 from row key '4'
> assert_equal(num_rows, 5,
> 'Num rows scanned without RAW/VERSIONS are not 5')
> {noformat}
> Which sometimes (almost always on fast machines) fails, because it only finds
> 3 versions out of 4. I believe this due to commands are running too fast and
> inserts data with same timestamp, so HBase cannot distinguish them. I'd like
> to add some sleep between puts to fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
Apache-HBase commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1865552399 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 40s | master passed | | +1 :green_heart: | compile | 0m 33s | master passed | | +1 :green_heart: | checkstyle | 0m 15s | master passed | | +1 :green_heart: | spotless | 0m 41s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 30s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 25s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 9s | The patch does not generate ASF License warnings. | | | | 25m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5576 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux dc21ebe1c0ff 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/4/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
Apache-HBase commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1865542255 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 47s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 55s | master passed | | +1 :green_heart: | compile | 0m 19s | master passed | | +1 :green_heart: | shadedjars | 4m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 0m 17s | the patch passed | | +1 :green_heart: | javac | 0m 17s | the patch passed | | +1 :green_heart: | shadedjars | 4m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 41s | hbase-common in the patch passed. | | | | 21m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5576 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2ee0af8dcbaf 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/4/testReport/ | | Max. process+thread count | 373 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
Apache-HBase commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1865536893 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 14s | master passed | | +1 :green_heart: | compile | 0m 14s | master passed | | +1 :green_heart: | shadedjars | 5m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 12s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 16s | the patch passed | | +1 :green_heart: | compile | 0m 15s | the patch passed | | +1 :green_heart: | javac | 0m 15s | the patch passed | | +1 :green_heart: | shadedjars | 5m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 48s | hbase-common in the patch passed. | | | | 18m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5576 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fc15cf211260 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/4/testReport/ | | Max. process+thread count | 367 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28151 Option to allow/disallow bypassing pre transit check for assing/unassign [hbase]
rahulLiving commented on PR #5493: URL: https://github.com/apache/hbase/pull/5493#issuecomment-1865536827 @rvaleti @virajjasani I have updated the PR to keep backward compatibility i.e default is `force=True` now. Please have a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28254) Flaky test: TestTableShell
[
https://issues.apache.org/jira/browse/HBASE-28254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799255#comment-17799255
]
Hudson commented on HBASE-28254:
Results for branch branch-2.4
[build #669 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/669/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/669/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/669/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/669/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/669/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Flaky test: TestTableShell
> --
>
> Key: HBASE-28254
> URL: https://issues.apache.org/jira/browse/HBASE-28254
> Project: HBase
> Issue Type: Test
> Components: flakies, integration tests
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.8
>
>
> The test is running the following Ruby commands:
> {noformat}
> # Instert test data
> @test_table.put(1, "x:a", 1)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> {noformat}
> And validate the versions with:
> {noformat}
> args = { VERSIONS => 10, RAW => true } # Since 4 versions of row with rowkey
> 2 is been added, we can use any number >= 4 for VERSIONS to scan all 4
> versions.
> num_rows = 0
> @test_table._scan_internal(args) do # Raw Scan
> num_rows += 1
> end
> # 5 since , 1 from row key '1' and other 4 from row key '4'
> assert_equal(num_rows, 5,
> 'Num rows scanned without RAW/VERSIONS are not 5')
> {noformat}
> Which sometimes (almost always on fast machines) fails, because it only finds
> 3 versions out of 4. I believe this due to commands are running too fast and
> inserts data with same timestamp, so HBase cannot distinguish them. I'd like
> to add some sleep between puts to fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28254) Flaky test: TestTableShell
[
https://issues.apache.org/jira/browse/HBASE-28254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799253#comment-17799253
]
Hudson commented on HBASE-28254:
Results for branch branch-2.6
[build #15 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/15/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/15/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/15/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/15/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/15/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Flaky test: TestTableShell
> --
>
> Key: HBASE-28254
> URL: https://issues.apache.org/jira/browse/HBASE-28254
> Project: HBase
> Issue Type: Test
> Components: flakies, integration tests
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.8
>
>
> The test is running the following Ruby commands:
> {noformat}
> # Instert test data
> @test_table.put(1, "x:a", 1)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> {noformat}
> And validate the versions with:
> {noformat}
> args = { VERSIONS => 10, RAW => true } # Since 4 versions of row with rowkey
> 2 is been added, we can use any number >= 4 for VERSIONS to scan all 4
> versions.
> num_rows = 0
> @test_table._scan_internal(args) do # Raw Scan
> num_rows += 1
> end
> # 5 since , 1 from row key '1' and other 4 from row key '4'
> assert_equal(num_rows, 5,
> 'Num rows scanned without RAW/VERSIONS are not 5')
> {noformat}
> Which sometimes (almost always on fast machines) fails, because it only finds
> 3 versions out of 4. I believe this due to commands are running too fast and
> inserts data with same timestamp, so HBase cannot distinguish them. I'd like
> to add some sleep between puts to fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
Apache-HBase commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1865463652 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | master passed | | +1 :green_heart: | compile | 0m 34s | master passed | | +1 :green_heart: | checkstyle | 0m 17s | master passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 0m 32s | the patch passed | | +1 :green_heart: | javac | 0m 32s | the patch passed | | +1 :green_heart: | checkstyle | 0m 15s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 29s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 25m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5576 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux c84cf5e572e5 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/3/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
Apache-HBase commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1865461008 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 56s | master passed | | +1 :green_heart: | compile | 0m 19s | master passed | | +1 :green_heart: | shadedjars | 4m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 0m 18s | the patch passed | | +1 :green_heart: | javac | 0m 18s | the patch passed | | +1 :green_heart: | shadedjars | 4m 53s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 19s | hbase-common in the patch passed. | | | | 20m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5576 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 319b1393414b 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/3/testReport/ | | Max. process+thread count | 373 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
Apache-HBase commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1865459813 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 14s | master passed | | +1 :green_heart: | compile | 0m 14s | master passed | | +1 :green_heart: | shadedjars | 5m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 11s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 16s | the patch passed | | +1 :green_heart: | compile | 0m 14s | the patch passed | | +1 :green_heart: | javac | 0m 14s | the patch passed | | +1 :green_heart: | shadedjars | 5m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 48s | hbase-common in the patch passed. | | | | 18m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5576 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f5a22b5aa79b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/3/testReport/ | | Max. process+thread count | 369 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-27730) Optimize reference counting in off-heap ByteBuff
[ https://issues.apache.org/jira/browse/HBASE-27730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Becker Ewing reassigned HBASE-27730: Assignee: Becker Ewing > Optimize reference counting in off-heap ByteBuff > > > Key: HBASE-27730 > URL: https://issues.apache.org/jira/browse/HBASE-27730 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Becker Ewing >Priority: Major > > In HBASE-27710 we uncovered a performance regression in reference counting of > ByteBuff. This was especially prominent in on-heap buffers when doing a > simple HFile.Reader iteration of a file. For that case, we saw a 4x > regression when reference counting was in play. > It stands to reason that this same regression exists in off-heap buffers, and > I've run a microbenchmark which indeed shows the same issue. With existing > reference counting, scanning a 20gb hfile takes 40s. With an optimized > version, scanning the same file takes 20s. We don't typically see this in > profiling live regionservers where so much else goes on, but optimizing this > would eliminate some cpu cycles. > It's worth noting that netty saw this same regression a few years ago: > [https://github.com/netty/netty/pull/8895]. Hat tip to [~zhangduo] for > pointing this out. > One of the fixes there was to copy some internal code from deeper in the ref > counting, so that the call stack was smaller and inlining was possible. We > can't really do that. > Another thing they did was add a boolean field in their CompositeByteBuffer, > which gets set to true when the buffer is recycled. So they don't need to do > reference counting on every operation, instead they can just check a boolean. > I tried adding a boolean to our RefCnt.java, and it indeed fixes the > regression. The problem is, due to class alignment issues in java, adding > this boolean field increases the heap size of RefCnt from 24 to 32 bytes. > This seems non-trivial given it's used in bucket cache where there could be > many millions of them. > I think we can get around this by simply nulling out the recycler in RefCnt > after it has been called. Then, instead of doing a boolean check we can do a > null check. This performs similarly to the boolean, but without any extra > memory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
jbewing commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1865448798 Based on the most recent benchmarking results ([see JIRA comment](https://issues.apache.org/jira/browse/HBASE-28256?focusedCommentId=17799200=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17799200)), I went and converted the approach here to use an approach adapted from [HBASE-14186](https://issues.apache.org/jira/browse/HBASE-14186). This approach has microbenchmarks that outperform our current setup. Additionally, there's no regression in the smaller vLong range like there is with the first approach that I used in this patch and performance is similar to or slightly worse in the higher vLong range than the approach I originally presented. However, after I implement [HBASE-27730](https://issues.apache.org/jira/browse/HBASE-27730), this newer approach drastically outperforms my original approach. Additionally, since there is no performance regression I think that it's safe to just replace the body of the existing readVLong method with this new implementation. Thus, this patch can be back-ported to branch-3 & branch-2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
Apache-HBase commented on PR #5593: URL: https://github.com/apache/hbase/pull/5593#issuecomment-1865446477 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 42s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 4m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 46s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 4m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 232m 38s | hbase-server in the patch failed. | | | | 255m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5593 | | JIRA Issue | HBASE-26268 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a59da3cee735 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/2/testReport/ | | Max. process+thread count | 4919 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28269 Ruby scripts are broken as they import non-existent classes [hbase]
Apache9 commented on code in PR #5588:
URL: https://github.com/apache/hbase/pull/5588#discussion_r1433450197
##
bin/get-active-master.rb:
##
@@ -24,9 +24,9 @@
java_import org.apache.hadoop.hbase.zookeeper.MasterAddressTracker
# disable debug/info logging on this script for clarity
-log_level = org.apache.log4j.Level::ERROR
-org.apache.log4j.Logger.getLogger('org.apache.hadoop.hbase').setLevel(log_level)
-org.apache.log4j.Logger.getLogger('org.apache.zookeeper').setLevel(log_level)
+log_level = org.apache.logging.log4j.Level::ERROR
+org.apache.logging.log4j.core.config.Configurator.setAllLevels('org.apache.hadoop.hbase',
log_level)
Review Comment:
I think you can use the methods here
https://github.com/apache/hbase/blob/master/hbase-logging/src/main/java/org/apache/hadoop/hbase/logging/Log4jUtils.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
Apache-HBase commented on PR #5593: URL: https://github.com/apache/hbase/pull/5593#issuecomment-1865443545 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 34s | master passed | | +1 :green_heart: | compile | 0m 41s | master passed | | +1 :green_heart: | shadedjars | 4m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 23s | the patch passed | | +1 :green_heart: | compile | 0m 41s | the patch passed | | +1 :green_heart: | javac | 0m 41s | the patch passed | | +1 :green_heart: | shadedjars | 4m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 230m 39s | hbase-server in the patch failed. | | | | 251m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5593 | | JIRA Issue | HBASE-26268 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a24ce712dcaa 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/2/testReport/ | | Max. process+thread count | 4553 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28271) Infinite waiting on lock acquisition by snapshot can result in unresponsive master
[ https://issues.apache.org/jira/browse/HBASE-28271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799241#comment-17799241 ] ruanhui commented on HBASE-28271: - Yes, you are right, in fact, setting 'hbase.snapshot.zk.coordinated' doesn't make any difference. In the original design, the client could decide to make a zk-based snapshot or a proc-v2-based snapshot by setting 'hbase.snapshot.zk.coordinated', but in later designs, this choice was given to the master. So setting this won't make any difference and this line of code should be removed. Thanks. [~vjasani] > Infinite waiting on lock acquisition by snapshot can result in unresponsive > master > -- > > Key: HBASE-28271 > URL: https://issues.apache.org/jira/browse/HBASE-28271 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.7 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Attachments: image.png > > > When a region is stuck in transition for significant time, any attempt to > take snapshot on the table would keep master handler thread in forever > waiting state. As part of the creating snapshot on enabled or disabled table, > in order to get the table level lock, LockProcedure is executed but if any > region of the table is in transition, LockProcedure could not be executed by > the snapshot handler, resulting in forever waiting until the region > transition is completed, allowing the table level lock to be acquired by the > snapshot handler. > In cases where a region stays in RIT for considerable time, if enough > attempts are made by the client to create snapshots on the table, it can > easily exhaust all handler threads, leading to potentially unresponsive > master. Attached a sample thread dump. > Proposal: The snapshot handler should not stay stuck forever if it cannot > take table level lock, it should fail-fast. > !image.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
Apache-HBase commented on PR #5593: URL: https://github.com/apache/hbase/pull/5593#issuecomment-1865314075 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 18s | master passed | | +1 :green_heart: | compile | 2m 56s | master passed | | +1 :green_heart: | checkstyle | 0m 44s | master passed | | +1 :green_heart: | spotless | 0m 53s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 50s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 3m 4s | the patch passed | | +1 :green_heart: | javac | 3m 4s | the patch passed | | +1 :green_heart: | checkstyle | 0m 41s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 22s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 52s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 39m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5593 | | JIRA Issue | HBASE-26268 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 1f0be471adbb 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
Apache-HBase commented on PR #5593: URL: https://github.com/apache/hbase/pull/5593#issuecomment-1865285529 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 18s | master passed | | +1 :green_heart: | compile | 0m 44s | master passed | | +1 :green_heart: | shadedjars | 6m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 19s | the patch passed | | +1 :green_heart: | compile | 0m 38s | the patch passed | | +1 :green_heart: | javac | 0m 38s | the patch passed | | +1 :green_heart: | shadedjars | 5m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 254m 6s | hbase-server in the patch failed. | | | | 277m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5593 | | JIRA Issue | HBASE-26268 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d595f47e8d08 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/1/testReport/ | | Max. process+thread count | 4724 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28275 Flaky test: Fix 'test_list_decommissioned_regionservers' … [hbase]
NihalJain commented on PR #5594: URL: https://github.com/apache/hbase/pull/5594#issuecomment-1865278046 FYI @bbeaudreault -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-28216) HDFS erasure coding support for table data dirs
[
https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799206#comment-17799206
]
Nihal Jain edited comment on HBASE-28216 at 12/20/23 11:34 PM:
---
Ah I deleted the comment thinking I am mistaken and creating noise, as it
didn't fail when I checked out latest master. Possibly some change in my local
which brought undefined constant error.
But later found the other test was failing since some time in flaky board and
looked into it. I have made a fix for the test TestAdmin2, I still think the
fix should be fine enough as the test is to just verify listing works good for
decommissioning. But given the new context that RS count increased from 1 to 3,
makes me wonder why 3 servers got decommissioned if we are passing 1, see
sample logs in PR. This could be some issue, will analyze more.
FYR HBASE-28275
was (Author: nihaljain.cs):
Ah I deleted the comment thinking I am mistaken and creating noise, as it
didn't fail when I checked out latest master. Possibly some change in my local
which brought undefined constant error.
But later found the other test was failing since some time in flaky board and
looked into it. I have made a fix for the test TestAdmin2, I still think the
fix should be fine enough as the test is to just verify listing works good for
decommissioning. But given the new context that RS count increased from 3 to 1,
makes me wonder why 3 servers got decommissioned if we are passing 1, see
sample logs in PR. This could be some issue, will analyze more.
FYR HBASE-28275
> HDFS erasure coding support for table data dirs
> ---
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy {}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the requested policy is available and enabled via
> DistributedFileSystem.
> getErasureCodingPolicies().
> * During ModifyTableProcedure, add a new state for
> MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY.
> ** When adding or changing a policy, use DistributedFileSystem.
> setErasureCodingPolicy to sync it for the data and archive dir of that table
> (or column in table)
> ** When removing the property or setting it to empty, use
> DistributedFileSystem.
> unsetErasureCodingPolicy to remove it from the data and archive dir.
> Since this new API is in hadoop-3 only, we'll need to add a reflection
> wrapper class for managing the calls and verifying that the API is available.
> We'll similarly do that API check in preflightChecks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28216) HDFS erasure coding support for table data dirs
[
https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799206#comment-17799206
]
Nihal Jain commented on HBASE-28216:
Ah I deleted the comment thinking I am mistaken and creating noise, as it
didn't fail when I checked out latest master. Possibly some change in my local
which brought undefined constant error.
But later found the other test was failing since some time in flaky board and
looked into it. I have made a fix for the test TestAdmin2, I still think the
fix should be fine enough as the test is to just verify listing works good for
decommissioning. But given the new context that RS count increased from 3 to 1,
makes me wonder why 3 servers got decommissioned if we are passing 1, see
sample logs in PR. This could be some issue, will analyze more.
FYR HBASE-28275
> HDFS erasure coding support for table data dirs
> ---
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy {}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the requested policy is available and enabled via
> DistributedFileSystem.
> getErasureCodingPolicies().
> * During ModifyTableProcedure, add a new state for
> MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY.
> ** When adding or changing a policy, use DistributedFileSystem.
> setErasureCodingPolicy to sync it for the data and archive dir of that table
> (or column in table)
> ** When removing the property or setting it to empty, use
> DistributedFileSystem.
> unsetErasureCodingPolicy to remove it from the data and archive dir.
> Since this new API is in hadoop-3 only, we'll need to add a reflection
> wrapper class for managing the calls and verifying that the API is available.
> We'll similarly do that API check in preflightChecks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
Apache-HBase commented on PR #5593: URL: https://github.com/apache/hbase/pull/5593#issuecomment-1865272112 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 0s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 4m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 4m 53s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 233m 40s | hbase-server in the patch failed. | | | | 257m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5593 | | JIRA Issue | HBASE-26268 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d1334b2e260d 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/1/testReport/ | | Max. process+thread count | 4952 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28275 Flaky test: Fix 'test_list_decommissioned_regionservers' … [hbase]
Apache-HBase commented on PR #5594: URL: https://github.com/apache/hbase/pull/5594#issuecomment-1865271533 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 13s | hbase-shell in the patch passed. | | | | 16m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5594/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5594 | | Optional Tests | unit | | uname | Linux 3f336748d8c6 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5594/1/testReport/ | | Max. process+thread count | 1895 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5594/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28275 Flaky test: Fix 'test_list_decommissioned_regionservers' … [hbase]
Apache-HBase commented on PR #5594: URL: https://github.com/apache/hbase/pull/5594#issuecomment-1865269707 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 58s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 6m 40s | hbase-shell in the patch passed. | | | | 13m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5594/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5594 | | Optional Tests | unit | | uname | Linux 8828933c71b7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5594/1/testReport/ | | Max. process+thread count | 1927 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5594/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28275 Flaky test: Fix 'test_list_decommissioned_regionservers' … [hbase]
Apache-HBase commented on PR #5594: URL: https://github.com/apache/hbase/pull/5594#issuecomment-1865262586 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | -0 :warning: | rubocop | 0m 4s | The patch generated 2 new + 148 unchanged - 1 fixed = 150 total (was 149) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | spotless | 0m 38s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 3m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5594/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5594 | | Optional Tests | dupname asflicense spotless rubocop | | uname | Linux 0ffe2c2a5ae6 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | rubocop | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5594/1/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 45 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5594/1/console | | versions | git=2.34.1 maven=3.8.6 rubocop=1.37.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28275 Flaky test: Fix 'test_list_decommissioned_regionservers' … [hbase]
NihalJain commented on PR #5594:
URL: https://github.com/apache/hbase/pull/5594#issuecomment-1865258878
Logs (with extra debug info) to demonstrate why the tests were failing. It
can be seen the count was not 1 as it was expected in test assertion.
Changed logic to do counting relatively instead of absolutely, which was
causing flakiness, due to test case ordering.
```
server_name=HOST_NAME,63596,1703112668148
2023-12-21T04:22:20,258 DEBUG [Listener at localhost/63569-EventThread {}]
zookeeper.ZKWatcher(604): master:63585-0x14905c3,
quorum=127.0.0.1:64405, baseZNode=/hbase Received ZooKeeper Event,
type=NodeChildrenChanged, state=SyncConnected, path=/hbase/draining
2023-12-21T04:22:20,258 INFO
[RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=63585 {}]
master.ServerManager(681): Server HOST_NAME,63596,1703112668148 added to
draining server list.
Took 0.0068 seconds
2023-12-21T04:22:20,259 DEBUG [zk-event-processor-pool-0 {}]
zookeeper.ZKUtil(111): master:63585-0x14905c3, quorum=127.0.0.1:64405,
baseZNode=/hbase Set watcher on existing
znode=/hbase/draining/HOST_NAME,63602,1703112668656
2023-12-21T04:22:20,259 DEBUG [zk-event-processor-pool-0 {}]
zookeeper.ZKUtil(111): master:63585-0x14905c3, quorum=127.0.0.1:64405,
baseZNode=/hbase Set watcher on existing
znode=/hbase/draining/HOST_NAME,63599,1703112668493
2023-12-21T04:22:20,259 DEBUG [zk-event-processor-pool-0 {}]
zookeeper.ZKUtil(111): master:63585-0x14905c3, quorum=127.0.0.1:64405,
baseZNode=/hbase Set watcher on existing
znode=/hbase/draining/HOST_NAME,63596,1703112668148
2023-12-21T04:22:20,259 WARN [zk-event-processor-pool-0 {}]
master.ServerManager(677): Server HOST_NAME,63602,1703112668656 is already in
the draining server list.Ignoring request to add it again.
2023-12-21T04:22:20,259 INFO [zk-event-processor-pool-0 {}]
master.DrainingServerTracker(92): Draining RS node created, adding to list
[HOST_NAME,63602,1703112668656]
2023-12-21T04:22:20,259 WARN [zk-event-processor-pool-0 {}]
master.ServerManager(677): Server HOST_NAME,63599,1703112668493 is already in
the draining server list.Ignoring request to add it again.
2023-12-21T04:22:20,259 INFO [zk-event-processor-pool-0 {}]
master.DrainingServerTracker(92): Draining RS node created, adding to list
[HOST_NAME,63599,1703112668493]
2023-12-21T04:22:20,259 WARN [zk-event-processor-pool-0 {}]
master.ServerManager(677): Server HOST_NAME,63596,1703112668148 is already in
the draining server list.Ignoring request to add it again.
2023-12-21T04:22:20,259 INFO [zk-event-processor-pool-0 {}]
master.DrainingServerTracker(92): Draining RS node created, adding to list
[HOST_NAME,63596,1703112668148]
DECOMMISSIONED REGION SERVERS
HOST_NAME,63599,1703112668493
HOST_NAME,63602,1703112668656
HOST_NAME,63596,1703112668148
3 row(s)
Took 0.0015 seconds
initial_number_of_rows=3
2023-12-21T04:22:20,267 DEBUG [Listener at localhost/63569-EventThread {}]
zookeeper.ZKWatcher(604): master:63585-0x14905c3,
quorum=127.0.0.1:64405, baseZNode=/hbase Received ZooKeeper Event,
type=NodeDeleted, state=SyncConnected,
path=/hbase/draining/HOST_NAME,63596,1703112668148
2023-12-21T04:22:20,267 DEBUG [Listener at localhost/63569-EventThread {}]
zookeeper.ZKWatcher(604): master:63585-0x14905c3,
quorum=127.0.0.1:64405, baseZNode=/hbase Received ZooKeeper Event,
type=NodeChildrenChanged, state=SyncConnected, path=/hbase/draining
2023-12-21T04:22:20,268 INFO [zk-event-processor-pool-0 {}]
master.DrainingServerTracker(109): Draining RS node deleted, removing from list
[HOST_NAME,63596,1703112668148]
Took 0.0070 seconds
2023-12-21T04:22:20,268 DEBUG [zk-event-processor-pool-0 {}]
zookeeper.ZKUtil(111): master:63585-0x14905c3, quorum=127.0.0.1:64405,
baseZNode=/hbase Set watcher on existing
znode=/hbase/draining/HOST_NAME,63602,1703112668656
2023-12-21T04:22:20,269 DEBUG [zk-event-processor-pool-0 {}]
zookeeper.ZKUtil(111): master:63585-0x14905c3, quorum=127.0.0.1:64405,
baseZNode=/hbase Set watcher on existing
znode=/hbase/draining/HOST_NAME,63599,1703112668493
2023-12-21T04:22:20,269 WARN [zk-event-processor-pool-0 {}]
master.ServerManager(677): Server HOST_NAME,63602,1703112668656 is already in
the draining server list.Ignoring request to add it again.
2023-12-21T04:22:20,269 INFO [zk-event-processor-pool-0 {}]
master.DrainingServerTracker(92): Draining RS node created, adding to list
[HOST_NAME,63602,1703112668656]
[jira] [Updated] (HBASE-28275) [Flaky test] Fix 'test_list_decommissioned_regionservers' in TestAdminShell2.java
[ https://issues.apache.org/jira/browse/HBASE-28275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-28275: --- Status: Patch Available (was: Open) > [Flaky test] Fix 'test_list_decommissioned_regionservers' in > TestAdminShell2.java > - > > Key: HBASE-28275 > URL: https://issues.apache.org/jira/browse/HBASE-28275 > Project: HBase > Issue Type: Test >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > > See flaky board: > [https://nightlies.apache.org/hbase/HBase-Find-Flaky-Tests/master/1345/output/dashboard.html] > Failed : > [14675|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14675] > [14676|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14676] > [14677|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14677] > [14678|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14678] > [14679|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14679] > [14680|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14680] > [14681|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14681] > [14682|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14682] > [14683|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14683] > [14684|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14684] > [14685|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14685] > [14686|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14686] > [14687|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14687] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28275) [Flaky test] Fix 'test_list_decommissioned_regionservers' in TestAdminShell2.java
[ https://issues.apache.org/jira/browse/HBASE-28275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-28275: --- Issue Type: Test (was: Bug) > [Flaky test] Fix 'test_list_decommissioned_regionservers' in > TestAdminShell2.java > - > > Key: HBASE-28275 > URL: https://issues.apache.org/jira/browse/HBASE-28275 > Project: HBase > Issue Type: Test >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > > See flaky board: > [https://nightlies.apache.org/hbase/HBase-Find-Flaky-Tests/master/1345/output/dashboard.html] > Failed : > [14675|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14675] > [14676|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14676] > [14677|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14677] > [14678|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14678] > [14679|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14679] > [14680|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14680] > [14681|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14681] > [14682|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14682] > [14683|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14683] > [14684|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14684] > [14685|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14685] > [14686|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14686] > [14687|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14687] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
jbewing commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433255643
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
I also think after quick pass of the benchmarking numbers that the approach
taken in https://issues.apache.org/jira/browse/HBASE-14186 is likely also the
one we should adopt here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
jbewing commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433252640
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
Just posted the raw benchmarking results [in the JIRA in this
comment](https://issues.apache.org/jira/browse/HBASE-28256?focusedCommentId=17799200=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17799200).
I haven't had the chance to go fully over it yet, but my preliminary thought
is that the optimization here and the recycler fix are worth it. It seems like
they each yield like a 50% performance improvement over our current setup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
jbewing commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433252912
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
I can tackle https://issues.apache.org/jira/browse/HBASE-27730 after this
issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-28256) Enhance ByteBufferUtils.readVLong to read 8 bytes at a time
[
https://issues.apache.org/jira/browse/HBASE-28256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799200#comment-17799200
]
Becker Ewing commented on HBASE-28256:
--
I've run some updated benchmarks that factor in a few more variables:
# vLongs of every size are tested
# An additional test parameter for whether to use the NONE recycler or not is
introduced (to account for the refCnt checking overhead)
# An additional readVLong method based off the one introduced in
https://issues.apache.org/jira/browse/HBASE-14186 is added to the benchmarks
The benchmarks and results are available at:
[https://gist.github.com/jbewing/449063eda1dde2c872846ec76c7a10f7]
Results
{noformat}
Benchmark
(shouldUseNoneRecycler) (vlong) Mode Cnt Score Error Units
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
true 9 avgt 5 2.238 ± 0.037 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
true 512 avgt 5 2.693 ± 0.034 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
true 8 avgt 5 2.911 ± 0.198 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
true 2146483640 avgt 5 2.692 ± 0.092 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
true 548755813887 avgt 5 2.914 ± 0.024 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
true 1700104028981 avgt 5 3.302 ± 0.338 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
true 35028797018963967 avgt 5 3.828 ± 0.081 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
true 9123372036854775807 avgt 5 4.066 ± 0.424 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
false 9 avgt 5 2.412 ± 0.027 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
false 512 avgt 5 4.180 ± 0.023 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
false 8 avgt 5 6.041 ± 0.272 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
false 2146483640 avgt 5 4.284 ± 0.054 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
false 548755813887 avgt 5 6.019 ± 0.237 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
false 1700104028981 avgt 5 6.782 ± 0.095 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
false 35028797018963967 avgt 5 8.371 ± 0.215 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB
false 9123372036854775807 avgt 5 10.275 ± 1.541 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
true 9 avgt 5 2.608 ± 2.881 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
true 512 avgt 5 2.679 ± 0.679 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
true 8 avgt 5 2.867 ± 0.191 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
true 2146483640 avgt 5 2.629 ± 0.029 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
true 548755813887 avgt 5 2.906 ± 0.016 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
true 1700104028981 avgt 5 3.243 ± 0.096 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
true 35028797018963967 avgt 5 3.750 ± 0.032 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
true 9123372036854775807 avgt 5 3.985 ± 0.073 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
false 9 avgt 5 2.367 ± 0.054 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
false 512 avgt 5 4.277 ± 0.337 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
false 8 avgt 5 5.984 ± 0.089 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
false 2146483640 avgt 5 4.216 ± 0.040 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
false 548755813887 avgt 5 6.084 ± 0.750 ns/op
ReadVLongBenchmark.readVLongHBase14186_OffHeapBB_Padded
false 1700104028981 avgt 5 6.632 ±
[jira] [Commented] (HBASE-28216) HDFS erasure coding support for table data dirs
[
https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799198#comment-17799198
]
Bryan Beaudreault commented on HBASE-28216:
---
I'm happy to take a look. Can you tell me what branch and test class you are
seeing failing and maybe the exception? I see the branch-3 failure above, but I
took a look at the output and it's not an undefined constant error.
I do think this might have broken the CommissioningTests because it changed the
number of RS in the cluster from 1 to 3, and the test expects 1. I will look at
fixing that, but want to make sure I catch what you're seeing as well. I can't
reproduce an undefined constant issue.
> HDFS erasure coding support for table data dirs
> ---
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy {}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the requested policy is available and enabled via
> DistributedFileSystem.
> getErasureCodingPolicies().
> * During ModifyTableProcedure, add a new state for
> MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY.
> ** When adding or changing a policy, use DistributedFileSystem.
> setErasureCodingPolicy to sync it for the data and archive dir of that table
> (or column in table)
> ** When removing the property or setting it to empty, use
> DistributedFileSystem.
> unsetErasureCodingPolicy to remove it from the data and archive dir.
> Since this new API is in hadoop-3 only, we'll need to add a reflection
> wrapper class for managing the calls and verifying that the API is available.
> We'll similarly do that API check in preflightChecks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
bbeaudreault commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433242660
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
I havent had time to implement the idea in HBASE-27730, but it should be a
pretty easy change. Would you want to tackle it?
Is the optimization here still worth it on top of the recycler fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-28275) [Flaky test] Fix 'test_list_decommissioned_regionservers' in TestAdminShell2.java
[ https://issues.apache.org/jira/browse/HBASE-28275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-28275: --- Description: See flaky board: [https://nightlies.apache.org/hbase/HBase-Find-Flaky-Tests/master/1345/output/dashboard.html] Failed : [14675|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14675] [14676|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14676] [14677|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14677] [14678|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14678] [14679|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14679] [14680|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14680] [14681|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14681] [14682|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14682] [14683|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14683] [14684|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14684] [14685|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14685] [14686|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14686] [14687|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14687] was:See flaky board: https://nightlies.apache.org/hbase/HBase-Find-Flaky-Tests/master/1345/output/dashboard.html > [Flaky test] Fix 'test_list_decommissioned_regionservers' in > TestAdminShell2.java > - > > Key: HBASE-28275 > URL: https://issues.apache.org/jira/browse/HBASE-28275 > Project: HBase > Issue Type: Bug >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > > See flaky board: > [https://nightlies.apache.org/hbase/HBase-Find-Flaky-Tests/master/1345/output/dashboard.html] > Failed : > [14675|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14675] > [14676|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14676] > [14677|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14677] > [14678|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14678] > [14679|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14679] > [14680|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14680] > [14681|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14681] > [14682|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14682] > [14683|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14683] > [14684|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14684] > [14685|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14685] > [14686|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14686] > [14687|https://ci-hbase.apache.org//job/HBase-Flaky-Tests/job/master/14687] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28275) Flaky test: Fix 'list decommissioned regionservers' in admin2_test.rb
[ https://issues.apache.org/jira/browse/HBASE-28275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-28275: --- Description: See flaky board: https://nightlies.apache.org/hbase/HBase-Find-Flaky-Tests/master/1345/output/dashboard.html > Flaky test: Fix 'list decommissioned regionservers' in admin2_test.rb > - > > Key: HBASE-28275 > URL: https://issues.apache.org/jira/browse/HBASE-28275 > Project: HBase > Issue Type: Bug >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > > See flaky board: > https://nightlies.apache.org/hbase/HBase-Find-Flaky-Tests/master/1345/output/dashboard.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28275) [Flaky test] Fix 'test_list_decommissioned_regionservers' in TestAdminShell2.java
[ https://issues.apache.org/jira/browse/HBASE-28275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-28275: --- Summary: [Flaky test] Fix 'test_list_decommissioned_regionservers' in TestAdminShell2.java (was: Flaky test: Fix 'list decommissioned regionservers' in admin2_test.rb) > [Flaky test] Fix 'test_list_decommissioned_regionservers' in > TestAdminShell2.java > - > > Key: HBASE-28275 > URL: https://issues.apache.org/jira/browse/HBASE-28275 > Project: HBase > Issue Type: Bug >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Minor > > See flaky board: > https://nightlies.apache.org/hbase/HBase-Find-Flaky-Tests/master/1345/output/dashboard.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28275) Flaky test: Fix 'list decommissioned regionservers' in admin2_test.rb
Nihal Jain created HBASE-28275: -- Summary: Flaky test: Fix 'list decommissioned regionservers' in admin2_test.rb Key: HBASE-28275 URL: https://issues.apache.org/jira/browse/HBASE-28275 Project: HBase Issue Type: Bug Reporter: Nihal Jain Assignee: Nihal Jain -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
jbewing commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433224938
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
The benchmarks are about halfway through running, so the results aren't
totally settled yet. Just wanted to reiterate what a good find that was
@bbeaudreault as the results when using ByteBuff's with the NONE recycler look
to be substantially faster so far (in the 50% range).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] (HBASE-28216) HDFS erasure coding support for table data dirs
[ https://issues.apache.org/jira/browse/HBASE-28216 ]
Nihal Jain deleted comment on HBASE-28216:
was (Author: nihaljain.cs):
Hey [~bbeaudreault] I think there are failures in hbase-shell after this
commit. I see tests failing locally due to undefined constant. Also checked
commit build is also failing. Mind having a look?
> HDFS erasure coding support for table data dirs
> ---
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy {}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the requested policy is available and enabled via
> DistributedFileSystem.
> getErasureCodingPolicies().
> * During ModifyTableProcedure, add a new state for
> MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY.
> ** When adding or changing a policy, use DistributedFileSystem.
> setErasureCodingPolicy to sync it for the data and archive dir of that table
> (or column in table)
> ** When removing the property or setting it to empty, use
> DistributedFileSystem.
> unsetErasureCodingPolicy to remove it from the data and archive dir.
> Since this new API is in hadoop-3 only, we'll need to add a reflection
> wrapper class for managing the calls and verifying that the API is available.
> We'll similarly do that API check in preflightChecks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
jbewing commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433196386
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
That's a good callout. In lieu of that I'm going to update my
microbenchmarks to also test on ByteBuff's with a NONE recycler (I'm
intentionally not using the NONE recycler in my current microbenchmarks to
simulate real region server operation).
Here's the part of the flamegraph for `next()` that shows where the time is
going for `readVLong` et al
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (HBASE-28216) HDFS erasure coding support for table data dirs
[
https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799175#comment-17799175
]
Nihal Jain edited comment on HBASE-28216 at 12/20/23 9:18 PM:
--
Hey [~bbeaudreault] I think there are failures in hbase-shell after this
commit. I see tests failing locally due to undefined constant. Also checked
commit build is also failing. Mind having a look?
was (Author: nihaljain.cs):
Hey [~bbeaudreault] I think there are failures in hbase-shell after this
commit. I see tests failing locally due to undefined constant. Also checked
commit build os also failing. Mind having a look?
> HDFS erasure coding support for table data dirs
> ---
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy {}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the requested policy is available and enabled via
> DistributedFileSystem.
> getErasureCodingPolicies().
> * During ModifyTableProcedure, add a new state for
> MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY.
> ** When adding or changing a policy, use DistributedFileSystem.
> setErasureCodingPolicy to sync it for the data and archive dir of that table
> (or column in table)
> ** When removing the property or setting it to empty, use
> DistributedFileSystem.
> unsetErasureCodingPolicy to remove it from the data and archive dir.
> Since this new API is in hadoop-3 only, we'll need to add a reflection
> wrapper class for managing the calls and verifying that the API is available.
> We'll similarly do that API check in preflightChecks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28216) HDFS erasure coding support for table data dirs
[
https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799175#comment-17799175
]
Nihal Jain commented on HBASE-28216:
Hey [~bbeaudreault] I think there are failures in hbase-shell after this
commit. I see tests failing locally due to undefined constant. Also checked
commit build os also failing. Mind having a look?
> HDFS erasure coding support for table data dirs
> ---
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy {}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the requested policy is available and enabled via
> DistributedFileSystem.
> getErasureCodingPolicies().
> * During ModifyTableProcedure, add a new state for
> MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY.
> ** When adding or changing a policy, use DistributedFileSystem.
> setErasureCodingPolicy to sync it for the data and archive dir of that table
> (or column in table)
> ** When removing the property or setting it to empty, use
> DistributedFileSystem.
> unsetErasureCodingPolicy to remove it from the data and archive dir.
> Since this new API is in hadoop-3 only, we'll need to add a reflection
> wrapper class for managing the calls and verifying that the API is available.
> We'll similarly do that API check in preflightChecks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28216) HDFS erasure coding support for table data dirs
[
https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799170#comment-17799170
]
Hudson commented on HBASE-28216:
Results for branch master
[build #968 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/968/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/968/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/968/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/968/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HDFS erasure coding support for table data dirs
> ---
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy {}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the requested policy is available and enabled via
> DistributedFileSystem.
> getErasureCodingPolicies().
> * During ModifyTableProcedure, add a new state for
> MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY.
> ** When adding or changing a policy, use DistributedFileSystem.
> setErasureCodingPolicy to sync it for the data and archive dir of that table
> (or column in table)
> ** When removing the property or setting it to empty, use
> DistributedFileSystem.
> unsetErasureCodingPolicy to remove it from the data and archive dir.
> Since this new API is in hadoop-3 only, we'll need to add a reflection
> wrapper class for managing the calls and verifying that the API is available.
> We'll similarly do that API check in preflightChecks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28254) Flaky test: TestTableShell
[
https://issues.apache.org/jira/browse/HBASE-28254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799171#comment-17799171
]
Hudson commented on HBASE-28254:
Results for branch master
[build #968 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/968/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/968/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/968/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/968/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Flaky test: TestTableShell
> --
>
> Key: HBASE-28254
> URL: https://issues.apache.org/jira/browse/HBASE-28254
> Project: HBase
> Issue Type: Test
> Components: flakies, integration tests
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.8
>
>
> The test is running the following Ruby commands:
> {noformat}
> # Instert test data
> @test_table.put(1, "x:a", 1)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> {noformat}
> And validate the versions with:
> {noformat}
> args = { VERSIONS => 10, RAW => true } # Since 4 versions of row with rowkey
> 2 is been added, we can use any number >= 4 for VERSIONS to scan all 4
> versions.
> num_rows = 0
> @test_table._scan_internal(args) do # Raw Scan
> num_rows += 1
> end
> # 5 since , 1 from row key '1' and other 4 from row key '4'
> assert_equal(num_rows, 5,
> 'Num rows scanned without RAW/VERSIONS are not 5')
> {noformat}
> Which sometimes (almost always on fast machines) fails, because it only finds
> 3 versions out of 4. I believe this due to commands are running too fast and
> inserts data with same timestamp, so HBase cannot distinguish them. I'd like
> to add some sleep between puts to fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28270) Remove the xml-maven-plugin in root pom on branch-3
[
https://issues.apache.org/jira/browse/HBASE-28270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799169#comment-17799169
]
Hudson commented on HBASE-28270:
Results for branch branch-3
[build #107 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/107/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/107/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/107/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/107/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove the xml-maven-plugin in root pom on branch-3
> ---
>
> Key: HBASE-28270
> URL: https://issues.apache.org/jira/browse/HBASE-28270
> Project: HBase
> Issue Type: Bug
> Components: build, pom
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 3.0.0-beta-1
>
>
> We only generate ref guide on master branch, so we have already deleted
> related resources on other branches.
> But on branch-3, we still have xml=maven-plugin for including
> hbase-default.xml in ref guide so it will fail the build.
> Let's remove this plugin.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28216) HDFS erasure coding support for table data dirs
[
https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799168#comment-17799168
]
Hudson commented on HBASE-28216:
Results for branch branch-3
[build #107 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/107/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/107/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/107/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/107/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HDFS erasure coding support for table data dirs
> ---
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Labels: patch-available
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy {}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the requested policy is available and enabled via
> DistributedFileSystem.
> getErasureCodingPolicies().
> * During ModifyTableProcedure, add a new state for
> MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY.
> ** When adding or changing a policy, use DistributedFileSystem.
> setErasureCodingPolicy to sync it for the data and archive dir of that table
> (or column in table)
> ** When removing the property or setting it to empty, use
> DistributedFileSystem.
> unsetErasureCodingPolicy to remove it from the data and archive dir.
> Since this new API is in hadoop-3 only, we'll need to add a reflection
> wrapper class for managing the calls and verifying that the API is available.
> We'll similarly do that API check in preflightChecks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
bbeaudreault commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433154777
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
Oh, if the actual problem is the refcnt stuff, Duo and I had an idea for
that a while ago: https://issues.apache.org/jira/browse/HBASE-27730. It was
never implemented only because I was focused on the snapshot scan problem and
we hadn't actually seen evidence of the problem elsewhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-28254) Flaky test: TestTableShell
[
https://issues.apache.org/jira/browse/HBASE-28254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799165#comment-17799165
]
Hudson commented on HBASE-28254:
Results for branch branch-2.5
[build #453 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/453/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/453/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/453/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/453/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/453/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Flaky test: TestTableShell
> --
>
> Key: HBASE-28254
> URL: https://issues.apache.org/jira/browse/HBASE-28254
> Project: HBase
> Issue Type: Test
> Components: flakies, integration tests
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-1, 2.5.8
>
>
> The test is running the following Ruby commands:
> {noformat}
> # Instert test data
> @test_table.put(1, "x:a", 1)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> @test_table.put(2, "x:raw1", 11)
> {noformat}
> And validate the versions with:
> {noformat}
> args = { VERSIONS => 10, RAW => true } # Since 4 versions of row with rowkey
> 2 is been added, we can use any number >= 4 for VERSIONS to scan all 4
> versions.
> num_rows = 0
> @test_table._scan_internal(args) do # Raw Scan
> num_rows += 1
> end
> # 5 since , 1 from row key '1' and other 4 from row key '4'
> assert_equal(num_rows, 5,
> 'Num rows scanned without RAW/VERSIONS are not 5')
> {noformat}
> Which sometimes (almost always on fast machines) fails, because it only finds
> 3 versions out of 4. I believe this due to commands are running too fast and
> inserts data with same timestamp, so HBase cannot distinguish them. I'd like
> to add some sleep between puts to fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
Apache-HBase commented on PR #5593: URL: https://github.com/apache/hbase/pull/5593#issuecomment-1865047541 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | master passed | | +1 :green_heart: | compile | 3m 27s | master passed | | +1 :green_heart: | checkstyle | 1m 2s | master passed | | +1 :green_heart: | spotless | 1m 7s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 14s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 13s | the patch passed | | +1 :green_heart: | compile | 3m 4s | the patch passed | | +1 :green_heart: | javac | 3m 4s | the patch passed | | +1 :green_heart: | checkstyle | 0m 42s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 53s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 57s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 40m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5593 | | JIRA Issue | HBASE-26268 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 6904f1763f78 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5593/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
jbewing commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433097154
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
I'm not sure what approach is better between
[HBASE-14186](https://issues.apache.org/jira/browse/HBASE-14186) and this
approach. This approach has the benefit of in the average case (where the
remaining size of the buffer contains 8 bytes) doing less operations on the
underlying ByteBuff after checking the size byte regardless of the size of the
vLong (3 ops). The ByteBuff operations are typically slow in flamegraphs not
because the actual methods are expensive to evaluate, but because
`checkRefCount()` is typically quite expensive.
I can do some microbenchmarks comparing the current approach,
[HBASE-14186](https://issues.apache.org/jira/browse/HBASE-14186)-like approach,
and the approach taken in this patch against each other for vLongs of all sizes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
jbewing commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433090992
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
Yeah it looks like mvcc, memstoreTs, and sequenceId are all pretty much used
interchangeably from a quick read. I've dug up a bit of literature on the topic
in [HBASE-13389](https://issues.apache.org/jira/browse/HBASE-13389). To
summarize: MVCC values stick around for a while and the performance hit
associated with that is known. They're not timestamps, but they're also not
likely to be super small.
This was a known issue and the performance impact was previously solved with
[HBASE-14861](https://issues.apache.org/jira/browse/HBASE-14186). That patch
affects only HFiles without a data block encoding applied and works by
attempting to eagerly read 4 bytes, then 2 bytes, and then falling back to
single byte reads as opposed to the approach taken in this PR where we attempt
to eagerly read 8 bytes. It's hard to say which approach is better, but I'm
thinking we should at least apply similar optimizations to data block
encodings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-26268) AccessController is missing a couple AdminService methods
[ https://issues.apache.org/jira/browse/HBASE-26268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Connell updated HBASE-26268: Status: Patch Available (was: Open) > AccessController is missing a couple AdminService methods > - > > Key: HBASE-26268 > URL: https://issues.apache.org/jira/browse/HBASE-26268 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Assignee: Charles Connell >Priority: Major > > Sorry for the vague title, not sure if it'd be better to create individual > jiras. I discovered that the following two AdminService endpoints are not > covered by AccessController: > > * updateConfiguration > * clearRegionBlockCache > There may be others, I haven't done a full audit. We should add coprocessor > hooks for these and wrap with AccessController, especially since they can > both have an affect on the cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] HBASE-26268: Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache [hbase]
charlesconnell opened a new pull request, #5593: URL: https://github.com/apache/hbase/pull/5593 Provide coprocessor hooks for updateConfiguration and clearRegionBlockCache. Also, use these new hooks in AccessController to gate access to these operations. This closes a minor security hole. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
Apache-HBase commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1864978265 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | master passed | | +1 :green_heart: | compile | 0m 35s | master passed | | +1 :green_heart: | checkstyle | 0m 17s | master passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 37s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed | | +1 :green_heart: | javac | 0m 33s | the patch passed | | +1 :green_heart: | checkstyle | 0m 14s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 28s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 25m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5576 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 3be5ffe49a30 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
Apache-HBase commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1864972846 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 0s | master passed | | +1 :green_heart: | compile | 0m 15s | master passed | | +1 :green_heart: | shadedjars | 5m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 0m 15s | the patch passed | | +1 :green_heart: | javac | 0m 15s | the patch passed | | +1 :green_heart: | shadedjars | 5m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 12s | hbase-common in the patch passed. | | | | 20m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5576 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 21f2c3586ed3 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/2/testReport/ | | Max. process+thread count | 380 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
bbeaudreault commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433067760
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
It seems like mvcc version and memstoreTs are sort of used interchangeably
for the same thing. So I guess the above indicates that this patch is not going
to only affect "timestamp-like" values. That said I do think that in most
production environments the mvcc is probably pretty large (as above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
Apache-HBase commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1864971776 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 38s | master passed | | +1 :green_heart: | compile | 0m 14s | master passed | | +1 :green_heart: | shadedjars | 5m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 14s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 22s | the patch passed | | +1 :green_heart: | compile | 0m 14s | the patch passed | | +1 :green_heart: | javac | 0m 14s | the patch passed | | +1 :green_heart: | shadedjars | 5m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 52s | hbase-common in the patch passed. | | | | 19m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5576 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 57028c8ccd1c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/2/testReport/ | | Max. process+thread count | 359 (vs. ulimit of 3) | | modules | C: hbase-common U: hbase-common | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5576/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
bbeaudreault commented on code in PR #5576:
URL: https://github.com/apache/hbase/pull/5576#discussion_r1433063478
##
hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java:
##
@@ -226,7 +226,7 @@ public static KeyValue nextShallowCopy(final ByteBuffer bb,
final boolean includ
int kvLength = (int) KeyValue.getKeyValueDataStructureSize(keyLength,
valueLength, tagsLength);
KeyValue keyValue = new KeyValue(bb.array(), underlyingArrayOffset,
kvLength);
if (includesMvccVersion) {
- long mvccVersion = ByteBufferUtils.readVLong(bb);
+ long mvccVersion = ByteBufferUtils.readVLongTimestamp(bb);
Review Comment:
FWIW I don't think the mvcc version is a timestamp. Just taking a look at a
random server that just opened a region (where it logs the next mvcc version),
the value was `564069676`
Maybe we keep readVLong and only use the new method for actual memstoreTs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
jbewing commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1864921833 I've done the `readVLong` -> `readVLongTimestamp` refactor in https://github.com/apache/hbase/pull/5576/commits/9ad49c6623ce6f6d28609d72ac9010f3197e105c. For the reasons listed above, this PR is now only safe to apply to master and branch-3. After this PR is approved, I'll open up a backport PR for branch-2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28271) Infinite waiting on lock acquisition by snapshot can result in unresponsive master
[ https://issues.apache.org/jira/browse/HBASE-28271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799124#comment-17799124 ] Viraj Jasani commented on HBASE-28271: -- Thank you [~frostruan]! I also wonder if setting "hbase.snapshot.zk.coordinated" in any test would even make any difference since we no longer use that config? > Infinite waiting on lock acquisition by snapshot can result in unresponsive > master > -- > > Key: HBASE-28271 > URL: https://issues.apache.org/jira/browse/HBASE-28271 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.7 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Attachments: image.png > > > When a region is stuck in transition for significant time, any attempt to > take snapshot on the table would keep master handler thread in forever > waiting state. As part of the creating snapshot on enabled or disabled table, > in order to get the table level lock, LockProcedure is executed but if any > region of the table is in transition, LockProcedure could not be executed by > the snapshot handler, resulting in forever waiting until the region > transition is completed, allowing the table level lock to be acquired by the > snapshot handler. > In cases where a region stays in RIT for considerable time, if enough > attempts are made by the client to create snapshots on the table, it can > easily exhaust all handler threads, leading to potentially unresponsive > master. Attached a sample thread dump. > Proposal: The snapshot handler should not stay stuck forever if it cannot > take table level lock, it should fail-fast. > !image.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28249 Bump jruby to 9.3.13.0 and related joni and jcodings to 2… [hbase]
chrajeshbabu commented on PR #5568: URL: https://github.com/apache/hbase/pull/5568#issuecomment-1864862381 +1 lgtm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28151 Option to allow/disallow bypassing pre transit check for assing/unassign [hbase]
Apache-HBase commented on PR #5493: URL: https://github.com/apache/hbase/pull/5493#issuecomment-1864859976 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 47s | master passed | | +1 :green_heart: | compile | 1m 36s | master passed | | +1 :green_heart: | shadedjars | 6m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | master passed | | -0 :warning: | patch | 7m 13s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 27s | the patch passed | | +1 :green_heart: | compile | 1m 42s | the patch passed | | +1 :green_heart: | javac | 1m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 25s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 21s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 224m 17s | hbase-server in the patch passed. | | | | 252m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/12/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5493 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9d20758f2f32 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/12/testReport/ | | Max. process+thread count | 4690 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/12/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28151 Option to allow/disallow bypassing pre transit check for assing/unassign [hbase]
Apache-HBase commented on PR #5493: URL: https://github.com/apache/hbase/pull/5493#issuecomment-1864858088 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 14s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 49s | master passed | | +1 :green_heart: | compile | 1m 46s | master passed | | +1 :green_heart: | shadedjars | 4m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | master passed | | -0 :warning: | patch | 6m 13s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 1m 48s | the patch passed | | +1 :green_heart: | javac | 1m 48s | the patch passed | | +1 :green_heart: | shadedjars | 4m 51s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 38s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 30s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 223m 46s | hbase-server in the patch passed. | | | | 251m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/12/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5493 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b4f7494eae6e 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/12/testReport/ | | Max. process+thread count | 4677 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/12/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
bbeaudreault commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1864802424 For branch-2 in that case, I'd probably just create a new readVLongTimestamp method and annotate the old readVLong method as Deprecated. If a use-case comes up down the line, we can un-deprecate it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28274 Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2) [hbase]
Apache-HBase commented on PR #5592: URL: https://github.com/apache/hbase/pull/5592#issuecomment-1864801548 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 7s | master passed | | +1 :green_heart: | compile | 0m 23s | master passed | | +1 :green_heart: | checkstyle | 0m 10s | master passed | | +1 :green_heart: | spotless | 0m 55s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 0m 19s | the patch passed | | +1 :green_heart: | javac | 0m 19s | the patch passed | | +1 :green_heart: | checkstyle | 0m 9s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 22s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | spotless | 0m 53s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 32m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5592/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5592 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 8df0e6d2f265 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-asyncfs U: hbase-asyncfs | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5592/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
jbewing commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1864799163 > I think rename it to readVLongTimestamp is good. That sounds good—I'll go down that path. @Apache9 & @bbeaudreault I have a question before proceeding with making those changes: in branch 2, ByteBufferUtils is an IA.Public and in branch-3/master it's IA.Private (see [HBASE-22044](https://issues.apache.org/jira/browse/HBASE-22044). That means that we can rename the method without worry in branch-3/master, but to backport to branch-2 (which I see no reason not to do) we would probably need to either: keep the existing readVLong methods in branch-2 and make the readVLongTimestamp methods completely separate (or possibly just have readVLong delegate to the new readVLongTimestamp methods. Any preference on how I approach these changes for branch-2 compatibility? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28274 Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2) [hbase]
Apache-HBase commented on PR #5592: URL: https://github.com/apache/hbase/pull/5592#issuecomment-1864781197 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 37s | master passed | | +1 :green_heart: | compile | 0m 14s | master passed | | +1 :green_heart: | shadedjars | 4m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 14s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 22s | the patch passed | | +1 :green_heart: | compile | 0m 15s | the patch passed | | +1 :green_heart: | javac | 0m 15s | the patch passed | | +1 :green_heart: | shadedjars | 4m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 20s | hbase-asyncfs in the patch passed. | | | | 18m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5592/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5592 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c600091e1b93 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5592/1/testReport/ | | Max. process+thread count | 601 (vs. ulimit of 3) | | modules | C: hbase-asyncfs U: hbase-asyncfs | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5592/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28274 Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2) [hbase]
Apache-HBase commented on PR #5592: URL: https://github.com/apache/hbase/pull/5592#issuecomment-1864782683 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 55s | master passed | | +1 :green_heart: | compile | 0m 15s | master passed | | +1 :green_heart: | shadedjars | 4m 55s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 12s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 44s | the patch passed | | +1 :green_heart: | compile | 0m 15s | the patch passed | | +1 :green_heart: | javac | 0m 15s | the patch passed | | +1 :green_heart: | shadedjars | 4m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 23s | hbase-asyncfs in the patch passed. | | | | 19m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5592/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5592 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c64bf27b51e3 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5592/1/testReport/ | | Max. process+thread count | 626 (vs. ulimit of 3) | | modules | C: hbase-asyncfs U: hbase-asyncfs | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5592/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] HBASE-28274 Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2) [hbase]
anmolnar opened a new pull request, #5592: URL: https://github.com/apache/hbase/pull/5592 This is not included in Flaky Test reports, but happens quite frequently for me locally. `testRecover()` passes the expected failure and throws error: ``` java.lang.AssertionError: flush should fail at org.junit.Assert.fail(Assert.java:89) at org.apache.hadoop.hbase.io.asyncfs.TestFanOutOneBlockAsyncDFSOutput.testRecover(TestFanOutOneBlockAsyncDFSOutput.java:154) ``` Looks like it happens only if the other test `testConnectToDatanodeFailed()` precedes it. They do similar thing: restart one of the DataNodes to generate failure in FS connection. I've tried the following things to fix it: - changed the ID of DataNode to be restarted to be unique in the 2 tests, - added `CLUSTER.waitDatanodeFullyStarted()` at the end of testConnectToDatanodeFailed. None of them helped unfortunately, so I ended up with this ugly solution of strict test ordering. Open for any ideas from experts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work started] (HBASE-28274) Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2)
[
https://issues.apache.org/jira/browse/HBASE-28274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-28274 started by Andor Molnar.
> Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2)
> -
>
> Key: HBASE-28274
> URL: https://issues.apache.org/jira/browse/HBASE-28274
> Project: HBase
> Issue Type: Test
> Components: flakies, integration tests, test
>Reporter: Andor Molnar
>Assignee: Andor Molnar
>Priority: Major
>
> The following test sometimes fails for me when running locally with Maven:
> TestFanOutOneBlockAsyncDFSOutput.testRecover()
> I can't really figure out the reason, but it's probably a side effect of the
> preceding test: testConnectToDatanodeFailed(). This test also restarts one of
> the datanodes in the MiniDFS cluster just like testRecover() and it somehow
> causes the failure.
> {noformat}
> java.lang.AssertionError: flush should fail
> at org.junit.Assert.fail(Assert.java:89)
> at
> org.apache.hadoop.hbase.io.asyncfs.TestFanOutOneBlockAsyncDFSOutput.testRecover(TestFanOutOneBlockAsyncDFSOutput.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method){noformat}
> The flush() call is testRecover() should fail, because we restart one of the
> DN in the DFS cluster which is expected to brake connection. It succeeds
> though if the preceding test already restarted a DN. No matter which DN we
> restart, even if they're different, the error occurs.
> I also tried to add CLUSTER.waitDatanodeFullyStarted() at the end of
> testConnectToDatanodeFailed(), looks like it made the tests slightly more
> stable, but didn't help fully.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28274) Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2)
Andor Molnar created HBASE-28274:
Summary: Flaky test: TestFanOutOneBlockAsyncDFSOutput (Part 2)
Key: HBASE-28274
URL: https://issues.apache.org/jira/browse/HBASE-28274
Project: HBase
Issue Type: Test
Components: flakies, integration tests, test
Reporter: Andor Molnar
Assignee: Andor Molnar
The following test sometimes fails for me when running locally with Maven:
TestFanOutOneBlockAsyncDFSOutput.testRecover()
I can't really figure out the reason, but it's probably a side effect of the
preceding test: testConnectToDatanodeFailed(). This test also restarts one of
the datanodes in the MiniDFS cluster just like testRecover() and it somehow
causes the failure.
{noformat}
java.lang.AssertionError: flush should fail
at org.junit.Assert.fail(Assert.java:89)
at
org.apache.hadoop.hbase.io.asyncfs.TestFanOutOneBlockAsyncDFSOutput.testRecover(TestFanOutOneBlockAsyncDFSOutput.java:154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method){noformat}
The flush() call is testRecover() should fail, because we restart one of the DN
in the DFS cluster which is expected to brake connection. It succeeds though if
the preceding test already restarted a DN. No matter which DN we restart, even
if they're different, the error occurs.
I also tried to add CLUSTER.waitDatanodeFullyStarted() at the end of
testConnectToDatanodeFailed(), looks like it made the tests slightly more
stable, but didn't help fully.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28261) Sync jvm11 module flags from hbase-surefire.jdk11.flags to bin/hbase
[ https://issues.apache.org/jira/browse/HBASE-28261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799071#comment-17799071 ] Istvan Toth commented on HBASE-28261: - Technically, and we could probably write a a meven enforcer plugin script for this, but I still think that comments are sufficient. Hopefully this is mostly shaken out by now. > Sync jvm11 module flags from hbase-surefire.jdk11.flags to bin/hbase > > > Key: HBASE-28261 > URL: https://issues.apache.org/jira/browse/HBASE-28261 > Project: HBase > Issue Type: Bug >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Trivial > > All the JDK11 seems to be coming from the runtime code, and we have found > that the missing options are required for the runtime when testing JDK 17. > Only the JDK17 "--add-opens java.base/jdk.internal.util.random=ALL-UNNAMED" > is known to be used by the test code, and probably not by the runtime. > Specifically, these ones are missing: > java.base/java.io=ALL-UNNAMED (pending in HBASE-28259. If that one doesn't > add it to surefire, we need to add that here) > java.base/java.util=ALL-UNNAMED > java.base/java.util.concurrent=ALL-UNNAMED -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HBASE-28261) Sync jvm11 module flags from hbase-surefire.jdk11.flags to bin/hbase
[ https://issues.apache.org/jira/browse/HBASE-28261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799071#comment-17799071 ] Istvan Toth edited comment on HBASE-28261 at 12/20/23 3:39 PM: --- Technically, we could probably write a a meven enforcer plugin script for this, but I still think that comments are sufficient. Hopefully this is mostly shaken out by now. was (Author: stoty): Technically, and we could probably write a a meven enforcer plugin script for this, but I still think that comments are sufficient. Hopefully this is mostly shaken out by now. > Sync jvm11 module flags from hbase-surefire.jdk11.flags to bin/hbase > > > Key: HBASE-28261 > URL: https://issues.apache.org/jira/browse/HBASE-28261 > Project: HBase > Issue Type: Bug >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Trivial > > All the JDK11 seems to be coming from the runtime code, and we have found > that the missing options are required for the runtime when testing JDK 17. > Only the JDK17 "--add-opens java.base/jdk.internal.util.random=ALL-UNNAMED" > is known to be used by the test code, and probably not by the runtime. > Specifically, these ones are missing: > java.base/java.io=ALL-UNNAMED (pending in HBASE-28259. If that one doesn't > add it to surefire, we need to add that here) > java.base/java.util=ALL-UNNAMED > java.base/java.util.concurrent=ALL-UNNAMED -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28261) Sync jvm11 module flags from hbase-surefire.jdk11.flags to bin/hbase
[ https://issues.apache.org/jira/browse/HBASE-28261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799069#comment-17799069 ] Istvan Toth commented on HBASE-28261: - Yes, I was thinking of adding comments in both places. Any other mechanism would be an overkill, and more trouble than it's worth. > Sync jvm11 module flags from hbase-surefire.jdk11.flags to bin/hbase > > > Key: HBASE-28261 > URL: https://issues.apache.org/jira/browse/HBASE-28261 > Project: HBase > Issue Type: Bug >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Trivial > > All the JDK11 seems to be coming from the runtime code, and we have found > that the missing options are required for the runtime when testing JDK 17. > Only the JDK17 "--add-opens java.base/jdk.internal.util.random=ALL-UNNAMED" > is known to be used by the test code, and probably not by the runtime. > Specifically, these ones are missing: > java.base/java.io=ALL-UNNAMED (pending in HBASE-28259. If that one doesn't > add it to surefire, we need to add that here) > java.base/java.util=ALL-UNNAMED > java.base/java.util.concurrent=ALL-UNNAMED -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28240 Phase II: Suspend TRSP and SCP when they can not get regi… [hbase]
GeorryHuang commented on code in PR #5559:
URL: https://github.com/apache/hbase/pull/5559#discussion_r1432815568
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNodeLock.java:
##
@@ -40,30 +50,40 @@ class RegionStateNodeLock {
private final Lock lock = new ReentrantLock();
- private final Condition cond = lock.newCondition();
+ private final Queue waitingQueue = new ArrayDeque<>();
private Object owner;
private int count;
+ private interface QueueEntry {
+
+Object getOwner();
+
+void await() throws ProcedureSuspendedException;
+
+void signal();
Review Comment:
nit: same as above
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java:
##
@@ -342,8 +343,8 @@ public void unlock() {
// Notice that, this does not mean you must use these 3 methods when locking
region state node in
// procedure, you are free to use the above 3 methods if you do not want to
hold the lock when
// suspending the procedure.
- public void lock(Procedure proc) {
-lock.lock(proc);
+ public void lock(Procedure proc, Runnable wakeUp) throws
ProcedureSuspendedException {
Review Comment:
Nit: Could add docs here to describe `wakeup`? I was puzzled when I first
read it, though I eventually understood your purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28151 Option to allow/disallow bypassing pre transit check for assing/unassign [hbase]
Apache-HBase commented on PR #5493: URL: https://github.com/apache/hbase/pull/5493#issuecomment-1864532113 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 20s | master passed | | +1 :green_heart: | compile | 5m 7s | master passed | | +1 :green_heart: | checkstyle | 1m 14s | master passed | | +1 :green_heart: | spotless | 0m 47s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 5m 26s | master passed | | -0 :warning: | patch | 2m 7s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 9s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 24s | the patch passed | | +1 :green_heart: | compile | 4m 50s | the patch passed | | +1 :green_heart: | cc | 4m 50s | the patch passed | | +1 :green_heart: | javac | 4m 50s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 9s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | hbaseprotoc | 1m 35s | the patch passed | | +1 :green_heart: | spotless | 0m 46s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 6m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 54m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/12/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5493 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool | | uname | Linux 405db22916c0 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 83 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/12/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
Apache9 commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1864505325 > It seems like some of the concern or confusion here is related to the generic name readVLong. If that method was used to read a _good use case_ for vlong, the performance at small byte sizes would matter. As it stands, it's only used in one _bad usecase_ the memstore ts. It's questionable whether a timestamp should be a vlong in the first place. > > Since this is only used for a timestamp use case, maybe we can move on from the small byte concerns. Would it make people feel better if we renamed the method to `readVLongTimestamp` with javadoc? @Apache9 > > (Edit: it may not be fair to call it a bad use case. What I mean is that if we care about saving bytes it feels like a timestamp could have a special encoding given the range of possible values, rather than vlong which is designed for all values) I think rename it to readVLongTimestamp is good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28151 Option to allow/disallow bypassing pre transit check for assing/unassign [hbase]
Apache-HBase commented on PR #5493: URL: https://github.com/apache/hbase/pull/5493#issuecomment-1864448444 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 49s | master passed | | +1 :green_heart: | compile | 1m 48s | master passed | | +1 :green_heart: | shadedjars | 4m 51s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | master passed | | -0 :warning: | patch | 6m 10s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 40s | the patch passed | | +1 :green_heart: | compile | 1m 48s | the patch passed | | +1 :green_heart: | javac | 1m 48s | the patch passed | | +1 :green_heart: | shadedjars | 4m 51s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 38s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 31s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 255m 55s | hbase-server in the patch passed. | | | | 284m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/11/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5493 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 083e19ee36c3 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/11/testReport/ | | Max. process+thread count | 4924 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/11/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28256 Enhance ByteBufferUtils.readVLong to read 8 bytes at a time [hbase]
bbeaudreault commented on PR #5576: URL: https://github.com/apache/hbase/pull/5576#issuecomment-1864420597 It seems like some of the concern or confusion here is related to the generic name readVLong. If that method was used to read a _good use case_ for vlong, the performance at small byte sizes would matter. As it stands, it's only used in one _bad usecase_ the memstore ts. It's questionable whether a timestamp should be a vlong in the first place. Since this is only used for a timestamp use case, maybe we can move on from the small byte concerns. Would it make people feel better if we renamed the method to `readVLongTimestamp` with javadoc? @Apache9 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28151 Option to allow/disallow bypassing pre transit check for assing/unassign [hbase]
Apache-HBase commented on PR #5493: URL: https://github.com/apache/hbase/pull/5493#issuecomment-1864402842 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 34s | master passed | | +1 :green_heart: | compile | 1m 16s | master passed | | +1 :green_heart: | shadedjars | 5m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | | -0 :warning: | patch | 6m 15s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 19s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 5m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 26s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 21s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 224m 7s | hbase-server in the patch passed. | | | | 250m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/11/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5493 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 434fbe695755 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/11/testReport/ | | Max. process+thread count | 4683 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/11/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28269 Ruby scripts are broken as they import non-existent classes [hbase]
Apache-HBase commented on PR #5588: URL: https://github.com/apache/hbase/pull/5588#issuecomment-1864197795 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 8s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 256m 40s | root in the patch failed. | | | | 272m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5588/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5588 | | Optional Tests | unit | | uname | Linux 384985adcd2c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5588/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5588/3/testReport/ | | Max. process+thread count | 7513 (vs. ulimit of 3) | | modules | C: hbase-shell . U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5588/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28151 Option to allow/disallow bypassing pre transit check for assing/unassign [hbase]
Apache-HBase commented on PR #5493: URL: https://github.com/apache/hbase/pull/5493#issuecomment-1864115146 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 50s | master passed | | +1 :green_heart: | compile | 3m 46s | master passed | | +1 :green_heart: | checkstyle | 1m 0s | master passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 4m 28s | master passed | | -0 :warning: | patch | 1m 47s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 39s | the patch passed | | +1 :green_heart: | compile | 3m 45s | the patch passed | | +1 :green_heart: | cc | 3m 45s | the patch passed | | +1 :green_heart: | javac | 3m 45s | the patch passed | | +1 :green_heart: | checkstyle | 0m 59s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 9m 46s | Patch does not cause any errors with Hadoop 3.2.4 3.3.6. | | +1 :green_heart: | hbaseprotoc | 1m 34s | the patch passed | | -1 :x: | spotless | 0m 21s | patch has 65 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 6m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 47m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/11/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5493 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool | | uname | Linux 13c6e1470e16 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c07847656 | | Default Java | Eclipse Adoptium-11.0.17+8 | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/11/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5493/11/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28259 Add java.base/java.io=ALL-UNNAMED open to jdk11_jvm_flags [hbase]
zhaomoran commented on code in PR #5581:
URL: https://github.com/apache/hbase/pull/5581#discussion_r1432410587
##
bin/hbase:
##
@@ -496,7 +496,7 @@ add_jdk11_deps_to_classpath() {
}
add_jdk11_jvm_flags() {
- HBASE_OPTS="$HBASE_OPTS
-Dorg.apache.hbase.thirdparty.io.netty.tryReflectionSetAccessible=true
--add-modules jdk.unsupported --add-opens java.base/java.nio=ALL-UNNAMED
--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens
java.base/java.lang=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED --add-opens
java.base/java.lang.reflect=ALL-UNNAMED --add-exports
java.base/jdk.internal.misc=ALL-UNNAMED --add-exports
java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-exports
java.base/sun.net.dns=ALL-UNNAMED --add-exports
java.base/sun.net.util=ALL-UNNAMED"
+ HBASE_OPTS="$HBASE_OPTS
-Dorg.apache.hbase.thirdparty.io.netty.tryReflectionSetAccessible=true
--add-modules jdk.unsupported --add-opens java.base/java.io=ALL-UNNAMED
--add-opens java.base/java.nio=ALL-UNNAMED --add-opens
java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED
--add-opens java.base/jdk.internal.ref=ALL-UNNAMED --add-opens
java.base/java.lang.reflect=ALL-UNNAMED --add-exports
java.base/jdk.internal.misc=ALL-UNNAMED --add-exports
java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-exports
java.base/sun.net.dns=ALL-UNNAMED --add-exports
java.base/sun.net.util=ALL-UNNAMED"
Review Comment:
No similar warning was found when running the test，If consistency is needed,
we can resolved in https://issues.apache.org/jira/browse/HBASE-28261.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-28270) Remove the xml-maven-plugin in root pom on branch-3
[ https://issues.apache.org/jira/browse/HBASE-28270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-28270. --- Hadoop Flags: Reviewed Resolution: Fixed Merged to branch-3. Thanks [~nihaljain.cs] for reviewing! > Remove the xml-maven-plugin in root pom on branch-3 > --- > > Key: HBASE-28270 > URL: https://issues.apache.org/jira/browse/HBASE-28270 > Project: HBase > Issue Type: Bug > Components: build, pom >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-beta-1 > > > We only generate ref guide on master branch, so we have already deleted > related resources on other branches. > But on branch-3, we still have xml=maven-plugin for including > hbase-default.xml in ref guide so it will fail the build. > Let's remove this plugin. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28270 Remove the xml-maven-plugin in root pom on branch-3 [hbase]
Apache9 merged PR #5587: URL: https://github.com/apache/hbase/pull/5587 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
