[GitHub] [hbase] Apache-HBase commented on pull request #5099: HBASE-27704 Quotas can drastically overflow configured limit
Apache-HBase commented on PR #5099: URL: https://github.com/apache/hbase/pull/5099#issuecomment-1467402076 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 23s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | master passed | | +1 :green_heart: | compile | 0m 40s | master passed | | +1 :green_heart: | shadedjars | 4m 27s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 13s | the patch passed | | +1 :green_heart: | compile | 0m 41s | the patch passed | | +1 :green_heart: | javac | 0m 41s | the patch passed | | +1 :green_heart: | shadedjars | 4m 28s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 199m 39s | hbase-server in the patch passed. | | | | 222m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5099/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5099 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5b2431fe935f 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c2b64e7caa | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5099/1/testReport/ | | Max. process+thread count | 2437 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5099/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5080: HBASE-27686: Recovery of BucketCache and Prefetched data after RS Crash
Apache-HBase commented on PR #5080: URL: https://github.com/apache/hbase/pull/5080#issuecomment-1467341366 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 23s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 42s | master passed | | +1 :green_heart: | compile | 0m 42s | master passed | | +1 :green_heart: | shadedjars | 4m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 18s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 4m 35s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 220m 56s | hbase-server in the patch failed. | | | | 243m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5080 | | JIRA Issue | HBASE-27686 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 88a1a2904d30 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c2b64e7caa | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/testReport/ | | Max. process+thread count | 2703 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5080: HBASE-27686: Recovery of BucketCache and Prefetched data after RS Crash
Apache-HBase commented on PR #5080: URL: https://github.com/apache/hbase/pull/5080#issuecomment-1467336694 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 31s | master passed | | +1 :green_heart: | compile | 0m 38s | master passed | | +1 :green_heart: | shadedjars | 4m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 2s | the patch passed | | +1 :green_heart: | compile | 0m 38s | the patch passed | | +1 :green_heart: | javac | 0m 38s | the patch passed | | +1 :green_heart: | shadedjars | 4m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 212m 29s | hbase-server in the patch passed. | | | | 234m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5080 | | JIRA Issue | HBASE-27686 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fbc9596464db 5.4.0-1093-aws #102~18.04.2-Ubuntu SMP Wed Dec 7 00:31:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c2b64e7caa | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/testReport/ | | Max. process+thread count | 2644 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-27712) Remove unused params in region metrics
tianhang tang created HBASE-27712: - Summary: Remove unused params in region metrics Key: HBASE-27712 URL: https://issues.apache.org/jira/browse/HBASE-27712 Project: HBase Issue Type: Bug Reporter: tianhang tang Assignee: tianhang tang Histogram metrics in region have been removed in HBASE-17017, but there are some time cost params still left. Need to remove them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [hbase] sunhelly merged pull request #5098: HBASE-27615 Add group of regionServer on Master webUI
sunhelly merged PR #5098: URL: https://github.com/apache/hbase/pull/5098 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5099: HBASE-27704 Quotas can drastically overflow configured limit
Apache-HBase commented on PR #5099: URL: https://github.com/apache/hbase/pull/5099#issuecomment-1467262179 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 42s | master passed | | +1 :green_heart: | compile | 2m 24s | master passed | | +1 :green_heart: | checkstyle | 0m 34s | master passed | | +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 13s | the patch passed | | +1 :green_heart: | compile | 2m 20s | the patch passed | | +1 :green_heart: | javac | 2m 20s | the patch passed | | +1 :green_heart: | checkstyle | 0m 30s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 46s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. | | +1 :green_heart: | spotless | 0m 37s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 27s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 37m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5099/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5099 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 836966f77f32 5.4.0-1093-aws #102~18.04.2-Ubuntu SMP Wed Dec 7 00:31:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c2b64e7caa | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 82 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5099/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-27704) Quotas can drastically overflow configured limit
[
https://issues.apache.org/jira/browse/HBASE-27704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault reassigned HBASE-27704:
-
Assignee: Bryan Beaudreault
> Quotas can drastically overflow configured limit
>
>
> Key: HBASE-27704
> URL: https://issues.apache.org/jira/browse/HBASE-27704
> Project: HBase
> Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Attachments: Screenshot 2023-03-10 at 5.17.51 PM.png
>
>
> The original implementation did not allow exceeding quota. For example, you
> specify a limit of 10 resource/sec and consume 20 resources, it takes 1.1
> seconds to be able submit another request. This was covered by the
> [testOverconsumption in
> TestRateLimiter|https://github.com/apache/hbase/blame/587b0b4f20bdc0415b6541023e611b69c87dba15/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRateLimiter.java#L97].
> As an incidental part of HBASE-13686, that logic was changed. There is no
> mention of the reasoning behind the change in the issue comments or review
> board, I think it was missed. The goal of that issue was to add different
> refill strategies, but it also modified the over consumption. The
> testOverconsumption was [split out for both refill
> strategies|https://github.com/apache/hbase/blame/master/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRateLimiter.java#L104-L159],
> but the core reasoning was lost. The comment says:
> {code:java}
> // 10 resources are available, but we need to consume 20 resources109
> // Verify that we have to wait at least 1.1sec to have 1 resource available
> {code}
> But the actual test was updated to only require a new resource after 100ms.
> This is incorrect.
> The problem is, when consuming if you go negative it sets to 0
> [here|https://github.com/apache/hbase/blame/master/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java#L187-L191].
> Additionally, when refilling the new logic does a Math.max(0, available +
> refillAmount):
> [here|https://github.com/apache/hbase/blame/master/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java#L159-L163].
> So it's really impossible to get below 0, which is impractical for a rate
> limiter.
> With this setup it's very easy to drastically overconsume the rate limiter.
> See attached screenshot, which shows two humps. The first one has the current
> logic, the second hump has my fix which removes both of those problems. The
> rate limit was set to 500mb/s, but I was easily able to go over 700 mb/s
> without the fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] bbeaudreault opened a new pull request, #5099: HBASE-27704 Quotas can drastically overflow configured limit
bbeaudreault opened a new pull request, #5099: URL: https://github.com/apache/hbase/pull/5099 Removes limits in `refill` which did not allow to go negative. FixedIntervalRateLimiter also needed some changes in getWaitInterval to properly match with how refilling really works. Updated the tests to revive the original intended logic for quotas, where we really test for overconsumption. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] haohao0103 commented on pull request #5097: HBASE-27688 HFile splitting occurs during bulkload, the CREATE_TIME_TS of hfileinfo is 0
haohao0103 commented on PR #5097: URL: https://github.com/apache/hbase/pull/5097#issuecomment-1467215308 > > 2023-03-13T04:16:20,762 ERROR [RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2] regionserver.HRegionServer(2747): * ABORTING region server df65750736d7,35391,1678680878451: Unexpected exception in RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2 * > > java.lang.NullPointerException: null > > at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.switched(ReplicationSourceWALReader.java:224) ~[classes/:?] > > at org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.readWALEntries(SerialReplicationSourceWALReader.java:114) ~[classes/:?] > > at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:178) ~[classes/:?] > > at org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.run(SerialReplicationSourceWALReader.java:35) ~[classes/:?] > > Should not be your problem. Likely a bug introduced by [HBASE-27632](https://issues.apache.org/jira/browse/HBASE-27632). > > Will merge this and file an new issue to solve the problem. ok,i got it .thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5080: HBASE-27686: Recovery of BucketCache and Prefetched data after RS Crash
Apache-HBase commented on PR #5080: URL: https://github.com/apache/hbase/pull/5080#issuecomment-1467191454 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 24s | master passed | | +1 :green_heart: | compile | 2m 25s | master passed | | +1 :green_heart: | checkstyle | 0m 33s | master passed | | +1 :green_heart: | spotless | 0m 38s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 21s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | +1 :green_heart: | compile | 2m 20s | the patch passed | | +1 :green_heart: | javac | 2m 20s | the patch passed | | -0 :warning: | checkstyle | 0m 33s | hbase-server: The patch generated 11 new + 5 unchanged - 0 fixed = 16 total (was 5) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 49s | Patch does not cause any errors with Hadoop 3.2.4 3.3.4. | | -1 :x: | spotless | 0m 18s | patch has 30 errors when running spotless:check, run spotless:apply to fix. | | -1 :x: | spotbugs | 1m 31s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 8s | The patch does not generate ASF License warnings. | | | | 37m 27s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | new org.apache.hadoop.hbase.io.hfile.bucket.BucketCache(String, long, int, int[], int, int, String, int, Configuration) invokes org.apache.hadoop.hbase.io.hfile.bucket.BucketCachePersister.start() At BucketCache.java:int, int, String, int, Configuration) invokes org.apache.hadoop.hbase.io.hfile.bucket.BucketCachePersister.start() At BucketCache.java:[line 323] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5080 | | JIRA Issue | HBASE-27686 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 49b1abc97bff 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c2b64e7caa | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/artifact/yetus-general-check/output/patch-spotless.txt | | spotbugs | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 85 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5080/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27109) Move replication queue storage from zookeeper to a separated HBase table
[
https://issues.apache.org/jira/browse/HBASE-27109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699879#comment-17699879
]
Hudson commented on HBASE-27109:
Results for branch HBASE-27109/table_based_rqs
[build #53 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/53/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/53/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/53/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/HBASE-27109%252Ftable_based_rqs/53/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move replication queue storage from zookeeper to a separated HBase table
>
>
> Key: HBASE-27109
> URL: https://issues.apache.org/jira/browse/HBASE-27109
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> This is a more specific issue based on the works which are already done in
> HBASE-15867.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-27701) ZStdCodec codec implementation class documentation typo
[
https://issues.apache.org/jira/browse/HBASE-27701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699857#comment-17699857
]
Frens Jan Rumph commented on HBASE-27701:
-
Literally no biggie! :D
> ZStdCodec codec implementation class documentation typo
> ---
>
> Key: HBASE-27701
> URL: https://issues.apache.org/jira/browse/HBASE-27701
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Frens Jan Rumph
>Assignee: Frens Jan Rumph
>Priority: Minor
> Fix For: 3.0.0-alpha-4
>
>
> As mentioned in the [u...@hbase.apache.org|mailto:u...@hbase.apache.org]
> mailing list I noticed a small typo in the documentation on compression for
> Zstd. The codec implementation class in the documentation is listed as
> {{org.apache.hadoop.hbase.io.compress.zstd.ZStdCodec}} while the actual class
> is written with a lower case S: {{ZStdCodec.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-27711) Regions permanently stuck in unknown_server state
[
https://issues.apache.org/jira/browse/HBASE-27711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Beitch updated HBASE-27711:
-
Description:
We see this log message and the regions listed are never put back into service
without manual intervention:
{code:java}
NodeC hbasemaster-0 hbasemaster 2023-02-15 14:15:56,149 WARN
[master/NodeC:16000.Chore.1] janitor.CatalogJanitor:
unknown_server=NodeA,16201,1676468874221/__test-table_NodeA__,,1672786676251.a3cac9159205d7611c85dd5c4feeded7.,
unknown_server=NodeA,16201,1676468874221/__test-table_NodeB__,,1672786676579.50e948f0a5bc962aabfe27e9ea4227a5.,
unknown_server=NodeA,16201,1676468874221/aeris_v2,,1672786736251.6ab0292cca294784bce8415cc69c30d4.,
unknown_server=NodeA,16201,1676468874221/aeris_v2,\x06,1672786736251.15d958805892370907a47f31a6e08db1.,
unknown_server=NodeA,16201,1676468874221/aeris_v2,\x12,1672786736251.ac3c78ff6903f52d9e2acf80b8436085.{code}
Normally when we see these unknown_server logs, they do get resolved by
reassigning the regions, however we have a reproducible case where this doesn't
happen.
When this occurs we also see the following log messages related to the regions:
{code:java}
NodeC hbasemaster-0 hbasemaster 2023-02-15 14:10:59,810 WARN
[RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16000]
assignment.AssignmentManager: Reporting NodeC,16201,1676469549542 server does
not match state=OPEN, location=NodeA,16201,1676468874221, table=aeris_v2,
region=6ab0292cca294784bce8415cc69c30d4 (time since last update=3749ms);
closing…
NodeC hbasemaster-0 hbasemaster 2023-02-15 14:11:00,323 WARN
[RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16000]
assignment.AssignmentManager: No matching procedure found for
C,16201,1676469549542 transition on state=OPEN,
location=NodeA,16201,1676468874221, table=aeris_v2,
region=6ab0292cca294784bce8415cc69c30d4 to CLOSED
{code}
This suggests that the master has a different mapping of region to region
server than is expected so it closes the region. We would expect that the
regions get assigned somewhere else and then reopened, but we are not seeing
that.
This log message comes from here:
[https://github.com/apache/hbase/blob/branch-2.4/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java#L1292]
The next thing that is done is calling AssignmentManager's
closeRegionServerSilently method.
Our setup:
We have a three server cluster that runs a full HBASE stack: 3 zookeeper nodes,
an HBASE master active and standby, 3 region servers, 3 HDFS data nodes. For
reliability testing we are running a script that will restart one of the three
nodes, which will have running on it a region server, zookeeper and HDFS
process, and possibly also the HBASE master primary or standby.
In this test we saw the issue after NodeB had been killed at 14:08:33, which
had been running the active master, so the master did switchover to NodeC. Then
at 14:12:56 we saw a "STUCK Region-In-Transition" log for a region on NodeA
(this is another common reproducible issue we plan to open a ticket for) and
then restarted just the region server process on NodeA to get that region
reassigned.
was:
We see this log message and the regions listed are never put back into service
without manual intervention:
{code:java}
NodeC hbasemaster-0 hbasemaster 2023-02-15 14:15:56,149 WARN
[master/NodeC:16000.Chore.1] janitor.CatalogJanitor:
unknown_server=NodeA,16201,1676468874221/__test-table_NodeA__,,1672786676251.a3cac9159205d7611c85dd5c4feeded7.,
unknown_server=NodeA,16201,1676468874221/__test-table_NodeB__,,1672786676579.50e948f0a5bc962aabfe27e9ea4227a5.,
unknown_server=NodeA,16201,1676468874221/aeris_v2,,1672786736251.6ab0292cca294784bce8415cc69c30d4.,
unknown_server=NodeA,16201,1676468874221/aeris_v2,\x06,1672786736251.15d958805892370907a47f31a6e08db1.,
unknown_server=NodeA,16201,1676468874221/aeris_v2,\x12,1672786736251.ac3c78ff6903f52d9e2acf80b8436085.{code}
Normally when we see these unknown_server logs, they do get resolved by
reassigning the regions, however we have a reproducible case where this doesn't
happen.
When this occurs we also see the following log messages related to the regions:
{code:java}
NodeC hbasemaster-0 hbasemaster 2023-02-15 14:10:59,810 WARN
[RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16000]
assignment.AssignmentManager: Reporting NodeC,16201,1676469549542 server does
not match state=OPEN, location=NodeA,16201,1676468874221, table=aeris_v2,
region=6ab0292cca294784bce8415cc69c30d4 (time since last update=3749ms);
closing…
NodeC hbasemaster-0 hbasemaster 2023-02-15 14:11:00,323 WARN
[RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16000]
assignment.AssignmentManager: No matching procedure found for
C,16201,1676469549542 transition on state=OPEN,
location=NodeA,16201,1676468874221, table=aeris_v2,
[jira] [Updated] (HBASE-27711) Regions permanently stuck in unknown_server state
[
https://issues.apache.org/jira/browse/HBASE-27711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Beitch updated HBASE-27711:
-
Environment:
HBase: 2.4.11
Hadoop: 3.2.4
ZooKeeper: 3.7.1
was:
HBase: 2.4.11
Hadoop: 3.2.4
ZooKeeper: 3.7.1
> Regions permanently stuck in unknown_server state
> -
>
> Key: HBASE-27711
> URL: https://issues.apache.org/jira/browse/HBASE-27711
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.4.11
> Environment: HBase: 2.4.11
> Hadoop: 3.2.4
> ZooKeeper: 3.7.1
>Reporter: Aaron Beitch
>Priority: Major
>
> We see this log message and the regions listed are never put back into
> service without manual intervention:
> {code:java}
> NodeC hbasemaster-0 hbasemaster 2023-02-15 14:15:56,149 WARN
> [master/NodeC:16000.Chore.1] janitor.CatalogJanitor:
> unknown_server=NodeA,16201,1676468874221/__test-table_NodeA__,,1672786676251.a3cac9159205d7611c85dd5c4feeded7.,
>
> unknown_server=NodeA,16201,1676468874221/__test-table_NodeB__,,1672786676579.50e948f0a5bc962aabfe27e9ea4227a5.,
>
> unknown_server=NodeA,16201,1676468874221/aeris_v2,,1672786736251.6ab0292cca294784bce8415cc69c30d4.,
>
> unknown_server=NodeA,16201,1676468874221/aeris_v2,\x06,1672786736251.15d958805892370907a47f31a6e08db1.,
>
> unknown_server=NodeA,16201,1676468874221/aeris_v2,\x12,1672786736251.ac3c78ff6903f52d9e2acf80b8436085.{code}
>
> Normally when we see these unknown_server logs, they do get resolved by
> reassigning the regions, however we have a reproducible case where this
> doesn't happen.
> When this occurs we also see the following log messages related to the
> regions:
> {code:java}
> NodeC hbasemaster-0 hbasemaster 2023-02-15 14:10:59,810 WARN
> [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16000]
> assignment.AssignmentManager: Reporting NodeC,16201,1676469549542 server does
> not match state=OPEN, location=NodeA,16201,1676468874221, table=aeris_v2,
> region=6ab0292cca294784bce8415cc69c30d4 (time since last update=3749ms);
> closing…
> NodeC hbasemaster-0 hbasemaster 2023-02-15 14:11:00,323 WARN
> [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16000]
> assignment.AssignmentManager: No matching procedure found for
> C,16201,1676469549542 transition on state=OPEN,
> location=NodeA,16201,1676468874221, table=aeris_v2,
> region=6ab0292cca294784bce8415cc69c30d4 to CLOSED
> {code}
>
> This suggests that the master has a different mapping of region to region
> server than is expected so it closes the region. We would expect that the
> regions get assigned somewhere else and then reopened, but we are not seeing
> that.
> This log message comes from here:
> [https://github.com/apache/hbase/blob/branch-2.4/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java#L1292]
> The next thing that is done is calling AssignmentManager's
> closeRegionServerSilently method.
> Our setup:
> We have a three server cluster that runs a full HBASE stack: 3 zookeeper
> nodes, an HBASE master active and standby, 3 region servers, 3 HDFS data
> nodes. For reliability testing we are running a script that will restart one
> of the three nodes, which will have running on it a region server, zookeeper
> and HDFS process, and possibly also the HBASE master primary or standby.
> In this test we saw the issue after NodeB had been killed at 14:08:33, which
> had been running the active master, so the master did switchover to NodeC.
> Then at 14:12:56 we saw a "STUCK Region-In-Transition" log for a region on
> NodeA (this is another common reproducible issue we plan to open a ticket
> for) and then restarted just the region server process on NodeA to get that
> region reassigned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27711) Regions permanently stuck in unknown_server state
Aaron Beitch created HBASE-27711:
Summary: Regions permanently stuck in unknown_server state
Key: HBASE-27711
URL: https://issues.apache.org/jira/browse/HBASE-27711
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 2.4.11
Environment: HBase: 2.4.11
Hadoop: 3.2.4
ZooKeeper: 3.7.1
Reporter: Aaron Beitch
We see this log message and the regions listed are never put back into service
without manual intervention:
{code:java}
NodeC hbasemaster-0 hbasemaster 2023-02-15 14:15:56,149 WARN
[master/NodeC:16000.Chore.1] janitor.CatalogJanitor:
unknown_server=NodeA,16201,1676468874221/__test-table_NodeA__,,1672786676251.a3cac9159205d7611c85dd5c4feeded7.,
unknown_server=NodeA,16201,1676468874221/__test-table_NodeB__,,1672786676579.50e948f0a5bc962aabfe27e9ea4227a5.,
unknown_server=NodeA,16201,1676468874221/aeris_v2,,1672786736251.6ab0292cca294784bce8415cc69c30d4.,
unknown_server=NodeA,16201,1676468874221/aeris_v2,\x06,1672786736251.15d958805892370907a47f31a6e08db1.,
unknown_server=NodeA,16201,1676468874221/aeris_v2,\x12,1672786736251.ac3c78ff6903f52d9e2acf80b8436085.{code}
Normally when we see these unknown_server logs, they do get resolved by
reassigning the regions, however we have a reproducible case where this doesn't
happen.
When this occurs we also see the following log messages related to the regions:
{code:java}
NodeC hbasemaster-0 hbasemaster 2023-02-15 14:10:59,810 WARN
[RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16000]
assignment.AssignmentManager: Reporting NodeC,16201,1676469549542 server does
not match state=OPEN, location=NodeA,16201,1676468874221, table=aeris_v2,
region=6ab0292cca294784bce8415cc69c30d4 (time since last update=3749ms);
closing…
NodeC hbasemaster-0 hbasemaster 2023-02-15 14:11:00,323 WARN
[RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16000]
assignment.AssignmentManager: No matching procedure found for
C,16201,1676469549542 transition on state=OPEN,
location=NodeA,16201,1676468874221, table=aeris_v2,
region=6ab0292cca294784bce8415cc69c30d4 to CLOSED
{code}
This suggests that the master has a different mapping of region to region
server than is expected so it closes the region. We would expect that the
regions get assigned somewhere else and then reopened, but we are not seeing
that.
This log message comes from here:
[https://github.com/apache/hbase/blob/branch-2.4/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java#L1292]
The next thing that is done is calling AssignmentManager's
closeRegionServerSilently method.
Our setup:
We have a three server cluster that runs a full HBASE stack: 3 zookeeper nodes,
an HBASE master active and standby, 3 region servers, 3 HDFS data nodes. For
reliability testing we are running a script that will restart one of the three
nodes, which will have running on it a region server, zookeeper and HDFS
process, and possibly also the HBASE master primary or standby.
In this test we saw the issue after NodeB had been killed at 14:08:33, which
had been running the active master, so the master did switchover to NodeC. Then
at 14:12:56 we saw a "STUCK Region-In-Transition" log for a region on NodeA
(this is another common reproducible issue we plan to open a ticket for) and
then restarted just the region server process on NodeA to get that region
reassigned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] hgromer commented on a diff in pull request #5051: HBASE-26874 VerifyReplication recompare async
hgromer commented on code in PR #5051:
URL: https://github.com/apache/hbase/pull/5051#discussion_r1134465456
##
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java:
##
@@ -112,6 +123,12 @@ public class VerifyReplication extends Configured
implements Tool {
// Peer Table Name
String peerTableName = null;
+ // we use a static retry/backoff schedule for this part because
+ // we have an opportunity rerun the fetching again when we retry the
+ // higher level recompare as well.
+ private static final int FETCH_LATEST_ROWS_TRIES = 3;
Review Comment:
Yeah that makes sense, I can implement these changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (HBASE-27710) ByteBuff ref counting is too expensive for on-heap buffers
Bryan Beaudreault created HBASE-27710: - Summary: ByteBuff ref counting is too expensive for on-heap buffers Key: HBASE-27710 URL: https://issues.apache.org/jira/browse/HBASE-27710 Project: HBase Issue Type: Improvement Reporter: Bryan Beaudreault We recently were upgrading an internal project's hbase client from 1.2 to 2.5. The project does raw HFile.Reader reading of some hfiles. We noticed an over 4x performance regression with the new client. Profiling showed that the majority of the time was spent in SingleByteBuff.checkRefCount. That ref counting is necessary for off-heap reads, but this case was using on-heap. Removing checkRefCount eliminated the regression. I asked on the dev list and we agreed to remove checkRefCount calls for on-heap buffers: https://lists.apache.org/thread/2qm0g8fbxb0no4ozfx7bw8xmkryt4zz0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27709) Fix UT failures after HBASE-27632 goes in
[
https://issues.apache.org/jira/browse/HBASE-27709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-27709:
--
Description:
Need to fix this.
{noformat}
2023-03-13T04:16:20,762 ERROR
[RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2]
regionserver.HRegionServer(2747): * ABORTING region server
df65750736d7,35391,1678680878451: Unexpected exception in
RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2
*
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.switched(ReplicationSourceWALReader.java:224)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.readWALEntries(SerialReplicationSourceWALReader.java:114)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:178)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.run(SerialReplicationSourceWALReader.java:35)
~[classes/:?]
{noformat}
And this
{noformat}
java.io.EOFException: EOF while reading message size
at
org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.parseDelimitedFrom(ProtobufUtil.java:3730)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufWALStreamReader.next(ProtobufWALStreamReader.java:56)
at
org.apache.hadoop.hbase.wal.WALStreamReader.next(WALStreamReader.java:42)
at
org.apache.hadoop.hbase.wal.TestWALOpenAfterDNRollingStart.test(TestWALOpenAfterDNRollingStart.java:126)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
{noformat}
was:
Need to fix this.
{noformat}
2023-03-13T04:16:20,762 ERROR
[RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2]
regionserver.HRegionServer(2747): *
[jira] [Updated] (HBASE-27709) Fix UT failures after HBASE-27632 goes in
[
https://issues.apache.org/jira/browse/HBASE-27709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-27709:
--
Summary: Fix UT failures after HBASE-27632 goes in (was: NPE in
SerialReplicationSourceWALReader.readWALEntries)
> Fix UT failures after HBASE-27632 goes in
> -
>
> Key: HBASE-27709
> URL: https://issues.apache.org/jira/browse/HBASE-27709
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> Need to fix this.
> {noformat}
> 2023-03-13T04:16:20,762 ERROR
> [RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2]
> regionserver.HRegionServer(2747): * ABORTING region server
> df65750736d7,35391,1678680878451: Unexpected exception in
> RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2
> *
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.switched(ReplicationSourceWALReader.java:224)
> ~[classes/:?]
> at
> org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.readWALEntries(SerialReplicationSourceWALReader.java:114)
> ~[classes/:?]
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:178)
> ~[classes/:?]
> at
> org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.run(SerialReplicationSourceWALReader.java:35)
> ~[classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #5098: HBASE-27615 Add group of regionServer on Master webUI
Apache-HBase commented on PR #5098: URL: https://github.com/apache/hbase/pull/5098#issuecomment-1466261454 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 58s | branch-2 passed | | +1 :green_heart: | javadoc | 0m 26s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 218m 51s | hbase-server in the patch failed. | | | | 230m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5098/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5098 | | Optional Tests | javac javadoc unit | | uname | Linux 74817c4e5568 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 0f67503b9d | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5098/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5098/1/testReport/ | | Max. process+thread count | 2240 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5098/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27707) Region replica replication sometimes orphans WAL queue entries during recovery
[
https://issues.apache.org/jira/browse/HBASE-27707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699676#comment-17699676
]
Nick Dimiduk commented on HBASE-27707:
--
So during reclamation, all queues for the peer named
{{region_replica_replication}} should be ignored? Deleted from the source?
> Region replica replication sometimes orphans WAL queue entries during recovery
> --
>
> Key: HBASE-27707
> URL: https://issues.apache.org/jira/browse/HBASE-27707
> Project: HBase
> Issue Type: Bug
> Components: read replicas, Replication
>Affects Versions: 2.5.0
>Reporter: Nick Dimiduk
>Priority: Critical
>
> Running with timeline-consistent read replicas and
> {{hbase.region.replica.replication.enabled=true}}, we're seeing some region
> servers have WAL queue entires that never clear. This appears to correlate
> with SCP and recovery of replication queues. The result is WALs that build
> up, consuming dangerous amounts of space on HDFS. Remediation requires
> disabling and removing the {{region_replica_replication}} peer, which forces
> an impacted region server to abort with the message "Failed to operate on
> replication queue". We then delete the zk entry, which unlocks the WAL and
> the cleaner chore can sweep them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #5098: HBASE-27615 Add group of regionServer on Master webUI
Apache-HBase commented on PR #5098: URL: https://github.com/apache/hbase/pull/5098#issuecomment-1466218005 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | branch-2 passed | | +1 :green_heart: | javadoc | 0m 22s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 15s | the patch passed | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 193m 29s | hbase-server in the patch passed. | | | | 205m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5098/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5098 | | Optional Tests | javac javadoc unit | | uname | Linux 56c22980588d 5.4.0-1093-aws #102~18.04.2-Ubuntu SMP Wed Dec 7 00:31:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 0f67503b9d | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5098/1/testReport/ | | Max. process+thread count | 2748 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5098/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-27709) NPE in SerialReplicationSourceWALReader.readWALEntries
Duo Zhang created HBASE-27709:
-
Summary: NPE in SerialReplicationSourceWALReader.readWALEntries
Key: HBASE-27709
URL: https://issues.apache.org/jira/browse/HBASE-27709
Project: HBase
Issue Type: Bug
Components: Replication
Reporter: Duo Zhang
Assignee: Duo Zhang
Fix For: 2.6.0, 3.0.0-alpha-4
Need to fix this.
{noformat}
2023-03-13T04:16:20,762 ERROR
[RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2]
regionserver.HRegionServer(2747): * ABORTING region server
df65750736d7,35391,1678680878451: Unexpected exception in
RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2
*
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.switched(ReplicationSourceWALReader.java:224)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.readWALEntries(SerialReplicationSourceWALReader.java:114)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:178)
~[classes/:?]
at
org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.run(SerialReplicationSourceWALReader.java:35)
~[classes/:?]
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] Apache9 commented on pull request #5097: HBASE-27688 HFile splitting occurs during bulkload, the CREATE_TIME_TS of hfileinfo is 0
Apache9 commented on PR #5097: URL: https://github.com/apache/hbase/pull/5097#issuecomment-1466207109 > 2023-03-13T04:16:20,762 ERROR [RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2] regionserver.HRegionServer(2747): * ABORTING region server df65750736d7,35391,1678680878451: Unexpected exception in RS_REFRESH_PEER-regionserver/df65750736d7:0-0.replicationSource,2.replicationSource.wal-reader.df65750736d7%2C35391%2C1678680878451,2 * java.lang.NullPointerException: null at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.switched(ReplicationSourceWALReader.java:224) ~[classes/:?] at org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.readWALEntries(SerialReplicationSourceWALReader.java:114) ~[classes/:?] at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:178) ~[classes/:?] at org.apache.hadoop.hbase.replication.regionserver.SerialReplicationSourceWALReader.run(SerialReplicationSourceWALReader.java:35) ~[classes/:?] Should not be your problem. Likely a bug introduced by HBASE-27632. Will merge this and file an new issue to solve the problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27706) Possible Zstd incompatibility
[
https://issues.apache.org/jira/browse/HBASE-27706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699648#comment-17699648
]
Duo Zhang commented on HBASE-27706:
---
[~apurtell] FYI.
> Possible Zstd incompatibility
> -
>
> Key: HBASE-27706
> URL: https://issues.apache.org/jira/browse/HBASE-27706
> Project: HBase
> Issue Type: Bug
> Components: compatibility
>Affects Versions: 2.5.3
>Reporter: Frens Jan Rumph
>Priority: Major
>
>
> We're in the process of upgrading a HBase installation from 2.2.4 to 2.5.3.
> We're currently using Zstd compression from our Hadoop installation. Due to
> some other class path issues (Netty issues in relation to the async WAL
> provider), we would like to remove Hadoop from the class path.
> However, using the Zstd compression from HBase (which uses
> [https://github.com/luben/zstd-jni]) we seem to hit some incompatibility.
> When restarting a node to use this implementation we had errors like the
> following:
> {code:java}
> 2023-03-10 16:33:01,925 WARN [RS_OPEN_REGION-regionserver/n2:16020-0]
> handler.AssignRegionHandler: Failed to open region
> NAMESPACE:TABLE,,1673888962751.cdb726dad4eaabf765969f195e91c737., will report
> to master
> java.io.IOException: java.io.IOException:
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data
> index and meta index from file
> hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1148)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1091)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:994)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:941)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7228)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7183)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7159)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7118)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7074)
> at
> org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:147)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:100)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.io.IOException:
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data
> index and meta index from file
> hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
> at
> org.apache.hadoop.hbase.regionserver.StoreEngine.openStoreFiles(StoreEngine.java:288)
> at
> org.apache.hadoop.hbase.regionserver.StoreEngine.initialize(StoreEngine.java:338)
> at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:297)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6359)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1114)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> ... 3 more
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading data index and meta index from file
> hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
> at
> org.apache.hadoop.hbase.io.hfile.HFileInfo.initMetaAndIndex(HFileInfo.java:392)
> at
> org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:394)
> at
> org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:518)
> at
> org.apache.hadoop.hbase.regionserver.StoreEngine.createStoreFileAndReader(StoreEngine.java:225)
> at
> org.apache.hadoop.hbase.regionserver.StoreEngine.lambda$openStoreFiles$0(StoreEngine.java:266)
> ... 6 more
> Caused by: java.io.IOException: Premature EOF from inputStream, but still
> need 2883 bytes
> at
>
[jira] [Commented] (HBASE-27707) Region replica replication sometimes orphans WAL queue entries during recovery
[
https://issues.apache.org/jira/browse/HBASE-27707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699647#comment-17699647
]
Duo Zhang commented on HBASE-27707:
---
{quote}
2023-03-09T18:53:18,042 INFO
[RS_CLAIM_REPLICATION_QUEUE-regionserver/datanode-40:16020-0]
regionserver.ReplicationSourceManager: Not transferring queue since the
replication peer region_replica_replication for queue
region_replica_replication does not exist
{quote}
I think this is the problem? For region replica replication, I do not think we
need to actually claim the queue, and we even do not actually record the
progress on zk? When a primary replica is online, it will flush everything to
HDFS so we do not need to replicate the previous WAL files any more...
There must be something wrong with the implementation, which causes unnecessary
claiming operation and cause trouble...
> Region replica replication sometimes orphans WAL queue entries during recovery
> --
>
> Key: HBASE-27707
> URL: https://issues.apache.org/jira/browse/HBASE-27707
> Project: HBase
> Issue Type: Bug
> Components: read replicas, Replication
>Affects Versions: 2.5.0
>Reporter: Nick Dimiduk
>Priority: Critical
>
> Running with timeline-consistent read replicas and
> {{hbase.region.replica.replication.enabled=true}}, we're seeing some region
> servers have WAL queue entires that never clear. This appears to correlate
> with SCP and recovery of replication queues. The result is WALs that build
> up, consuming dangerous amounts of space on HDFS. Remediation requires
> disabling and removing the {{region_replica_replication}} peer, which forces
> an impacted region server to abort with the message "Failed to operate on
> replication queue". We then delete the zk entry, which unlocks the WAL and
> the cleaner chore can sweep them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-27708) CPU hot-spot resolving User subject with tracing disabled
[ https://issues.apache.org/jira/browse/HBASE-27708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-27708: - Summary: CPU hot-spot resolving User subject with tracing disabled (was: Thread contention resolving User subject with tracing disabled) > CPU hot-spot resolving User subject with tracing disabled > - > > Key: HBASE-27708 > URL: https://issues.apache.org/jira/browse/HBASE-27708 > Project: HBase > Issue Type: Bug > Components: Client, tracing >Affects Versions: 2.5.0 >Reporter: Nick Dimiduk >Priority: Major > Attachments: 27708.jpg > > > Even with OpenTelemetry tracing disabled, we see contention related to > populating the string representation of the User principle on the client > side. Can HBase connection cache this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27708) Thread contention resolving User subject with tracing disabled
[ https://issues.apache.org/jira/browse/HBASE-27708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-27708: - Attachment: 27708.jpg > Thread contention resolving User subject with tracing disabled > -- > > Key: HBASE-27708 > URL: https://issues.apache.org/jira/browse/HBASE-27708 > Project: HBase > Issue Type: Bug > Components: Client, tracing >Affects Versions: 2.5.0 >Reporter: Nick Dimiduk >Priority: Major > Attachments: 27708.jpg > > > Even with OpenTelemetry tracing disabled, we see contention related to > populating the string representation of the User principle on the client > side. Can HBase connection cache this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27708) Thread contention resolving User subject with tracing disabled
Nick Dimiduk created HBASE-27708: Summary: Thread contention resolving User subject with tracing disabled Key: HBASE-27708 URL: https://issues.apache.org/jira/browse/HBASE-27708 Project: HBase Issue Type: Bug Components: Client, tracing Affects Versions: 2.5.0 Reporter: Nick Dimiduk Even with OpenTelemetry tracing disabled, we see contention related to populating the string representation of the User principle on the client side. Can HBase connection cache this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27707) Region replica replication sometimes orphans WAL queue entries during recovery
[
https://issues.apache.org/jira/browse/HBASE-27707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699624#comment-17699624
]
Nick Dimiduk commented on HBASE-27707:
--
When a region server aborts due to deleting the replication peer out from under
it, it looks like,
{noformat}
2023-03-08T21:42:09,867 ERROR
[RS_REFRESH_PEER-regionserver/datanode-71:16020-0]
zookeeper.RecoverableZooKeeper: ZooKeeper multi failed after 4 attempts
2023-03-08T21:42:09,868 ERROR
[RS_REFRESH_PEER-regionserver/datanode-71:16020-0] regionserver.HRegionServer:
* ABORTING region server datanode-71.example.com,16020,1677729204016:
Failed to operate on replication queue *
org.apache.hadoop.hbase.replication.ReplicationException: Failed to delete
queue (serverName=datanode-71.example.com,16020,1677729204016,
queueId=region_replica_replication)
at
org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.removeQueue(ZKReplicationQueueStorage.java:176)
~[hbase-replication-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.lambda$deleteQueue$7(ReplicationSourceManager.java:448)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.abortWhenFail(ReplicationSourceManager.java:483)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.deleteQueue(ReplicationSourceManager.java:448)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.removeSource(ReplicationSourceManager.java:439)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.removePeer(ReplicationSourceManager.java:295)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.replication.regionserver.PeerProcedureHandlerImpl.removePeer(PeerProcedureHandlerImpl.java:57)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.replication.regionserver.RefreshPeerCallable.doCall(RefreshPeerCallable.java:52)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.procedure2.BaseRSProcedureCallable.call(BaseRSProcedureCallable.java:35)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.procedure2.BaseRSProcedureCallable.call(BaseRSProcedureCallable.java:23)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:100)
~[hbase-server-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
~[zookeeper-3.5.7.jar:3.5.7]
at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1911)
~[zookeeper-3.5.7.jar:3.5.7]
at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1830)
~[zookeeper-3.5.7.jar:3.5.7]
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:760)
~[hbase-zookeeper-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1314)
~[hbase-zookeeper-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.submitBatchedMultiOrSequential(ZKUtil.java:1006)
~[hbase-zookeeper-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:980)
~[hbase-zookeeper-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:890)
~[hbase-zookeeper-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
at
org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.removeQueue(ZKReplicationQueueStorage.java:174)
~[hbase-replication-2.5.0.0-3.3.3.2-example.jar:2.5.0.0-3.3.3.2-example]
... 14 more
{noformat}
> Region replica
[jira] [Updated] (HBASE-27707) Region replica replication sometimes orphans WAL queue entries during recovery
[
https://issues.apache.org/jira/browse/HBASE-27707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-27707:
-
Priority: Critical (was: Major)
> Region replica replication sometimes orphans WAL queue entries during recovery
> --
>
> Key: HBASE-27707
> URL: https://issues.apache.org/jira/browse/HBASE-27707
> Project: HBase
> Issue Type: Bug
> Components: read replicas, Replication
>Affects Versions: 2.5.0
>Reporter: Nick Dimiduk
>Priority: Critical
>
> Running with timeline-consistent read replicas and
> {{hbase.region.replica.replication.enabled=true}}, we're seeing some region
> servers have WAL queue entires that never clear. This appears to correlate
> with SCP and recovery of replication queues. The result is WALs that build
> up, consuming dangerous amounts of space on HDFS. Remediation requires
> disabling and removing the {{region_replica_replication}} peer, which forces
> an impacted region server to abort with the message "Failed to operate on
> replication queue". We then delete the zk entry, which unlocks the WAL and
> the cleaner chore can sweep them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-27707) Region replica replication sometimes orphans WAL queue entries during recovery
[
https://issues.apache.org/jira/browse/HBASE-27707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699616#comment-17699616
]
Nick Dimiduk commented on HBASE-27707:
--
Here are some notes from tracing pids through the logs.
{noformat}
Procedures And Locks screen shows,
181951 181104 WAITING nonroot
org.apache.hadoop.hbase.master.replication.ClaimReplicationQueuesProcedure
182060 181951 RUNNABLE
org.apache.hadoop.hbase.master.replication.ClaimReplicationQueueRemoteProcedure
2023-03-09T18:35:13,865 DEBUG [RegionServerTracker-0]
procedure2.ProcedureExecutor: Stored pid=181104,
state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
datanode-39.example.com,16020,1677724535336, splitWal=true, meta=false
2023-03-09T18:35:13,865 INFO [RegionServerTracker-0]
assignment.AssignmentManager: Scheduled ServerCrashProcedure pid=181104 for
datanode-39.example.com,16020,1677724535336 (carryingMeta=false)
datanode-39.example.com,16020,1677724535336/CRASHED/regionCount=299/lock=java.util.concurrent.locks.ReentrantReadWriteLock@2900bd9f[Write
locks = 1, Read locks = 0], oldState=ONLINE.
2023-03-09T18:35:13,866 INFO [PEWorker-8] procedure.ServerCrashProcedure:
Start pid=181104, state=RUNNABLE:SERVER_CRASH_START, locked=true;
ServerCrashProcedure datanode-39.example.com,16020,1677724535336,
splitWal=true, meta=false
2023-03-09T18:35:13,867 INFO [PEWorker-8] procedure.ServerCrashProcedure:
datanode-39.example.com,16020,1677724535336 had 299 regions
2023-03-09T18:35:13,868 INFO [PEWorker-8] procedure.ServerCrashProcedure:
Splitting WALs pid=181104, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true;
ServerCrashProcedure datanode-39.example.com,16020,1677724535336,
splitWal=true, meta=false, isMeta: false
2023-03-09T18:35:13,871 DEBUG [PEWorker-8] master.MasterWalManager: Renamed
region directory:
hdfs://hadoop:8020/hbase/WALs/datanode-39.example.com,16020,1677724535336-splitting
2023-03-09T18:35:13,874 INFO [PEWorker-8] master.SplitWALManager:
datanode-39.example.com,16020,1677724535336 WAL count=124, meta=false
... 124 events matching Initialized AND SplitWALRemoteProcedure AND
datanode-39.example.com
... 44 events matching Finished AND SplitWALRemoteProcedure AND
datanode-39.example.com
tracing one of the lost pids:
2023-03-09T18:35:13,884 INFO [PEWorker-4] procedure2.ProcedureExecutor:
Initialized subprocedures=[{pid=181234, ppid=181118, state=RUNNABLE;
SplitWALRemoteProcedure datanode-39.example.com.1678383484540,
worker=datanode-42.example.com,16020,1677724534736}]
2023-03-09T18:35:14,071 INFO
[RS_LOG_REPLAY_OPS-regionserver/datanode-42:16020-0] wal.WALSplitter: Splitting
hdfs://hadoop:8020/hbase/WALs/datanode-39.example.com,16020,1677724535336-splitting/datanode-39.example.com.1678383484540,
size=256.2 M (268665707bytes)
2023-03-09T18:35:17,092 INFO
[RS_LOG_REPLAY_OPS-regionserver/datanode-42:16020-0] wal.WALSplitter: Processed
3459 edits across 16 Regions in 3014 ms; skipped=3018;
WAL=hdfs://hadoop:8020/hbase/WALs/datanode-39.example.com,16020,1677724535336-splitting/datanode-39.example.com.1678383484540,
size=256.2 M, length=268665707, corrupted=false, cancelled=false
2023-03-09T18:35:17,093 DEBUG
[RS_LOG_REPLAY_OPS-regionserver/datanode-42:16020-0]
regionserver.SplitLogWorker: Done splitting WAL
hdfs://hadoop:8020/hbase/WALs/datanode-39.example.com,16020,1677724535336-splitting/datanode-39.example.com.1678383484540
2023-03-09T18:35:17,093 DEBUG
[RS_LOG_REPLAY_OPS-regionserver/datanode-42:16020-0]
regionserver.RemoteProcedureResultReporter: Successfully complete execution of
pid=181234
... no log of an associated "Remote procedure done, pid={}" or Remote procedure
failed, pid={}"
... the master seems to not receive the RPC but RS thinks it sent and
acknowledged?
... pbuf response is an empty object, exceptions checked, and no indication
that procedureResultReporter thread died
what happened to all of these? they all completed successfully
=> 80x RemoteProcedureResultReporter: Successfully complete execution of pid=...
SCP decides to move on...
2023-03-09T18:35:35,798 INFO [PEWorker-14] master.SplitLogManager:
hdfs://hadoop:8020/hbase/WALs/datanode-39.example.com,16020,1677724535336-splitting
dir is empty, no logs to split.
2023-03-09T18:35:35,798 DEBUG [PEWorker-14] procedure.ServerCrashProcedure:
Check if datanode-39.example.com,16020,1677724535336 WAL splitting is done?
wals=0, meta=false
2023-03-09T18:35:35,802 INFO [PEWorker-14] procedure.ServerCrashProcedure:
Remove WAL directory for datanode-39.example.com,16020,1677724535336 failed,
ignore...File
hdfs://hadoop:8020/hbase/WALs/datanode-39.example.com,16020,1677724535336-splitting
does not exist.
now trying to reclaim the queues -- these repeat about every 45 seconds
2023-03-09T18:36:18,297 DEBUG [PEWorker-2]
replication.ClaimReplicationQueuesProcedure: There are 1 replication queues
need to be
[jira] [Created] (HBASE-27707) Region replica replication sometimes orphans WAL queue entries during recovery
Nick Dimiduk created HBASE-27707:
Summary: Region replica replication sometimes orphans WAL queue
entries during recovery
Key: HBASE-27707
URL: https://issues.apache.org/jira/browse/HBASE-27707
Project: HBase
Issue Type: Bug
Components: read replicas, Replication
Affects Versions: 2.5.0
Reporter: Nick Dimiduk
Running with timeline-consistent read replicas and
{{hbase.region.replica.replication.enabled=true}}, we're seeing some region
servers have WAL queue entires that never clear. This appears to correlate with
SCP and recovery of replication queues. The result is WALs that build up,
consuming dangerous amounts of space on HDFS. Remediation requires disabling
and removing the {{region_replica_replication}} peer, which forces an impacted
region server to abort with the message "Failed to operate on replication
queue". We then delete the zk entry, which unlocks the WAL and the cleaner
chore can sweep them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-27615) Add group of regionServer on Master webUI
[ https://issues.apache.org/jira/browse/HBASE-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699584#comment-17699584 ] tianhang tang commented on HBASE-27615: --- [~Xiaolin Ha] Mind help me to review the backport PR? Thank you. > Add group of regionServer on Master webUI > - > > Key: HBASE-27615 > URL: https://issues.apache.org/jira/browse/HBASE-27615 > Project: HBase > Issue Type: Improvement >Reporter: tianhang tang >Assignee: tianhang tang >Priority: Major > Attachments: image-2023-02-06-12-04-03-503.png > > > We do have a RSGroupList on webUI now, but it is still a little inconvenient > if I just want to know which group a specific regionServer belongs to. > So add this info on webUI: > !image-2023-02-06-12-04-03-503.png|width=889,height=174! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HBASE-27700) rolling-restart.sh stop all masters at the same time
[
https://issues.apache.org/jira/browse/HBASE-27700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jack Yang reassigned HBASE-27700:
-
Assignee: Jack Yang
> rolling-restart.sh stop all masters at the same time
>
>
> Key: HBASE-27700
> URL: https://issues.apache.org/jira/browse/HBASE-27700
> Project: HBase
> Issue Type: Improvement
>Reporter: Jack Yang
>Assignee: Jack Yang
>Priority: Minor
>
> The rolling-restart.sh in $HBASE_HOME/bin would stop all master service
> (including the backup ones) at the same time, and then restart them at the
> same time:
> {code:java}
> # The content of rolling-restart.sh
> ...
> # stop all masters before re-start to avoid races for master znode
> "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}" stop master
> "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" \
> --hosts "${HBASE_BACKUP_MASTERS}" stop master-backup
> # make sure the master znode has been deleted before continuing
> zmaster=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool
> zookeeper.znode.master`
> ...
> # all masters are down, now restart
> "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}"
> ${START_CMD_DIST_MODE} master
> "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" \
> --hosts "${HBASE_BACKUP_MASTERS}" ${START_CMD_DIST_MODE} master-backup {code}
> In this way the HMaster service would be unavailable during this period. We
> can restart them in a more graceful way, like this:
> * Stop the backup masters, and then restart them one by one
> * Stop the active master, then one of the backup master would become active
> * Start the original active master, now it's the backup one
> Will upload patch soon.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-27700) rolling-restart.sh stop all masters at the same time
[
https://issues.apache.org/jira/browse/HBASE-27700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699572#comment-17699572
]
Jack Yang commented on HBASE-27700:
---
Thanks! [~psomogyi]
> rolling-restart.sh stop all masters at the same time
>
>
> Key: HBASE-27700
> URL: https://issues.apache.org/jira/browse/HBASE-27700
> Project: HBase
> Issue Type: Improvement
>Reporter: Jack Yang
>Priority: Minor
>
> The rolling-restart.sh in $HBASE_HOME/bin would stop all master service
> (including the backup ones) at the same time, and then restart them at the
> same time:
> {code:java}
> # The content of rolling-restart.sh
> ...
> # stop all masters before re-start to avoid races for master znode
> "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}" stop master
> "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" \
> --hosts "${HBASE_BACKUP_MASTERS}" stop master-backup
> # make sure the master znode has been deleted before continuing
> zmaster=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool
> zookeeper.znode.master`
> ...
> # all masters are down, now restart
> "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}"
> ${START_CMD_DIST_MODE} master
> "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" \
> --hosts "${HBASE_BACKUP_MASTERS}" ${START_CMD_DIST_MODE} master-backup {code}
> In this way the HMaster service would be unavailable during this period. We
> can restart them in a more graceful way, like this:
> * Stop the backup masters, and then restart them one by one
> * Stop the active master, then one of the backup master would become active
> * Start the original active master, now it's the backup one
> Will upload patch soon.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #5098: HBASE-27615 Add group of regionServer on Master webUI
Apache-HBase commented on PR #5098: URL: https://github.com/apache/hbase/pull/5098#issuecomment-1465922411 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 33s | branch-2 passed | | +1 :green_heart: | spotless | 0m 53s | branch has no errors when running spotless:check. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | spotless | 0m 46s | patch has no errors when running spotless:check. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 13m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5098/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5098 | | Optional Tests | dupname asflicense javac spotless | | uname | Linux ee79869dd113 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 0f67503b9d | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5098/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27698) Migrate meta locations from zookeeper to master data may not always possible if we migrate from 1.x HBase
[
https://issues.apache.org/jira/browse/HBASE-27698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699552#comment-17699552
]
Rajeshbabu Chintaguntla commented on HBASE-27698:
-
Here are the steps and reasons leading to meta initialisation failures during
master initialisation.
1) migrate zookeeper meta location to master data: This will not happen because
no znode of meta location
2) Assignment manager startup : This reads the meta location from master data
but could not find because step 1 not created meta location in master data. So
region states is empty.
3) Server manager creating the server crash procedures for dead servers: Since
the region states is empty carrying meta is false all the time for all the dead
servers as well even though a server was carrying meta.
4) Splitting wals: this will not split meta wals because carrying meta is false
from server manager.
5) Checking table region state before init meta: Since the region states is
empty it's considering as fresh cluster setup and trying to create the meta
layout etc in the filesystem.
6) Checking for hbase:meta table in file system: Step 5 considers fresh cluster
initialisation but there is hbase:meta table so there's inconsistency because
of that init meta got failed.
> Migrate meta locations from zookeeper to master data may not always possible
> if we migrate from 1.x HBase
> -
>
> Key: HBASE-27698
> URL: https://issues.apache.org/jira/browse/HBASE-27698
> Project: HBase
> Issue Type: Bug
> Components: migration
>Affects Versions: 2.5.0
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
>
> In HBase 1.x versions meta server location from zookeeper will be removed
> when the server stopped. In such cases migrating to 2.5.x branches may not
> create any meta entries in master data. So in case if we could not find the
> meta location from zookeeper we can get meta location from wal directories
> with .meta extension and add to master data.
> {noformat}
> private void tryMigrateMetaLocationsFromZooKeeper() throws IOException,
> KeeperException {
> // try migrate data from zookeeper
> try (ResultScanner scanner =
> masterRegion.getScanner(new
> Scan().addFamily(HConstants.CATALOG_FAMILY))) {
> if (scanner.next() != null) {
> // notice that all replicas for a region are in the same row, so the
> migration can be
> // done with in a one row put, which means if we have data in catalog
> family then we can
> // make sure that the migration is done.
> LOG.info("The {} family in master local region already has data in
> it, skip migrating...",
> HConstants.CATALOG_FAMILY_STR);
> return;
> }
> }
> // start migrating
> byte[] row =
> CatalogFamilyFormat.getMetaKeyForRegion(RegionInfoBuilder.FIRST_META_REGIONINFO);
> Put put = new Put(row);
> List metaReplicaNodes = zooKeeper.getMetaReplicaNodes();
> StringBuilder info = new StringBuilder("Migrating meta locations:");
> for (String metaReplicaNode : metaReplicaNodes) {
> int replicaId =
> zooKeeper.getZNodePaths().getMetaReplicaIdFromZNode(metaReplicaNode);
> RegionState state = MetaTableLocator.getMetaRegionState(zooKeeper,
> replicaId);
> info.append(" ").append(state);
> put.setTimestamp(state.getStamp());
> MetaTableAccessor.addRegionInfo(put, state.getRegion());
> if (state.getServerName() != null) {
> MetaTableAccessor.addLocation(put, state.getServerName(),
> HConstants.NO_SEQNUM, replicaId);
> }
>
> put.add(CellBuilderFactory.create(CellBuilderType.SHALLOW_COPY).setRow(put.getRow())
> .setFamily(HConstants.CATALOG_FAMILY)
>
> .setQualifier(RegionStateStore.getStateColumn(replicaId)).setTimestamp(put.getTimestamp())
>
> .setType(Cell.Type.Put).setValue(Bytes.toBytes(state.getState().name())).build());
> }
> if (!put.isEmpty()) {
> LOG.info(info.toString());
> masterRegion.update(r -> r.put(put));
> } else {
> LOG.info("No meta location available on zookeeper, skip migrating...");
> }
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-27706) Possible Zstd incompatibility
[
https://issues.apache.org/jira/browse/HBASE-27706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Frens Jan Rumph updated HBASE-27706:
Description:
We're in the process of upgrading a HBase installation from 2.2.4 to 2.5.3.
We're currently using Zstd compression from our Hadoop installation. Due to
some other class path issues (Netty issues in relation to the async WAL
provider), we would like to remove Hadoop from the class path.
However, using the Zstd compression from HBase (which uses
[https://github.com/luben/zstd-jni]) we seem to hit some incompatibility. When
restarting a node to use this implementation we had errors like the following:
{code:java}
2023-03-10 16:33:01,925 WARN [RS_OPEN_REGION-regionserver/n2:16020-0]
handler.AssignRegionHandler: Failed to open region
NAMESPACE:TABLE,,1673888962751.cdb726dad4eaabf765969f195e91c737., will report
to master
java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data
index and meta index from file
hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1148)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1091)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:994)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:941)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7228)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7183)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7159)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7118)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7074)
at
org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:147)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:100)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data
index and meta index from file
hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
at
org.apache.hadoop.hbase.regionserver.StoreEngine.openStoreFiles(StoreEngine.java:288)
at
org.apache.hadoop.hbase.regionserver.StoreEngine.initialize(StoreEngine.java:338)
at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:297)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6359)
at
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1114)
at
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
... 3 more
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading data index and meta index from file
hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
at
org.apache.hadoop.hbase.io.hfile.HFileInfo.initMetaAndIndex(HFileInfo.java:392)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:394)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:518)
at
org.apache.hadoop.hbase.regionserver.StoreEngine.createStoreFileAndReader(StoreEngine.java:225)
at
org.apache.hadoop.hbase.regionserver.StoreEngine.lambda$openStoreFiles$0(StoreEngine.java:266)
... 6 more
Caused by: java.io.IOException: Premature EOF from inputStream, but still need
2883 bytes
at
org.apache.hadoop.hbase.io.util.BlockIOUtils.readFullyWithHeapBuffer(BlockIOUtils.java:153)
at
org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:104)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:644)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$1.nextBlock(HFileBlock.java:1397)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$1.nextBlockWithBlockType(HFileBlock.java:1407)
at
[jira] [Created] (HBASE-27706) Possible Zstd incompatibility
Frens Jan Rumph created HBASE-27706:
---
Summary: Possible Zstd incompatibility
Key: HBASE-27706
URL: https://issues.apache.org/jira/browse/HBASE-27706
Project: HBase
Issue Type: Bug
Components: compatibility
Affects Versions: 2.5.3
Reporter: Frens Jan Rumph
We're in the process of upgrading a HBase installation from 2.2.4 to 2.5.3.
We're currently using Zstd compression from our Hadoop installation. Due to
some other class path issues (Netty issues in relation to the async WAL
provider), we would like to remove Hadoop from the class path.
However, using the Zstd compression from HBase (which uses
[https://github.com/luben/zstd-jni]) we seem to hit some incompatibility. When
restarting a node to use this implementation we had errors like the following:
{code:java}
2023-03-10 16:33:01,925 WARN [RS_OPEN_REGION-regionserver/n2:16020-0]
handler.AssignRegionHandler: Failed to open region
NAMESPACE:TABLE,,1673888962751.cdb726dad4eaabf765969f195e91c737., will report
to master
java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data
index and meta index from file
hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1148)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1091)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:994)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:941)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7228)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7183)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7159)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7118)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7074)
at
org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:147)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:100)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data
index and meta index from file
hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
at
org.apache.hadoop.hbase.regionserver.StoreEngine.openStoreFiles(StoreEngine.java:288)
at
org.apache.hadoop.hbase.regionserver.StoreEngine.initialize(StoreEngine.java:338)
at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:297)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6359)
at
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1114)
at
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
... 3 more
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading data index and meta index from file
hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
at
org.apache.hadoop.hbase.io.hfile.HFileInfo.initMetaAndIndex(HFileInfo.java:392)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:394)
at
org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:518)
at
org.apache.hadoop.hbase.regionserver.StoreEngine.createStoreFileAndReader(StoreEngine.java:225)
at
org.apache.hadoop.hbase.regionserver.StoreEngine.lambda$openStoreFiles$0(StoreEngine.java:266)
... 6 more
Caused by: java.io.IOException: Premature EOF from inputStream, but still need
2883 bytes
at
org.apache.hadoop.hbase.io.util.BlockIOUtils.readFullyWithHeapBuffer(BlockIOUtils.java:153)
at
org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:104)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:644)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$1.nextBlock(HFileBlock.java:1397)
at
[jira] [Resolved] (HBASE-27701) ZStdCodec codec implementation class documentation typo
[
https://issues.apache.org/jira/browse/HBASE-27701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi resolved HBASE-27701.
---
Fix Version/s: 3.0.0-alpha-4
Resolution: Fixed
Thanks for the contribution, [~frensjan]!
> ZStdCodec codec implementation class documentation typo
> ---
>
> Key: HBASE-27701
> URL: https://issues.apache.org/jira/browse/HBASE-27701
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Frens Jan Rumph
>Assignee: Frens Jan Rumph
>Priority: Minor
> Fix For: 3.0.0-alpha-4
>
>
> As mentioned in the [u...@hbase.apache.org|mailto:u...@hbase.apache.org]
> mailing list I noticed a small typo in the documentation on compression for
> Zstd. The codec implementation class in the documentation is listed as
> {{org.apache.hadoop.hbase.io.compress.zstd.ZStdCodec}} while the actual class
> is written with a lower case S: {{ZStdCodec.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HBASE-27701) ZStdCodec codec implementation class documentation typo
[
https://issues.apache.org/jira/browse/HBASE-27701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Somogyi reassigned HBASE-27701:
-
Assignee: Frens Jan Rumph
> ZStdCodec codec implementation class documentation typo
> ---
>
> Key: HBASE-27701
> URL: https://issues.apache.org/jira/browse/HBASE-27701
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Frens Jan Rumph
>Assignee: Frens Jan Rumph
>Priority: Minor
>
> As mentioned in the [u...@hbase.apache.org|mailto:u...@hbase.apache.org]
> mailing list I noticed a small typo in the documentation on compression for
> Zstd. The codec implementation class in the documentation is listed as
> {{org.apache.hadoop.hbase.io.compress.zstd.ZStdCodec}} while the actual class
> is written with a lower case S: {{ZStdCodec.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] petersomogyi merged pull request #5095: HBASE-27701 Fix a misspelling of the ZstdCodec
petersomogyi merged PR #5095: URL: https://github.com/apache/hbase/pull/5095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-27700) rolling-restart.sh stop all masters at the same time
[
https://issues.apache.org/jira/browse/HBASE-27700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699534#comment-17699534
]
Peter Somogyi commented on HBASE-27700:
---
Hi [~jacklove2run], I've added you to the contributor list so you're able to
assign HBASE tickets to yourself in Jira.
> rolling-restart.sh stop all masters at the same time
>
>
> Key: HBASE-27700
> URL: https://issues.apache.org/jira/browse/HBASE-27700
> Project: HBase
> Issue Type: Improvement
>Reporter: Jack Yang
>Priority: Minor
>
> The rolling-restart.sh in $HBASE_HOME/bin would stop all master service
> (including the backup ones) at the same time, and then restart them at the
> same time:
> {code:java}
> # The content of rolling-restart.sh
> ...
> # stop all masters before re-start to avoid races for master znode
> "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}" stop master
> "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" \
> --hosts "${HBASE_BACKUP_MASTERS}" stop master-backup
> # make sure the master znode has been deleted before continuing
> zmaster=`$bin/hbase org.apache.hadoop.hbase.util.HBaseConfTool
> zookeeper.znode.master`
> ...
> # all masters are down, now restart
> "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}"
> ${START_CMD_DIST_MODE} master
> "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" \
> --hosts "${HBASE_BACKUP_MASTERS}" ${START_CMD_DIST_MODE} master-backup {code}
> In this way the HMaster service would be unavailable during this period. We
> can restart them in a more graceful way, like this:
> * Stop the backup masters, and then restart them one by one
> * Stop the active master, then one of the backup master would become active
> * Start the original active master, now it's the backup one
> Will upload patch soon.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [hbase] Apache-HBase commented on pull request #4966: HBASE-27216 Revisit the ReplicationSyncUp tool
Apache-HBase commented on PR #4966: URL: https://github.com/apache/hbase/pull/4966#issuecomment-1465646568 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-27109/table_based_rqs Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 56s | HBASE-27109/table_based_rqs passed | | +1 :green_heart: | compile | 1m 24s | HBASE-27109/table_based_rqs passed | | +1 :green_heart: | shadedjars | 4m 36s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 50s | HBASE-27109/table_based_rqs passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 1m 24s | the patch passed | | +1 :green_heart: | javac | 1m 24s | the patch passed | | +1 :green_heart: | shadedjars | 4m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 24s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 7s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 21s | hbase-replication in the patch passed. | | -1 :x: | unit | 255m 23s | hbase-server in the patch failed. | | | | 282m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4966/14/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/4966 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4c9aea2e1d2b 5.4.0-1093-aws #102~18.04.2-Ubuntu SMP Wed Dec 7 00:31:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-27109/table_based_rqs / 9f95b0e2b6 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4966/14/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4966/14/testReport/ | | Max. process+thread count | 2854 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-replication hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4966/14/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] haohao0103 commented on pull request #5097: HBASE-27688 HFile splitting occurs during bulkload, the CREATE_TIME_TS of hfileinfo is 0
haohao0103 commented on PR #5097: URL: https://github.com/apache/hbase/pull/5097#issuecomment-1465591194 @Apache9 hello ,I have checked the error log, the error content is mainly the connection timeout, I am at a loss to understand this error, could you please guide me? Thank you. The error stack is as follows: ## org.apache.hadoop.hbase.replication.TestReplicationSmallTests.testLoading[0: serialPeer=true] Error Message Failed to get result within timeout, timeout=6ms Stacktrace java.io.IOException: Failed to get result within timeout, timeout=6ms at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:53) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:271) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:440) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:314) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:612) at org.apache.hadoop.hbase.client.ResultScanner.next(ResultScanner.java:95) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5096: HBASE-27702 Remove 'hbase.regionserver.hlog.writer.impl' config
Apache-HBase commented on PR #5096: URL: https://github.com/apache/hbase/pull/5096#issuecomment-1465579507 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 5s | master passed | | +1 :green_heart: | compile | 0m 51s | master passed | | +1 :green_heart: | shadedjars | 4m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 44s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 4m 35s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 214m 4s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 0m 41s | hbase-it in the patch passed. | | | | 237m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5096/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5096 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c3dc8f34df63 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 72d5a46899 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5096/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5096/1/testReport/ | | Max. process+thread count | 2456 (vs. ulimit of 3) | | modules | C: hbase-server hbase-it U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5096/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-27698) Migrate meta locations from zookeeper to master data may not always possible if we migrate from 1.x HBase
[
https://issues.apache.org/jira/browse/HBASE-27698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699457#comment-17699457
]
Rajeshbabu Chintaguntla edited comment on HBASE-27698 at 3/13/23 6:21 AM:
--
[~vjasani]
We are getting following exception when we do express upgrade from HBase 1.1.2
to 2.5.2 which expects zookeeper znode to be present to create the master data
for meta and region info in master cache otherwise going through the meta init
flow and failing.
{noformat}
2023-03-10 09:49:10,828 WARN [PEWorker-3] procedure.InitMetaProcedure: Failed
to init meta, suspend 1000secs
java.io.IOException: Meta table is not partial, please sideline this meta
directory or run HBCK to fix this meta table, e.g. rebuild the server hostname
in ZNode for the meta region
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.deleteMetaTableDirectoryIfPartial(InitMetaProcedure.java:199)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.writeFsLayout(InitMetaProcedure.java:78)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:102)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:54)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1394)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1960)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1987)
{noformat}
As of now there is no way or tool to create the meta location znode.
[~nihaljain.cs] created a script to create the zookeeper node and able to
perform the migration. He is going to contribute it.
Or else other way to migrate is rebuild the meta using HBCK2 which will be
clumsy as we need to perform lot of steps to adding missing regions in meta,
run assignments or restart the services etc.
was (Author: rajeshbabu):
[~vjasani]
We are getting following exception when we do express upgrade from HBase 1.1.2
to 2.5.2 which expects zookeeper znode to be present to create the master data
for meta and region info in master cache.
{noformat}
2023-03-10 09:49:10,828 WARN [PEWorker-3] procedure.InitMetaProcedure: Failed
to init meta, suspend 1000secs
java.io.IOException: Meta table is not partial, please sideline this meta
directory or run HBCK to fix this meta table, e.g. rebuild the server hostname
in ZNode for the meta region
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.deleteMetaTableDirectoryIfPartial(InitMetaProcedure.java:199)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.writeFsLayout(InitMetaProcedure.java:78)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:102)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:54)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1394)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1960)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1987)
{noformat}
As of now there is no way or tool to create the meta location znode.
[~nihaljain.cs] created a script to create the zookeeper node and able to
perform the migration. He is going to contribute it.
Or else other way to migrate is rebuild the meta using HBCK2 which will be
clumsy as we need to perform lot of steps to adding missing regions in meta,
run assignments or restart the services etc.
> Migrate meta locations from zookeeper to master data may not always possible
> if we migrate from 1.x HBase
>
[GitHub] [hbase] Apache-HBase commented on pull request #5096: HBASE-27702 Remove 'hbase.regionserver.hlog.writer.impl' config
Apache-HBase commented on PR #5096: URL: https://github.com/apache/hbase/pull/5096#issuecomment-1465577521 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 4m 53s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 4m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 208m 21s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 0m 45s | hbase-it in the patch passed. | | | | 234m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5096/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5096 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 88d20c8c40cc 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 72d5a46899 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5096/1/testReport/ | | Max. process+thread count | 2722 (vs. ulimit of 3) | | modules | C: hbase-server hbase-it U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5096/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-27698) Migrate meta locations from zookeeper to master data may not always possible if we migrate from 1.x HBase
[
https://issues.apache.org/jira/browse/HBASE-27698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699457#comment-17699457
]
Rajeshbabu Chintaguntla edited comment on HBASE-27698 at 3/13/23 6:19 AM:
--
[~vjasani]
We are getting following exception when we do express upgrade from HBase 1.1.2
to 2.5.2 which expects zookeeper znode to be present to create the master data
for meta and region info in master cache.
{noformat}
2023-03-10 09:49:10,828 WARN [PEWorker-3] procedure.InitMetaProcedure: Failed
to init meta, suspend 1000secs
java.io.IOException: Meta table is not partial, please sideline this meta
directory or run HBCK to fix this meta table, e.g. rebuild the server hostname
in ZNode for the meta region
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.deleteMetaTableDirectoryIfPartial(InitMetaProcedure.java:199)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.writeFsLayout(InitMetaProcedure.java:78)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:102)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:54)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1394)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1960)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1987)
{noformat}
As of now there is no way or tool to create the meta location znode.
[~nihaljain.cs] created a script to create the zookeeper node and able to
perform the migration. He is going to contribute it.
Or else other way to migrate is rebuild the meta using HBCK2 which will be
clumsy as we need to perform lot of steps to adding missing regions in meta,
run assignments or restart the services etc.
was (Author: rajeshbabu):
[~vjasani]
We are getting following exception when we do express upgrade which expects
zookeeper znode to be created.
{noformat}
2023-03-10 09:49:10,828 WARN [PEWorker-3] procedure.InitMetaProcedure: Failed
to init meta, suspend 1000secs
java.io.IOException: Meta table is not partial, please sideline this meta
directory or run HBCK to fix this meta table, e.g. rebuild the server hostname
in ZNode for the meta region
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.deleteMetaTableDirectoryIfPartial(InitMetaProcedure.java:199)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.writeFsLayout(InitMetaProcedure.java:78)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:102)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:54)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1394)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1960)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1987)
{noformat}
As of now there is no way or tool to create the meta location znode.
[~nihaljain.cs] created a script to create the zookeeper node and able to
perform the migration. He is going to contribute it.
Or else other way to migrate is rebuild the meta using HBCK2 which will be
clumsy as we need to perform lot of steps to adding missing regions in meta,
run assignments or restart the services etc.
> Migrate meta locations from zookeeper to master data may not always possible
> if we migrate from 1.x HBase
> -
>
> Key: HBASE-27698
> URL:
[GitHub] [hbase] Apache-HBase commented on pull request #5097: HBASE-27688 HFile splitting occurs during bulkload, the CREATE_TIME_TS of hfileinfo is 0
Apache-HBase commented on PR #5097: URL: https://github.com/apache/hbase/pull/5097#issuecomment-1465573552 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 59s | branch-2 passed | | +1 :green_heart: | compile | 0m 34s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 46s | the patch passed | | +1 :green_heart: | compile | 0m 35s | the patch passed | | +1 :green_heart: | javac | 0m 35s | the patch passed | | +1 :green_heart: | shadedjars | 4m 24s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 208m 49s | hbase-server in the patch failed. | | | | 230m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5097/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5097 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux df65750736d7 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 0f67503b9d | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5097/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5097/1/testReport/ | | Max. process+thread count | 2607 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5097/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #5097: HBASE-27688 HFile splitting occurs during bulkload, the CREATE_TIME_TS of hfileinfo is 0
Apache-HBase commented on PR #5097: URL: https://github.com/apache/hbase/pull/5097#issuecomment-1465567149 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 17s | branch-2 passed | | +1 :green_heart: | compile | 0m 45s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 4m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 196m 33s | hbase-server in the patch passed. | | | | 221m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5097/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5097 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 192710e48fe7 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 0f67503b9d | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5097/1/testReport/ | | Max. process+thread count | 2821 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5097/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
