Re: [PR] HBASE-28456 HBase Restore restores old data if data for the same timestamp is in different hfiles [hbase]
Apache-HBase commented on PR #5775: URL: https://github.com/apache/hbase/pull/5775#issuecomment-2019797454 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 58s | master passed | | +1 :green_heart: | compile | 1m 28s | master passed | | +1 :green_heart: | shadedjars | 6m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | +1 :green_heart: | compile | 1m 24s | the patch passed | | +1 :green_heart: | javac | 1m 24s | the patch passed | | +1 :green_heart: | shadedjars | 6m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 15s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 273m 6s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 22m 43s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 13m 26s | hbase-backup in the patch passed. | | | | 340m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c3f38f403494 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 38aef80ffb | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/testReport/ | | Max. process+thread count | 4881 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce hbase-backup U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28456 HBase Restore restores old data if data for the same timestamp is in different hfiles [hbase]
Apache-HBase commented on PR #5775: URL: https://github.com/apache/hbase/pull/5775#issuecomment-2019544586 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | compile | 2m 8s | master passed | | +1 :green_heart: | shadedjars | 6m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 9s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | compile | 1m 47s | the patch passed | | +1 :green_heart: | javac | 1m 47s | the patch passed | | +1 :green_heart: | shadedjars | 5m 26s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 218m 53s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 13m 59s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 10m 1s | hbase-backup in the patch passed. | | | | 274m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 49e769fb4895 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 38aef80ffb | | Default Java | Eclipse Adoptium-17.0.10+7 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/testReport/ | | Max. process+thread count | 4546 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce hbase-backup U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28453) Support a middle ground between the Average and Fixed interval rate limiters
[
https://issues.apache.org/jira/browse/HBASE-28453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830829#comment-17830829
]
Hudson commented on HBASE-28453:
Results for branch branch-2
[build #1017 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1017/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1017/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1017/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1017/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1017/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Support a middle ground between the Average and Fixed interval rate limiters
>
>
> Key: HBASE-28453
> URL: https://issues.apache.org/jira/browse/HBASE-28453
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Ray Mattingly
>Assignee: Ray Mattingly
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
> Attachments: Screenshot 2024-03-21 at 2.08.51 PM.png, Screenshot
> 2024-03-21 at 2.30.01 PM.png
>
>
> h3. Background
> HBase quotas support two rate limiters: a "fixed" and an "average" interval
> rate limiter.
> h4. FixedIntervalRateLimiter
> The fixed interval rate limiter is simpler: it has a TimeUnit, say 1 second,
> and it refills a resource allotment on the recurring interval. So you may get
> 10 resources every second, and if you exhaust all 10 resources in the first
> millisecond of an interval then you will need to wait 999ms to acquire even 1
> more resource.
> h4. AverageIntervalRateLimiter
> The average interval rate limiter, HBase's default, allows for more flexibly
> timed refilling of the resource allotment. Extending our previous example,
> say you have a 10 reads/sec quota and you have exhausted all 10 resources
> within 1ms of the last full refill. If you request 1 more read then, rather
> than returning a 999ms wait interval indicating the next full refill time,
> the rate limiter will recognize that you only need to wait 99ms before 1 read
> can be available. After 100ms has passed in aggregate since the last full
> refill, it will support the refilling of 1/10th the limit to facilitate the
> request for 1/10th the resources.
> h3. The Problems with Current RateLimiters
> The problem with the fixed interval rate limiter is that it is too strict
> from a latency perspective. It results in quota limits to which we cannot
> fully subscribe with any consistency.
> The problem with the average interval rate limiter is that, in practice, it
> is far too optimistic. For example, a real rate limiter might limit to
> 100MB/sec of read IO per machine. Any multigets that come in will require
> only a tiny fraction of this limit; for example, a 64kb block is only 0.06%
> of the total. As a result, the vast majority of wait intervals end up being
> tiny — like <5ms. This can actually cause an inverse of your intention, where
> setting up a throttle causes a DDOS of your RPC layer via continuous
> throttling and ~immediate retrying. I've discussed this problem in
> https://issues.apache.org/jira/browse/HBASE-28429 and proposed a minimum wait
> interval as the solution there; after some more thinking, I believe this new
> rate limiter would be a less hacky solution to this deficit so I'd like to
> close that Jira in favor of this one.
> See the attached chart where I put in place a 10k req/sec/machine throttle
> for this user at 10:43 to try to curb this high traffic, and it resulted in a
> huge spike of req/sec due to the throttle/retry loop created by the
> AverageIntervalRateLimiter.
> h3. Original Proposal: PartialIntervalRateLimiter as a Solution
> I've implemented a RateLimiter which allows for partial chunks of the overall
> interval to be refilled, by default these chunks are 10% (or 100ms of a 1s
> interval). I've deployed this to a test cluster at my day job and have seen
> this really help our ability to full subscribe to a quota
[jira] [Commented] (HBASE-28449) Fix BackupSystemTable Scans
[
https://issues.apache.org/jira/browse/HBASE-28449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830830#comment-17830830
]
Hudson commented on HBASE-28449:
Results for branch branch-2
[build #1017 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1017/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1017/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1017/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1017/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1017/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix BackupSystemTable Scans
>
>
> Key: HBASE-28449
> URL: https://issues.apache.org/jira/browse/HBASE-28449
> Project: HBase
> Issue Type: Bug
>Reporter: Briana Augenreich
>Assignee: Briana Augenreich
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
>
> When calculating which WALs should be included in an incremental backup the
> backup system does a prefix scan for the last roll log timestamp. This uses
> the backup root in the prefix (.) If you happen have
> multiple backup roots where one is a root of the other you'll get inaccurate
> results.
>
> Since the rowkey is let's modify
> the prefix scan to be .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28456 HBase Restore restores old data if data for the same timestamp is in different hfiles [hbase]
Apache-HBase commented on PR #5775: URL: https://github.com/apache/hbase/pull/5775#issuecomment-2019538318 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 3s | master passed | | +1 :green_heart: | compile | 1m 19s | master passed | | +1 :green_heart: | shadedjars | 5m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 46s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 5m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 47s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 218m 9s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 14m 19s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 9m 30s | hbase-backup in the patch passed. | | | | 268m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7c894cb1c895 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 38aef80ffb | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/testReport/ | | Max. process+thread count | 4646 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce hbase-backup U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
rubenvw-ngdata commented on code in PR #5776:
URL: https://github.com/apache/hbase/pull/5776#discussion_r1539650301
##
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/BackupTestUtil.java:
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+
+public class BackupTestUtil {
Review Comment:
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28456 HBase Restore restores old data if data for the same timestamp is in different hfiles [hbase]
Apache-HBase commented on PR #5775: URL: https://github.com/apache/hbase/pull/5775#issuecomment-2021108682 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 49s | master passed | | +1 :green_heart: | compile | 1m 32s | master passed | | +1 :green_heart: | shadedjars | 5m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 53s | the patch passed | | +1 :green_heart: | compile | 1m 33s | the patch passed | | +1 :green_heart: | javac | 1m 33s | the patch passed | | +1 :green_heart: | shadedjars | 5m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 219m 24s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 13m 49s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 9m 42s | hbase-backup in the patch passed. | | | | 270m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ebddb03258ee 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 38aef80ffb | | Default Java | Eclipse Adoptium-17.0.10+7 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/testReport/ | | Max. process+thread count | 4460 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce hbase-backup U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-28412) Restoring incremental backups to mapped table requires existence of original table
[
https://issues.apache.org/jira/browse/HBASE-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault resolved HBASE-28412.
---
Fix Version/s: 2.6.0
3.0.0-beta-2
Resolution: Fixed
Pushed to branch-2.6+. Thanks [~rubenvw] for the contribution!
I also added you and [~dieterdp_ng] as contributors to the project so that you
can be assigned jiras.
> Restoring incremental backups to mapped table requires existence of original
> table
> --
>
> Key: HBASE-28412
> URL: https://issues.apache.org/jira/browse/HBASE-28412
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Reporter: Dieter De Paepe
>Assignee: Ruben Van Wanzeele
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
>
> It appears that restoring a non-existing table from an incremental backup
> with the "-m" parameter results in an error in the restore client.
> Reproduction steps:
> Build & start hbase:
> {code:java}
> mvn clean install -Phadoop-3.0 -DskipTests
> bin/start-hbase.sh{code}
> In HBase shell: create table and some values:
> {code:java}
> create 'test', 'cf'
> put 'test', 'row1', 'cf:a', 'value1'
> put 'test', 'row2', 'cf:b', 'value2'
> put 'test', 'row3', 'cf:c', 'value3'
> scan 'test' {code}
> Create a full backup:
> {code:java}
> bin/hbase backup create full file:/tmp/hbase-backup{code}
> Adjust some data through HBase shell:
> {code:java}
> put 'test', 'row1', 'cf:a', 'value1-new'
> scan 'test' {code}
> Create an incremental backup:
> {code:java}
> bin/hbase backup create incremental file:/tmp/hbase-backup {code}
> Delete the original table in HBase shell:
> {code:java}
> disable 'test'
> drop 'test' {code}
> Restore the incremental backup under a new table name:
> {code:java}
> bin/hbase backup history
> bin/hbase restore file:/tmp/hbase-backup -t "test" -m
> "test-restored" {code}
> This results in the following output / error:
> {code:java}
> ...
> 2024-03-25T13:38:53,062 WARN [main {}] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2024-03-25T13:38:53,174 INFO [main {}] Configuration.deprecation:
> hbase.client.pause.cqtbe is deprecated. Instead, use
> hbase.client.pause.server.overloaded
> 2024-03-25T13:38:53,554 INFO [main {}] impl.RestoreTablesClient: HBase table
> test-restored does not exist. It will be created during restore process
> 2024-03-25T13:38:53,593 INFO [main {}] impl.RestoreTablesClient: Restoring
> 'test' to 'test-restored' from full backup image
> file:/tmp/hbase-backup/backup_1711370230143/default/test
> 2024-03-25T13:38:53,707 INFO [main {}] util.BackupUtils: Creating target
> table 'test-restored'
> 2024-03-25T13:38:54,546 INFO [main {}] mapreduce.MapReduceRestoreJob:
> Restore test into test-restored
> 2024-03-25T13:38:54,646 INFO [main {}] mapreduce.HFileOutputFormat2:
> bulkload locality sensitive enabled
> 2024-03-25T13:38:54,647 INFO [main {}] mapreduce.HFileOutputFormat2: Looking
> up current regions for table test-restored
> 2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2:
> Configuring 1 reduce partitions to match current region count for all tables
> 2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2: Writing
> partition information to
> file:/tmp/hbase-tmp/partitions_0667b6e2-79ef-4cfe-97e1-abb204ee420d
> 2024-03-25T13:38:54,687 INFO [main {}] compress.CodecPool: Got brand-new
> compressor [.deflate]
> 2024-03-25T13:38:54,713 INFO [main {}] mapreduce.HFileOutputFormat2:
> Incremental output configured for tables: test-restored
> 2024-03-25T13:38:54,715 WARN [main {}] mapreduce.TableMapReduceUtil: The
> addDependencyJars(Configuration, Class...) method has been deprecated
> since it is easy to use incorrectly. Most users should rely on
> addDependencyJars(Job) instead. See HBASE-8386 for more details.
> 2024-03-25T13:38:54,742 WARN [main {}] impl.MetricsConfig: Cannot locate
> configuration: tried
> hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties
> 2024-03-25T13:38:54,834 INFO [main {}] input.FileInputFormat: Total input
> files to process : 1
> 2024-03-25T13:38:54,853 INFO [main {}] mapreduce.JobSubmitter: number of
> splits:1
> 2024-03-25T13:38:54,964 INFO [main {}] mapreduce.JobSubmitter: Submitting
> tokens for job: job_local748155768_0001
> 2024-03-25T13:38:54,967 INFO [main {}] mapreduce.JobSubmitter: Executing
> with tokens: []
> 2024-03-25T13:38:55,076 INFO [main {}] mapred.LocalDistributedCacheManager:
> Creating symlink:
> /tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
> <-
[jira] [Updated] (HBASE-28412) Restoring incremental backups to mapped table requires existence of original table
[
https://issues.apache.org/jira/browse/HBASE-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault updated HBASE-28412:
--
Affects Version/s: (was: 2.6.0)
(was: 4.0.0-alpha-1)
> Restoring incremental backups to mapped table requires existence of original
> table
> --
>
> Key: HBASE-28412
> URL: https://issues.apache.org/jira/browse/HBASE-28412
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Reporter: Dieter De Paepe
>Assignee: Ruben Van Wanzeele
>Priority: Major
> Labels: pull-request-available
>
> It appears that restoring a non-existing table from an incremental backup
> with the "-m" parameter results in an error in the restore client.
> Reproduction steps:
> Build & start hbase:
> {code:java}
> mvn clean install -Phadoop-3.0 -DskipTests
> bin/start-hbase.sh{code}
> In HBase shell: create table and some values:
> {code:java}
> create 'test', 'cf'
> put 'test', 'row1', 'cf:a', 'value1'
> put 'test', 'row2', 'cf:b', 'value2'
> put 'test', 'row3', 'cf:c', 'value3'
> scan 'test' {code}
> Create a full backup:
> {code:java}
> bin/hbase backup create full file:/tmp/hbase-backup{code}
> Adjust some data through HBase shell:
> {code:java}
> put 'test', 'row1', 'cf:a', 'value1-new'
> scan 'test' {code}
> Create an incremental backup:
> {code:java}
> bin/hbase backup create incremental file:/tmp/hbase-backup {code}
> Delete the original table in HBase shell:
> {code:java}
> disable 'test'
> drop 'test' {code}
> Restore the incremental backup under a new table name:
> {code:java}
> bin/hbase backup history
> bin/hbase restore file:/tmp/hbase-backup -t "test" -m
> "test-restored" {code}
> This results in the following output / error:
> {code:java}
> ...
> 2024-03-25T13:38:53,062 WARN [main {}] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2024-03-25T13:38:53,174 INFO [main {}] Configuration.deprecation:
> hbase.client.pause.cqtbe is deprecated. Instead, use
> hbase.client.pause.server.overloaded
> 2024-03-25T13:38:53,554 INFO [main {}] impl.RestoreTablesClient: HBase table
> test-restored does not exist. It will be created during restore process
> 2024-03-25T13:38:53,593 INFO [main {}] impl.RestoreTablesClient: Restoring
> 'test' to 'test-restored' from full backup image
> file:/tmp/hbase-backup/backup_1711370230143/default/test
> 2024-03-25T13:38:53,707 INFO [main {}] util.BackupUtils: Creating target
> table 'test-restored'
> 2024-03-25T13:38:54,546 INFO [main {}] mapreduce.MapReduceRestoreJob:
> Restore test into test-restored
> 2024-03-25T13:38:54,646 INFO [main {}] mapreduce.HFileOutputFormat2:
> bulkload locality sensitive enabled
> 2024-03-25T13:38:54,647 INFO [main {}] mapreduce.HFileOutputFormat2: Looking
> up current regions for table test-restored
> 2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2:
> Configuring 1 reduce partitions to match current region count for all tables
> 2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2: Writing
> partition information to
> file:/tmp/hbase-tmp/partitions_0667b6e2-79ef-4cfe-97e1-abb204ee420d
> 2024-03-25T13:38:54,687 INFO [main {}] compress.CodecPool: Got brand-new
> compressor [.deflate]
> 2024-03-25T13:38:54,713 INFO [main {}] mapreduce.HFileOutputFormat2:
> Incremental output configured for tables: test-restored
> 2024-03-25T13:38:54,715 WARN [main {}] mapreduce.TableMapReduceUtil: The
> addDependencyJars(Configuration, Class...) method has been deprecated
> since it is easy to use incorrectly. Most users should rely on
> addDependencyJars(Job) instead. See HBASE-8386 for more details.
> 2024-03-25T13:38:54,742 WARN [main {}] impl.MetricsConfig: Cannot locate
> configuration: tried
> hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties
> 2024-03-25T13:38:54,834 INFO [main {}] input.FileInputFormat: Total input
> files to process : 1
> 2024-03-25T13:38:54,853 INFO [main {}] mapreduce.JobSubmitter: number of
> splits:1
> 2024-03-25T13:38:54,964 INFO [main {}] mapreduce.JobSubmitter: Submitting
> tokens for job: job_local748155768_0001
> 2024-03-25T13:38:54,967 INFO [main {}] mapreduce.JobSubmitter: Executing
> with tokens: []
> 2024-03-25T13:38:55,076 INFO [main {}] mapred.LocalDistributedCacheManager:
> Creating symlink:
> /tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
> <- /home/dieter/code/hbase/libjars/*
> 2024-03-25T13:38:55,079 WARN [main {}] fs.FileUtil: Command 'ln -s
> /tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
>
Re: [PR] HBASE-28458 BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached [hbase]
Apache-HBase commented on PR #5777: URL: https://github.com/apache/hbase/pull/5777#issuecomment-2021400781 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 45s | master passed | | +1 :green_heart: | compile | 1m 25s | master passed | | +1 :green_heart: | shadedjars | 8m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | the patch passed | | +1 :green_heart: | compile | 1m 17s | the patch passed | | +1 :green_heart: | javac | 1m 17s | the patch passed | | +1 :green_heart: | shadedjars | 8m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 256m 3s | hbase-server in the patch passed. | | | | 291m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5777 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 486d5eed0aca 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/testReport/ | | Max. process+thread count | 4507 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28458 BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached [hbase]
Apache-HBase commented on PR #5777: URL: https://github.com/apache/hbase/pull/5777#issuecomment-2020878066 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 56s | master passed | | +1 :green_heart: | compile | 3m 38s | master passed | | +1 :green_heart: | checkstyle | 0m 59s | master passed | | +1 :green_heart: | spotless | 1m 8s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 22s | the patch passed | | +1 :green_heart: | compile | 3m 35s | the patch passed | | +1 :green_heart: | javac | 3m 35s | the patch passed | | +1 :green_heart: | checkstyle | 0m 51s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 7m 44s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 1m 5s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 29s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 42m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5777 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 6137de877d21 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
rubenvw-ngdata commented on code in PR #5776: URL: https://github.com/apache/hbase/pull/5776#discussion_r1539649431 ## hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupRestoreWithModifications.java: ## @@ -18,6 +18,7 @@ package org.apache.hadoop.hbase.backup; import static org.apache.hadoop.hbase.backup.BackupInfo.BackupState.COMPLETE; +import static org.apache.hadoop.hbase.backup.BackupTestUtil.*; Review Comment: resolved -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
Apache-HBase commented on PR #5776: URL: https://github.com/apache/hbase/pull/5776#issuecomment-2021093977 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 7s | master passed | | +1 :green_heart: | compile | 0m 31s | master passed | | +1 :green_heart: | shadedjars | 7m 36s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 0m 22s | the patch passed | | +1 :green_heart: | javac | 0m 22s | the patch passed | | +1 :green_heart: | shadedjars | 7m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 13m 28s | hbase-backup in the patch passed. | | | | 39m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5776 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0574124ff9e5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/testReport/ | | Max. process+thread count | 3303 (vs. ulimit of 3) | | modules | C: hbase-backup U: hbase-backup | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
Apache-HBase commented on PR #5776: URL: https://github.com/apache/hbase/pull/5776#issuecomment-2021094695 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 22s | master passed | | +1 :green_heart: | compile | 0m 28s | master passed | | +1 :green_heart: | shadedjars | 8m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 32s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | shadedjars | 7m 57s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 11m 41s | hbase-backup in the patch passed. | | | | 40m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5776 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e09f71f374c3 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/testReport/ | | Max. process+thread count | 3234 (vs. ulimit of 3) | | modules | C: hbase-backup U: hbase-backup | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28450 BuckeCache.evictBlocksByHfileName won't work after a cache recovery from file [hbase]
Apache-HBase commented on PR #5769: URL: https://github.com/apache/hbase/pull/5769#issuecomment-2021266305 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | master passed | | +1 :green_heart: | compile | 0m 39s | master passed | | +1 :green_heart: | shadedjars | 5m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 26s | the patch passed | | +1 :green_heart: | compile | 0m 40s | the patch passed | | +1 :green_heart: | javac | 0m 40s | the patch passed | | +1 :green_heart: | shadedjars | 5m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 292m 43s | hbase-server in the patch failed. | | | | 317m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5769 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5669af81f555 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 38aef80ffb | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/testReport/ | | Max. process+thread count | 4683 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-27986 Null Pointer Exception when running testLruBlockCache with a misconfiguration [hbase]
Apache-HBase commented on PR #5779: URL: https://github.com/apache/hbase/pull/5779#issuecomment-2021324202 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 3m 19s | master passed | | +1 :green_heart: | checkstyle | 0m 42s | master passed | | +1 :green_heart: | spotless | 0m 57s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 8s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 3m 3s | the patch passed | | +1 :green_heart: | javac | 3m 3s | the patch passed | | +1 :green_heart: | checkstyle | 0m 41s | the patch passed | | -0 :warning: | whitespace | 0m 1s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 6m 12s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 52s | patch has 24 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 36m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5779 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux cc47dc5fc9dc 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Eclipse Adoptium-11.0.17+8 | | whitespace | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/artifact/yetus-general-check/output/whitespace-eol.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26192 Master UI hbck should provide a JSON formatted output option [hbase]
apurtell commented on code in PR #5772:
URL: https://github.com/apache/hbase/pull/5772#discussion_r1539847142
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/http/hbck/HbckConfigFactory.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.http.hbck;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.http.jersey.ResponseEntityMapper;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.master.http.gson.GsonSerializationFeature;
+import org.apache.hadoop.hbase.master.http.jersey.MasterFeature;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.org.glassfish.jersey.server.ResourceConfig;
+import
org.apache.hbase.thirdparty.org.glassfish.jersey.server.ServerProperties;
+import org.apache.hbase.thirdparty.org.glassfish.jersey.server.TracingConfig;
+
+@InterfaceAudience.Private
+public final class HbckConfigFactory {
+ private HbckConfigFactory() {
+ }
+
+ public static ResourceConfig createResourceConfig(Configuration conf,
HMaster master) {
+return new ResourceConfig().setApplicationName("hbck")
+ .packages(HbckConfigFactory.class.getPackage().getName())
+ // TODO: anything registered here that does not have necessary bindings
won't inject properly
+ // at annotation sites and will result in a WARN logged by
o.a.h.t.o.g.j.i.inject.Providers.
+ // These warnings should be treated by the service as fatal errors, but
I have not found a
+ // callback API for registering a failed binding handler.
+
.register(ResponseEntityMapper.class).register(GsonSerializationFeature.class)
+ .register(new MasterFeature(master))
+
+ // devs: enable TRACING to see how jersey is dispatching to resources.
Review Comment:
Nice, thanks for adding this note.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/http/hbck/resource/HbckMetricsResource.java:
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.http.hbck.resource;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.inject.Inject;
+import org.apache.hadoop.hbase.HbckEmptyRegionInfo;
+import org.apache.hadoop.hbase.HbckInconsistentRegions;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnFS;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnRS;
+import org.apache.hadoop.hbase.HbckOverlapRegions;
+import org.apache.hadoop.hbase.HbckRegionDetails;
+import org.apache.hadoop.hbase.HbckRegionHoles;
+import org.apache.hadoop.hbase.HbckServerName;
+import org.apache.hadoop.hbase.HbckUnknownServers;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.hbck.HbckReport;
+import org.apache.hadoop.hbase.master.http.hbck.model.HbckMetrics;
+import org.apache.hadoop.hbase.master.janitor.CatalogJanitorReport;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.javax.ws.rs.GET;
+import org.apache.hbase.thirdparty.javax.ws.rs.Path;
+import org.apache.hbase.thirdparty.javax.ws.rs.Produces;
+import org.apache.hbase.thirdparty.javax.ws.rs.core.MediaType;
+
+/**
+ * The root object exposing hbck.jsp page as
Re: [PR] HBASE-28458 BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached [hbase]
Apache-HBase commented on PR #5777: URL: https://github.com/apache/hbase/pull/5777#issuecomment-2021387643 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 5m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 0s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 5m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 257m 48s | hbase-server in the patch passed. | | | | 283m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5777 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2e8b26ff0225 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/testReport/ | | Max. process+thread count | 4492 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26192 Master UI hbck should provide a JSON formatted output option [hbase]
mihir6692 commented on code in PR #5772:
URL: https://github.com/apache/hbase/pull/5772#discussion_r1540072864
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/http/hbck/resource/HbckMetricsResource.java:
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.http.hbck.resource;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.inject.Inject;
+import org.apache.hadoop.hbase.HbckEmptyRegionInfo;
+import org.apache.hadoop.hbase.HbckInconsistentRegions;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnFS;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnRS;
+import org.apache.hadoop.hbase.HbckOverlapRegions;
+import org.apache.hadoop.hbase.HbckRegionDetails;
+import org.apache.hadoop.hbase.HbckRegionHoles;
+import org.apache.hadoop.hbase.HbckServerName;
+import org.apache.hadoop.hbase.HbckUnknownServers;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.hbck.HbckReport;
+import org.apache.hadoop.hbase.master.http.hbck.model.HbckMetrics;
+import org.apache.hadoop.hbase.master.janitor.CatalogJanitorReport;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.javax.ws.rs.GET;
+import org.apache.hbase.thirdparty.javax.ws.rs.Path;
+import org.apache.hbase.thirdparty.javax.ws.rs.Produces;
+import org.apache.hbase.thirdparty.javax.ws.rs.core.MediaType;
+
+/**
+ * The root object exposing hbck.jsp page as JSON Output.
+ */
+@Path("hbck-metrics")
+@Produces({ MediaType.APPLICATION_JSON })
Review Comment:
Based on my understanding from https://github.com/apache/hbase/pull/4177 by
@ndimiduk and limited understanding of JERSEY, We would have to implement
something similar to
https://github.com/apache/hbase/tree/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/http/gson
for XML too. And that would be an another JIRA itself.
@ndimiduk What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28458 BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached [hbase]
Apache-HBase commented on PR #5777: URL: https://github.com/apache/hbase/pull/5777#issuecomment-2021527640 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | shadedjars | 6m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 46s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 8m 57s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 46s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 337m 0s | hbase-server in the patch failed. | | | | 369m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5777 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0405894c202a 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/testReport/ | | Max. process+thread count | 5138 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5777/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
bbeaudreault commented on code in PR #5776: URL: https://github.com/apache/hbase/pull/5776#discussion_r1539624430 ## hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupRestoreWithModifications.java: ## @@ -18,6 +18,7 @@ package org.apache.hadoop.hbase.backup; import static org.apache.hadoop.hbase.backup.BackupInfo.BackupState.COMPLETE; +import static org.apache.hadoop.hbase.backup.BackupTestUtil.*; Review Comment: Avoid star imports -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
Apache-HBase commented on PR #5776: URL: https://github.com/apache/hbase/pull/5776#issuecomment-2020970457 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 23s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 25s | master passed | | +1 :green_heart: | compile | 0m 15s | master passed | | +1 :green_heart: | shadedjars | 5m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 11s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 24s | the patch passed | | +1 :green_heart: | compile | 0m 15s | the patch passed | | +1 :green_heart: | javac | 0m 15s | the patch passed | | +1 :green_heart: | shadedjars | 5m 35s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 11m 9s | hbase-backup in the patch passed. | | | | 29m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5776 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5ef238829f04 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/testReport/ | | Max. process+thread count | 3653 (vs. ulimit of 3) | | modules | C: hbase-backup U: hbase-backup | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
Apache-HBase commented on PR #5776: URL: https://github.com/apache/hbase/pull/5776#issuecomment-2020972463 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 22s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 0m 34s | master passed | | +1 :green_heart: | checkstyle | 0m 13s | master passed | | +1 :green_heart: | spotless | 1m 4s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 2s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | compile | 0m 37s | the patch passed | | +1 :green_heart: | javac | 0m 37s | the patch passed | | -0 :warning: | checkstyle | 0m 14s | hbase-backup: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 6m 54s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 57s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 44s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 30m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5776 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 3ea072ad1a7c 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-backup.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-backup U: hbase-backup | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
Apache-HBase commented on PR #5776: URL: https://github.com/apache/hbase/pull/5776#issuecomment-2021082313 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 0m 17s | master passed | | +1 :green_heart: | shadedjars | 6m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 11s | the patch passed | | +1 :green_heart: | compile | 0m 18s | the patch passed | | +1 :green_heart: | javac | 0m 18s | the patch passed | | +1 :green_heart: | shadedjars | 6m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 11m 54s | hbase-backup in the patch passed. | | | | 34m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5776 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b467dda34254 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/testReport/ | | Max. process+thread count | 3651 (vs. ulimit of 3) | | modules | C: hbase-backup U: hbase-backup | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
bbeaudreault closed pull request #5776: HBASE-28412 Select correct target table for incremental backup URL: https://github.com/apache/hbase/pull/5776 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
bbeaudreault commented on PR #5776: URL: https://github.com/apache/hbase/pull/5776#issuecomment-2021115402 Pre-commit looks good. There is 1 checkstyle issue. Branch permissions don't allow me to commit to this PR, so I'm going to cherry-pick it rather than merge. Closing this and cherry-picking the commit from your branch with the checkstyle warning fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28456 HBase Restore restores old data if data for the same timestamp is in different hfiles [hbase]
Apache-HBase commented on PR #5775: URL: https://github.com/apache/hbase/pull/5775#issuecomment-2021138268 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 9s | master passed | | +1 :green_heart: | compile | 1m 17s | master passed | | +1 :green_heart: | shadedjars | 5m 36s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 1m 19s | the patch passed | | +1 :green_heart: | javac | 1m 19s | the patch passed | | +1 :green_heart: | shadedjars | 5m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 229m 30s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 14m 52s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 10m 16s | hbase-backup in the patch passed. | | | | 281m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 301ff65adad1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 38aef80ffb | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/testReport/ | | Max. process+thread count | 4467 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce hbase-backup U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26192 Master UI hbck should provide a JSON formatted output option [hbase]
apurtell closed pull request #4470: HBASE-26192 Master UI hbck should provide a JSON formatted output option URL: https://github.com/apache/hbase/pull/4470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-27986) Null Pointer Exception when running testLruBlockCache with a misconfiguration
[
https://issues.apache.org/jira/browse/HBASE-27986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HBASE-27986:
---
Labels: pull-request-available (was: )
> Null Pointer Exception when running testLruBlockCache with a misconfiguration
> -
>
> Key: HBASE-27986
> URL: https://issues.apache.org/jira/browse/HBASE-27986
> Project: HBase
> Issue Type: Sub-task
>Reporter: ConfX
>Priority: Critical
> Labels: pull-request-available
> Attachments: reproduce.sh
>
>
> h2. What happened
> After setting {{{}hfile.block.cache.size=1.1175870895385742E-8{}}}, running
> test
> {{org.apache.hadoop.hbase.io.hfile.TestBlockCacheReporting#testLruBlockCache}}
> results in a {{{}NullPointerException{}}}.
> h2. Where's the bug
> In the test
> {{{}org.apache.hadoop.hbase.io.hfile.TestBlockCacheReporting#testLruBlockCache{}}}:
> {code:java}
> BlockCache blockCache = BlockCacheFactory.createBlockCache(this.conf);
> logPerBlock(blockCache);
> addDataAndHits(blockCache, 3); {code}
> Here the returned blockCache could be null, but is directly used as the
> parameter of {{logPerBlock}} without checking.
> h2. How to reproduce
> # Set {{hfile.block.cache.size=1.1175870895385742E-8}}
> # Run
> {{org.apache.hadoop.hbase.io.hfile.TestBlockCacheReporting#testLruBlockCache}}
> and the following exception should be observed:
> {code:java}
> java.lang.NullPointerExceptionat
> org.apache.hadoop.hbase.io.hfile.TestBlockCacheReporting.logPerBlock(TestBlockCacheReporting.java:147)
>at
> org.apache.hadoop.hbase.io.hfile.TestBlockCacheReporting.testLruBlockCache(TestBlockCacheReporting.java:100)
> {code}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28453) Support a middle ground between the Average and Fixed interval rate limiters
[
https://issues.apache.org/jira/browse/HBASE-28453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831061#comment-17831061
]
Hudson commented on HBASE-28453:
Results for branch branch-3
[build #172 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/172/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/172/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/172/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/172/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Support a middle ground between the Average and Fixed interval rate limiters
>
>
> Key: HBASE-28453
> URL: https://issues.apache.org/jira/browse/HBASE-28453
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Ray Mattingly
>Assignee: Ray Mattingly
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
> Attachments: Screenshot 2024-03-21 at 2.08.51 PM.png, Screenshot
> 2024-03-21 at 2.30.01 PM.png
>
>
> h3. Background
> HBase quotas support two rate limiters: a "fixed" and an "average" interval
> rate limiter.
> h4. FixedIntervalRateLimiter
> The fixed interval rate limiter is simpler: it has a TimeUnit, say 1 second,
> and it refills a resource allotment on the recurring interval. So you may get
> 10 resources every second, and if you exhaust all 10 resources in the first
> millisecond of an interval then you will need to wait 999ms to acquire even 1
> more resource.
> h4. AverageIntervalRateLimiter
> The average interval rate limiter, HBase's default, allows for more flexibly
> timed refilling of the resource allotment. Extending our previous example,
> say you have a 10 reads/sec quota and you have exhausted all 10 resources
> within 1ms of the last full refill. If you request 1 more read then, rather
> than returning a 999ms wait interval indicating the next full refill time,
> the rate limiter will recognize that you only need to wait 99ms before 1 read
> can be available. After 100ms has passed in aggregate since the last full
> refill, it will support the refilling of 1/10th the limit to facilitate the
> request for 1/10th the resources.
> h3. The Problems with Current RateLimiters
> The problem with the fixed interval rate limiter is that it is too strict
> from a latency perspective. It results in quota limits to which we cannot
> fully subscribe with any consistency.
> The problem with the average interval rate limiter is that, in practice, it
> is far too optimistic. For example, a real rate limiter might limit to
> 100MB/sec of read IO per machine. Any multigets that come in will require
> only a tiny fraction of this limit; for example, a 64kb block is only 0.06%
> of the total. As a result, the vast majority of wait intervals end up being
> tiny — like <5ms. This can actually cause an inverse of your intention, where
> setting up a throttle causes a DDOS of your RPC layer via continuous
> throttling and ~immediate retrying. I've discussed this problem in
> https://issues.apache.org/jira/browse/HBASE-28429 and proposed a minimum wait
> interval as the solution there; after some more thinking, I believe this new
> rate limiter would be a less hacky solution to this deficit so I'd like to
> close that Jira in favor of this one.
> See the attached chart where I put in place a 10k req/sec/machine throttle
> for this user at 10:43 to try to curb this high traffic, and it resulted in a
> huge spike of req/sec due to the throttle/retry loop created by the
> AverageIntervalRateLimiter.
> h3. Original Proposal: PartialIntervalRateLimiter as a Solution
> I've implemented a RateLimiter which allows for partial chunks of the overall
> interval to be refilled, by default these chunks are 10% (or 100ms of a 1s
> interval). I've deployed this to a test cluster at my day job and have seen
> this really help our ability to full subscribe to a quota limit without
> executing superfluous retries. See the other attached chart which shows a
> cluster undergoing a rolling restart from using FixedIntervalRateLimiter to
> my new PartialIntervalRateLimiter and how it is
[jira] [Commented] (HBASE-28449) Fix BackupSystemTable Scans
[
https://issues.apache.org/jira/browse/HBASE-28449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831062#comment-17831062
]
Hudson commented on HBASE-28449:
Results for branch branch-3
[build #172 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/172/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/172/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/172/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/172/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix BackupSystemTable Scans
>
>
> Key: HBASE-28449
> URL: https://issues.apache.org/jira/browse/HBASE-28449
> Project: HBase
> Issue Type: Bug
>Reporter: Briana Augenreich
>Assignee: Briana Augenreich
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
>
> When calculating which WALs should be included in an incremental backup the
> backup system does a prefix scan for the last roll log timestamp. This uses
> the backup root in the prefix (.) If you happen have
> multiple backup roots where one is a root of the other you'll get inaccurate
> results.
>
> Since the rowkey is let's modify
> the prefix scan to be .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
rubenvw-ngdata commented on code in PR #5776:
URL: https://github.com/apache/hbase/pull/5776#discussion_r1539479695
##
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupRestoreWithModifications.java:
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupInfo.BackupState.COMPLETE;
+import static org.apache.hadoop.hbase.backup.BackupType.FULL;
+import static org.apache.hadoop.hbase.backup.BackupType.INCREMENTAL;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testing.TestingHBaseCluster;
+import org.apache.hadoop.hbase.testing.TestingHBaseClusterOption;
+import org.apache.hadoop.hbase.tool.BulkLoadHFiles;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Category(MediumTests.class)
+@RunWith(Parameterized.class)
+public class TestBackupRestoreWithModifications {
Review Comment:
Just confirming that the test has timing related issues at my end too. We're
trying to find out where the timing issue is.
What I'll do for this ticket, I'll create a separate test for restoring a
backup (full or incremental) on an empty and non empty environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
Apache-HBase commented on PR #5776: URL: https://github.com/apache/hbase/pull/5776#issuecomment-2020995259 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 0m 31s | master passed | | +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 32s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | shadedjars | 7m 17s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 13m 27s | hbase-backup in the patch passed. | | | | 41m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5776 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 98d4ceb2b515 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/testReport/ | | Max. process+thread count | 3808 (vs. ulimit of 3) | | modules | C: hbase-backup U: hbase-backup | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26192 Master UI hbck should provide a JSON formatted output option [hbase]
apurtell commented on code in PR #5772:
URL: https://github.com/apache/hbase/pull/5772#discussion_r1539849184
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/http/hbck/resource/HbckMetricsResource.java:
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.http.hbck.resource;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.inject.Inject;
+import org.apache.hadoop.hbase.HbckEmptyRegionInfo;
+import org.apache.hadoop.hbase.HbckInconsistentRegions;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnFS;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnRS;
+import org.apache.hadoop.hbase.HbckOverlapRegions;
+import org.apache.hadoop.hbase.HbckRegionDetails;
+import org.apache.hadoop.hbase.HbckRegionHoles;
+import org.apache.hadoop.hbase.HbckServerName;
+import org.apache.hadoop.hbase.HbckUnknownServers;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.hbck.HbckReport;
+import org.apache.hadoop.hbase.master.http.hbck.model.HbckMetrics;
+import org.apache.hadoop.hbase.master.janitor.CatalogJanitorReport;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.javax.ws.rs.GET;
+import org.apache.hbase.thirdparty.javax.ws.rs.Path;
+import org.apache.hbase.thirdparty.javax.ws.rs.Produces;
+import org.apache.hbase.thirdparty.javax.ws.rs.core.MediaType;
+
+/**
+ * The root object exposing hbck.jsp page as JSON Output.
+ */
+@Path("hbck-metrics")
+@Produces({ MediaType.APPLICATION_JSON })
Review Comment:
Can we add support for XML here too? (Is that `MediaType.APPLICATION_XML`?)
Should be as simple as that. Jersey will choose according to the client's
`Accept` header.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/http/hbck/resource/HbckMetricsResource.java:
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.http.hbck.resource;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.inject.Inject;
+import org.apache.hadoop.hbase.HbckEmptyRegionInfo;
+import org.apache.hadoop.hbase.HbckInconsistentRegions;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnFS;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnRS;
+import org.apache.hadoop.hbase.HbckOverlapRegions;
+import org.apache.hadoop.hbase.HbckRegionDetails;
+import org.apache.hadoop.hbase.HbckRegionHoles;
+import org.apache.hadoop.hbase.HbckServerName;
+import org.apache.hadoop.hbase.HbckUnknownServers;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.hbck.HbckReport;
+import org.apache.hadoop.hbase.master.http.hbck.model.HbckMetrics;
+import org.apache.hadoop.hbase.master.janitor.CatalogJanitorReport;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.javax.ws.rs.GET;
+import org.apache.hbase.thirdparty.javax.ws.rs.Path;
+import org.apache.hbase.thirdparty.javax.ws.rs.Produces;
+import org.apache.hbase.thirdparty.javax.ws.rs.core.MediaType;
+
+/**
+ * The root object
Re: [PR] HBASE-28450 BuckeCache.evictBlocksByHfileName won't work after a cache recovery from file [hbase]
Apache-HBase commented on PR #5769: URL: https://github.com/apache/hbase/pull/5769#issuecomment-2021177590 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 34s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 6m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 9s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 6m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 244m 0s | hbase-server in the patch passed. | | | | 272m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5769 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ef9cbe0b52a6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 38aef80ffb | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/testReport/ | | Max. process+thread count | 4683 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-28412) Restoring incremental backups to mapped table requires existence of original table
[
https://issues.apache.org/jira/browse/HBASE-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault reassigned HBASE-28412:
-
Assignee: Ruben Van Wanzeele
> Restoring incremental backups to mapped table requires existence of original
> table
> --
>
> Key: HBASE-28412
> URL: https://issues.apache.org/jira/browse/HBASE-28412
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.6.0, 4.0.0-alpha-1
>Reporter: Dieter De Paepe
>Assignee: Ruben Van Wanzeele
>Priority: Major
> Labels: pull-request-available
>
> It appears that restoring a non-existing table from an incremental backup
> with the "-m" parameter results in an error in the restore client.
> Reproduction steps:
> Build & start hbase:
> {code:java}
> mvn clean install -Phadoop-3.0 -DskipTests
> bin/start-hbase.sh{code}
> In HBase shell: create table and some values:
> {code:java}
> create 'test', 'cf'
> put 'test', 'row1', 'cf:a', 'value1'
> put 'test', 'row2', 'cf:b', 'value2'
> put 'test', 'row3', 'cf:c', 'value3'
> scan 'test' {code}
> Create a full backup:
> {code:java}
> bin/hbase backup create full file:/tmp/hbase-backup{code}
> Adjust some data through HBase shell:
> {code:java}
> put 'test', 'row1', 'cf:a', 'value1-new'
> scan 'test' {code}
> Create an incremental backup:
> {code:java}
> bin/hbase backup create incremental file:/tmp/hbase-backup {code}
> Delete the original table in HBase shell:
> {code:java}
> disable 'test'
> drop 'test' {code}
> Restore the incremental backup under a new table name:
> {code:java}
> bin/hbase backup history
> bin/hbase restore file:/tmp/hbase-backup -t "test" -m
> "test-restored" {code}
> This results in the following output / error:
> {code:java}
> ...
> 2024-03-25T13:38:53,062 WARN [main {}] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2024-03-25T13:38:53,174 INFO [main {}] Configuration.deprecation:
> hbase.client.pause.cqtbe is deprecated. Instead, use
> hbase.client.pause.server.overloaded
> 2024-03-25T13:38:53,554 INFO [main {}] impl.RestoreTablesClient: HBase table
> test-restored does not exist. It will be created during restore process
> 2024-03-25T13:38:53,593 INFO [main {}] impl.RestoreTablesClient: Restoring
> 'test' to 'test-restored' from full backup image
> file:/tmp/hbase-backup/backup_1711370230143/default/test
> 2024-03-25T13:38:53,707 INFO [main {}] util.BackupUtils: Creating target
> table 'test-restored'
> 2024-03-25T13:38:54,546 INFO [main {}] mapreduce.MapReduceRestoreJob:
> Restore test into test-restored
> 2024-03-25T13:38:54,646 INFO [main {}] mapreduce.HFileOutputFormat2:
> bulkload locality sensitive enabled
> 2024-03-25T13:38:54,647 INFO [main {}] mapreduce.HFileOutputFormat2: Looking
> up current regions for table test-restored
> 2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2:
> Configuring 1 reduce partitions to match current region count for all tables
> 2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2: Writing
> partition information to
> file:/tmp/hbase-tmp/partitions_0667b6e2-79ef-4cfe-97e1-abb204ee420d
> 2024-03-25T13:38:54,687 INFO [main {}] compress.CodecPool: Got brand-new
> compressor [.deflate]
> 2024-03-25T13:38:54,713 INFO [main {}] mapreduce.HFileOutputFormat2:
> Incremental output configured for tables: test-restored
> 2024-03-25T13:38:54,715 WARN [main {}] mapreduce.TableMapReduceUtil: The
> addDependencyJars(Configuration, Class...) method has been deprecated
> since it is easy to use incorrectly. Most users should rely on
> addDependencyJars(Job) instead. See HBASE-8386 for more details.
> 2024-03-25T13:38:54,742 WARN [main {}] impl.MetricsConfig: Cannot locate
> configuration: tried
> hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties
> 2024-03-25T13:38:54,834 INFO [main {}] input.FileInputFormat: Total input
> files to process : 1
> 2024-03-25T13:38:54,853 INFO [main {}] mapreduce.JobSubmitter: number of
> splits:1
> 2024-03-25T13:38:54,964 INFO [main {}] mapreduce.JobSubmitter: Submitting
> tokens for job: job_local748155768_0001
> 2024-03-25T13:38:54,967 INFO [main {}] mapreduce.JobSubmitter: Executing
> with tokens: []
> 2024-03-25T13:38:55,076 INFO [main {}] mapred.LocalDistributedCacheManager:
> Creating symlink:
> /tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
> <- /home/dieter/code/hbase/libjars/*
> 2024-03-25T13:38:55,079 WARN [main {}] fs.FileUtil: Command 'ln -s
> /tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
>
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
Apache-HBase commented on PR #5776: URL: https://github.com/apache/hbase/pull/5776#issuecomment-2021002223 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 47s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 59s | master passed | | +1 :green_heart: | compile | 0m 28s | master passed | | +1 :green_heart: | shadedjars | 8m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 30s | the patch passed | | +1 :green_heart: | compile | 0m 39s | the patch passed | | +1 :green_heart: | javac | 0m 39s | the patch passed | | +1 :green_heart: | shadedjars | 8m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 15m 53s | hbase-backup in the patch passed. | | | | 44m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5776 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bca0a9269491 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/testReport/ | | Max. process+thread count | 3352 (vs. ulimit of 3) | | modules | C: hbase-backup U: hbase-backup | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28450 BuckeCache.evictBlocksByHfileName won't work after a cache recovery from file [hbase]
Apache-HBase commented on PR #5769: URL: https://github.com/apache/hbase/pull/5769#issuecomment-2021200540 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 53s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 28s | master passed | | +1 :green_heart: | compile | 1m 10s | master passed | | +1 :green_heart: | shadedjars | 6m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 7s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 5m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 256m 3s | hbase-server in the patch failed. | | | | 285m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5769 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 07bcdc2277f2 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 38aef80ffb | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/testReport/ | | Max. process+thread count | 4668 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5769/7/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28443 Return too slow when scanning a table with non-existing REGION_REPLICA_ID [hbase]
Apache-HBase commented on PR #5767: URL: https://github.com/apache/hbase/pull/5767#issuecomment-2021372824 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 0s | master passed | | +1 :green_heart: | compile | 1m 12s | master passed | | +1 :green_heart: | shadedjars | 5m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | | -0 :warning: | patch | 6m 22s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 51s | the patch passed | | +1 :green_heart: | compile | 1m 12s | the patch passed | | +1 :green_heart: | javac | 1m 12s | the patch passed | | +1 :green_heart: | shadedjars | 5m 16s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 34s | hbase-client in the patch passed. | | -1 :x: | unit | 260m 52s | hbase-server in the patch failed. | | | | 288m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5767 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cbb2e5a8648c 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/testReport/ | | Max. process+thread count | 5059 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28443 Return too slow when scanning a table with non-existing REGION_REPLICA_ID [hbase]
Apache-HBase commented on PR #5767: URL: https://github.com/apache/hbase/pull/5767#issuecomment-2021413168 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 1s | master passed | | +1 :green_heart: | compile | 1m 10s | master passed | | +1 :green_heart: | shadedjars | 7m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | master passed | | -0 :warning: | patch | 8m 13s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 6s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 6m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 33s | hbase-client in the patch passed. | | -1 :x: | unit | 279m 49s | hbase-server in the patch failed. | | | | 310m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5767 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dcb56b7983e7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/testReport/ | | Max. process+thread count | 4412 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28443 Return too slow when scanning a table with non-existing REGION_REPLICA_ID [hbase]
Apache-HBase commented on PR #5767: URL: https://github.com/apache/hbase/pull/5767#issuecomment-2021477976 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 0s | master passed | | +1 :green_heart: | compile | 1m 17s | master passed | | +1 :green_heart: | shadedjars | 5m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | master passed | | -0 :warning: | patch | 6m 21s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 57s | the patch passed | | +1 :green_heart: | compile | 1m 22s | the patch passed | | +1 :green_heart: | javac | 1m 22s | the patch passed | | +1 :green_heart: | shadedjars | 6m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 50s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 43s | hbase-client in the patch passed. | | -1 :x: | unit | 317m 42s | hbase-server in the patch failed. | | | | 347m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5767 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5da8d34b6c40 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-17.0.10+7 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/testReport/ | | Max. process+thread count | 4820 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] HBASE-27986 Null Pointer Exception when running testLruBlockCache with a misconfiguration [hbase]
teamconfx opened a new pull request, #5779: URL: https://github.com/apache/hbase/pull/5779 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28452: Missing null check of rpcServer.scheduler.executor causes NPE with invalid value of hbase.client.default.rpc.codec [hbase]
Apache-HBase commented on PR #5778: URL: https://github.com/apache/hbase/pull/5778#issuecomment-2021227320 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 13s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 6s | master passed | | +1 :green_heart: | compile | 2m 30s | master passed | | +1 :green_heart: | checkstyle | 0m 39s | master passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 33s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 2m 26s | the patch passed | | +1 :green_heart: | javac | 2m 26s | the patch passed | | +1 :green_heart: | checkstyle | 0m 37s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 55s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 28m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5778 | | JIRA Issue | HBASE-28452 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux c85073e61286 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
Apache-HBase commented on PR #5776: URL: https://github.com/apache/hbase/pull/5776#issuecomment-2021066328 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 27s | master passed | | +1 :green_heart: | compile | 0m 31s | master passed | | +1 :green_heart: | checkstyle | 0m 11s | master passed | | +1 :green_heart: | spotless | 0m 52s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 0m 34s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 15s | the patch passed | | +1 :green_heart: | compile | 0m 25s | the patch passed | | +1 :green_heart: | javac | 0m 25s | the patch passed | | -0 :warning: | checkstyle | 0m 9s | hbase-backup: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 6m 19s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 1m 4s | patch has 1 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 0m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 27m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5776 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 9c2abb3f3907 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-backup.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-backup U: hbase-backup | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5776/3/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28456 HBase Restore restores old data if data for the same timestamp is in different hfiles [hbase]
Apache-HBase commented on PR #5775: URL: https://github.com/apache/hbase/pull/5775#issuecomment-2021131115 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 45s | master passed | | +1 :green_heart: | compile | 1m 9s | master passed | | +1 :green_heart: | shadedjars | 5m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 27s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed | | +1 :green_heart: | javac | 1m 9s | the patch passed | | +1 :green_heart: | shadedjars | 5m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 227m 45s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 14m 13s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 10m 24s | hbase-backup in the patch passed. | | | | 277m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5775 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a953142d17d5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 38aef80ffb | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/testReport/ | | Max. process+thread count | 5653 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce hbase-backup U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28449) Fix BackupSystemTable Scans
[
https://issues.apache.org/jira/browse/HBASE-28449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831084#comment-17831084
]
Hudson commented on HBASE-28449:
Results for branch master
[build #1037 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1037/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1037/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1037/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1037/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix BackupSystemTable Scans
>
>
> Key: HBASE-28449
> URL: https://issues.apache.org/jira/browse/HBASE-28449
> Project: HBase
> Issue Type: Bug
>Reporter: Briana Augenreich
>Assignee: Briana Augenreich
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
>
> When calculating which WALs should be included in an incremental backup the
> backup system does a prefix scan for the last roll log timestamp. This uses
> the backup root in the prefix (.) If you happen have
> multiple backup roots where one is a root of the other you'll get inaccurate
> results.
>
> Since the rowkey is let's modify
> the prefix scan to be .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28453) Support a middle ground between the Average and Fixed interval rate limiters
[
https://issues.apache.org/jira/browse/HBASE-28453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831083#comment-17831083
]
Hudson commented on HBASE-28453:
Results for branch master
[build #1037 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1037/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1037/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1037/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1037/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Support a middle ground between the Average and Fixed interval rate limiters
>
>
> Key: HBASE-28453
> URL: https://issues.apache.org/jira/browse/HBASE-28453
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Ray Mattingly
>Assignee: Ray Mattingly
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
> Attachments: Screenshot 2024-03-21 at 2.08.51 PM.png, Screenshot
> 2024-03-21 at 2.30.01 PM.png
>
>
> h3. Background
> HBase quotas support two rate limiters: a "fixed" and an "average" interval
> rate limiter.
> h4. FixedIntervalRateLimiter
> The fixed interval rate limiter is simpler: it has a TimeUnit, say 1 second,
> and it refills a resource allotment on the recurring interval. So you may get
> 10 resources every second, and if you exhaust all 10 resources in the first
> millisecond of an interval then you will need to wait 999ms to acquire even 1
> more resource.
> h4. AverageIntervalRateLimiter
> The average interval rate limiter, HBase's default, allows for more flexibly
> timed refilling of the resource allotment. Extending our previous example,
> say you have a 10 reads/sec quota and you have exhausted all 10 resources
> within 1ms of the last full refill. If you request 1 more read then, rather
> than returning a 999ms wait interval indicating the next full refill time,
> the rate limiter will recognize that you only need to wait 99ms before 1 read
> can be available. After 100ms has passed in aggregate since the last full
> refill, it will support the refilling of 1/10th the limit to facilitate the
> request for 1/10th the resources.
> h3. The Problems with Current RateLimiters
> The problem with the fixed interval rate limiter is that it is too strict
> from a latency perspective. It results in quota limits to which we cannot
> fully subscribe with any consistency.
> The problem with the average interval rate limiter is that, in practice, it
> is far too optimistic. For example, a real rate limiter might limit to
> 100MB/sec of read IO per machine. Any multigets that come in will require
> only a tiny fraction of this limit; for example, a 64kb block is only 0.06%
> of the total. As a result, the vast majority of wait intervals end up being
> tiny — like <5ms. This can actually cause an inverse of your intention, where
> setting up a throttle causes a DDOS of your RPC layer via continuous
> throttling and ~immediate retrying. I've discussed this problem in
> https://issues.apache.org/jira/browse/HBASE-28429 and proposed a minimum wait
> interval as the solution there; after some more thinking, I believe this new
> rate limiter would be a less hacky solution to this deficit so I'd like to
> close that Jira in favor of this one.
> See the attached chart where I put in place a 10k req/sec/machine throttle
> for this user at 10:43 to try to curb this high traffic, and it resulted in a
> huge spike of req/sec due to the throttle/retry loop created by the
> AverageIntervalRateLimiter.
> h3. Original Proposal: PartialIntervalRateLimiter as a Solution
> I've implemented a RateLimiter which allows for partial chunks of the overall
> interval to be refilled, by default these chunks are 10% (or 100ms of a 1s
> interval). I've deployed this to a test cluster at my day job and have seen
> this really help our ability to full subscribe to a quota limit without
> executing superfluous retries. See the other attached chart which shows a
> cluster undergoing a rolling restart from using FixedIntervalRateLimiter to
> my new PartialIntervalRateLimiter and how it is then
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
rubenvw-ngdata commented on code in PR #5776:
URL: https://github.com/apache/hbase/pull/5776#discussion_r1539574336
##
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupRestoreWithModifications.java:
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupInfo.BackupState.COMPLETE;
+import static org.apache.hadoop.hbase.backup.BackupType.FULL;
+import static org.apache.hadoop.hbase.backup.BackupType.INCREMENTAL;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testing.TestingHBaseCluster;
+import org.apache.hadoop.hbase.testing.TestingHBaseClusterOption;
+import org.apache.hadoop.hbase.tool.BulkLoadHFiles;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Category(MediumTests.class)
+@RunWith(Parameterized.class)
+public class TestBackupRestoreWithModifications {
Review Comment:
@bbeaudreault should be good now. I'm still here for 1 hour if there is
anything left to change. Tomorrow, I'm not working, so hope it is ok :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
bbeaudreault commented on code in PR #5776:
URL: https://github.com/apache/hbase/pull/5776#discussion_r1539621891
##
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/BackupTestUtil.java:
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+
+public class BackupTestUtil {
Review Comment:
@rubenvw-ngdata we can pre-empt the pre-commit hooks here -- You're going to
need to add `@InterfaceAudience.Private` to this class.
Also just make sure you ran spotless
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-26192 Master UI hbck should provide a JSON formatted output option [hbase]
apurtell commented on code in PR #5772:
URL: https://github.com/apache/hbase/pull/5772#discussion_r1539849184
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/http/hbck/resource/HbckMetricsResource.java:
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.http.hbck.resource;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.inject.Inject;
+import org.apache.hadoop.hbase.HbckEmptyRegionInfo;
+import org.apache.hadoop.hbase.HbckInconsistentRegions;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnFS;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnRS;
+import org.apache.hadoop.hbase.HbckOverlapRegions;
+import org.apache.hadoop.hbase.HbckRegionDetails;
+import org.apache.hadoop.hbase.HbckRegionHoles;
+import org.apache.hadoop.hbase.HbckServerName;
+import org.apache.hadoop.hbase.HbckUnknownServers;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.hbck.HbckReport;
+import org.apache.hadoop.hbase.master.http.hbck.model.HbckMetrics;
+import org.apache.hadoop.hbase.master.janitor.CatalogJanitorReport;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.javax.ws.rs.GET;
+import org.apache.hbase.thirdparty.javax.ws.rs.Path;
+import org.apache.hbase.thirdparty.javax.ws.rs.Produces;
+import org.apache.hbase.thirdparty.javax.ws.rs.core.MediaType;
+
+/**
+ * The root object exposing hbck.jsp page as JSON Output.
+ */
+@Path("hbck-metrics")
+@Produces({ MediaType.APPLICATION_JSON })
Review Comment:
Can we add support for XML here too? (Is that `MediaType.APPLICATION_XML`?)
Should be as simple as that. Jersey will dispatch according to the clients
`Accept` header.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/http/hbck/resource/HbckMetricsResource.java:
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.http.hbck.resource;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.inject.Inject;
+import org.apache.hadoop.hbase.HbckEmptyRegionInfo;
+import org.apache.hadoop.hbase.HbckInconsistentRegions;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnFS;
+import org.apache.hadoop.hbase.HbckOrphanRegionsOnRS;
+import org.apache.hadoop.hbase.HbckOverlapRegions;
+import org.apache.hadoop.hbase.HbckRegionDetails;
+import org.apache.hadoop.hbase.HbckRegionHoles;
+import org.apache.hadoop.hbase.HbckServerName;
+import org.apache.hadoop.hbase.HbckUnknownServers;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.hbck.HbckReport;
+import org.apache.hadoop.hbase.master.http.hbck.model.HbckMetrics;
+import org.apache.hadoop.hbase.master.janitor.CatalogJanitorReport;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.javax.ws.rs.GET;
+import org.apache.hbase.thirdparty.javax.ws.rs.Path;
+import org.apache.hbase.thirdparty.javax.ws.rs.Produces;
+import org.apache.hbase.thirdparty.javax.ws.rs.core.MediaType;
+
+/**
+ * The root object
[jira] [Updated] (HBASE-26192) Master UI hbck should provide a JSON formatted output option
[
https://issues.apache.org/jira/browse/HBASE-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26192:
Fix Version/s: 2.4.18
2.7.0
2.6.1
2.5.9
> Master UI hbck should provide a JSON formatted output option
>
>
> Key: HBASE-26192
> URL: https://issues.apache.org/jira/browse/HBASE-26192
> Project: HBase
> Issue Type: New Feature
>Reporter: Andrew Kyle Purtell
>Assignee: Mihir Monani
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9
>
> Attachments: HBCK Report in JSON Format.png, Screen Shot 2022-05-31
> at 5.18.15 PM.png
>
>
> It used to be possible to get hbck's verdict of cluster status from the
> command line, especially useful for headless deployments, i.e. without
> requiring a browser with sufficient connectivity to load a UI, or scrape
> information out of raw HTML, or write regex to comb over log4j output. The
> hbck tool's output wasn't particularly convenient to parse but it was
> straightforward to extract the desired information with a handful of regular
> expressions.
> HBCK2 has a different design philosophy than the old hbck, which is to serve
> as a collection of small and discrete recovery and repair functions, rather
> than attempt to be a universal repair tool. This makes a lot of sense and
> isn't the issue at hand. Unfortunately the old hbck's utility for reporting
> the current cluster health assessment has not been replaced either in whole
> or in part. Instead:
> {quote}
> HBCK2 is for fixes. For listings of inconsistencies or blockages in the
> running cluster, you go elsewhere, to the logs and UI of the running cluster
> Master. Once an issue has been identified, you use the HBCK2 tool to ask the
> Master to effect fixes or to skip-over bad state. Asking the Master to make
> the fixes rather than try and effect the repair locally in a fix-it tool's
> context is another important difference between HBCK2 and hbck1.
> {quote}
> Developing custom tooling to mine logs and scrape UI simply to gain a top
> level assessment of system health is unsatisfying. There should be a
> convenient means for querying the system if issues that rise to the level of
> _inconsistency_, in the hbck parlance, are believed to be present. It would
> be relatively simple to bring back the experience of invoking a command line
> tool to deliver a verdict. This could be added to the hbck2 tool itself but
> given that hbase-operator-tools is a separate project an intrinsic solution
> is desirable.
> An option that immediately comes to mind is modification of the Master's
> hbck.jsp page to provide a JSON formatted output option if the HTTP Accept
> header asks for text/json. However, looking at the source of hbck.jsp, it
> makes more sense to leave it as is and implement a convenient machine
> parseable output format elsewhere. This can be trivially accomplished with a
> new servlet. Like hbck.jsp the servlet implementation would get a reference
> to HbckChore and present the information this class makes available via its
> various getters.
> The machine parseable output is sufficient to enable headless hbck status
> checking but it still would be nice if we could provide operators a command
> line tool that formats the information for convenient viewing in a terminal.
> That part could be implemented in the hbck2 tool after this proposal is
> implemented.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-26192) Master UI hbck should provide a JSON formatted output option
[
https://issues.apache.org/jira/browse/HBASE-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26192:
Fix Version/s: (was: 2.4.18)
> Master UI hbck should provide a JSON formatted output option
>
>
> Key: HBASE-26192
> URL: https://issues.apache.org/jira/browse/HBASE-26192
> Project: HBase
> Issue Type: New Feature
>Reporter: Andrew Kyle Purtell
>Assignee: Mihir Monani
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9
>
> Attachments: HBCK Report in JSON Format.png, Screen Shot 2022-05-31
> at 5.18.15 PM.png
>
>
> It used to be possible to get hbck's verdict of cluster status from the
> command line, especially useful for headless deployments, i.e. without
> requiring a browser with sufficient connectivity to load a UI, or scrape
> information out of raw HTML, or write regex to comb over log4j output. The
> hbck tool's output wasn't particularly convenient to parse but it was
> straightforward to extract the desired information with a handful of regular
> expressions.
> HBCK2 has a different design philosophy than the old hbck, which is to serve
> as a collection of small and discrete recovery and repair functions, rather
> than attempt to be a universal repair tool. This makes a lot of sense and
> isn't the issue at hand. Unfortunately the old hbck's utility for reporting
> the current cluster health assessment has not been replaced either in whole
> or in part. Instead:
> {quote}
> HBCK2 is for fixes. For listings of inconsistencies or blockages in the
> running cluster, you go elsewhere, to the logs and UI of the running cluster
> Master. Once an issue has been identified, you use the HBCK2 tool to ask the
> Master to effect fixes or to skip-over bad state. Asking the Master to make
> the fixes rather than try and effect the repair locally in a fix-it tool's
> context is another important difference between HBCK2 and hbck1.
> {quote}
> Developing custom tooling to mine logs and scrape UI simply to gain a top
> level assessment of system health is unsatisfying. There should be a
> convenient means for querying the system if issues that rise to the level of
> _inconsistency_, in the hbck parlance, are believed to be present. It would
> be relatively simple to bring back the experience of invoking a command line
> tool to deliver a verdict. This could be added to the hbck2 tool itself but
> given that hbase-operator-tools is a separate project an intrinsic solution
> is desirable.
> An option that immediately comes to mind is modification of the Master's
> hbck.jsp page to provide a JSON formatted output option if the HTTP Accept
> header asks for text/json. However, looking at the source of hbck.jsp, it
> makes more sense to leave it as is and implement a convenient machine
> parseable output format elsewhere. This can be trivially accomplished with a
> new servlet. Like hbck.jsp the servlet implementation would get a reference
> to HbckChore and present the information this class makes available via its
> various getters.
> The machine parseable output is sufficient to enable headless hbck status
> checking but it still would be nice if we could provide operators a command
> line tool that formats the information for convenient viewing in a terminal.
> That part could be implemented in the hbck2 tool after this proposal is
> implemented.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-28452) Missing null check of rpcServer.scheduler.executor causes NPE with invalid value of hbase.client.default.rpc.codec
[
https://issues.apache.org/jira/browse/HBASE-28452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HBASE-28452:
---
Labels: pull-request-available (was: )
> Missing null check of rpcServer.scheduler.executor causes NPE with invalid
> value of hbase.client.default.rpc.codec
> --
>
> Key: HBASE-28452
> URL: https://issues.apache.org/jira/browse/HBASE-28452
> Project: HBase
> Issue Type: Bug
>Reporter: ConfX
>Priority: Major
> Labels: pull-request-available
>
> When `hbase.client.default.rpc.codec` is set to
> `org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec` the method
> `createRpcClientRTEDuringConnectionSetup` gives a NoSuchMethodException for
> the provided default codec value.
>
> Since there was an exception, the code inside try block is not executed which
> contained the below code line
> {code:java}
> try (AbstractRpcClient client =
> createRpcClientRTEDuringConnectionSetup(clientConf))
> { rpcServer.start(); . . . }
> {code}
>
> When finally block is executed which contains
> {code:java}
> finally
> { rpcServer.stop(); }
> {code}
>
> The method `rpcServer.stop()` eventually makes call to `executor.shutdown()`.
> A null pointer exception is returned as `rpcServer.scheduler.executor` was
> never instantiated, which happens inside `rpcServer.start()`
>
> {code:java}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.ipc.FifoRpcScheduler.stop(FifoRpcScheduler.java:71)
> at org.apache.hadoop.hbase.ipc.SimpleRpcServer.stop(SimpleRpcServer.java:441)
> at
> org.apache.hadoop.hbase.ipc.AbstractTestIPC.testRTEDuringConnectionSetup(AbstractTestIPC.java:194){code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-26192 Master UI hbck should provide a JSON formatted output option [hbase]
mihir6692 commented on PR #5772: URL: https://github.com/apache/hbase/pull/5772#issuecomment-2021561414 PR for branch-2 and branch-2.x https://github.com/apache/hbase/pull/5780 (It only required one manual change, using `org.apache.hadoop.hbase.StartMiniClusterOption` in branch-2 in `TestHbckMetricsResource`, while `master` and `branch-3` uses `org.apache.hadoop.hbase.StartTestingClusterOption` ) @apurtell -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-27931. [DO NOT MERGE] Update hadoop.version to 3.3.6 [hbase]
jojochuang closed pull request #5289: HBASE-27931. [DO NOT MERGE] Update hadoop.version to 3.3.6 URL: https://github.com/apache/hbase/pull/5289 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25920) Support Hadoop 3.3.1
[ https://issues.apache.org/jira/browse/HBASE-25920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HBASE-25920: --- Labels: pull-request-available (was: ) > Support Hadoop 3.3.1 > > > Key: HBASE-25920 > URL: https://issues.apache.org/jira/browse/HBASE-25920 > Project: HBase > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0-alpha-1, 2.5.0 > > > The Hadoop 3.3.1 is a big release, quite different from 3.3.0. > File this jira to track the support for Hadoop 3.3.1. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-25920 WIP. Test patch for Hadoop 3.3.1 support [hbase]
jojochuang closed pull request #3310: HBASE-25920 WIP. Test patch for Hadoop 3.3.1 support URL: https://github.com/apache/hbase/pull/3310 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-27986 Null Pointer Exception when running testLruBlockCache with a misconfiguration [hbase]
Apache-HBase commented on PR #5779: URL: https://github.com/apache/hbase/pull/5779#issuecomment-2021712358 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 6m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 6m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 300m 37s | hbase-server in the patch failed. | | | | 329m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5779 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7a423bf8a638 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Eclipse Adoptium-17.0.10+7 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/testReport/ | | Max. process+thread count | 4474 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26192 Master UI hbck should provide a JSON formatted output option [hbase]
mihir6692 commented on PR #5780: URL: https://github.com/apache/hbase/pull/5780#issuecomment-2021921186 Test failure in `TestBlockEvictionOnRegionMovement` is not related. I ran this test multiple times in local and it doesn't fail. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28366 Mis-order of SCP and regionServerReport results into region inconsistencies [hbase]
apurtell commented on code in PR #5774:
URL: https://github.com/apache/hbase/pull/5774#discussion_r1540422199
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java:
##
@@ -324,8 +324,23 @@ public void regionServerReport(ServerName sn,
ServerMetrics sl) throws YouAreDea
// the ServerName to use. Here we presume a master has already done
// that so we'll press on with whatever it gave us for ServerName.
if (!checkAndRecordNewServer(sn, sl)) {
-LOG.info("RegionServerReport ignored, could not record the server: " +
sn);
-return; // Not recorded, so no need to move on
+// Master already registered server with same (host + port) and higher
startcode.
+// This can happen if regionserver report comes late from old server
+// (possible race condition), by that time master has already
processed SCP for that
+// server and started accepting regionserver report from new server
i.e. server with
+// same (host + port) and higher startcode.
+// The exception thrown here is not meant to tell the region server it
is dead because if
+// there is a new server on the same host port, the old server should
have already been
+// dead in ideal situation.
+// The exception thrown here is to skip the later steps of the whole
regionServerReport
+// request processing. Usually, after recording it in ServerManager,
we will call the
+// related methods in AssignmentManager to record region states. If
the region server
+// is already dead, we should not do these steps anymore, so here we
throw an exception
+// to let the upper layer know that they should not continue
processing anymore.
+final String errorMsg = "RegionServerReport ignored, could not record
the server: " + sn
+ + " . Consider yourself dead as server with higher startcode is
already registered.";
Review Comment:
How about
```
final String errorMsg = "RegionServerReport received from " + sn
+ ", but another server with the same name and higher startcode is already
registered, ignoring";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28452: Missing null check of rpcServer.scheduler.executor causes NPE with invalid value of hbase.client.default.rpc.codec [hbase]
Apache-HBase commented on PR #5778: URL: https://github.com/apache/hbase/pull/5778#issuecomment-2021574910 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | master passed | | +1 :green_heart: | compile | 0m 47s | master passed | | +1 :green_heart: | shadedjars | 5m 55s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 59s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | shadedjars | 5m 48s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 220m 18s | hbase-server in the patch passed. | | | | 245m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5778 | | JIRA Issue | HBASE-28452 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 089c66982971 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/testReport/ | | Max. process+thread count | 4741 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28448 CompressionTest hangs when run over a Ozone ofs path [hbase]
Apache-HBase commented on PR #5771: URL: https://github.com/apache/hbase/pull/5771#issuecomment-2021626531 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 49s | master passed | | +1 :green_heart: | compile | 2m 31s | master passed | | +1 :green_heart: | checkstyle | 0m 38s | master passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 34s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 2m 29s | the patch passed | | +1 :green_heart: | javac | 2m 29s | the patch passed | | +1 :green_heart: | checkstyle | 0m 37s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 57s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 28m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5771 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 5f45a507eb01 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 82 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-27986 Null Pointer Exception when running testLruBlockCache with a misconfiguration [hbase]
Apache-HBase commented on PR #5779: URL: https://github.com/apache/hbase/pull/5779#issuecomment-2021637515 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | master passed | | +1 :green_heart: | compile | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 5m 49s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 1s | the patch passed | | +1 :green_heart: | compile | 0m 51s | the patch passed | | +1 :green_heart: | javac | 0m 51s | the patch passed | | +1 :green_heart: | shadedjars | 5m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 238m 43s | hbase-server in the patch failed. | | | | 263m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5779 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7eb158ad6fd0 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/testReport/ | | Max. process+thread count | 5982 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28448 CompressionTest hangs when run over a Ozone ofs path [hbase]
Apache-HBase commented on PR #5771: URL: https://github.com/apache/hbase/pull/5771#issuecomment-2021860840 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 13s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 5m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 6m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 32s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 254m 33s | hbase-server in the patch passed. | | | | 282m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5771 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ec43f4176df1 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/testReport/ | | Max. process+thread count | 4445 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-27315 Add timeout to JavaRegexEngine [hbase]
mwkang commented on code in PR #4720:
URL: https://github.com/apache/hbase/pull/4720#discussion_r1540375136
##
hbase-client/src/main/java/org/apache/hadoop/hbase/filter/RegexStringComparator.java:
##
@@ -255,11 +258,20 @@ static interface Engine {
* This is the default engine.
*/
static class JavaRegexEngine implements Engine {
+private static final Configuration conf = HBaseConfiguration.create();
Review Comment:
To pass configurations to RegexStringComparator, a new class
ConfigurationHolder was created. It holds the configuration as a member
variable. When HMaster and HRegionserver are started, it registers itself as an
observer to the ConfigurationManager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28448 CompressionTest hangs when run over a Ozone ofs path [hbase]
Apache-HBase commented on PR #5771: URL: https://github.com/apache/hbase/pull/5771#issuecomment-2021933581 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 55s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 56s | master passed | | +1 :green_heart: | compile | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 6m 29s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 24s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | shadedjars | 6m 22s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 335m 7s | hbase-server in the patch failed. | | | | 364m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5771 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a47c33f7fd56 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/testReport/ | | Max. process+thread count | 4616 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28452: Missing null check of rpcServer.scheduler.executor causes NPE with invalid value of hbase.client.default.rpc.codec [hbase]
Apache-HBase commented on PR #5778: URL: https://github.com/apache/hbase/pull/5778#issuecomment-2021557546 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 5s | master passed | | +1 :green_heart: | compile | 0m 52s | master passed | | +1 :green_heart: | shadedjars | 5m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 0m 52s | the patch passed | | +1 :green_heart: | javac | 0m 52s | the patch passed | | +1 :green_heart: | shadedjars | 5m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 205m 52s | hbase-server in the patch passed. | | | | 230m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5778 | | JIRA Issue | HBASE-28452 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0491b9e24eb3 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/testReport/ | | Max. process+thread count | 5302 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26192 Master UI hbck should provide a JSON formatted output option [hbase]
Apache-HBase commented on PR #5780: URL: https://github.com/apache/hbase/pull/5780#issuecomment-2021800157 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 38s | branch-2 passed | | +1 :green_heart: | compile | 0m 57s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 20s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | shadedjars | 5m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 52s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 208m 58s | hbase-server in the patch passed. | | | | 241m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5780/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5780 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5d84dfe0ec31 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 54276566c6 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5780/1/testReport/ | | Max. process+thread count | 4248 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5780/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28453) Support a middle ground between the Average and Fixed interval rate limiters
[
https://issues.apache.org/jira/browse/HBASE-28453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831164#comment-17831164
]
Hudson commented on HBASE-28453:
Results for branch branch-2.6
[build #82 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Support a middle ground between the Average and Fixed interval rate limiters
>
>
> Key: HBASE-28453
> URL: https://issues.apache.org/jira/browse/HBASE-28453
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Ray Mattingly
>Assignee: Ray Mattingly
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
> Attachments: Screenshot 2024-03-21 at 2.08.51 PM.png, Screenshot
> 2024-03-21 at 2.30.01 PM.png
>
>
> h3. Background
> HBase quotas support two rate limiters: a "fixed" and an "average" interval
> rate limiter.
> h4. FixedIntervalRateLimiter
> The fixed interval rate limiter is simpler: it has a TimeUnit, say 1 second,
> and it refills a resource allotment on the recurring interval. So you may get
> 10 resources every second, and if you exhaust all 10 resources in the first
> millisecond of an interval then you will need to wait 999ms to acquire even 1
> more resource.
> h4. AverageIntervalRateLimiter
> The average interval rate limiter, HBase's default, allows for more flexibly
> timed refilling of the resource allotment. Extending our previous example,
> say you have a 10 reads/sec quota and you have exhausted all 10 resources
> within 1ms of the last full refill. If you request 1 more read then, rather
> than returning a 999ms wait interval indicating the next full refill time,
> the rate limiter will recognize that you only need to wait 99ms before 1 read
> can be available. After 100ms has passed in aggregate since the last full
> refill, it will support the refilling of 1/10th the limit to facilitate the
> request for 1/10th the resources.
> h3. The Problems with Current RateLimiters
> The problem with the fixed interval rate limiter is that it is too strict
> from a latency perspective. It results in quota limits to which we cannot
> fully subscribe with any consistency.
> The problem with the average interval rate limiter is that, in practice, it
> is far too optimistic. For example, a real rate limiter might limit to
> 100MB/sec of read IO per machine. Any multigets that come in will require
> only a tiny fraction of this limit; for example, a 64kb block is only 0.06%
> of the total. As a result, the vast majority of wait intervals end up being
> tiny — like <5ms. This can actually cause an inverse of your intention, where
> setting up a throttle causes a DDOS of your RPC layer via continuous
> throttling and ~immediate retrying. I've discussed this problem in
> https://issues.apache.org/jira/browse/HBASE-28429 and proposed a minimum wait
> interval as the solution there; after some more thinking, I believe this new
> rate limiter would be a less hacky solution to this deficit so I'd like to
> close that Jira in favor of this one.
> See the attached chart where I put in place a 10k req/sec/machine throttle
> for this user at 10:43 to try to curb this high traffic, and it resulted in a
> huge spike of req/sec due to the throttle/retry loop created by the
> AverageIntervalRateLimiter.
> h3. Original Proposal: PartialIntervalRateLimiter as a Solution
> I've implemented a RateLimiter which allows for partial chunks of the overall
> interval to be refilled, by default these chunks are 10% (or 100ms of a 1s
> interval). I've deployed this to a test cluster at my day job and have seen
> this really help our ability to full subscribe to a quota
[jira] [Commented] (HBASE-28456) HBase Restore restores old data if data for the same timestamp is in different hfiles
[
https://issues.apache.org/jira/browse/HBASE-28456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831166#comment-17831166
]
Hudson commented on HBASE-28456:
Results for branch branch-2.6
[build #82 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase Restore restores old data if data for the same timestamp is in
> different hfiles
> -
>
> Key: HBASE-28456
> URL: https://issues.apache.org/jira/browse/HBASE-28456
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Reporter: Ruben Van Wanzeele
>Assignee: Bryan Beaudreault
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
> Attachments:
> Add_incremental_test_for_HBASE-28456_Fix_HBASE-28412_for_incremental_test.patch,
> ChangesOnHFilesOnSameTimestampAreNotCorrectlyRestored.java
>
>
> The restore brings back 'old' data when executing restore.
> It feels like the hfile sequence id is not respected during the restore.
> See testing code attached. The workaround solution is to trigger major
> compaction before doing the backup (not really feasible for daily backups)
> We didn't investigate this yet, but this might also impact the merge of
> multiple incremental backups (since that follows a similar code path merging
> hfiles).
> This currently blocks our support for HBase backup and restore.
> Willing to participate in a solution if necessary.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28449) Fix BackupSystemTable Scans
[
https://issues.apache.org/jira/browse/HBASE-28449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831165#comment-17831165
]
Hudson commented on HBASE-28449:
Results for branch branch-2.6
[build #82 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/82/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix BackupSystemTable Scans
>
>
> Key: HBASE-28449
> URL: https://issues.apache.org/jira/browse/HBASE-28449
> Project: HBase
> Issue Type: Bug
>Reporter: Briana Augenreich
>Assignee: Briana Augenreich
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
>
> When calculating which WALs should be included in an incremental backup the
> backup system does a prefix scan for the last roll log timestamp. This uses
> the backup root in the prefix (.) If you happen have
> multiple backup roots where one is a root of the other you'll get inaccurate
> results.
>
> Since the rowkey is let's modify
> the prefix scan to be .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] HBASE-28452: Missing null check of rpcServer.scheduler.executor causes NPE with invalid value of hbase.client.default.rpc.codec [hbase]
Apache-HBase commented on PR #5778: URL: https://github.com/apache/hbase/pull/5778#issuecomment-2021593916 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 46s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 46s | master passed | | +1 :green_heart: | compile | 0m 43s | master passed | | +1 :green_heart: | shadedjars | 5m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 28s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 240m 31s | hbase-server in the patch failed. | | | | 264m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5778 | | JIRA Issue | HBASE-28452 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a1ceb722ad1f 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/testReport/ | | Max. process+thread count | 5650 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5778/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-27986 Null Pointer Exception when running testLruBlockCache with a misconfiguration [hbase]
Apache-HBase commented on PR #5779: URL: https://github.com/apache/hbase/pull/5779#issuecomment-2021721812 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 59s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 8s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | shadedjars | 6m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 311m 46s | hbase-server in the patch passed. | | | | 341m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5779 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d542417773c0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/testReport/ | | Max. process+thread count | 4489 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5779/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28292 Make Delay prefetch property to be dynamically configured [hbase]
kabhishek4 commented on code in PR #5605:
URL: https://github.com/apache/hbase/pull/5605#discussion_r1540512005
##
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestPrefetch.java:
##
@@ -295,10 +302,23 @@ private void readStoreFile(Path storeFilePath,
throws Exception {
// Open the file
HFile.Reader reader = HFile.createReader(fs, storeFilePath, cacheConfig,
true, conf);
+startTimer();
while (!reader.prefetchComplete()) {
// Sleep for a bit
Thread.sleep(1000);
+ if (getComputeTiming()) {
+// After task is scheduled and before the delay expires, prefetch
should not start
+// if prefetchFutures contains entry (which means it's not cancelled
or completed)
+// and wait time remaining is below delay expiry watermark, it can be
deduced that
+// the prefetch is not started yet.
+if (getElapsedTime() >= (conf.getLong(PREFETCH_DELAY, 1000))) {
+ assertTrue("Prefetch should be started at this point",
reader.prefetchStarted());
+ setComputeTiming(false);
+} else {
+ assertFalse("Prefetch Should not start at this point",
reader.prefetchStarted());
+}
+ }
Review Comment:
In prefetch executor, we further compute passed in delay using variation and
a random
multiplier to get 'effective delay'. Hence, in the test, for delay of
25000 milli-secs
checking that prefetch is started after 2 milli-sec and prefetch
started after that.
However, prefetch should not start after configured delay.
In previous change set, I had introduced new class variable i.e.
computedDealy which could
have helped to get the exact value however, later removed due to
previous comments.
Tried to disable the PREFETCH_DELAY_VARIATION in the tests but looks
like it's not possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-27931) Update hadoop.version to 3.3.6
[ https://issues.apache.org/jira/browse/HBASE-27931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HBASE-27931: --- Labels: pull-request-available (was: ) > Update hadoop.version to 3.3.6 > -- > > Key: HBASE-27931 > URL: https://issues.apache.org/jira/browse/HBASE-27931 > Project: HBase > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > > HBase's default Hadoop3 version is 3.2.4 but HBase already supports Haddoop > 3.3.x. > Hadoop 3.2 line has not been updated for over a year. It is perhaps the time > to update the Hadoop dependency to the 3.3.x line. (I'll start a DISCUSS > thread if the test goes well) > 3.3.6 RC is out which fixed a bunch of CVEs and I'd like to test HBase > against it. Additionally, Hadoop 3.3.6 will permit us to use non-HDFS as WAL > storage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-26192 Master UI hbck should provide a JSON formatted output option [hbase]
Apache-HBase commented on PR #5780: URL: https://github.com/apache/hbase/pull/5780#issuecomment-2021603545 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 22s | branch-2 passed | | +1 :green_heart: | compile | 3m 15s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 52s | branch-2 passed | | +1 :green_heart: | spotless | 0m 46s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 20s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 58s | the patch passed | | +1 :green_heart: | compile | 3m 15s | the patch passed | | +1 :green_heart: | javac | 3m 15s | the patch passed | | +1 :green_heart: | checkstyle | 0m 50s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 10m 7s | Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6. | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 34m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5780/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5780 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 3fb4ee0c1b88 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 54276566c6 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5780/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-26192 Master UI hbck should provide a JSON formatted output option [hbase]
Apache-HBase commented on PR #5780: URL: https://github.com/apache/hbase/pull/5780#issuecomment-2021802269 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 11s | branch-2 passed | | +1 :green_heart: | compile | 1m 5s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 51s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 5m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 8m 8s | hbase-client in the patch passed. | | -1 :x: | unit | 208m 55s | hbase-server in the patch failed. | | | | 243m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5780/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5780 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5e692e8f120d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 54276566c6 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5780/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5780/1/testReport/ | | Max. process+thread count | 4395 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5780/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28448 CompressionTest hangs when run over a Ozone ofs path [hbase]
Apache-HBase commented on PR #5771: URL: https://github.com/apache/hbase/pull/5771#issuecomment-2021923461 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 53s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 18s | master passed | | +1 :green_heart: | compile | 1m 3s | master passed | | +1 :green_heart: | shadedjars | 6m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 48s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 321m 27s | hbase-server in the patch passed. | | | | 354m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5771 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e14b7f420ef9 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78923b75e4 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/testReport/ | | Max. process+thread count | 4590 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5771/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28458) BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached
[
https://issues.apache.org/jira/browse/HBASE-28458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HBASE-28458:
---
Labels: pull-request-available (was: )
> BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully
> cached
> ---
>
> Key: HBASE-28458
> URL: https://issues.apache.org/jira/browse/HBASE-28458
> Project: HBase
> Issue Type: Bug
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Labels: pull-request-available
>
> Noticed that
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was
> flakey, failing whenever the block eviction happened while prefetch was still
> ongoing.
> In the test, we pass an instance of BucketCache directly to the cache config,
> so the test is actually placing both data and meta blocks in the bucket
> cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted
> after the it has already evicted two blocks.
> Inside BucketCache.notifyFileCachingCompleted, we iterate through the
> backingMap entry set, counting number of blocks for the given file. Then, to
> consider whether the file is fully cached or not, we do the following
> validation:
> {noformat}
> if (dataBlockCount == count.getValue() || totalBlockCount ==
> count.getValue()) {
> LOG.debug("File {} has now been fully cached.", fileName);
> fileCacheCompleted(fileName, size);
> } {noformat}
> But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts
> two blocks and asserts that the file hasn't been considered fully cached.
> When these evictions happen while prefetch is still going, we'll pass that
> check, as the the number of blocks for the file in the backingMap would still
> be 55, which is what we pass as dataBlockCount.
> As BucketCache is intended for storing data blocks only, I believe we should
> make sure BucketCache.notifyFileCachingCompleted only accounts for data
> blocks. Also, the
> TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should
> be updated to consistently reproduce the eviction concurrent to the prefetch.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] HBASE-28458 BucketCache.notifyFileCachingCompleted may incorrectly consider a file fully cached [hbase]
wchevreuil opened a new pull request, #5777: URL: https://github.com/apache/hbase/pull/5777 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
bbeaudreault commented on code in PR #5776:
URL: https://github.com/apache/hbase/pull/5776#discussion_r1539518711
##
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupRestoreWithModifications.java:
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupInfo.BackupState.COMPLETE;
+import static org.apache.hadoop.hbase.backup.BackupType.FULL;
+import static org.apache.hadoop.hbase.backup.BackupType.INCREMENTAL;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testing.TestingHBaseCluster;
+import org.apache.hadoop.hbase.testing.TestingHBaseClusterOption;
+import org.apache.hadoop.hbase.tool.BulkLoadHFiles;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Category(MediumTests.class)
+@RunWith(Parameterized.class)
+public class TestBackupRestoreWithModifications {
Review Comment:
Sounds great, thanks! I'll keep an eye out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28412 Select correct target table for incremental backup [hbase]
bbeaudreault commented on code in PR #5776:
URL: https://github.com/apache/hbase/pull/5776#discussion_r1539531767
##
hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupRestoreWithModifications.java:
##
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.backup;
+
+import static org.apache.hadoop.hbase.backup.BackupInfo.BackupState.COMPLETE;
+import static org.apache.hadoop.hbase.backup.BackupType.FULL;
+import static org.apache.hadoop.hbase.backup.BackupType.INCREMENTAL;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.impl.BackupAdminImpl;
+import org.apache.hadoop.hbase.backup.impl.BackupManager;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testing.TestingHBaseCluster;
+import org.apache.hadoop.hbase.testing.TestingHBaseClusterOption;
+import org.apache.hadoop.hbase.tool.BulkLoadHFiles;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@Category(MediumTests.class)
+@RunWith(Parameterized.class)
+public class TestBackupRestoreWithModifications {
Review Comment:
@rubenvw-ngdata one last thing -- i know it's getting late over there in
Ghent (based on your profile), but I'd like to start the 2.6.0 release tomorrow
morning my time (Eastern Time). Happy to include this if we can get the test
fixed up before then. No pressure, just letting you know of my timeline and
acknowledging the timezone differences. Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28443 Return too slow when scanning a table with non-existing REGION_REPLICA_ID [hbase]
Apache-HBase commented on PR #5767: URL: https://github.com/apache/hbase/pull/5767#issuecomment-2020830781 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 33s | master passed | | +1 :green_heart: | compile | 3m 43s | master passed | | +1 :green_heart: | checkstyle | 0m 57s | master passed | | +1 :green_heart: | spotless | 0m 47s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 29s | master passed | | -0 :warning: | patch | 1m 52s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 3m 28s | the patch passed | | +1 :green_heart: | javac | 3m 28s | the patch passed | | +1 :green_heart: | checkstyle | 0m 54s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 47s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 56s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 3m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 39m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5767 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 9c33c3d44bce 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / feef7446f2 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5767/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28420) Aborting Active HMaster is not rejecting remote Procedure Reports
[
https://issues.apache.org/jira/browse/HBASE-28420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830850#comment-17830850
]
Aman Poonia commented on HBASE-28420:
-
[~umesh9414] i think the issue here is that new master doesn't know that a
procedure is scheduled as it is not stored in procedure store. A simple check
might not fix it. A proper solution is to store this
type(SplitWALRemoteProcedure) of procedure in proc store so when a new master
comes up it reads the proc store for current ongoing procs and gets to know
that there is a remote proc schedued and it needs to check for the progress of
that. In the above case new master doesn't even know that a procedure was
scheduled by old master.
> Aborting Active HMaster is not rejecting remote Procedure Reports
> -
>
> Key: HBASE-28420
> URL: https://issues.apache.org/jira/browse/HBASE-28420
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 2.5.7
>Reporter: Umesh Kumar Kumawat
>Assignee: Umesh Kumar Kumawat
>Priority: Critical
>
> When the Active Hmaster is in the process of abortion and another HMaster is
> becoming Active HMaster,at the same time if any region server reports the
> completion of the remote procedure, it generally goes to the old active
> HMaster because of the cached value of rssStub ->
> [code|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L2829]
> ([caller
> method|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L3941]).
> On the Master side
> ([code|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L2381]),
> It did check if the service is started but that returns true if the master
> is in the process of abortion(I didn't see when we are setting this flag
> false while abortion).
> This issue becomes *critical* when *ServerCrash of meta hosting RS and master
> failover* happens at the same time and hbase:meta got stuck in the offline
> state.
> Logs for abortion start of HMaster
> {noformat}
> 2024-02-02 07:33:11,581 ERROR [PEWorker-6] master.HMaster - * ABORTING
> master server4-1xxx,61000,1705169084562:
> FAILED persisting region=52d36581218e00a2668776cfea897132 state=CLOSING
> *{noformat}
> {noformat}
> 2024-02-02 07:33:40,999 INFO [master/server4-1xxx:61000]
> regionserver.HRegionServer - Exiting;
> stopping=hbase2b-mnds4-1-ia2.ops.sfdc.net,61000,1705169084562; zookeeper
> connection closed.{noformat}
> it took almost 30 seconds to abort the HMaster.
>
> Logs of starting SCP for meta carrying host. (This SCP is started by the new
> active HMaster)
> {noformat}
> 2024-02-02 07:33:32,622 INFO [aster/server3-1xxx61000:becomeActiveMaster]
> assignment.AssignmentManager - Scheduled
> ServerCrashProcedure pid=3305546 for server5-1xxx61020,1706857451955
> (carryingMeta=true) server5-1-
> xxx61020,1706857451955/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1b0a5293[Write
>
> locks = 1, Read locks = 0], oldState=ONLINE.{noformat}
> initialization of remote procedure
> {noformat}
> 2024-02-02 07:33:33,178 INFO [PEWorker-4] procedure2.ProcedureExecutor -
> Initialized subprocedures=[{pid=3305548,
> ppid=3305547, state=RUNNABLE; SplitWALRemoteProcedure server5-1-
> t%2C61020%2C1706857451955.meta.1706858156058.meta,
> worker=server4-1-,61020,1705169180881}]{noformat}
> Logs of remote procedure handling on Old Active Hmaster(server4-1xxx,61000)
> (in the process of abortion)
> {noformat}
> 2024-02-02 07:33:37,990 DEBUG
> [r.default.FPBQ.Fifo.handler=243,queue=9,port=61000] master.HMaster - Remote
> procedure
> done, pid=3305548{noformat}
> This should be handled by the new active HMaster so that it can wake up the
> suspended Procedure on the new Active Hmaster. As the new ActiveHMaster was
> not able to wake that up, SCP procedure got stuck thus meta stayed OFFLINE.
>
> Logs of Hmaster trying to becomeActivehmaster but stuck-
> {noformat}
> 2024-02-02 07:33:43,159 WARN [aster/server3-1-ia2:61000:becomeActiveMaster]
> master.HMaster - hbase:meta,,1.1588230740
> is NOT online; state={1588230740 state=OPEN, ts=1706859212481,
> server=server5-1-xxx,61020,1706857451955};
> ServerCrashProcedures=true. Master startup cannot progress, in
> holding-pattern until region onlined.{noformat}
> After this master was stuck till we did hmaster failover to come out of this
> situation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-28412) TableNotFoundException while restoring out-of-system incremental backup with mapping
[
https://issues.apache.org/jira/browse/HBASE-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dieter De Paepe updated HBASE-28412:
Description:
It appears that restoring a non-existing table from an incremental backup with
the "-m" parameter results in an error in the restore client.
Reproduction steps:
Build & start hbase:
{code:java}
mvn clean install -Phadoop-3.0 -DskipTests
bin/start-hbase.sh{code}
In HBase shell: create table and some values:
{code:java}
create 'test', 'cf'
put 'test', 'row1', 'cf:a', 'value1'
put 'test', 'row2', 'cf:b', 'value2'
put 'test', 'row3', 'cf:c', 'value3'
scan 'test' {code}
Create a full backup:
{code:java}
bin/hbase backup create full file:/tmp/hbase-backup{code}
Adjust some data through HBase shell:
{code:java}
put 'test', 'row1', 'cf:a', 'value1-new'
scan 'test' {code}
Create an incremental backup:
{code:java}
bin/hbase backup create incremental file:/tmp/hbase-backup {code}
Delete the original table in HBase shell:
{code:java}
disable 'test'
drop 'test' {code}
Restore the incremental backup under a new table name:
{code:java}
bin/hbase backup history
bin/hbase restore file:/tmp/hbase-backup -t "test" -m
"test-restored" {code}
This results in the following output / error:
{code:java}
...
{code}
was:
It appears that restoring a non-existing table from an incremental backup with
the "-m" parameter results in an error in the restore client.
Reproduction steps:
Build & start hbase:
{code:java}
mvn clean install -Phadoop-3.0 -DskipTests
bin/start-hbase.sh{code}
In HBase shell: create table and some values:
{code:java}
create 'test', 'cf'
put 'test', 'row1', 'cf:a', 'value1'
put 'test', 'row2', 'cf:b', 'value2'
put 'test', 'row3', 'cf:c', 'value3'
scan 'test' {code}
Create a full backup:
{code:java}
bin/hbase backup create full file:/tmp/hbase-backup{code}
Adjust some data through HBase shell:
{code:java}
put 'test', 'row1', 'cf:a', 'value1-new'
scan 'test' {code}
Create an incremental backup:
{code:java}
bin/hbase backup create incremental file:/tmp/hbase-backup {code}
Delete the original table in HBase shell:
{code:java}
disable 'test'
drop 'test' {code}
Restore the incremental backup under a new table name:
{code:java}
bin/hbase backup history
bin/hbase restore file:/tmp/hbase-backup -t "test" -m
"test-restored" {code}
This results in the following output / error:
{code:java}
// code placeholder
{code}
> TableNotFoundException while restoring out-of-system incremental backup with
> mapping
>
>
> Key: HBASE-28412
> URL: https://issues.apache.org/jira/browse/HBASE-28412
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.6.0
>Reporter: Dieter De Paepe
>Priority: Major
>
> It appears that restoring a non-existing table from an incremental backup
> with the "-m" parameter results in an error in the restore client.
> Reproduction steps:
> Build & start hbase:
> {code:java}
> mvn clean install -Phadoop-3.0 -DskipTests
> bin/start-hbase.sh{code}
> In HBase shell: create table and some values:
> {code:java}
> create 'test', 'cf'
> put 'test', 'row1', 'cf:a', 'value1'
> put 'test', 'row2', 'cf:b', 'value2'
> put 'test', 'row3', 'cf:c', 'value3'
> scan 'test' {code}
> Create a full backup:
> {code:java}
> bin/hbase backup create full file:/tmp/hbase-backup{code}
> Adjust some data through HBase shell:
> {code:java}
> put 'test', 'row1', 'cf:a', 'value1-new'
> scan 'test' {code}
> Create an incremental backup:
> {code:java}
> bin/hbase backup create incremental file:/tmp/hbase-backup {code}
> Delete the original table in HBase shell:
> {code:java}
> disable 'test'
> drop 'test' {code}
> Restore the incremental backup under a new table name:
> {code:java}
> bin/hbase backup history
> bin/hbase restore file:/tmp/hbase-backup -t "test" -m
> "test-restored" {code}
> This results in the following output / error:
> {code:java}
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-28412) Restoring incremental backups to mapped table requires existence of original table
[
https://issues.apache.org/jira/browse/HBASE-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dieter De Paepe updated HBASE-28412:
Affects Version/s: 4.0.0-alpha-1
> Restoring incremental backups to mapped table requires existence of original
> table
> --
>
> Key: HBASE-28412
> URL: https://issues.apache.org/jira/browse/HBASE-28412
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.6.0, 4.0.0-alpha-1
>Reporter: Dieter De Paepe
>Priority: Major
>
> It appears that restoring a non-existing table from an incremental backup
> with the "-m" parameter results in an error in the restore client.
> Reproduction steps:
> Build & start hbase:
> {code:java}
> mvn clean install -Phadoop-3.0 -DskipTests
> bin/start-hbase.sh{code}
> In HBase shell: create table and some values:
> {code:java}
> create 'test', 'cf'
> put 'test', 'row1', 'cf:a', 'value1'
> put 'test', 'row2', 'cf:b', 'value2'
> put 'test', 'row3', 'cf:c', 'value3'
> scan 'test' {code}
> Create a full backup:
> {code:java}
> bin/hbase backup create full file:/tmp/hbase-backup{code}
> Adjust some data through HBase shell:
> {code:java}
> put 'test', 'row1', 'cf:a', 'value1-new'
> scan 'test' {code}
> Create an incremental backup:
> {code:java}
> bin/hbase backup create incremental file:/tmp/hbase-backup {code}
> Delete the original table in HBase shell:
> {code:java}
> disable 'test'
> drop 'test' {code}
> Restore the incremental backup under a new table name:
> {code:java}
> bin/hbase backup history
> bin/hbase restore file:/tmp/hbase-backup -t "test" -m
> "test-restored" {code}
> This results in the following output / error:
> {code:java}
> ...
> 2024-03-25T13:38:53,062 WARN [main {}] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2024-03-25T13:38:53,174 INFO [main {}] Configuration.deprecation:
> hbase.client.pause.cqtbe is deprecated. Instead, use
> hbase.client.pause.server.overloaded
> 2024-03-25T13:38:53,554 INFO [main {}] impl.RestoreTablesClient: HBase table
> test-restored does not exist. It will be created during restore process
> 2024-03-25T13:38:53,593 INFO [main {}] impl.RestoreTablesClient: Restoring
> 'test' to 'test-restored' from full backup image
> file:/tmp/hbase-backup/backup_1711370230143/default/test
> 2024-03-25T13:38:53,707 INFO [main {}] util.BackupUtils: Creating target
> table 'test-restored'
> 2024-03-25T13:38:54,546 INFO [main {}] mapreduce.MapReduceRestoreJob:
> Restore test into test-restored
> 2024-03-25T13:38:54,646 INFO [main {}] mapreduce.HFileOutputFormat2:
> bulkload locality sensitive enabled
> 2024-03-25T13:38:54,647 INFO [main {}] mapreduce.HFileOutputFormat2: Looking
> up current regions for table test-restored
> 2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2:
> Configuring 1 reduce partitions to match current region count for all tables
> 2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2: Writing
> partition information to
> file:/tmp/hbase-tmp/partitions_0667b6e2-79ef-4cfe-97e1-abb204ee420d
> 2024-03-25T13:38:54,687 INFO [main {}] compress.CodecPool: Got brand-new
> compressor [.deflate]
> 2024-03-25T13:38:54,713 INFO [main {}] mapreduce.HFileOutputFormat2:
> Incremental output configured for tables: test-restored
> 2024-03-25T13:38:54,715 WARN [main {}] mapreduce.TableMapReduceUtil: The
> addDependencyJars(Configuration, Class...) method has been deprecated
> since it is easy to use incorrectly. Most users should rely on
> addDependencyJars(Job) instead. See HBASE-8386 for more details.
> 2024-03-25T13:38:54,742 WARN [main {}] impl.MetricsConfig: Cannot locate
> configuration: tried
> hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties
> 2024-03-25T13:38:54,834 INFO [main {}] input.FileInputFormat: Total input
> files to process : 1
> 2024-03-25T13:38:54,853 INFO [main {}] mapreduce.JobSubmitter: number of
> splits:1
> 2024-03-25T13:38:54,964 INFO [main {}] mapreduce.JobSubmitter: Submitting
> tokens for job: job_local748155768_0001
> 2024-03-25T13:38:54,967 INFO [main {}] mapreduce.JobSubmitter: Executing
> with tokens: []
> 2024-03-25T13:38:55,076 INFO [main {}] mapred.LocalDistributedCacheManager:
> Creating symlink:
> /tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
> <- /home/dieter/code/hbase/libjars/*
> 2024-03-25T13:38:55,079 WARN [main {}] fs.FileUtil: Command 'ln -s
> /tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
> /home/dieter/code/hbase/libjars/*' failed 1 with: ln: failed to create
> symbolic link '/home/dieter/code/hbase/libjars/*':
Re: [PR] HBASE-28366 Mis-order of SCP and regionServerReport results into region inconsistencies [hbase]
bbeaudreault commented on code in PR #5774:
URL: https://github.com/apache/hbase/pull/5774#discussion_r1539000878
##
hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java:
##
@@ -324,8 +324,23 @@ public void regionServerReport(ServerName sn,
ServerMetrics sl) throws YouAreDea
// the ServerName to use. Here we presume a master has already done
// that so we'll press on with whatever it gave us for ServerName.
if (!checkAndRecordNewServer(sn, sl)) {
-LOG.info("RegionServerReport ignored, could not record the server: " +
sn);
-return; // Not recorded, so no need to move on
+// Master already registered server with same (host + port) and higher
startcode.
+// This can happen if regionserver report comes late from old server
+// (possible race condition), by that time master has already
processed SCP for that
+// server and started accepting regionserver report from new server
i.e. server with
+// same (host + port) and higher startcode.
+// The exception thrown here is not meant to tell the region server it
is dead because if
+// there is a new server on the same host port, the old server should
have already been
+// dead in ideal situation.
+// The exception thrown here is to skip the later steps of the whole
regionServerReport
+// request processing. Usually, after recording it in ServerManager,
we will call the
+// related methods in AssignmentManager to record region states. If
the region server
+// is already dead, we should not do these steps anymore, so here we
throw an exception
+// to let the upper layer know that they should not continue
processing anymore.
+final String errorMsg = "RegionServerReport ignored, could not record
the server: " + sn
+ + " . Consider yourself dead as server with higher startcode is
already registered.";
Review Comment:
I think the comment is clear. To me the log message needs to change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-28412) TableNotFoundException while restoring out-of-system incremental backup with mapping
[
https://issues.apache.org/jira/browse/HBASE-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dieter De Paepe updated HBASE-28412:
Description:
It appears that restoring a non-existing table from an incremental backup with
the "-m" parameter results in an error in the restore client.
Reproduction steps:
Build & start hbase:
{code:java}
mvn clean install -Phadoop-3.0 -DskipTests
bin/start-hbase.sh{code}
In HBase shell: create table and some values:
{code:java}
create 'test', 'cf'
put 'test', 'row1', 'cf:a', 'value1'
put 'test', 'row2', 'cf:b', 'value2'
put 'test', 'row3', 'cf:c', 'value3'
scan 'test' {code}
Create a full backup:
{code:java}
bin/hbase backup create full file:/tmp/hbase-backup{code}
Adjust some data through HBase shell:
{code:java}
put 'test', 'row1', 'cf:a', 'value1-new'
scan 'test' {code}
Create an incremental backup:
{code:java}
bin/hbase backup create incremental file:/tmp/hbase-backup {code}
Delete the original table in HBase shell:
{code:java}
disable 'test'
drop 'test' {code}
Restore the incremental backup under a new table name:
{code:java}
bin/hbase backup history
bin/hbase restore file:/tmp/hbase-backup -t "test" -m
"test-restored" {code}
This results in the following output / error:
{code:java}
...
2024-03-25T13:38:53,062 WARN [main {}] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2024-03-25T13:38:53,174 INFO [main {}] Configuration.deprecation:
hbase.client.pause.cqtbe is deprecated. Instead, use
hbase.client.pause.server.overloaded
2024-03-25T13:38:53,554 INFO [main {}] impl.RestoreTablesClient: HBase table
test-restored does not exist. It will be created during restore process
2024-03-25T13:38:53,593 INFO [main {}] impl.RestoreTablesClient: Restoring
'test' to 'test-restored' from full backup image
file:/tmp/hbase-backup/backup_1711370230143/default/test
2024-03-25T13:38:53,707 INFO [main {}] util.BackupUtils: Creating target table
'test-restored'
2024-03-25T13:38:54,546 INFO [main {}] mapreduce.MapReduceRestoreJob: Restore
test into test-restored
2024-03-25T13:38:54,646 INFO [main {}] mapreduce.HFileOutputFormat2: bulkload
locality sensitive enabled
2024-03-25T13:38:54,647 INFO [main {}] mapreduce.HFileOutputFormat2: Looking
up current regions for table test-restored
2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2:
Configuring 1 reduce partitions to match current region count for all tables
2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2: Writing
partition information to
file:/tmp/hbase-tmp/partitions_0667b6e2-79ef-4cfe-97e1-abb204ee420d
2024-03-25T13:38:54,687 INFO [main {}] compress.CodecPool: Got brand-new
compressor [.deflate]
2024-03-25T13:38:54,713 INFO [main {}] mapreduce.HFileOutputFormat2:
Incremental output configured for tables: test-restored
2024-03-25T13:38:54,715 WARN [main {}] mapreduce.TableMapReduceUtil: The
addDependencyJars(Configuration, Class...) method has been deprecated since
it is easy to use incorrectly. Most users should rely on addDependencyJars(Job)
instead. See HBASE-8386 for more details.
2024-03-25T13:38:54,742 WARN [main {}] impl.MetricsConfig: Cannot locate
configuration: tried
hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties
2024-03-25T13:38:54,834 INFO [main {}] input.FileInputFormat: Total input
files to process : 1
2024-03-25T13:38:54,853 INFO [main {}] mapreduce.JobSubmitter: number of
splits:1
2024-03-25T13:38:54,964 INFO [main {}] mapreduce.JobSubmitter: Submitting
tokens for job: job_local748155768_0001
2024-03-25T13:38:54,967 INFO [main {}] mapreduce.JobSubmitter: Executing with
tokens: []
2024-03-25T13:38:55,076 INFO [main {}] mapred.LocalDistributedCacheManager:
Creating symlink:
/tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
<- /home/dieter/code/hbase/libjars/*
2024-03-25T13:38:55,079 WARN [main {}] fs.FileUtil: Command 'ln -s
/tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
/home/dieter/code/hbase/libjars/*' failed 1 with: ln: failed to create
symbolic link '/home/dieter/code/hbase/libjars/*': No such file or
directory2024-03-25T13:38:55,079 WARN [main {}]
mapred.LocalDistributedCacheManager: Failed to create symlink:
/tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
<- /home/dieter/code/hbase/libjars/*
2024-03-25T13:38:55,079 INFO [main {}] mapred.LocalDistributedCacheManager:
Localized
file:/tmp/hadoop/mapred/staging/dieter748155768/.staging/job_local748155768_0001/libjars
as
file:/tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
2024-03-25T13:38:55,129 INFO [main {}] mapreduce.Job: The url to track the
job: http://localhost:8080/
[jira] [Updated] (HBASE-28412) TableNotFoundException while restoring out-of-system incremental backup with mapping
[
https://issues.apache.org/jira/browse/HBASE-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dieter De Paepe updated HBASE-28412:
Description:
It appears that restoring a non-existing table from an incremental backup with
the "-m" parameter results in an error in the restore client.
Reproduction steps:
Build & start hbase:
{code:java}
mvn clean install -Phadoop-3.0 -DskipTests
bin/start-hbase.sh{code}
In HBase shell: create table and some values:
{code:java}
create 'test', 'cf'
put 'test', 'row1', 'cf:a', 'value1'
put 'test', 'row2', 'cf:b', 'value2'
put 'test', 'row3', 'cf:c', 'value3'
scan 'test' {code}
Create a full backup:
{code:java}
bin/hbase backup create full file:/tmp/hbase-backup{code}
Adjust some data through HBase shell:
{code:java}
put 'test', 'row1', 'cf:a', 'value1-new'
scan 'test' {code}
Create an incremental backup:
{code:java}
bin/hbase backup create incremental file:/tmp/hbase-backup {code}
Delete the original table in HBase shell:
{code:java}
disable 'test'
drop 'test' {code}
Restore the incremental backup under a new table name:
{code:java}
bin/hbase backup history
bin/hbase restore file:/tmp/hbase-backup -t "test" -m
"test-restored" {code}
This results in the following output / error:
{code:java}
// code placeholder
{code}
was:
It appears that restoring a non-existing table from an incremental backup with
the "-m" parameter results in an error in the restore client.
Reproduction steps:
> TableNotFoundException while restoring out-of-system incremental backup with
> mapping
>
>
> Key: HBASE-28412
> URL: https://issues.apache.org/jira/browse/HBASE-28412
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.6.0
>Reporter: Dieter De Paepe
>Priority: Major
>
> It appears that restoring a non-existing table from an incremental backup
> with the "-m" parameter results in an error in the restore client.
> Reproduction steps:
> Build & start hbase:
> {code:java}
> mvn clean install -Phadoop-3.0 -DskipTests
> bin/start-hbase.sh{code}
> In HBase shell: create table and some values:
> {code:java}
> create 'test', 'cf'
> put 'test', 'row1', 'cf:a', 'value1'
> put 'test', 'row2', 'cf:b', 'value2'
> put 'test', 'row3', 'cf:c', 'value3'
> scan 'test' {code}
> Create a full backup:
> {code:java}
> bin/hbase backup create full file:/tmp/hbase-backup{code}
> Adjust some data through HBase shell:
> {code:java}
> put 'test', 'row1', 'cf:a', 'value1-new'
> scan 'test' {code}
> Create an incremental backup:
> {code:java}
> bin/hbase backup create incremental file:/tmp/hbase-backup {code}
> Delete the original table in HBase shell:
> {code:java}
> disable 'test'
> drop 'test' {code}
> Restore the incremental backup under a new table name:
> {code:java}
> bin/hbase backup history
> bin/hbase restore file:/tmp/hbase-backup -t "test" -m
> "test-restored" {code}
> This results in the following output / error:
> {code:java}
> // code placeholder
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-28412) TableNotFoundException while restoring out-of-system incremental backup with mapping
[
https://issues.apache.org/jira/browse/HBASE-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dieter De Paepe updated HBASE-28412:
Description:
It appears that restoring a non-existing table from an incremental backup with
the "-m" parameter results in an error in the restore client.
Reproduction steps:
Build & start hbase:
{code:java}
mvn clean install -Phadoop-3.0 -DskipTests
bin/start-hbase.sh{code}
In HBase shell: create table and some values:
{code:java}
create 'test', 'cf'
put 'test', 'row1', 'cf:a', 'value1'
put 'test', 'row2', 'cf:b', 'value2'
put 'test', 'row3', 'cf:c', 'value3'
scan 'test' {code}
Create a full backup:
{code:java}
bin/hbase backup create full file:/tmp/hbase-backup{code}
Adjust some data through HBase shell:
{code:java}
put 'test', 'row1', 'cf:a', 'value1-new'
scan 'test' {code}
Create an incremental backup:
{code:java}
bin/hbase backup create incremental file:/tmp/hbase-backup {code}
Delete the original table in HBase shell:
{code:java}
disable 'test'
drop 'test' {code}
Restore the incremental backup under a new table name:
{code:java}
bin/hbase backup history
bin/hbase restore file:/tmp/hbase-backup -t "test" -m
"test-restored" {code}
This results in the following output / error:
{code:java}
... org.apache.hadoop.hbase.TableNotFoundException: test
at
org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:635)
~[hbase-client-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:244)
~[hbase-client-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceHFileSplitterJob.createSubmittableJob(MapReduceHFileSplitterJob.java:117)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceHFileSplitterJob.run(MapReduceHFileSplitterJob.java:165)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceRestoreJob.run(MapReduceRestoreJob.java:84)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.util.RestoreTool.incrementalRestoreTable(RestoreTool.java:205)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restoreImages(RestoreTablesClient.java:186)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restore(RestoreTablesClient.java:229)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.execute(RestoreTablesClient.java:265)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.restore(BackupAdminImpl.java:518)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.RestoreDriver.parseAndRun(RestoreDriver.java:176)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.RestoreDriver.doWork(RestoreDriver.java:216)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at org.apache.hadoop.hbase.backup.RestoreDriver.run(RestoreDriver.java:252)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
~[hadoop-common-3.3.5.jar:?]
at
org.apache.hadoop.hbase.backup.RestoreDriver.main(RestoreDriver.java:224)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
2024-03-25T13:38:56,491 ERROR [main {}] backup.RestoreDriver: Error while
running restore backup
java.io.IOException: Can not restore from backup directory
file:/tmp/hbase-backup/backup_1711370245048/default/test/cf/860b10b854204226834b85212e529f29
(check Hadoop and HBase logs)
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceRestoreJob.run(MapReduceRestoreJob.java:103)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.util.RestoreTool.incrementalRestoreTable(RestoreTool.java:205)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restoreImages(RestoreTablesClient.java:186)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restore(RestoreTablesClient.java:229)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.execute(RestoreTablesClient.java:265)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.restore(BackupAdminImpl.java:518)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.RestoreDriver.parseAndRun(RestoreDriver.java:176)
~[hbase-backup-2.6.1-SNAPSHOT.jar:2.6.1-SNAPSHOT]
at
org.apache.hadoop.hbase.backup.RestoreDriver.doWork(RestoreDriver.java:216)
[jira] [Commented] (HBASE-28420) Aborting Active HMaster is not rejecting remote Procedure Reports
[
https://issues.apache.org/jira/browse/HBASE-28420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830857#comment-17830857
]
Umesh Kumar Kumawat commented on HBASE-28420:
-
[~mnpoonia] in this case, the new master only scheduled all the procedures, so
unknown to the master is not the case here.
[~zhangduo] I am planning to solve this. I don't have any clear solution.
Should we add some timed checks on the master side or simply add some
persistence on the master side?
> Aborting Active HMaster is not rejecting remote Procedure Reports
> -
>
> Key: HBASE-28420
> URL: https://issues.apache.org/jira/browse/HBASE-28420
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 2.5.7
>Reporter: Umesh Kumar Kumawat
>Assignee: Umesh Kumar Kumawat
>Priority: Critical
>
> When the Active Hmaster is in the process of abortion and another HMaster is
> becoming Active HMaster,at the same time if any region server reports the
> completion of the remote procedure, it generally goes to the old active
> HMaster because of the cached value of rssStub ->
> [code|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L2829]
> ([caller
> method|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L3941]).
> On the Master side
> ([code|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L2381]),
> It did check if the service is started but that returns true if the master
> is in the process of abortion(I didn't see when we are setting this flag
> false while abortion).
> This issue becomes *critical* when *ServerCrash of meta hosting RS and master
> failover* happens at the same time and hbase:meta got stuck in the offline
> state.
> Logs for abortion start of HMaster
> {noformat}
> 2024-02-02 07:33:11,581 ERROR [PEWorker-6] master.HMaster - * ABORTING
> master server4-1xxx,61000,1705169084562:
> FAILED persisting region=52d36581218e00a2668776cfea897132 state=CLOSING
> *{noformat}
> {noformat}
> 2024-02-02 07:33:40,999 INFO [master/server4-1xxx:61000]
> regionserver.HRegionServer - Exiting;
> stopping=hbase2b-mnds4-1-ia2.ops.sfdc.net,61000,1705169084562; zookeeper
> connection closed.{noformat}
> it took almost 30 seconds to abort the HMaster.
>
> Logs of starting SCP for meta carrying host. (This SCP is started by the new
> active HMaster)
> {noformat}
> 2024-02-02 07:33:32,622 INFO [aster/server3-1xxx61000:becomeActiveMaster]
> assignment.AssignmentManager - Scheduled
> ServerCrashProcedure pid=3305546 for server5-1xxx61020,1706857451955
> (carryingMeta=true) server5-1-
> xxx61020,1706857451955/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1b0a5293[Write
>
> locks = 1, Read locks = 0], oldState=ONLINE.{noformat}
> initialization of remote procedure
> {noformat}
> 2024-02-02 07:33:33,178 INFO [PEWorker-4] procedure2.ProcedureExecutor -
> Initialized subprocedures=[{pid=3305548,
> ppid=3305547, state=RUNNABLE; SplitWALRemoteProcedure server5-1-
> t%2C61020%2C1706857451955.meta.1706858156058.meta,
> worker=server4-1-,61020,1705169180881}]{noformat}
> Logs of remote procedure handling on Old Active Hmaster(server4-1xxx,61000)
> (in the process of abortion)
> {noformat}
> 2024-02-02 07:33:37,990 DEBUG
> [r.default.FPBQ.Fifo.handler=243,queue=9,port=61000] master.HMaster - Remote
> procedure
> done, pid=3305548{noformat}
> This should be handled by the new active HMaster so that it can wake up the
> suspended Procedure on the new Active Hmaster. As the new ActiveHMaster was
> not able to wake that up, SCP procedure got stuck thus meta stayed OFFLINE.
>
> Logs of Hmaster trying to becomeActivehmaster but stuck-
> {noformat}
> 2024-02-02 07:33:43,159 WARN [aster/server3-1-ia2:61000:becomeActiveMaster]
> master.HMaster - hbase:meta,,1.1588230740
> is NOT online; state={1588230740 state=OPEN, ts=1706859212481,
> server=server5-1-xxx,61020,1706857451955};
> ServerCrashProcedures=true. Master startup cannot progress, in
> holding-pattern until region onlined.{noformat}
> After this master was stuck till we did hmaster failover to come out of this
> situation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28456) HBase Restore restores old data if data for the same timestamp is in different hfiles
[ https://issues.apache.org/jira/browse/HBASE-28456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830868#comment-17830868 ] Ruben Van Wanzeele commented on HBASE-28456: Hi Bryan, The good news is, your fix is valid for the incremental issue too I did add a fix for HBASE-28412 where there is an issue for the incremental restore See patch attached. Hope it helps. > HBase Restore restores old data if data for the same timestamp is in > different hfiles > - > > Key: HBASE-28456 > URL: https://issues.apache.org/jira/browse/HBASE-28456 > Project: HBase > Issue Type: Bug > Components: backuprestore >Affects Versions: 2.6.0, 3.0.0 >Reporter: Ruben Van Wanzeele >Assignee: Bryan Beaudreault >Priority: Blocker > Labels: pull-request-available > Attachments: > Add_incremental_test_for_HBASE-28456_Fix_HBASE-28412_for_incremental_test.patch, > ChangesOnHFilesOnSameTimestampAreNotCorrectlyRestored.java > > > The restore brings back 'old' data when executing restore. > It feels like the hfile sequence id is not respected during the restore. > See testing code attached. The workaround solution is to trigger major > compaction before doing the backup (not really feasible for daily backups) > We didn't investigate this yet, but this might also impact the merge of > multiple incremental backups (since that follows a similar code path merging > hfiles). > This currently blocks our support for HBase backup and restore. > Willing to participate in a solution if necessary. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28456) HBase Restore restores old data if data for the same timestamp is in different hfiles
[ https://issues.apache.org/jira/browse/HBASE-28456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruben Van Wanzeele updated HBASE-28456: --- Attachment: Add_incremental_test_for_HBASE-28456_Fix_HBASE-28412_for_incremental_test.patch > HBase Restore restores old data if data for the same timestamp is in > different hfiles > - > > Key: HBASE-28456 > URL: https://issues.apache.org/jira/browse/HBASE-28456 > Project: HBase > Issue Type: Bug > Components: backuprestore >Affects Versions: 2.6.0, 3.0.0 >Reporter: Ruben Van Wanzeele >Assignee: Bryan Beaudreault >Priority: Blocker > Labels: pull-request-available > Attachments: > Add_incremental_test_for_HBASE-28456_Fix_HBASE-28412_for_incremental_test.patch, > ChangesOnHFilesOnSameTimestampAreNotCorrectlyRestored.java > > > The restore brings back 'old' data when executing restore. > It feels like the hfile sequence id is not respected during the restore. > See testing code attached. The workaround solution is to trigger major > compaction before doing the backup (not really feasible for daily backups) > We didn't investigate this yet, but this might also impact the merge of > multiple incremental backups (since that follows a similar code path merging > hfiles). > This currently blocks our support for HBase backup and restore. > Willing to participate in a solution if necessary. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28412) TableNotFoundException while restoring out-of-system incremental backup with mapping
[
https://issues.apache.org/jira/browse/HBASE-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dieter De Paepe updated HBASE-28412:
Description:
It appears that restoring a non-existing table from an incremental backup with
the "-m" parameter results in an error in the restore client.
Reproduction steps:
was:
It appears that restoring a non-existing table from an incremental backup with
the "-m" parameter results in an error in the restore client.
Reproduction steps:
Start from a table with some data
{code:java}
hbase:006:0> scan 'lily_tenant_acme:LILY_SETTINGS'
ROW COLUMN+CELL
row1 column=0:a,
timestamp=2024-02-27T18:49:46.489, value=val1
row1 column=0:test,
timestamp=2024-02-29T15:46:29.956, value=value2 {code}
Create a full backup.
{code:java}
hbase backup create full /tmp/backup_hbase2 {code}
Change the table (eg using hbase shell):
{code:java}
put 'lily_tenant_acme:LILY_SETTINGS', 'row1', '0:test', 'value3' {code}
Create an incremental backup
{code:java}
hbase backup create incremental /tmp/backup_hbase2 {code}
Delete the table:
{code:java}
disable 'lily_tenant_acme:LILY_SETTINGS'
drop 'lily_tenant_acme:LILY_SETTINGS' {code}
Restore of the incremental backup with the "-m" parameter results in an error:
{code:java}
hbase restore hdfs:///tmp/backup_hbase2 backup_1709222166873 -t
"lily_tenant_acme:LILY_SETTINGS" -m "lily_tenant_acme:LILY_SETTINGS-restored"
24/02/29 15:59:20 WARN org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil:
The addDependencyJars(Configuration, Class...) method has been deprecated
since it is easy to use incorrectly. Most users should rely on
addDependencyJars(Job) instead. See HBASE-8386 for more details.
24/02/29 15:59:35 WARN org.apache.hadoop.hbase.tool.LoadIncrementalHFiles:
Skipping non-directory
hdfs://hdfsns/user/lily/hbase-staging/bulk_output-lily_tenant_acme-LILY_SETTINGS-restored-1709222360030/_SUCCESS
24/02/29 15:59:36 ERROR
org.apache.hadoop.hbase.backup.mapreduce.MapReduceRestoreJob:
org.apache.hadoop.hbase.TableNotFoundException: lily_tenant_acme:LILY_SETTINGS
org.apache.hadoop.hbase.TableNotFoundException: lily_tenant_acme:LILY_SETTINGS
at
org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:635)
at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:244)
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceHFileSplitterJob.createSubmittableJob(MapReduceHFileSplitterJob.java:117)
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceHFileSplitterJob.run(MapReduceHFileSplitterJob.java:165)
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceRestoreJob.run(MapReduceRestoreJob.java:84)
at
org.apache.hadoop.hbase.backup.util.RestoreTool.incrementalRestoreTable(RestoreTool.java:205)
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restoreImages(RestoreTablesClient.java:185)
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restore(RestoreTablesClient.java:228)
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.execute(RestoreTablesClient.java:264)
at
org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.restore(BackupAdminImpl.java:518)
at
org.apache.hadoop.hbase.backup.RestoreDriver.parseAndRun(RestoreDriver.java:176)
at
org.apache.hadoop.hbase.backup.RestoreDriver.doWork(RestoreDriver.java:216)
at org.apache.hadoop.hbase.backup.RestoreDriver.run(RestoreDriver.java:252)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.hbase.backup.RestoreDriver.main(RestoreDriver.java:224)
24/02/29 15:59:36 ERROR org.apache.hadoop.hbase.backup.RestoreDriver: Error
while running restore backup
java.io.IOException: Can not restore from backup directory
hdfs://hdfsns/tmp/backup_hbase2/backup_1709222166873/lily_tenant_acme/LILY_SETTINGS/0/f011305e78cd4196bf67d4dcca859ea1,hdfs://hdfsns/tmp/backup_hbase2/backup_1709222166873/lily_tenant_acme/LILY_SETTINGS/c6f5124f51d797fde1a43093747d04fa/0/800bad45ccd049fda6a439b60954cd06_SeqId_4_,hdfs://hdfsns/tmp/backup_hbase2/backup_1709222166873/lily_tenant_acme/LILY_SETTINGS/c6f5124f51d797fde1a43093747d04fa/0/e334bbc7089e4c3f8dab84b9b704824c_SeqId_6_
(check Hadoop and HBase logs)
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceRestoreJob.run(MapReduceRestoreJob.java:103)
at
org.apache.hadoop.hbase.backup.util.RestoreTool.incrementalRestoreTable(RestoreTool.java:205)
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restoreImages(RestoreTablesClient.java:185)
at
org.apache.hadoop.hbase.backup.impl.RestoreTablesClient.restore(RestoreTablesClient.java:228)
at
[jira] [Updated] (HBASE-28412) Restoring incremental backups to mapped table requires existence of original table
[
https://issues.apache.org/jira/browse/HBASE-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dieter De Paepe updated HBASE-28412:
Summary: Restoring incremental backups to mapped table requires existence
of original table (was: TableNotFoundException while restoring out-of-system
incremental backup with mapping)
> Restoring incremental backups to mapped table requires existence of original
> table
> --
>
> Key: HBASE-28412
> URL: https://issues.apache.org/jira/browse/HBASE-28412
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Affects Versions: 2.6.0
>Reporter: Dieter De Paepe
>Priority: Major
>
> It appears that restoring a non-existing table from an incremental backup
> with the "-m" parameter results in an error in the restore client.
> Reproduction steps:
> Build & start hbase:
> {code:java}
> mvn clean install -Phadoop-3.0 -DskipTests
> bin/start-hbase.sh{code}
> In HBase shell: create table and some values:
> {code:java}
> create 'test', 'cf'
> put 'test', 'row1', 'cf:a', 'value1'
> put 'test', 'row2', 'cf:b', 'value2'
> put 'test', 'row3', 'cf:c', 'value3'
> scan 'test' {code}
> Create a full backup:
> {code:java}
> bin/hbase backup create full file:/tmp/hbase-backup{code}
> Adjust some data through HBase shell:
> {code:java}
> put 'test', 'row1', 'cf:a', 'value1-new'
> scan 'test' {code}
> Create an incremental backup:
> {code:java}
> bin/hbase backup create incremental file:/tmp/hbase-backup {code}
> Delete the original table in HBase shell:
> {code:java}
> disable 'test'
> drop 'test' {code}
> Restore the incremental backup under a new table name:
> {code:java}
> bin/hbase backup history
> bin/hbase restore file:/tmp/hbase-backup -t "test" -m
> "test-restored" {code}
> This results in the following output / error:
> {code:java}
> ...
> 2024-03-25T13:38:53,062 WARN [main {}] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2024-03-25T13:38:53,174 INFO [main {}] Configuration.deprecation:
> hbase.client.pause.cqtbe is deprecated. Instead, use
> hbase.client.pause.server.overloaded
> 2024-03-25T13:38:53,554 INFO [main {}] impl.RestoreTablesClient: HBase table
> test-restored does not exist. It will be created during restore process
> 2024-03-25T13:38:53,593 INFO [main {}] impl.RestoreTablesClient: Restoring
> 'test' to 'test-restored' from full backup image
> file:/tmp/hbase-backup/backup_1711370230143/default/test
> 2024-03-25T13:38:53,707 INFO [main {}] util.BackupUtils: Creating target
> table 'test-restored'
> 2024-03-25T13:38:54,546 INFO [main {}] mapreduce.MapReduceRestoreJob:
> Restore test into test-restored
> 2024-03-25T13:38:54,646 INFO [main {}] mapreduce.HFileOutputFormat2:
> bulkload locality sensitive enabled
> 2024-03-25T13:38:54,647 INFO [main {}] mapreduce.HFileOutputFormat2: Looking
> up current regions for table test-restored
> 2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2:
> Configuring 1 reduce partitions to match current region count for all tables
> 2024-03-25T13:38:54,669 INFO [main {}] mapreduce.HFileOutputFormat2: Writing
> partition information to
> file:/tmp/hbase-tmp/partitions_0667b6e2-79ef-4cfe-97e1-abb204ee420d
> 2024-03-25T13:38:54,687 INFO [main {}] compress.CodecPool: Got brand-new
> compressor [.deflate]
> 2024-03-25T13:38:54,713 INFO [main {}] mapreduce.HFileOutputFormat2:
> Incremental output configured for tables: test-restored
> 2024-03-25T13:38:54,715 WARN [main {}] mapreduce.TableMapReduceUtil: The
> addDependencyJars(Configuration, Class...) method has been deprecated
> since it is easy to use incorrectly. Most users should rely on
> addDependencyJars(Job) instead. See HBASE-8386 for more details.
> 2024-03-25T13:38:54,742 WARN [main {}] impl.MetricsConfig: Cannot locate
> configuration: tried
> hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties
> 2024-03-25T13:38:54,834 INFO [main {}] input.FileInputFormat: Total input
> files to process : 1
> 2024-03-25T13:38:54,853 INFO [main {}] mapreduce.JobSubmitter: number of
> splits:1
> 2024-03-25T13:38:54,964 INFO [main {}] mapreduce.JobSubmitter: Submitting
> tokens for job: job_local748155768_0001
> 2024-03-25T13:38:54,967 INFO [main {}] mapreduce.JobSubmitter: Executing
> with tokens: []
> 2024-03-25T13:38:55,076 INFO [main {}] mapred.LocalDistributedCacheManager:
> Creating symlink:
> /tmp/hadoop-dieter/mapred/local/job_local748155768_0001_0768a243-06e8-4524-8a6d-016ddd75df52/libjars
> <- /home/dieter/code/hbase/libjars/*
> 2024-03-25T13:38:55,079 WARN [main {}] fs.FileUtil: Command 'ln -s
>
[jira] [Commented] (HBASE-28420) Aborting Active HMaster is not rejecting remote Procedure Reports
[
https://issues.apache.org/jira/browse/HBASE-28420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830863#comment-17830863
]
Aman Poonia commented on HBASE-28420:
-
[~umesh9414] Thank you for the clarification.
I was wondering what happens if the new master also restarts in between and
another host becomes master or is in process of becoming active master?
> Aborting Active HMaster is not rejecting remote Procedure Reports
> -
>
> Key: HBASE-28420
> URL: https://issues.apache.org/jira/browse/HBASE-28420
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 2.5.7
>Reporter: Umesh Kumar Kumawat
>Assignee: Umesh Kumar Kumawat
>Priority: Critical
>
> When the Active Hmaster is in the process of abortion and another HMaster is
> becoming Active HMaster,at the same time if any region server reports the
> completion of the remote procedure, it generally goes to the old active
> HMaster because of the cached value of rssStub ->
> [code|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L2829]
> ([caller
> method|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L3941]).
> On the Master side
> ([code|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L2381]),
> It did check if the service is started but that returns true if the master
> is in the process of abortion(I didn't see when we are setting this flag
> false while abortion).
> This issue becomes *critical* when *ServerCrash of meta hosting RS and master
> failover* happens at the same time and hbase:meta got stuck in the offline
> state.
> Logs for abortion start of HMaster
> {noformat}
> 2024-02-02 07:33:11,581 ERROR [PEWorker-6] master.HMaster - * ABORTING
> master server4-1xxx,61000,1705169084562:
> FAILED persisting region=52d36581218e00a2668776cfea897132 state=CLOSING
> *{noformat}
> {noformat}
> 2024-02-02 07:33:40,999 INFO [master/server4-1xxx:61000]
> regionserver.HRegionServer - Exiting;
> stopping=hbase2b-mnds4-1-ia2.ops.sfdc.net,61000,1705169084562; zookeeper
> connection closed.{noformat}
> it took almost 30 seconds to abort the HMaster.
>
> Logs of starting SCP for meta carrying host. (This SCP is started by the new
> active HMaster)
> {noformat}
> 2024-02-02 07:33:32,622 INFO [aster/server3-1xxx61000:becomeActiveMaster]
> assignment.AssignmentManager - Scheduled
> ServerCrashProcedure pid=3305546 for server5-1xxx61020,1706857451955
> (carryingMeta=true) server5-1-
> xxx61020,1706857451955/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1b0a5293[Write
>
> locks = 1, Read locks = 0], oldState=ONLINE.{noformat}
> initialization of remote procedure
> {noformat}
> 2024-02-02 07:33:33,178 INFO [PEWorker-4] procedure2.ProcedureExecutor -
> Initialized subprocedures=[{pid=3305548,
> ppid=3305547, state=RUNNABLE; SplitWALRemoteProcedure server5-1-
> t%2C61020%2C1706857451955.meta.1706858156058.meta,
> worker=server4-1-,61020,1705169180881}]{noformat}
> Logs of remote procedure handling on Old Active Hmaster(server4-1xxx,61000)
> (in the process of abortion)
> {noformat}
> 2024-02-02 07:33:37,990 DEBUG
> [r.default.FPBQ.Fifo.handler=243,queue=9,port=61000] master.HMaster - Remote
> procedure
> done, pid=3305548{noformat}
> This should be handled by the new active HMaster so that it can wake up the
> suspended Procedure on the new Active Hmaster. As the new ActiveHMaster was
> not able to wake that up, SCP procedure got stuck thus meta stayed OFFLINE.
>
> Logs of Hmaster trying to becomeActivehmaster but stuck-
> {noformat}
> 2024-02-02 07:33:43,159 WARN [aster/server3-1-ia2:61000:becomeActiveMaster]
> master.HMaster - hbase:meta,,1.1588230740
> is NOT online; state={1588230740 state=OPEN, ts=1706859212481,
> server=server5-1-xxx,61020,1706857451955};
> ServerCrashProcedures=true. Master startup cannot progress, in
> holding-pattern until region onlined.{noformat}
> After this master was stuck till we did hmaster failover to come out of this
> situation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HBASE-28456) HBase Restore restores old data if data for the same timestamp is in different hfiles
[ https://issues.apache.org/jira/browse/HBASE-28456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830868#comment-17830868 ] Ruben Van Wanzeele edited comment on HBASE-28456 at 3/26/24 10:29 AM: -- Hi Bryan, The good news is, your fix is valid for the incremental issue too, see test added. I did add a fix for HBASE-28412 where there is an issue for the incremental restore See patch attached. Hope it helps. was (Author: rubenvw): Hi Bryan, The good news is, your fix is valid for the incremental issue too I did add a fix for HBASE-28412 where there is an issue for the incremental restore See patch attached. Hope it helps. > HBase Restore restores old data if data for the same timestamp is in > different hfiles > - > > Key: HBASE-28456 > URL: https://issues.apache.org/jira/browse/HBASE-28456 > Project: HBase > Issue Type: Bug > Components: backuprestore >Affects Versions: 2.6.0, 3.0.0 >Reporter: Ruben Van Wanzeele >Assignee: Bryan Beaudreault >Priority: Blocker > Labels: pull-request-available > Attachments: > Add_incremental_test_for_HBASE-28456_Fix_HBASE-28412_for_incremental_test.patch, > ChangesOnHFilesOnSameTimestampAreNotCorrectlyRestored.java > > > The restore brings back 'old' data when executing restore. > It feels like the hfile sequence id is not respected during the restore. > See testing code attached. The workaround solution is to trigger major > compaction before doing the backup (not really feasible for daily backups) > We didn't investigate this yet, but this might also impact the merge of > multiple incremental backups (since that follows a similar code path merging > hfiles). > This currently blocks our support for HBase backup and restore. > Willing to participate in a solution if necessary. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28456) HBase Restore restores old data if data for the same timestamp is in different hfiles
[ https://issues.apache.org/jira/browse/HBASE-28456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830878#comment-17830878 ] Bryan Beaudreault commented on HBASE-28456: --- Thanks [~rubenvw]. Do you want to submit a PR for HBASE-28412 with your fix? > HBase Restore restores old data if data for the same timestamp is in > different hfiles > - > > Key: HBASE-28456 > URL: https://issues.apache.org/jira/browse/HBASE-28456 > Project: HBase > Issue Type: Bug > Components: backuprestore >Affects Versions: 2.6.0, 3.0.0 >Reporter: Ruben Van Wanzeele >Assignee: Bryan Beaudreault >Priority: Blocker > Labels: pull-request-available > Attachments: > Add_incremental_test_for_HBASE-28456_Fix_HBASE-28412_for_incremental_test.patch, > ChangesOnHFilesOnSameTimestampAreNotCorrectlyRestored.java > > > The restore brings back 'old' data when executing restore. > It feels like the hfile sequence id is not respected during the restore. > See testing code attached. The workaround solution is to trigger major > compaction before doing the backup (not really feasible for daily backups) > We didn't investigate this yet, but this might also impact the merge of > multiple incremental backups (since that follows a similar code path merging > hfiles). > This currently blocks our support for HBase backup and restore. > Willing to participate in a solution if necessary. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28420) Aborting Active HMaster is not rejecting remote Procedure Reports
[
https://issues.apache.org/jira/browse/HBASE-28420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830901#comment-17830901
]
Duo Zhang commented on HBASE-28420:
---
You can see the code in RegionRemoteProcedureBase, what we have done in
reportTransition. We should do the same in ServerRemoteProcedure, in the
remoteOperationDone method, we should update the procedure's field to store the
result, and then wake up the procedure to continue the later processing.
> Aborting Active HMaster is not rejecting remote Procedure Reports
> -
>
> Key: HBASE-28420
> URL: https://issues.apache.org/jira/browse/HBASE-28420
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 2.5.7
>Reporter: Umesh Kumar Kumawat
>Assignee: Umesh Kumar Kumawat
>Priority: Critical
>
> When the Active Hmaster is in the process of abortion and another HMaster is
> becoming Active HMaster,at the same time if any region server reports the
> completion of the remote procedure, it generally goes to the old active
> HMaster because of the cached value of rssStub ->
> [code|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L2829]
> ([caller
> method|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java#L3941]).
> On the Master side
> ([code|https://github.com/apache/hbase/blob/branch-2.5/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L2381]),
> It did check if the service is started but that returns true if the master
> is in the process of abortion(I didn't see when we are setting this flag
> false while abortion).
> This issue becomes *critical* when *ServerCrash of meta hosting RS and master
> failover* happens at the same time and hbase:meta got stuck in the offline
> state.
> Logs for abortion start of HMaster
> {noformat}
> 2024-02-02 07:33:11,581 ERROR [PEWorker-6] master.HMaster - * ABORTING
> master server4-1xxx,61000,1705169084562:
> FAILED persisting region=52d36581218e00a2668776cfea897132 state=CLOSING
> *{noformat}
> {noformat}
> 2024-02-02 07:33:40,999 INFO [master/server4-1xxx:61000]
> regionserver.HRegionServer - Exiting;
> stopping=hbase2b-mnds4-1-ia2.ops.sfdc.net,61000,1705169084562; zookeeper
> connection closed.{noformat}
> it took almost 30 seconds to abort the HMaster.
>
> Logs of starting SCP for meta carrying host. (This SCP is started by the new
> active HMaster)
> {noformat}
> 2024-02-02 07:33:32,622 INFO [aster/server3-1xxx61000:becomeActiveMaster]
> assignment.AssignmentManager - Scheduled
> ServerCrashProcedure pid=3305546 for server5-1xxx61020,1706857451955
> (carryingMeta=true) server5-1-
> xxx61020,1706857451955/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@1b0a5293[Write
>
> locks = 1, Read locks = 0], oldState=ONLINE.{noformat}
> initialization of remote procedure
> {noformat}
> 2024-02-02 07:33:33,178 INFO [PEWorker-4] procedure2.ProcedureExecutor -
> Initialized subprocedures=[{pid=3305548,
> ppid=3305547, state=RUNNABLE; SplitWALRemoteProcedure server5-1-
> t%2C61020%2C1706857451955.meta.1706858156058.meta,
> worker=server4-1-,61020,1705169180881}]{noformat}
> Logs of remote procedure handling on Old Active Hmaster(server4-1xxx,61000)
> (in the process of abortion)
> {noformat}
> 2024-02-02 07:33:37,990 DEBUG
> [r.default.FPBQ.Fifo.handler=243,queue=9,port=61000] master.HMaster - Remote
> procedure
> done, pid=3305548{noformat}
> This should be handled by the new active HMaster so that it can wake up the
> suspended Procedure on the new Active Hmaster. As the new ActiveHMaster was
> not able to wake that up, SCP procedure got stuck thus meta stayed OFFLINE.
>
> Logs of Hmaster trying to becomeActivehmaster but stuck-
> {noformat}
> 2024-02-02 07:33:43,159 WARN [aster/server3-1-ia2:61000:becomeActiveMaster]
> master.HMaster - hbase:meta,,1.1588230740
> is NOT online; state={1588230740 state=OPEN, ts=1706859212481,
> server=server5-1-xxx,61020,1706857451955};
> ServerCrashProcedures=true. Master startup cannot progress, in
> holding-pattern until region onlined.{noformat}
> After this master was stuck till we did hmaster failover to come out of this
> situation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28456) HBase Restore restores old data if data for the same timestamp is in different hfiles
[ https://issues.apache.org/jira/browse/HBASE-28456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830907#comment-17830907 ] Ruben Van Wanzeele commented on HBASE-28456: I created [https://github.com/apache/hbase/pull/5776] for that. > HBase Restore restores old data if data for the same timestamp is in > different hfiles > - > > Key: HBASE-28456 > URL: https://issues.apache.org/jira/browse/HBASE-28456 > Project: HBase > Issue Type: Bug > Components: backuprestore >Affects Versions: 2.6.0, 3.0.0 >Reporter: Ruben Van Wanzeele >Assignee: Bryan Beaudreault >Priority: Blocker > Labels: pull-request-available > Attachments: > Add_incremental_test_for_HBASE-28456_Fix_HBASE-28412_for_incremental_test.patch, > ChangesOnHFilesOnSameTimestampAreNotCorrectlyRestored.java > > > The restore brings back 'old' data when executing restore. > It feels like the hfile sequence id is not respected during the restore. > See testing code attached. The workaround solution is to trigger major > compaction before doing the backup (not really feasible for daily backups) > We didn't investigate this yet, but this might also impact the merge of > multiple incremental backups (since that follows a similar code path merging > hfiles). > This currently blocks our support for HBase backup and restore. > Willing to participate in a solution if necessary. -- This message was sent by Atlassian Jira (v8.20.10#820010)
