[GitHub] [hbase] huaxiangsun commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
huaxiangsun commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r509003475
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -71,38 +76,81 @@
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
- private Configuration conf;
private MasterServices masterServices;
- private boolean splitEnabled;
- private boolean mergeEnabled;
- private int minRegionCount;
- private Period mergeMinRegionAge;
- private int mergeMinRegionSizeMb;
+
+ /** Ensure configuration changes are applied atomically. */
+ private final ReadWriteLock configUpdateLock = new ReentrantReadWriteLock();
+ @GuardedBy("configUpdateLock") private Configuration conf;
+ @GuardedBy("configUpdateLock") private boolean splitEnabled;
+ @GuardedBy("configUpdateLock") private boolean mergeEnabled;
+ @GuardedBy("configUpdateLock") private int minRegionCount;
+ @GuardedBy("configUpdateLock") private Period mergeMinRegionAge;
+ @GuardedBy("configUpdateLock") private int mergeMinRegionSizeMb;
public SimpleRegionNormalizer() {
-splitEnabled = DEFAULT_SPLIT_ENABLED;
-mergeEnabled = DEFAULT_MERGE_ENABLED;
-minRegionCount = DEFAULT_MIN_REGION_COUNT;
-mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
-mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock();
+try {
+ splitEnabled = DEFAULT_SPLIT_ENABLED;
+ mergeEnabled = DEFAULT_MERGE_ENABLED;
+ minRegionCount = DEFAULT_MIN_REGION_COUNT;
+ mergeMinRegionAge = Period.ofDays(DEFAULT_MERGE_MIN_REGION_AGE_DAYS);
+ mergeMinRegionSizeMb = DEFAULT_MERGE_MIN_REGION_SIZE_MB;
+} finally {
+ writeLock.unlock();
+}
}
@Override
public Configuration getConf() {
-return conf;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return conf;
+} finally {
+ readLock.unlock();
+}
}
@Override
public void setConf(final Configuration conf) {
if (conf == null) {
return;
}
-this.conf = conf;
-splitEnabled = conf.getBoolean(SPLIT_ENABLED_KEY, DEFAULT_SPLIT_ENABLED);
-mergeEnabled = conf.getBoolean(MERGE_ENABLED_KEY, DEFAULT_MERGE_ENABLED);
-minRegionCount = parseMinRegionCount(conf);
-mergeMinRegionAge = parseMergeMinRegionAge(conf);
-mergeMinRegionSizeMb = parseMergeMinRegionSizeMb(conf);
+
+final Lock readLock = configUpdateLock.readLock();
+final Lock writeLock = configUpdateLock.writeLock();
+writeLock.lock(); // "a writer can acquire the read lock, but not
vice-versa."
+readLock.lock();
Review comment:
Why is readLock.lock() called here? Thought writeLock is enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Apache-HBase commented on pull request #2568: URL: https://github.com/apache/hbase/pull/2568#issuecomment-713318222 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 18s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | compile | 1m 20s | master passed | | +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 6m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 26s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 149m 40s | hbase-server in the patch passed. | | | | 180m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2568 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 18b5aab88c14 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/testReport/ | | Max. process+thread count | 4032 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Apache-HBase commented on pull request #2568: URL: https://github.com/apache/hbase/pull/2568#issuecomment-713312633 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 1m 34s | master passed | | +1 :green_heart: | shadedjars | 6m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 1m 33s | the patch passed | | +1 :green_heart: | javac | 1m 33s | the patch passed | | +1 :green_heart: | shadedjars | 6m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 41s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 132m 12s | hbase-server in the patch passed. | | | | 164m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2568 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 845062de16c2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 2020-01-14 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/testReport/ | | Max. process+thread count | 3937 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25210) RegionInfo.isOffline is now a duplication with RegionInfo.isSplit
Duo Zhang created HBASE-25210: - Summary: RegionInfo.isOffline is now a duplication with RegionInfo.isSplit Key: HBASE-25210 URL: https://issues.apache.org/jira/browse/HBASE-25210 Project: HBase Issue Type: Improvement Reporter: Duo Zhang The only place, where we set it to true is in splitRegion, and at the same time we will set split to true. So in general, I suggest that we deprecated isOffline and isSplitParent in RegionInfo, only leave the isSplit method. And in RegionInfoBuilder, we deprecated setOffline and only leave the setSplit method. This could make our code base cleaner. And for serialization compatibility, we'd better still keep the split and offline fields in the actual RegionInfo datastructure for a while. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2541: HBASE-25173 Remove owner related methods in TableDescriptor/TableDescriptorBuilder
Apache-HBase commented on pull request #2541: URL: https://github.com/apache/hbase/pull/2541#issuecomment-713282433 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 51s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | checkstyle | 2m 5s | master passed | | +1 :green_heart: | spotbugs | 3m 37s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | +1 :green_heart: | checkstyle | 0m 28s | The patch passed checkstyle in hbase-client | | +1 :green_heart: | checkstyle | 1m 13s | hbase-server: The patch generated 0 new + 188 unchanged - 1 fixed = 188 total (was 189) | | +1 :green_heart: | checkstyle | 0m 10s | The patch passed checkstyle in hbase-shell | | +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-endpoint | | -0 :warning: | rubocop | 0m 23s | The patch generated 4 new + 572 unchanged - 6 fixed = 576 total (was 578) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 49s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 4m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 42s | The patch does not generate ASF License warnings. | | | | 51m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2541 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle rubocop | | uname | Linux 40401d898ae7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | rubocop | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/4/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server hbase-shell hbase-endpoint U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25207) Revisit the implementation and usage of RegionStates.include
[ https://issues.apache.org/jira/browse/HBASE-25207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25207: -- Component/s: Region Assignment > Revisit the implementation and usage of RegionStates.include > > > Key: HBASE-25207 > URL: https://issues.apache.org/jira/browse/HBASE-25207 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Duo Zhang >Priority: Major > > After several round of refactoring and fixing, the method has been used in > lots of places and the implementation looks really confusing. > As in the first if condition for testing RegionStateNode and RegionInfo > state, we will always return false when split is true, which means we will > always filter out split parent, as a split parent, is split = true and also > offline = true. > I think the reason why there is no problem is that, only in > EnableTableProcedure we call this method with offline = true, and > EnableTableProcedure does not need to deal with split parent... > And now since we found a problem in HBASE-25206, where we need to get split > parent when deleting a table, I think it is time to revisit this method and > make logic less confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Apache-HBase commented on pull request #2568: URL: https://github.com/apache/hbase/pull/2568#issuecomment-713274238 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 18s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 40s | master passed | | +1 :green_heart: | checkstyle | 1m 36s | master passed | | +1 :green_heart: | spotbugs | 2m 46s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | checkstyle | 1m 30s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 35s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 27s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 46m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2568 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux b6fda8ae0ff6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache9 commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r508960285
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaSimpleChooser.java
##
@@ -0,0 +1,248 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import static org.apache.hadoop.hbase.HConstants.DEFAULT_META_REPLICA_NUM;
+import static org.apache.hadoop.hbase.HConstants.META_REPLICAS_NUM;
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+
+/**
+ * MetaReplicaLoadBalanceReplicaSimpleChooser implements a simple meta replica
load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up meta,
+ * it first check if the row is the stale location cache, if yes, this means
the the location from
+ * meta replica is stale, it will go to the primary meta to look up
update-to-date location;
+ * otherwise, it will randomly pick up a meta replica region for meta lookup.
When clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a chore.
+ */
+
+/**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+class StaleLocationCacheEntry {
Review comment:
Give it a separated file or make it an inner class. Having two classes
in a single file is not a good practise.
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaChooser.java
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * There are two modes with meta replica support.
+ * HighAvailable- Client sends requests to the primary meta region
first, within a
+ * configured amount of time, if there is no response
coming back,
+ * client sends requests to all replica regions and takes
the first
+ * response.
+ *
+ * LoadBalance - Client sends requests to meta replica regions in a
round-robin mode,
+ * if results from replica regions are stale, next time,
client sends requests for
+ * these stable locations to the primary meta region. In
this mode, scan
+ * requests are load
[GitHub] [hbase] Apache-HBase commented on pull request #2556: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Apache-HBase commented on pull request #2556: URL: https://github.com/apache/hbase/pull/2556#issuecomment-713267715 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 50s | master passed | | +1 :green_heart: | compile | 1m 10s | master passed | | +1 :green_heart: | shadedjars | 7m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 44s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 8m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 205m 38s | hbase-server in the patch passed. | | | | 237m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2556 | | JIRA Issue | HBASE-25193 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ff831f362daa 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 2020-01-14 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/2/testReport/ | | Max. process+thread count | 3358 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache9 commented on pull request #2570: URL: https://github.com/apache/hbase/pull/2570#issuecomment-713259579 On a high level design, we used to have a `hbase.meta.replicas.use` config to control whether to make use meta replicas at client side. Do we want to deprecate this config and let our new configs rule here? Just asking. For me, I think the scope of `hbase.meta.replicas.use` is too wide, as it will impact all the client operations on meta(not sure if we have hacked all the places but at least it is designed to). After reviewing the PR, I think our approach here is only for the locator related logic right? This is also what I expect. In general, I think there are 3 ways we access meta table at client side: 1. Locator related logic. This is the most critical path at client side and also makes the most pressure on meta table. 2. Admin related logic. We have delegated most of the operaations through master but there are still some places we will access meta directly. But admin operation is expected to be low frequent so I do not think we need to deal with it here. 3. Users access meta table directly by their own. This is controlled by user written code so I do not think we need to deal with it either, users should take care of it by their own. I think only 1 is what we really care here, so I suggest that, we just narrow the scope of the newly introduced configs to be locator only(maybe by adding 'locator' keyword in the config name), and consider it first before `hbase.meta.replicas.use` in the locator related logic. So users do not need to set `hbase.meta.replicas.use` to true to enable this feature, just select the LB mode. If the new config is disabled(set to None maybe), then we honor the old `hbase.meta.replicas.use` config. In general, I think the abstraction and trick here are both good. Setting the replica id directly on Query is a straight forward way to archive our goal here, and the chooser or selector is also a good abstraction. The only concern is how to implement the 'fallback to primary' logic as we need to pass down from the rpc retrying caller of the actual exception type, anyway, this can be done later. And I suggest we just make this PR against master branch, it is only client side change and whether we have implement the meta region replication should not impact the code. Why I suggest this is that, the code for master and branch-2 will be different, as on branch-2, the sync client has its own logic to do region locating, which is not well constructed I believe(we expose a bunch of locating methods in ClusterConnection interface and use it everywhere). So if we want to include this feature in 2.4.0, we'd better make this PR against master, and also backport it to branch-2 ASAP. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
anoopsjohn commented on a change in pull request #2483:
URL: https://github.com/apache/hbase/pull/2483#discussion_r508268448
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -9143,4 +9162,8 @@ public void setReadRequestsCount(long readRequestsCount) {
public void setWriteRequestsCount(long writeRequestsCount) {
this.writeRequestsCount.add(writeRequestsCount);
}
+
+ public void setFullScanRequestsCount(long fullScanRequestsCount) {
Review comment:
Not used. Do we need this setter?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
##
@@ -3498,6 +3501,12 @@ public ScanResponse scan(final RpcController controller,
final ScanRequest reque
throw new ServiceException(e);
}
HRegion region = rsh.r;
+// If the start row is the actual start Row of the region and end row is
actual end row of region
+// or
+// If the stop is not specified or the stop Row is greater than the end
key of the current region
+if(!region.getRegionInfo().getTable().isSystemTable()){
Review comment:
Nit : missing code formatting
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java
##
@@ -23,6 +23,7 @@
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.hbase.HConstants;
Review comment:
Unused import
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -7238,6 +7246,9 @@ public boolean nextRaw(List outResults,
ScannerContext scannerContext)
rsServices.getMetrics().updateReadQueryMeter(getRegionInfo().getTable());
}
+ if(isFullRegionScan()) {
Review comment:
nit format
if (isFullRegionScan()) {
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -7699,6 +7710,14 @@ public void run() throws IOException {
// callback
this.close();
}
+
+public boolean isFullRegionScan() {
Review comment:
Can be private now?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2556: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Apache-HBase commented on pull request #2556: URL: https://github.com/apache/hbase/pull/2556#issuecomment-713243775 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | master passed | | +1 :green_heart: | compile | 0m 55s | master passed | | +1 :green_heart: | shadedjars | 6m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 6m 28s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 140m 33s | hbase-server in the patch passed. | | | | 166m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2556 | | JIRA Issue | HBASE-25193 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 032eee0e7dd5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/2/testReport/ | | Max. process+thread count | 4514 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-713237969 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 46s | master passed | | +1 :green_heart: | compile | 1m 22s | master passed | | +1 :green_heart: | shadedjars | 6m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 6m 31s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hbase-common in the patch passed. | | -1 :x: | unit | 145m 33s | hbase-server in the patch failed. | | | | 175m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ac84e216b262 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/3/testReport/ | | Max. process+thread count | 4502 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-713236711 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 56s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 11s | master passed | | +1 :green_heart: | compile | 1m 31s | master passed | | +1 :green_heart: | shadedjars | 6m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 7s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 32s | the patch passed | | +1 :green_heart: | javac | 1m 32s | the patch passed | | +1 :green_heart: | shadedjars | 6m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 47s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 135m 24s | hbase-server in the patch passed. | | | | 170m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3c0321ff7036 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 2020-01-14 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/3/testReport/ | | Max. process+thread count | 4122 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2556: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Apache-HBase commented on pull request #2556: URL: https://github.com/apache/hbase/pull/2556#issuecomment-713207495 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 11s | master passed | | +1 :green_heart: | checkstyle | 1m 12s | master passed | | +1 :green_heart: | spotbugs | 2m 11s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | +1 :green_heart: | checkstyle | 1m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 47s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 42m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2556 | | JIRA Issue | HBASE-25193 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 096e96263c47 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2556/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2523: HBASE-25167 Normalizer support for hot config reloading
Apache-HBase commented on pull request #2523: URL: https://github.com/apache/hbase/pull/2523#issuecomment-713198337 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 33s | master passed | | +1 :green_heart: | checkstyle | 1m 30s | master passed | | +1 :green_heart: | spotbugs | 2m 43s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 23s | the patch passed | | +1 :green_heart: | checkstyle | 0m 25s | hbase-common: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) | | +1 :green_heart: | checkstyle | 1m 6s | The patch passed checkstyle in hbase-server | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 9s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 45m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2523 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 4fc970cab9da 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2523/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on pull request #2556: HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
sandeepvinayak commented on pull request #2556: URL: https://github.com/apache/hbase/pull/2556#issuecomment-713193016 @wchevreuil This makes sense, I added both the options keeping them mutually exclusive. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
ndimiduk commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r508884289
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -140,39 +188,76 @@ private static int parseMergeMinRegionSizeMb(final
Configuration conf) {
key, parsedValue, settledValue);
}
+ private static void logConfigurationUpdated(final String key, final T
oldValue,
+final T newValue) {
+if (!Objects.equals(oldValue, newValue)) {
+ LOG.info("Updated configuration for key '{}' from {} to {}", key,
oldValue, newValue);
+}
+ }
+
/**
* Return this instance's configured value for {@value #SPLIT_ENABLED_KEY}.
*/
public boolean isSplitEnabled() {
-return splitEnabled;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return splitEnabled;
+} finally {
+ readLock.unlock();
+}
}
/**
* Return this instance's configured value for {@value #MERGE_ENABLED_KEY}.
*/
public boolean isMergeEnabled() {
-return mergeEnabled;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return mergeEnabled;
+} finally {
+ readLock.unlock();
+}
}
/**
* Return this instance's configured value for {@value
#MIN_REGION_COUNT_KEY}.
*/
public int getMinRegionCount() {
-return minRegionCount;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return minRegionCount;
+} finally {
+ readLock.unlock();
+}
}
/**
* Return this instance's configured value for {@value
#MERGE_MIN_REGION_AGE_DAYS_KEY}.
*/
public Period getMergeMinRegionAge() {
-return mergeMinRegionAge;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return mergeMinRegionAge;
+} finally {
+ readLock.unlock();
+}
}
/**
* Return this instance's configured value for {@value
#MERGE_MIN_REGION_SIZE_MB_KEY}.
*/
public int getMergeMinRegionSizeMb() {
Review comment:
In light of
[HBASE-24640](https://issues.apache.org/jira/browse/HBASE-24640), I didn't want
to introduce any new uses of this annotation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
ndimiduk commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r508883897
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -71,38 +76,81 @@
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
- private Configuration conf;
private MasterServices masterServices;
- private boolean splitEnabled;
- private boolean mergeEnabled;
- private int minRegionCount;
- private Period mergeMinRegionAge;
- private int mergeMinRegionSizeMb;
+
+ /** Ensure configuration changes are applied atomically. */
+ private final ReadWriteLock configUpdateLock = new ReentrantReadWriteLock();
+ @GuardedBy("configUpdateLock") private Configuration conf;
Review comment:
I'm not sure about "all" uses, it just seemed prudent for this use. I'm
also not sure our static analysis tools honor this `GuardedBy` annotation, I
have a TODO for myself to track this down and see if it's really supported.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] brfrn169 commented on pull request #2569: HBASE-25206 Data loss can happen if a cloned table loses original spl…
brfrn169 commented on pull request #2569: URL: https://github.com/apache/hbase/pull/2569#issuecomment-713182063 Thank you for reviewing this! @Apache9 > If you do not mind, I plan to open an issue first to address the problem of the include method, and then we could come back to fix the bug here. Sure. Please go ahead. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] brfrn169 commented on pull request #2498: HBASE-24996 Support CheckAndMutate in Region.batchMutate()
brfrn169 commented on pull request #2498: URL: https://github.com/apache/hbase/pull/2498#issuecomment-713179853 Thank you very much! @Apache9 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-713154876 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 57s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 24s | master passed | | +1 :green_heart: | compile | 1m 26s | master passed | | +1 :green_heart: | shadedjars | 7m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 26s | the patch passed | | +1 :green_heart: | compile | 1m 32s | the patch passed | | +1 :green_heart: | javac | 1m 32s | the patch passed | | +1 :green_heart: | shadedjars | 7m 49s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 41s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 212m 9s | hbase-server in the patch failed. | | | | 246m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7ac16cb1f1c6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/testReport/ | | Max. process+thread count | 3378 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r508824027
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaSimpleChooser.java
##
@@ -0,0 +1,248 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import static org.apache.hadoop.hbase.HConstants.DEFAULT_META_REPLICA_NUM;
+import static org.apache.hadoop.hbase.HConstants.META_REPLICAS_NUM;
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+
+/**
+ * MetaReplicaLoadBalanceReplicaSimpleChooser implements a simple meta replica
load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up meta,
+ * it first check if the row is the stale location cache, if yes, this means
the the location from
+ * meta replica is stale, it will go to the primary meta to look up
update-to-date location;
+ * otherwise, it will randomly pick up a meta replica region for meta lookup.
When clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a chore.
+ */
+
+/**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+class StaleLocationCacheEntry {
+ // meta replica id where
+ private int metaReplicaId;
+
+ // timestamp in milliseconds
+ private long timestamp;
+
+ private byte[] endKey;
+
+ StaleLocationCacheEntry(final int metaReplicaId, final byte[] endKey) {
+this.metaReplicaId = metaReplicaId;
+this.endKey = endKey;
+timestamp = System.currentTimeMillis();
+ }
+
+ public byte[] getEndKey() {
+return this.endKey;
+ }
+
+ public int getMetaReplicaId() {
+return this.metaReplicaId;
+ }
+ public long getTimestamp() {
+return this.timestamp;
+ }
+
+ @Override
+ public String toString() {
+return new ToStringBuilder(this)
+ .append("endKey", endKey)
+ .append("metaReplicaId", metaReplicaId)
+ .append("timestamp", timestamp)
+ .toString();
+ }
+}
+
+/**
+ * A simple implementation of MetaReplicaLoadBalanceReplicaChooser.
+ *
+ * It follows a simple algorithm to choose a meta replica to go:
+ *
+ * 1. If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a meta replica region to do lookup.
+ * 2. If the location from meta replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * MetaReplicaLoadBalanceReplicaSimpleChooser.
+ * 3. When client tries to do meta lookup, it checks StaleLocationCache first
for rows it tries to
+ * lookup, if entry exists, it will go with primary meta region to do
lookup; otherwise, it
+ * will follow step 1.
+ * 4. A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ */
+class MetaReplicaLoadBalanceReplicaSimpleChooser implements
MetaReplicaLoadBalanceReplicaChooser {
+ private static final Logger LOG =
+LoggerFactory.getLogger(MetaReplicaLoadBalanceReplicaSimpleChooser.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL = 1500; // 1.5 seconds
+
+ private final
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570: URL: https://github.com/apache/hbase/pull/2570#discussion_r508823274 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaSimpleChooser.java ## @@ -0,0 +1,248 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.client; + +import org.apache.commons.lang3.builder.ToStringBuilder; +import org.apache.hadoop.hbase.HRegionLocation; +import org.apache.hadoop.hbase.ScheduledChore; +import org.apache.hadoop.hbase.Stoppable; +import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.util.Bytes; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import java.util.Iterator; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.ConcurrentNavigableMap; +import java.util.concurrent.ConcurrentSkipListMap; +import java.util.concurrent.ThreadLocalRandom; +import static org.apache.hadoop.hbase.HConstants.DEFAULT_META_REPLICA_NUM; +import static org.apache.hadoop.hbase.HConstants.META_REPLICAS_NUM; +import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow; +import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR; +import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent; + +/** + * MetaReplicaLoadBalanceReplicaSimpleChooser implements a simple meta replica load balancing + * algorithm. It maintains a stale location cache for each table. Whenever client looks up meta, + * it first check if the row is the stale location cache, if yes, this means the the location from + * meta replica is stale, it will go to the primary meta to look up update-to-date location; + * otherwise, it will randomly pick up a meta replica region for meta lookup. When clients receive + * RegionNotServedException from region servers, it will add these region locations to the stale + * location cache. The stale cache will be cleaned up periodically by a chore. Review comment: I need something to clean up the stale cache periodically. Let me do some research on guava cache. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r508821616
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaSimpleChooser.java
##
@@ -0,0 +1,248 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import static org.apache.hadoop.hbase.HConstants.DEFAULT_META_REPLICA_NUM;
+import static org.apache.hadoop.hbase.HConstants.META_REPLICAS_NUM;
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+
+/**
+ * MetaReplicaLoadBalanceReplicaSimpleChooser implements a simple meta replica
load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up meta,
+ * it first check if the row is the stale location cache, if yes, this means
the the location from
+ * meta replica is stale, it will go to the primary meta to look up
update-to-date location;
+ * otherwise, it will randomly pick up a meta replica region for meta lookup.
When clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a chore.
+ */
+
+/**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+class StaleLocationCacheEntry {
+ // meta replica id where
+ private int metaReplicaId;
+
+ // timestamp in milliseconds
+ private long timestamp;
+
+ private byte[] endKey;
+
+ StaleLocationCacheEntry(final int metaReplicaId, final byte[] endKey) {
+this.metaReplicaId = metaReplicaId;
+this.endKey = endKey;
+timestamp = System.currentTimeMillis();
+ }
+
+ public byte[] getEndKey() {
+return this.endKey;
+ }
+
+ public int getMetaReplicaId() {
+return this.metaReplicaId;
+ }
+ public long getTimestamp() {
+return this.timestamp;
+ }
+
+ @Override
+ public String toString() {
+return new ToStringBuilder(this)
+ .append("endKey", endKey)
+ .append("metaReplicaId", metaReplicaId)
+ .append("timestamp", timestamp)
+ .toString();
+ }
+}
+
+/**
+ * A simple implementation of MetaReplicaLoadBalanceReplicaChooser.
+ *
+ * It follows a simple algorithm to choose a meta replica to go:
+ *
+ * 1. If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a meta replica region to do lookup.
+ * 2. If the location from meta replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * MetaReplicaLoadBalanceReplicaSimpleChooser.
+ * 3. When client tries to do meta lookup, it checks StaleLocationCache first
for rows it tries to
+ * lookup, if entry exists, it will go with primary meta region to do
lookup; otherwise, it
+ * will follow step 1.
+ * 4. A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ */
+class MetaReplicaLoadBalanceReplicaSimpleChooser implements
MetaReplicaLoadBalanceReplicaChooser {
+ private static final Logger LOG =
+LoggerFactory.getLogger(MetaReplicaLoadBalanceReplicaSimpleChooser.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL = 1500; // 1.5 seconds
+
+ private final
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r508821616
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaSimpleChooser.java
##
@@ -0,0 +1,248 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.commons.lang3.builder.ToStringBuilder;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ThreadLocalRandom;
+import static org.apache.hadoop.hbase.HConstants.DEFAULT_META_REPLICA_NUM;
+import static org.apache.hadoop.hbase.HConstants.META_REPLICAS_NUM;
+import static org.apache.hadoop.hbase.client.ConnectionUtils.isEmptyStopRow;
+import static org.apache.hadoop.hbase.util.Bytes.BYTES_COMPARATOR;
+import static org.apache.hadoop.hbase.util.ConcurrentMapUtils.computeIfAbsent;
+
+/**
+ * MetaReplicaLoadBalanceReplicaSimpleChooser implements a simple meta replica
load balancing
+ * algorithm. It maintains a stale location cache for each table. Whenever
client looks up meta,
+ * it first check if the row is the stale location cache, if yes, this means
the the location from
+ * meta replica is stale, it will go to the primary meta to look up
update-to-date location;
+ * otherwise, it will randomly pick up a meta replica region for meta lookup.
When clients receive
+ * RegionNotServedException from region servers, it will add these region
locations to the stale
+ * location cache. The stale cache will be cleaned up periodically by a chore.
+ */
+
+/**
+ * StaleLocationCacheEntry is the entry when a stale location is reported by
an client.
+ */
+class StaleLocationCacheEntry {
+ // meta replica id where
+ private int metaReplicaId;
+
+ // timestamp in milliseconds
+ private long timestamp;
+
+ private byte[] endKey;
+
+ StaleLocationCacheEntry(final int metaReplicaId, final byte[] endKey) {
+this.metaReplicaId = metaReplicaId;
+this.endKey = endKey;
+timestamp = System.currentTimeMillis();
+ }
+
+ public byte[] getEndKey() {
+return this.endKey;
+ }
+
+ public int getMetaReplicaId() {
+return this.metaReplicaId;
+ }
+ public long getTimestamp() {
+return this.timestamp;
+ }
+
+ @Override
+ public String toString() {
+return new ToStringBuilder(this)
+ .append("endKey", endKey)
+ .append("metaReplicaId", metaReplicaId)
+ .append("timestamp", timestamp)
+ .toString();
+ }
+}
+
+/**
+ * A simple implementation of MetaReplicaLoadBalanceReplicaChooser.
+ *
+ * It follows a simple algorithm to choose a meta replica to go:
+ *
+ * 1. If there is no stale location entry for rows it looks up, it will
randomly
+ * pick a meta replica region to do lookup.
+ * 2. If the location from meta replica region is stale, client gets
RegionNotServedException
+ * from region server, in this case, it will create
StaleLocationCacheEntry in
+ * MetaReplicaLoadBalanceReplicaSimpleChooser.
+ * 3. When client tries to do meta lookup, it checks StaleLocationCache first
for rows it tries to
+ * lookup, if entry exists, it will go with primary meta region to do
lookup; otherwise, it
+ * will follow step 1.
+ * 4. A chore will periodically run to clean up cache entries in the
StaleLocationCache.
+ */
+class MetaReplicaLoadBalanceReplicaSimpleChooser implements
MetaReplicaLoadBalanceReplicaChooser {
+ private static final Logger LOG =
+LoggerFactory.getLogger(MetaReplicaLoadBalanceReplicaSimpleChooser.class);
+ private final long STALE_CACHE_TIMEOUT_IN_MILLISECONDS = 3000; // 3 seconds
+ private final int STALE_CACHE_CLEAN_CHORE_INTERVAL = 1500; // 1.5 seconds
+
+ private final
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r508821332
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaChooser.java
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * There are two modes with meta replica support.
+ * HighAvailable- Client sends requests to the primary meta region
first, within a
+ * configured amount of time, if there is no response
coming back,
+ * client sends requests to all replica regions and takes
the first
+ * response.
+ *
+ * LoadBalance - Client sends requests to meta replica regions in a
round-robin mode,
+ * if results from replica regions are stale, next time,
client sends requests for
+ * these stable locations to the primary meta region. In
this mode, scan
+ * requests are load balanced across all replica regions.
+ */
+enum MetaReplicaMode {
+ None,
+ HighAvailable,
+ LoadBalance
+}
+
+/**
+ * A Meta replica chooser decides which meta replica to go for scan requests.
+ */
+@InterfaceAudience.Private
+public interface MetaReplicaLoadBalanceReplicaChooser {
+
+ void updateCacheOnError(final HRegionLocation loc, final int
fromMetaReplicaId);
Review comment:
Is the cache here TableCache or StaleCache maintained in
ReplicaSelector? There is no need to pass it in.
Will take care of the rest comments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r508819430
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaChooser.java
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * There are two modes with meta replica support.
+ * HighAvailable- Client sends requests to the primary meta region
first, within a
+ * configured amount of time, if there is no response
coming back,
+ * client sends requests to all replica regions and takes
the first
+ * response.
+ *
+ * LoadBalance - Client sends requests to meta replica regions in a
round-robin mode,
+ * if results from replica regions are stale, next time,
client sends requests for
+ * these stable locations to the primary meta region. In
this mode, scan
+ * requests are load balanced across all replica regions.
+ */
+enum MetaReplicaMode {
+ None,
+ HighAvailable,
Review comment:
Yeah, META_REPLICAS_MODE_LOADBALANCE_REPILCA_CHOOSER(/SELECTOR) is
supposed to be the config change for a different implementation.
If it is changed to ReplicaMode, it sounds like this is supported for user
tables as well. A ReplicaMode only applies to some special system tables?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r508813012
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaChooser.java
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * There are two modes with meta replica support.
+ * HighAvailable- Client sends requests to the primary meta region
first, within a
+ * configured amount of time, if there is no response
coming back,
+ * client sends requests to all replica regions and takes
the first
+ * response.
+ *
+ * LoadBalance - Client sends requests to meta replica regions in a
round-robin mode,
+ * if results from replica regions are stale, next time,
client sends requests for
+ * these stable locations to the primary meta region. In
this mode, scan
+ * requests are load balanced across all replica regions.
+ */
+enum MetaReplicaMode {
+ None,
+ HighAvailable,
+ LoadBalance
+}
+
+/**
+ * A Meta replica chooser decides which meta replica to go for scan requests.
+ */
+@InterfaceAudience.Private
+public interface MetaReplicaLoadBalanceReplicaChooser {
Review comment:
Thanks, selector is a better fit, had a hard time to pick up a right
name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-713115395 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 38s | master passed | | +1 :green_heart: | compile | 1m 31s | master passed | | +1 :green_heart: | shadedjars | 7m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 3s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 29s | the patch passed | | +1 :green_heart: | compile | 1m 31s | the patch passed | | +1 :green_heart: | javac | 1m 31s | the patch passed | | +1 :green_heart: | shadedjars | 7m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 37s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 136m 10s | hbase-server in the patch failed. | | | | 168m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 36f44d5ff947 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/testReport/ | | Max. process+thread count | 4390 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache-HBase commented on pull request #2570: URL: https://github.com/apache/hbase/pull/2570#issuecomment-713089262 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-18070 Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 12s | HBASE-18070 passed | | +1 :green_heart: | compile | 2m 24s | HBASE-18070 passed | | +1 :green_heart: | shadedjars | 8m 11s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 32s | hbase-client in HBASE-18070 failed. | | -0 :warning: | javadoc | 0m 19s | hbase-common in HBASE-18070 failed. | | -0 :warning: | javadoc | 0m 47s | hbase-server in HBASE-18070 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 52s | the patch passed | | +1 :green_heart: | compile | 2m 25s | the patch passed | | +1 :green_heart: | javac | 2m 25s | the patch passed | | +1 :green_heart: | shadedjars | 8m 14s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 30s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 47s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 49s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 17s | hbase-client in the patch passed. | | -1 :x: | unit | 144m 52s | hbase-server in the patch failed. | | | | 186m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2570 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2c97bc4d46aa 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-18070 / 4e47554af8 | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/testReport/ | | Max. process+thread count | 4210 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache-HBase commented on pull request #2570: URL: https://github.com/apache/hbase/pull/2570#issuecomment-713087298 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-18070 Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 40s | HBASE-18070 passed | | +1 :green_heart: | compile | 1m 44s | HBASE-18070 passed | | +1 :green_heart: | shadedjars | 6m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 22s | HBASE-18070 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | compile | 1m 46s | the patch passed | | +1 :green_heart: | javac | 1m 46s | the patch passed | | +1 :green_heart: | shadedjars | 6m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 1s | hbase-client in the patch passed. | | -1 :x: | unit | 149m 33s | hbase-server in the patch failed. | | | | 183m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2570 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 12298d174364 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-18070 / 4e47554af8 | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/testReport/ | | Max. process+thread count | 4557 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25209) Add CLI support to remove a particular configuration from ReplicationPeerConfig.
[
https://issues.apache.org/jira/browse/HBASE-25209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ankit Jain resolved HBASE-25209.
Resolution: Not A Problem
> Add CLI support to remove a particular configuration from
> ReplicationPeerConfig.
>
>
> Key: HBASE-25209
> URL: https://issues.apache.org/jira/browse/HBASE-25209
> Project: HBase
> Issue Type: Improvement
> Components: Replication, shell
> Environment: {code:java}
> // code placeholder
> {code}
>Reporter: Ankit Jain
>Assignee: Ankit Jain
>Priority: Minor
>
> Currently, there is no easy way to remove a configuration from
> ReplicationPeerConfig after it is being set.
> This can come in handy at multiple instances when we want to remove a
> particular configuration when it's not needed anymore or it is causing
> unexpected behavior. For eg in case there is a need to remove a walFilter
> (hbase.replication.source.custom.walentryfilters) configuration.
> As part of this Jira we want to add that support.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25209) Add CLI support to remove a particular configuration from ReplicationPeerConfig.
[
https://issues.apache.org/jira/browse/HBASE-25209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217879#comment-17217879
]
Ankit Jain commented on HBASE-25209:
Closing this out as we can set an empty string as part of the config CONFIG =>
\{ "foo" => ""} using update_peer_configuration which will also work and there
is no immediate need for this request.
> Add CLI support to remove a particular configuration from
> ReplicationPeerConfig.
>
>
> Key: HBASE-25209
> URL: https://issues.apache.org/jira/browse/HBASE-25209
> Project: HBase
> Issue Type: Improvement
> Components: Replication, shell
> Environment: {code:java}
> // code placeholder
> {code}
>Reporter: Ankit Jain
>Assignee: Ankit Jain
>Priority: Minor
>
> Currently, there is no easy way to remove a configuration from
> ReplicationPeerConfig after it is being set.
> This can come in handy at multiple instances when we want to remove a
> particular configuration when it's not needed anymore or it is causing
> unexpected behavior. For eg in case there is a need to remove a walFilter
> (hbase.replication.source.custom.walentryfilters) configuration.
> As part of this Jira we want to add that support.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
saintstack commented on a change in pull request #2570:
URL: https://github.com/apache/hbase/pull/2570#discussion_r508765010
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaChooser.java
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * There are two modes with meta replica support.
+ * HighAvailable- Client sends requests to the primary meta region
first, within a
+ * configured amount of time, if there is no response
coming back,
+ * client sends requests to all replica regions and takes
the first
+ * response.
+ *
+ * LoadBalance - Client sends requests to meta replica regions in a
round-robin mode,
+ * if results from replica regions are stale, next time,
client sends requests for
+ * these stable locations to the primary meta region. In
this mode, scan
+ * requests are load balanced across all replica regions.
+ */
+enum MetaReplicaMode {
+ None,
+ HighAvailable,
+ LoadBalance
+}
+
+/**
+ * A Meta replica chooser decides which meta replica to go for scan requests.
+ */
+@InterfaceAudience.Private
+public interface MetaReplicaLoadBalanceReplicaChooser {
Review comment:
Is selector better than chooser?
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaChooser.java
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * There are two modes with meta replica support.
+ * HighAvailable- Client sends requests to the primary meta region
first, within a
+ * configured amount of time, if there is no response
coming back,
+ * client sends requests to all replica regions and takes
the first
+ * response.
+ *
+ * LoadBalance - Client sends requests to meta replica regions in a
round-robin mode,
+ * if results from replica regions are stale, next time,
client sends requests for
+ * these stable locations to the primary meta region. In
this mode, scan
+ * requests are load balanced across all replica regions.
+ */
+enum MetaReplicaMode {
+ None,
+ HighAvailable,
Review comment:
Does the enum have to be named 'MetaReplicaMode' Can it be named
ReplicaMode or ReadReplicaClientPolicy ?
When it is named for meta only, it implies this policy only works for meta
replica? Is this so?
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaReplicaLoadBalanceReplicaChooser.java
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License,
[jira] [Commented] (HBASE-25204) Nightly job failed as the name of jdk and maven changed
[
https://issues.apache.org/jira/browse/HBASE-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217866#comment-17217866
]
Hudson commented on HBASE-25204:
Results for branch master
[build #101 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Nightly job failed as the name of jdk and maven changed
>
>
> Key: HBASE-25204
> URL: https://issues.apache.org/jira/browse/HBASE-25204
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7
>
>
> See
> [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/85/console]
> [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/103/console]
>
> org.codehaus.groovy.control.MultipleCompilationErrorsException: startup
> failed: WorkflowScript: 508: Tool type "maven" does not have an install of
> "Maven (latest)" configured - did you mean "maven_latest"? @ line 508, column
> 19. maven 'Maven (latest)' ^ WorkflowScript: 510: Tool type "jdk" does not
> have an install of "JDK 1.8 (latest)" configured - did you mean
> "jdk_1.8_latest"? @ line 510, column 17. jdk "JDK 1.8 (latest)"
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25166) MobFileCompactionChore is closing the master's shared cluster connection
[
https://issues.apache.org/jira/browse/HBASE-25166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217865#comment-17217865
]
Hudson commented on HBASE-25166:
Results for branch master
[build #101 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MobFileCompactionChore is closing the master's shared cluster connection
>
>
> Key: HBASE-25166
> URL: https://issues.apache.org/jira/browse/HBASE-25166
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 3.0.0-alpha-1
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Code which is doing so in MobFileCompactionChore
> {code:java}
> try (Connection conn = master.getConnection();
> Admin admin = conn.getAdmin();) { {code}
> As master uses this connection to read the meta or other system tables, so
> none of the meta operations through master will work.
> Symptoms in master logs:-
> {code:java}
> s, events=841, succcessCount=123, totalEvents=12824192,
> totalSuccessCount=1891300
> 2020-10-05 16:34:25,062 INFO
> org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor: Unable
> to get remote Address
> 2020-10-05 16:34:25,062 ERROR
> org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to
> normalize regions.
> java.io.IOException: connection is closed
> at
> org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:797)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:768)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:727)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:215)
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189)
> at
> org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1821)
> at
> org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:188)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
> Symptoms at the client:-
> {code:java}
> RpcRetryingCaller{globalStartTime=1602099132430, pause=100, maxAttempts=11},
> java.io.IOException: java.io.IOException: connection is closed
> at
> org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1116)
> at
> org.apache.hadoop.hbase.master.TableStateManager.readMetaState(TableStateManager.java:258)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTablePresent(TableStateManager.java:175)
> at
> org.apache.hadoop.hbase.master.HMaster.getTableDescriptors(HMaster.java:3277)
> at
> org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:3221)
>
[jira] [Commented] (HBASE-25194) Do not publish workspace in flaky find job
[
https://issues.apache.org/jira/browse/HBASE-25194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217864#comment-17217864
]
Hudson commented on HBASE-25194:
Results for branch master
[build #101 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Do not publish workspace in flaky find job
> --
>
> Key: HBASE-25194
> URL: https://issues.apache.org/jira/browse/HBASE-25194
> Project: HBase
> Issue Type: Sub-task
> Components: jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7
>
>
> As said in the parent issue.
> And I tried to use the publishHTML target on jenkins job configuration page,
> the help message for reportDir is
> {noformat}
> The path to the HTML report directory relative to the workspace.
> {noformat}
> For the reportFiles is
> {noformat}
> The file(s) to provide links inside the report directory
> {noformat}
> I think this clearly means that the plugin will publish all the files under
> reportDir, the reportFiles is just the index page.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25203) Change the reference url to flaky list in our jenkins jobs
[
https://issues.apache.org/jira/browse/HBASE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217867#comment-17217867
]
Hudson commented on HBASE-25203:
Results for branch master
[build #101 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/101/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the reference url to flaky list in our jenkins jobs
> --
>
> Key: HBASE-25203
> URL: https://issues.apache.org/jira/browse/HBASE-25203
> Project: HBase
> Issue Type: Sub-task
> Components: flakies, jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7
>
>
> Nightly
> Flaky tests
> PreCommit
> personality
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25209) Add CLI support to remove a particular configuration from ReplicationPeerConfig.
Ankit Jain created HBASE-25209:
--
Summary: Add CLI support to remove a particular configuration from
ReplicationPeerConfig.
Key: HBASE-25209
URL: https://issues.apache.org/jira/browse/HBASE-25209
Project: HBase
Issue Type: Improvement
Components: Replication, shell
Environment: {code:java}
// code placeholder
{code}
Reporter: Ankit Jain
Assignee: Ankit Jain
Currently, there is no easy way to remove a configuration from
ReplicationPeerConfig after it is being set.
This can come in handy at multiple instances when we want to remove a
particular configuration when it's not needed anymore or it is causing
unexpected behavior. For eg in case there is a need to remove a walFilter
(hbase.replication.source.custom.walentryfilters) configuration.
As part of this Jira we want to add that support.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-713046815 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 51s | master passed | | +1 :green_heart: | checkstyle | 1m 31s | master passed | | +1 :green_heart: | spotbugs | 2m 43s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | -0 :warning: | checkstyle | 0m 14s | hbase-hadoop-compat: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | -0 :warning: | checkstyle | 1m 24s | hbase-server: The patch generated 8 new + 272 unchanged - 0 fixed = 280 total (was 272) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 19m 1s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. | | | | 45m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 95cc8ed21f09 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/artifact/yetus-general-check/output/diff-checkstyle-hbase-hadoop-compat.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/11/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pankaj72981 commented on pull request #2537: HBASE-24977 Meta table shouldn't be modified as read only
pankaj72981 commented on pull request #2537: URL: https://github.com/apache/hbase/pull/2537#issuecomment-713036605 @saintstack Sir, Do you know why continuous-integration/jenkins/pr-merge can't be build? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #2419: HBASE-25050 - We initialize Filesystems more than once.
busbey commented on a change in pull request #2419: URL: https://github.com/apache/hbase/pull/2419#discussion_r508709843 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java ## @@ -82,7 +82,10 @@ public HFileSystem(Configuration conf, boolean useHBaseChecksum) // Create the default filesystem with checksum verification switched on. // By default, any operation to this FilterFileSystem occurs on // the underlying filesystem that has checksums switched on. -this.fs = FileSystem.get(conf); +// This FS#get(URI, conf) clearly indicates in the javadoc that if the FS is +// not created it will initialize the FS and return that created FS. If it is +// already created it will just return the FS that was already created. Review comment: To Duo's request, add in something like: ``` // We take pains to funnel all of our FileSystem instantiation through this call to ensure // we never need to call FS.initialize ourself so that we do not have to track any state to // avoid calling initialize more than once. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
Apache-HBase commented on pull request #2570: URL: https://github.com/apache/hbase/pull/2570#issuecomment-713008034 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-18070 Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 1s | HBASE-18070 passed | | +1 :green_heart: | checkstyle | 2m 3s | HBASE-18070 passed | | +1 :green_heart: | spotbugs | 3m 47s | HBASE-18070 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | -0 :warning: | checkstyle | 0m 27s | hbase-client: The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 24s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | -1 :x: | spotbugs | 1m 12s | hbase-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 42m 52s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-client | | | Should org.apache.hadoop.hbase.client.MetaReplicaLoadBalanceReplicaSimpleChooser$StaleTableCache be a _static_ inner class? At MetaReplicaLoadBalanceReplicaSimpleChooser.java:inner class? At MetaReplicaLoadBalanceReplicaSimpleChooser.java:[lines 111-112] | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2570 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux b51aff078fd9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-18070 / 4e47554af8 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/artifact/yetus-general-check/output/new-spotbugs-hbase-client.html | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2570/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-25158) Enhance balancer to make sure no meta primary/replica regions are going to be assigned to one same region server.
[ https://issues.apache.org/jira/browse/HBASE-25158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun reassigned HBASE-25158: Assignee: Huaxiang Sun > Enhance balancer to make sure no meta primary/replica regions are going to be > assigned to one same region server. > - > > Key: HBASE-25158 > URL: https://issues.apache.org/jira/browse/HBASE-25158 > Project: HBase > Issue Type: Sub-task >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Minor > > Region replica has enhancement in balancer that primary region and its > replicas are not going to be assigned to the same region server. Today, there > is only one meta region, so this enhancement is still enough. With split meta > coming in, it needs to make sure that no meta regoin/replicas is going to be > assigned to the same region server in order to avoid hotspot issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25126) Add load balance logic in hbase-client to distribute read load over meta replica regions.
[ https://issues.apache.org/jira/browse/HBASE-25126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217749#comment-17217749 ] Huaxiang Sun commented on HBASE-25126: -- a patch is posted for review. > Add load balance logic in hbase-client to distribute read load over meta > replica regions. > - > > Key: HBASE-25126 > URL: https://issues.apache.org/jira/browse/HBASE-25126 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-1 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-25126) Add load balance logic in hbase-client to distribute read load over meta replica regions.
[ https://issues.apache.org/jira/browse/HBASE-25126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-25126 started by Huaxiang Sun. > Add load balance logic in hbase-client to distribute read load over meta > replica regions. > - > > Key: HBASE-25126 > URL: https://issues.apache.org/jira/browse/HBASE-25126 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-1 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25208) Running an hbase-shell command in non-interactive mode should not complain about missing files when it fails to talk to the cluster.
Sean Busbey created HBASE-25208:
---
Summary: Running an hbase-shell command in non-interactive mode
should not complain about missing files when it fails to talk to the cluster.
Key: HBASE-25208
URL: https://issues.apache.org/jira/browse/HBASE-25208
Project: HBase
Issue Type: Bug
Components: shell
Reporter: Sean Busbey
if you script out some shell commands, then run them via the shell in
non-interactive mode we sometimes give an erroneous "I couldn't find that file"
response when the master(s) are in a bad state instead of revealing the
underlying cluster issue.
e.g.
{code}
tmpfile=$(mktemp)
echo "balance_switch false" > $tmpfile
echo "exit" >> $tmpfile
$HBASE_BIN --config $CONF_DIR shell -n $tmpfile
RET=$?
rm $tmpfile
exit $RET
{code}
as an example, with a master stuck trying to initialize as active the failure
might look like
{code}
++ mktemp
+ tmpfile=/tmp/tmp.dMHTj0nVXx
+ echo balance_switch true
+ echo exit
+ bin/hbase --config /etc/hbase/conf shell -n /tmp/tmp.dMHTj0nVXx
LoadError: no such file to load -- /tmp/tmp.dMHTj0nVXx
load at org/jruby/RubyKernel.java:974
at /some/path/to/lib/hbase/bin/hirb.rb:186
+ RET=1
+ rm /tmp/tmp.dMHTj0nVXx
+ exit 1
{code}
which is very confusing. the shell should report something related to an issue
talking to the cluster rather than a problem with the passed script.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] huaxiangsun opened a new pull request #2570: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun opened a new pull request #2570: URL: https://github.com/apache/hbase/pull/2570 … load over meta replica regions It adds load balance support. With "hbase.meta.replicas.use" set to true, client support meta replica feature. The existing mode is called HighAvailable, client sends scan request to the primary meta replica region first, if response is not back within a configured amount of time, it will send scan requests to all meta replica regions and take the first response. On top of the existing mode, a new mode LoadBalance is added. In this mode, client first choose a meta replica region to send scan request. If the response is stale, it will send the scan request to the primary meta region. In this mode, all meta replica regions serve scan requests. Two new config knobs are added: 1. hbase.meta.replicas.mode Valid value is "HighAvailable" and "LoadBalance". 2. hbase.meta.replicas.mode.loadbalance.replica.chooser Implementation class for MetaReplicaChooser. Only org.apache.hadoop.hbase.client.MetaReplicaLoadBalanceReplicaSimpleChooser.class is supported for now, which is the default as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2569: HBASE-25206 Data loss can happen if a cloned table loses original spl…
Apache-HBase commented on pull request #2569: URL: https://github.com/apache/hbase/pull/2569#issuecomment-712971925 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 0m 59s | master passed | | +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 6m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 142m 25s | hbase-server in the patch passed. | | | | 169m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2569/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2569 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 763a1077a7fa 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2569/1/testReport/ | | Max. process+thread count | 5046 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2569/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2569: HBASE-25206 Data loss can happen if a cloned table loses original spl…
Apache-HBase commented on pull request #2569: URL: https://github.com/apache/hbase/pull/2569#issuecomment-712968715 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 0s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 6m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 6m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 139m 38s | hbase-server in the patch failed. | | | | 168m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2569/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2569 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dc4fdd31df8a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2569/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2569/1/testReport/ | | Max. process+thread count | 3997 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2569/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25200) Try enlarge the flaky test timeout for branch-2.2
[ https://issues.apache.org/jira/browse/HBASE-25200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217685#comment-17217685 ] Duo Zhang commented on HBASE-25200: --- https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//#showFailuresLink Seems we do have a lot of failures? Let's fix them ASAP? [~zghao] Thanks. > Try enlarge the flaky test timeout for branch-2.2 > - > > Key: HBASE-25200 > URL: https://issues.apache.org/jira/browse/HBASE-25200 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > > Now there are too many flaky tests to run. And the flaky test job cannot > finished in 2 hours. Then these tests will be marked flaky again. > > See > [https://ci-hadoop.apache.org/job/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/124/artifact/dashboard.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25204) Nightly job failed as the name of jdk and maven changed
[
https://issues.apache.org/jira/browse/HBASE-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217684#comment-17217684
]
Hudson commented on HBASE-25204:
Results for branch branch-2.2
[build #104 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Nightly job failed as the name of jdk and maven changed
>
>
> Key: HBASE-25204
> URL: https://issues.apache.org/jira/browse/HBASE-25204
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7
>
>
> See
> [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/85/console]
> [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/103/console]
>
> org.codehaus.groovy.control.MultipleCompilationErrorsException: startup
> failed: WorkflowScript: 508: Tool type "maven" does not have an install of
> "Maven (latest)" configured - did you mean "maven_latest"? @ line 508, column
> 19. maven 'Maven (latest)' ^ WorkflowScript: 510: Tool type "jdk" does not
> have an install of "JDK 1.8 (latest)" configured - did you mean
> "jdk_1.8_latest"? @ line 510, column 17. jdk "JDK 1.8 (latest)"
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25093) the RSGroupBasedLoadBalancer#retainAssignment throws NPE
[
https://issues.apache.org/jira/browse/HBASE-25093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217681#comment-17217681
]
Hudson commented on HBASE-25093:
Results for branch branch-2.2
[build #104 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> the RSGroupBasedLoadBalancer#retainAssignment throws NPE
>
>
> Key: HBASE-25093
> URL: https://issues.apache.org/jira/browse/HBASE-25093
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.2.3
>Reporter: Bo Cui
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.3, 2.4.0, 2.2.7
>
>
> when BaseLoadBalancer#
> https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java#L1433
> If the result of the BaseLoadBalancer#retainAssignment is null, the
> RSGroupBasedLoadBalancer#retainAssignment will throw NPE.
> https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L206
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25203) Change the reference url to flaky list in our jenkins jobs
[
https://issues.apache.org/jira/browse/HBASE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217683#comment-17217683
]
Hudson commented on HBASE-25203:
Results for branch branch-2.2
[build #104 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the reference url to flaky list in our jenkins jobs
> --
>
> Key: HBASE-25203
> URL: https://issues.apache.org/jira/browse/HBASE-25203
> Project: HBase
> Issue Type: Sub-task
> Components: flakies, jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7
>
>
> Nightly
> Flaky tests
> PreCommit
> personality
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25194) Do not publish workspace in flaky find job
[
https://issues.apache.org/jira/browse/HBASE-25194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217682#comment-17217682
]
Hudson commented on HBASE-25194:
Results for branch branch-2.2
[build #104 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/104//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Do not publish workspace in flaky find job
> --
>
> Key: HBASE-25194
> URL: https://issues.apache.org/jira/browse/HBASE-25194
> Project: HBase
> Issue Type: Sub-task
> Components: jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7
>
>
> As said in the parent issue.
> And I tried to use the publishHTML target on jenkins job configuration page,
> the help message for reportDir is
> {noformat}
> The path to the HTML report directory relative to the workspace.
> {noformat}
> For the reportFiles is
> {noformat}
> The file(s) to provide links inside the report directory
> {noformat}
> I think this clearly means that the plugin will publish all the files under
> reportDir, the reportFiles is just the index page.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25207) Revisit the implementation and usage of RegionStates.include
Duo Zhang created HBASE-25207: - Summary: Revisit the implementation and usage of RegionStates.include Key: HBASE-25207 URL: https://issues.apache.org/jira/browse/HBASE-25207 Project: HBase Issue Type: Bug Reporter: Duo Zhang After several round of refactoring and fixing, the method has been used in lots of places and the implementation looks really confusing. As in the first if condition for testing RegionStateNode and RegionInfo state, we will always return false when split is true, which means we will always filter out split parent, as a split parent, is split = true and also offline = true. I think the reason why there is no problem is that, only in EnableTableProcedure we call this method with offline = true, and EnableTableProcedure does not need to deal with split parent... And now since we found a problem in HBASE-25206, where we need to get split parent when deleting a table, I think it is time to revisit this method and make logic less confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #2569: HBASE-25206 Data loss can happen if a cloned table loses original spl…
Apache9 commented on pull request #2569: URL: https://github.com/apache/hbase/pull/2569#issuecomment-712919691 Oh it is in HBASE-14614, where the patch is several MB, which means we can not know the actual reason on why the code is like this... If you do not mind, I plan to open an issue first to address the problem of the include method, and then we could come back to fix the bug here. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2569: HBASE-25206 Data loss can happen if a cloned table loses original spl…
Apache9 commented on pull request #2569: URL: https://github.com/apache/hbase/pull/2569#issuecomment-712916667 Oh the logic is already there before HBASE-23187. In HBASE-23187 we just add checks for RegionInfo. Let me check for earlier changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2541: HBASE-25173 Remove owner related methods in TableDescriptor/TableDescriptorBuilder
Apache-HBase commented on pull request #2541: URL: https://github.com/apache/hbase/pull/2541#issuecomment-712914852 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +1 :green_heart: | compile | 2m 6s | master passed | | +1 :green_heart: | shadedjars | 7m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 24s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | +1 :green_heart: | compile | 2m 5s | the patch passed | | +1 :green_heart: | javac | 2m 5s | the patch passed | | +1 :green_heart: | shadedjars | 7m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 12s | hbase-client in the patch passed. | | -1 :x: | unit | 211m 36s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 9m 6s | hbase-shell in the patch passed. | | +1 :green_heart: | unit | 3m 24s | hbase-endpoint in the patch passed. | | | | 263m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2541 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7225079cde88 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/testReport/ | | Max. process+thread count | 3846 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server hbase-shell hbase-endpoint U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2569: HBASE-25206 Data loss can happen if a cloned table loses original spl…
Apache9 commented on pull request #2569: URL: https://github.com/apache/hbase/pull/2569#issuecomment-712914659 They are added by this issue. https://issues.apache.org/jira/browse/HBASE-23187 Let me take a look on the PR about why we add these checks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
wchevreuil commented on a change in pull request #2568:
URL: https://github.com/apache/hbase/pull/2568#discussion_r508578218
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -5301,7 +5301,7 @@ private long replayRecoveredEditsForPaths(long
minSeqIdForTheRegion, FileSystem
long maxSeqId;
String fileName = edits.getName();
- maxSeqId = Math.abs(Long.parseLong(fileName));
+ maxSeqId = Math.abs(Long.parseLong(fileName.split("\\.")[0]));
Review comment:
For files already with the `corrupted` suffix, shouldn't we just skip
the whole replay attempt from this point onwards? We already know it will not
work, so we can save some computation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2569: HBASE-25206 Data loss can happen if a cloned table loses original spl…
Apache9 commented on a change in pull request #2569:
URL: https://github.com/apache/hbase/pull/2569#discussion_r508577459
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
##
@@ -100,6 +101,12 @@ protected Flow executeFromState(final MasterProcedureEnv
env, DeleteTableState s
// TODO: Move out... in the acquireLock()
LOG.debug("Waiting for RIT for {}", this);
regions =
env.getAssignmentManager().getRegionStates().getRegionsOfTable(getTableName());
+ // We need to archive SPLIT regions as well, so add the SPLIT
regions to the region list
+ List splitRegions =
env.getAssignmentManager().getRegionStates()
Review comment:
I think here we could just get all the regions of the table.
And look at the code of getRegionsOfTable(TableName, boolean), I'm afraid
the implementation of the include method has something wrong?
```
if (node.isInState(State.SPLIT) || hri.isSplit()) {
return false;
}
if ((node.isInState(State.OFFLINE) || hri.isOffline()) && !offline) {
return false;
}
return (!hri.isOffline() && !hri.isSplit()) ||
((hri.isOffline() || hri.isSplit()) && offline);
```
I do not think we need the two extra if conditions? Especially the first
one, it will make us always exclude split parent?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2541: HBASE-25173 Remove owner related methods in TableDescriptor/TableDescriptorBuilder
Apache-HBase commented on pull request #2541: URL: https://github.com/apache/hbase/pull/2541#issuecomment-712901977 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 36s | master passed | | +1 :green_heart: | compile | 2m 28s | master passed | | +1 :green_heart: | shadedjars | 7m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 31s | the patch passed | | +1 :green_heart: | compile | 2m 23s | the patch passed | | +1 :green_heart: | javac | 2m 23s | the patch passed | | +1 :green_heart: | shadedjars | 7m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 16s | hbase-client in the patch passed. | | -1 :x: | unit | 196m 32s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 7m 3s | hbase-shell in the patch passed. | | +1 :green_heart: | unit | 3m 21s | hbase-endpoint in the patch passed. | | | | 245m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2541 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 58d96ecfcf68 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/testReport/ | | Max. process+thread count | 3344 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server hbase-shell hbase-endpoint U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2565: HBASE-25003 Backport HBASE-24350 and HBASE-24779 to branch-2.2
Apache-HBase commented on pull request #2565: URL: https://github.com/apache/hbase/pull/2565#issuecomment-712897746 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 50s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-2.2 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 35s | branch-2.2 passed | | +1 :green_heart: | compile | 1m 30s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 1m 51s | branch-2.2 passed | | +1 :green_heart: | shadedjars | 4m 27s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | branch-2.2 passed | | +0 :ok: | spotbugs | 3m 38s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 32s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 13s | the patch passed | | +1 :green_heart: | compile | 1m 31s | the patch passed | | +1 :green_heart: | javac | 1m 31s | the patch passed | | +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-hadoop-compat | | +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-hadoop2-compat | | +1 :green_heart: | checkstyle | 1m 29s | hbase-server: The patch generated 0 new + 9 unchanged - 1 fixed = 9 total (was 10) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 23s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 27m 45s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | javadoc | 1m 5s | the patch passed | | +1 :green_heart: | findbugs | 4m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 33s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 0m 38s | hbase-hadoop2-compat in the patch passed. | | +1 :green_heart: | unit | 269m 25s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 1m 16s | The patch does not generate ASF License warnings. | | | | 345m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2565/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2565 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 0042a63064e9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2565/out/precommit/personality/provided.sh | | git revision | branch-2.2 / 24265c56fe | | Default Java | 1.8.0_181 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2565/2/testReport/ | | Max. process+thread count | 2674 (vs. ulimit of 12500) | | modules | C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2565/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2498: HBASE-24996 Support CheckAndMutate in Region.batchMutate()
Apache9 commented on pull request #2498: URL: https://github.com/apache/hbase/pull/2498#issuecomment-712891186 Sorry for the late reply... This is a big change, need to find a suitable large block of time to review it... Will do this soon this week. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2569: HBASE-25206 Data loss can happen if a cloned table loses original spl…
Apache-HBase commented on pull request #2569: URL: https://github.com/apache/hbase/pull/2569#issuecomment-712882044 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 8s | master passed | | +1 :green_heart: | checkstyle | 1m 13s | master passed | | +1 :green_heart: | spotbugs | 2m 11s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 54s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 42m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2569/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2569 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 5c42cd868e96 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2569/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25206) Data loss can happen if a cloned table loses original split region(delete table)
[
https://issues.apache.org/jira/browse/HBASE-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-25206:
-
Description:
Steps to reproduce are as follows:
1. Create a table and put some data into the table:
{code:java}
create 'test1','cf'
put 'test1','r1','cf','v1'
put 'test1','r2','cf','v2'
put 'test1','r3','cf','v3'
put 'test1','r4','cf','v4'
put 'test1','r5','cf','v5'
{code}
2. Take a snapshot for the table:
{code:java}
snapshot 'test1','snap_test'
{code}
3. Clone the snapshot to another table
{code:java}
clone_snapshot 'snap_test','test2'
{code}
4. Delete the snapshot
{code:java}
delete_snapshot 'snap_test'
{code}
5. Split the original table
{code:java}
split 'test1','r3'
{code}
6. Drop the original table
{code:java}
disable 'test1'
drop 'test1'
{code}
After that, we see the error like the following in RS log when opening the
regions of the cloned table:
{code:java}
2020-10-20 13:32:18,415 WARN org.apache.hadoop.hbase.regionserver.HRegion:
Failed initialize of region=
test2,,1603200595702.bebdc4f740626206eeccad96b7643261., starting to roll back
memstore
java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable
to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89,
hdfs://:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89,
hdfs://:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89,
hdfs://:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1095)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:943)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:899)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7246)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7204)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7176)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7134)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7085)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:283)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.FileNotFoundException: Unable to open
link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89,
hdfs://:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89,
hdfs://:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89,
hdfs://:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
at
org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:590)
at
org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:557)
at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:303)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5731)
at
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1059)
at
org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1056)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.io.FileNotFoundException: Unable to open link:
org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://:8020/hbase/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89,
hdfs://:8020/hbase/.tmp/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89,
hdfs://:8020/hbase/mobdir/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89,
hdfs://:8020/hbase/archive/data/default/test1/349b766b1b38e21f627ed4e441ae643c/cf/b6e39865710345c8998dec0bcc94cc89]
at
[GitHub] [hbase] brfrn169 opened a new pull request #2569: HBASE-25206 Data loss can happen if a cloned table loses original spl…
brfrn169 opened a new pull request #2569: URL: https://github.com/apache/hbase/pull/2569 …it region(delete table) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2537: HBASE-24977 Meta table shouldn't be modified as read only
Apache-HBase commented on pull request #2537: URL: https://github.com/apache/hbase/pull/2537#issuecomment-712844065 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 1s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 48s | master passed | | +1 :green_heart: | compile | 0m 55s | master passed | | +1 :green_heart: | shadedjars | 6m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 6m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 148m 30s | hbase-server in the patch passed. | | | | 174m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2537 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9039e113cb80 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/4/testReport/ | | Max. process+thread count | 4202 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25206) Data loss can happen if a cloned table loses original split region(delete table)
Toshihiro Suzuki created HBASE-25206:
Summary: Data loss can happen if a cloned table loses original
split region(delete table)
Key: HBASE-25206
URL: https://issues.apache.org/jira/browse/HBASE-25206
Project: HBase
Issue Type: Bug
Reporter: Toshihiro Suzuki
Assignee: Toshihiro Suzuki
Steps to reproduce are as follows:
1. Create a table and put some data into the table:
{code:java}
create 'test1','cf'
put 'test1','r1','cf','v1'
put 'test1','r2','cf','v2'
put 'test1','r3','cf','v3'
put 'test1','r4','cf','v4'
put 'test1','r5','cf','v5'
{code}
2. Take a snapshot for the table:
{code:java}
snapshot 'test1','snap_test'
{code}
3. Clone the snapshot to another table
{code:java}
clone_snapshot 'snap_test','test2'
{code}
4. Delete the snapshot
{code:java}
delete_snapshot 'snap_test'
{code}
5. Split the original table
{code:java}
split 'test1','r3'
{code}
6. Drop the original table
{code:java}
disable 'test1'
drop 'test1'
{code}
After that, we see the error like the following in RS log when opening the
regions of the cloned table:
{code:java}
2020-10-20 22:15:47,554 WARN [RS_OPEN_REGION-regionserver/10.0.1.8:0-0]
regionserver.HRegion(965): Failed initialize of region=
testCloneSnapshotBeforeSplittingRegionAndDroppingTable_0__regionReplication_1_-1603199739880,,1603199732706.92f431fab12aaded92a23513901daa5a.,
starting to roll back memstore
java.io.IOException: java.io.IOException: java.io.FileNotFoundException:
HFileLink
locations=[hdfs://localhost:62716/user/tsuzuki/test-data/c00e6c6b-1c3b-5e40-4227-831ae42cf2f4/data/default/testCloneSnapshotBeforeSplittingRegionAndDroppingTable_0__regionReplication_1_1603199732705/f4658c2b6fb129d95f62e63d3742177d/cf/719b64120a0f4394ae7af8926bc56402,
hdfs://localhost:62716/user/tsuzuki/test-data/c00e6c6b-1c3b-5e40-4227-831ae42cf2f4/.tmp/data/default/testCloneSnapshotBeforeSplittingRegionAndDroppingTable_0__regionReplication_1_1603199732705/f4658c2b6fb129d95f62e63d3742177d/cf/719b64120a0f4394ae7af8926bc56402,
hdfs://localhost:62716/user/tsuzuki/test-data/c00e6c6b-1c3b-5e40-4227-831ae42cf2f4/mobdir/data/default/testCloneSnapshotBeforeSplittingRegionAndDroppingTable_0__regionReplication_1_1603199732705/f4658c2b6fb129d95f62e63d3742177d/cf/719b64120a0f4394ae7af8926bc56402,
hdfs://localhost:62716/user/tsuzuki/test-data/c00e6c6b-1c3b-5e40-4227-831ae42cf2f4/archive/data/default/testCloneSnapshotBeforeSplittingRegionAndDroppingTable_0__regionReplication_1_1603199732705/f4658c2b6fb129d95f62e63d3742177d/cf/719b64120a0f4394ae7af8926bc56402]
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1179)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1121)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1011)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:962)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7999)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7955)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7930)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7888)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7839)
at
org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:132)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.FileNotFoundException: HFileLink
locations=[hdfs://localhost:62716/user/tsuzuki/test-data/c00e6c6b-1c3b-5e40-4227-831ae42cf2f4/data/default/testCloneSnapshotBeforeSplittingRegionAndDroppingTable_0__regionReplication_1_1603199732705/f4658c2b6fb129d95f62e63d3742177d/cf/719b64120a0f4394ae7af8926bc56402,
hdfs://localhost:62716/user/tsuzuki/test-data/c00e6c6b-1c3b-5e40-4227-831ae42cf2f4/.tmp/data/default/testCloneSnapshotBeforeSplittingRegionAndDroppingTable_0__regionReplication_1_1603199732705/f4658c2b6fb129d95f62e63d3742177d/cf/719b64120a0f4394ae7af8926bc56402,
hdfs://localhost:62716/user/tsuzuki/test-data/c00e6c6b-1c3b-5e40-4227-831ae42cf2f4/mobdir/data/default/testCloneSnapshotBeforeSplittingRegionAndDroppingTable_0__regionReplication_1_1603199732705/f4658c2b6fb129d95f62e63d3742177d/cf/719b64120a0f4394ae7af8926bc56402,
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Apache-HBase commented on pull request #2568: URL: https://github.com/apache/hbase/pull/2568#issuecomment-712829053 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 0s | master passed | | +1 :green_heart: | compile | 1m 19s | master passed | | +1 :green_heart: | shadedjars | 7m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 55s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 7m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 54s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 208m 39s | hbase-server in the patch passed. | | | | 241m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2568 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f8dec23eac1b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7d00e43dce | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/testReport/ | | Max. process+thread count | 3476 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Apache-HBase commented on pull request #2568: URL: https://github.com/apache/hbase/pull/2568#issuecomment-712796440 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 24s | master passed | | +1 :green_heart: | compile | 1m 32s | master passed | | +1 :green_heart: | shadedjars | 7m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 7s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 48s | the patch passed | | +1 :green_heart: | compile | 1m 41s | the patch passed | | +1 :green_heart: | javac | 1m 41s | the patch passed | | +1 :green_heart: | shadedjars | 8m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 50s | hbase-common in the patch passed. | | -1 :x: | unit | 143m 9s | hbase-server in the patch failed. | | | | 179m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2568 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 949cb680364e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7d00e43dce | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/testReport/ | | Max. process+thread count | 3820 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2568/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2541: HBASE-25173 Remove owner related methods in TableDescriptor/TableDescriptorBuilder
Apache-HBase commented on pull request #2541: URL: https://github.com/apache/hbase/pull/2541#issuecomment-712780072 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 33s | master passed | | +1 :green_heart: | checkstyle | 2m 0s | master passed | | +1 :green_heart: | spotbugs | 3m 27s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | checkstyle | 0m 27s | The patch passed checkstyle in hbase-client | | +1 :green_heart: | checkstyle | 1m 10s | hbase-server: The patch generated 0 new + 188 unchanged - 1 fixed = 188 total (was 189) | | +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-shell | | +1 :green_heart: | checkstyle | 0m 13s | The patch passed checkstyle in hbase-endpoint | | -0 :warning: | rubocop | 0m 19s | The patch generated 1 new + 562 unchanged - 16 fixed = 563 total (was 578) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 2s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | | | | 46m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2541 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle rubocop | | uname | Linux 3c7032803505 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | rubocop | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server hbase-shell hbase-endpoint U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2541/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2537: HBASE-24977 Meta table shouldn't be modified as read only
Apache-HBase commented on pull request #2537: URL: https://github.com/apache/hbase/pull/2537#issuecomment-712776166 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 30s | master passed | | +1 :green_heart: | compile | 1m 26s | master passed | | +1 :green_heart: | shadedjars | 8m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 15s | the patch passed | | +1 :green_heart: | compile | 1m 30s | the patch passed | | +1 :green_heart: | javac | 1m 30s | the patch passed | | +1 :green_heart: | shadedjars | 8m 26s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 46s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 12m 54s | hbase-server in the patch failed. | | | | 47m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2537 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 54e58f8f2df8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/4/testReport/ | | Max. process+thread count | 615 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2537: HBASE-24977 Meta table shouldn't be modified as read only
Apache-HBase commented on pull request #2537: URL: https://github.com/apache/hbase/pull/2537#issuecomment-712772630 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | master passed | | +1 :green_heart: | checkstyle | 1m 5s | master passed | | +1 :green_heart: | spotbugs | 2m 11s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | checkstyle | 1m 3s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 40m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2537 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 7021f027638b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 200343c59c | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #2523: HBASE-25167 Normalizer support for hot config reloading
virajjasani commented on a change in pull request #2523:
URL: https://github.com/apache/hbase/pull/2523#discussion_r508380812
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -140,39 +188,76 @@ private static int parseMergeMinRegionSizeMb(final
Configuration conf) {
key, parsedValue, settledValue);
}
+ private static void logConfigurationUpdated(final String key, final T
oldValue,
+final T newValue) {
+if (!Objects.equals(oldValue, newValue)) {
+ LOG.info("Updated configuration for key '{}' from {} to {}", key,
oldValue, newValue);
+}
+ }
+
/**
* Return this instance's configured value for {@value #SPLIT_ENABLED_KEY}.
*/
public boolean isSplitEnabled() {
-return splitEnabled;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return splitEnabled;
+} finally {
+ readLock.unlock();
+}
}
/**
* Return this instance's configured value for {@value #MERGE_ENABLED_KEY}.
*/
public boolean isMergeEnabled() {
-return mergeEnabled;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return mergeEnabled;
+} finally {
+ readLock.unlock();
+}
}
/**
* Return this instance's configured value for {@value
#MIN_REGION_COUNT_KEY}.
*/
public int getMinRegionCount() {
-return minRegionCount;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return minRegionCount;
+} finally {
+ readLock.unlock();
+}
}
/**
* Return this instance's configured value for {@value
#MERGE_MIN_REGION_AGE_DAYS_KEY}.
*/
public Period getMergeMinRegionAge() {
-return mergeMinRegionAge;
+final Lock readLock = configUpdateLock.readLock();
+readLock.lock();
+try {
+ return mergeMinRegionAge;
+} finally {
+ readLock.unlock();
+}
}
/**
* Return this instance's configured value for {@value
#MERGE_MIN_REGION_SIZE_MB_KEY}.
*/
public int getMergeMinRegionSizeMb() {
Review comment:
Last 3 getters should have `@VisibleForTesting` ?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -71,38 +76,81 @@
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
- private Configuration conf;
private MasterServices masterServices;
- private boolean splitEnabled;
- private boolean mergeEnabled;
- private int minRegionCount;
- private Period mergeMinRegionAge;
- private int mergeMinRegionSizeMb;
+
+ /** Ensure configuration changes are applied atomically. */
+ private final ReadWriteLock configUpdateLock = new ReentrantReadWriteLock();
+ @GuardedBy("configUpdateLock") private Configuration conf;
Review comment:
Indeed ! Wondering if we can ensure all implementors of
`ConfigurationObserver` can start using such lock for atomic updates of
non-final fields (of course not as part of this Jira :) )
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2537: HBASE-24977 Meta table shouldn't be modified as read only
Apache-HBase commented on pull request #2537: URL: https://github.com/apache/hbase/pull/2537#issuecomment-712751518 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 0m 55s | master passed | | +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 154m 35s | hbase-server in the patch failed. | | | | 181m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2537 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux af78d4e21f71 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7d00e43dce | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/3/testReport/ | | Max. process+thread count | 4451 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25203) Change the reference url to flaky list in our jenkins jobs
[
https://issues.apache.org/jira/browse/HBASE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217485#comment-17217485
]
Hudson commented on HBASE-25203:
Results for branch branch-2
[build #79 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79//console].
> Change the reference url to flaky list in our jenkins jobs
> --
>
> Key: HBASE-25203
> URL: https://issues.apache.org/jira/browse/HBASE-25203
> Project: HBase
> Issue Type: Sub-task
> Components: flakies, jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7
>
>
> Nightly
> Flaky tests
> PreCommit
> personality
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25204) Nightly job failed as the name of jdk and maven changed
[
https://issues.apache.org/jira/browse/HBASE-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217486#comment-17217486
]
Hudson commented on HBASE-25204:
Results for branch branch-2
[build #79 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79//console].
> Nightly job failed as the name of jdk and maven changed
>
>
> Key: HBASE-25204
> URL: https://issues.apache.org/jira/browse/HBASE-25204
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7
>
>
> See
> [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/85/console]
> [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/103/console]
>
> org.codehaus.groovy.control.MultipleCompilationErrorsException: startup
> failed: WorkflowScript: 508: Tool type "maven" does not have an install of
> "Maven (latest)" configured - did you mean "maven_latest"? @ line 508, column
> 19. maven 'Maven (latest)' ^ WorkflowScript: 510: Tool type "jdk" does not
> have an install of "JDK 1.8 (latest)" configured - did you mean
> "jdk_1.8_latest"? @ line 510, column 17. jdk "JDK 1.8 (latest)"
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25194) Do not publish workspace in flaky find job
[
https://issues.apache.org/jira/browse/HBASE-25194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217484#comment-17217484
]
Hudson commented on HBASE-25194:
Results for branch branch-2
[build #79 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/79//console].
> Do not publish workspace in flaky find job
> --
>
> Key: HBASE-25194
> URL: https://issues.apache.org/jira/browse/HBASE-25194
> Project: HBase
> Issue Type: Sub-task
> Components: jenkins
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7
>
>
> As said in the parent issue.
> And I tried to use the publishHTML target on jenkins job configuration page,
> the help message for reportDir is
> {noformat}
> The path to the HTML report directory relative to the workspace.
> {noformat}
> For the reportFiles is
> {noformat}
> The file(s) to provide links inside the report directory
> {noformat}
> I think this clearly means that the plugin will publish all the files under
> reportDir, the reportFiles is just the index page.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25200) Try enlarge the flaky test timeout for branch-2.2
[ https://issues.apache.org/jira/browse/HBASE-25200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217481#comment-17217481 ] Duo Zhang commented on HBASE-25200: --- I've deleted the broken builds for both nightly and flaky, now the list is smaller. Let's see whether it works. > Try enlarge the flaky test timeout for branch-2.2 > - > > Key: HBASE-25200 > URL: https://issues.apache.org/jira/browse/HBASE-25200 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > > Now there are too many flaky tests to run. And the flaky test job cannot > finished in 2 hours. Then these tests will be marked flaky again. > > See > [https://ci-hadoop.apache.org/job/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/124/artifact/dashboard.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25204) Nightly job failed as the name of jdk and maven changed
[ https://issues.apache.org/jira/browse/HBASE-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-25204. Fix Version/s: 2.2.7 1.4.14 2.4.0 1.7.0 2.3.3 3.0.0-alpha-1 Resolution: Fixed Pushed to all active branchs. Thanks [~zhangduo] for reviewing. > Nightly job failed as the name of jdk and maven changed > > > Key: HBASE-25204 > URL: https://issues.apache.org/jira/browse/HBASE-25204 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7 > > > See > [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/85/console] > [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/103/console] > > org.codehaus.groovy.control.MultipleCompilationErrorsException: startup > failed: WorkflowScript: 508: Tool type "maven" does not have an install of > "Maven (latest)" configured - did you mean "maven_latest"? @ line 508, column > 19. maven 'Maven (latest)' ^ WorkflowScript: 510: Tool type "jdk" does not > have an install of "JDK 1.8 (latest)" configured - did you mean > "jdk_1.8_latest"? @ line 510, column 17. jdk "JDK 1.8 (latest)" > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25205) Corrupted hfiles append timestamp every time the region is trying to open
Junhong Xu created HBASE-25205:
--
Summary: Corrupted hfiles append timestamp every time the region
is trying to open
Key: HBASE-25205
URL: https://issues.apache.org/jira/browse/HBASE-25205
Project: HBase
Issue Type: Bug
Reporter: Junhong Xu
Assignee: Junhong Xu
When the RS crashed, we replay WALs to generate recover edits or HFile
directly. If the replaying WAL RS crashed again, the file just writing to may
be corrupted. In some cases, we may want to move on(e.g. in the case of sink to
hfile as we have WAL and replaying the WAL again is OK), and move the file with
extra timestamp as suffix.But if the region is opened again, the corrupted file
can't be opened, and renamed with an extra timestamp again.After some round
like this, the file name will be too long to rename.The log is like this:
{code:java}
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$PathComponentTooLongException):
The maximum path component name limit of 6537855
8b0444c27a9d21fb0f4e4293f.1602831270772.1602831291050.1602831296855.1602831408803.1602831493989.1602831584077.1602831600838.1602831659805.1602831736374.1602831738002.1
602831959867.1602831979707.1602832095288.1602832103908.1602832538224.1602833079431
in directory /hbase/XXX/data/default/IntegrationTestBigLinkedList/aa376ec
f026a5e63d0703384e34ec6aa/meta/recovered.hfiles is exceeded: limit=255
length=256
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxComponentLength(FSDirectory.java:1230)
at
org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.verifyFsLimitsForRename(FSDirRenameOp.java:98)
at
org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.unprotectedRenameTo(FSDirRenameOp.java:191)
at
org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.renameTo(FSDirRenameOp.java:493)
at
org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.renameToInt(FSDirRenameOp.java:62)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3080)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:1113)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:665)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:916)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:862)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1716)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2742)at
org.apache.hadoop.ipc.Client.call(Client.java:1504)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy17.rename(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.rename(ClientNamenodeProtocolTranslatorPB.java:504)
at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:249)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:107)
at com.sun.proxy.$Proxy18.rename(Unknown Source)
at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372)
at com.sun.proxy.$Proxy21.rename(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:1996)
at
org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:605)
at
org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:226)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] infraio merged pull request #2567: HBASE-25204 Nightly job failed as the name of jdk and maven changed
infraio merged pull request #2567: URL: https://github.com/apache/hbase/pull/2567 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24840) Avoid shows closing region task when create table
[ https://issues.apache.org/jira/browse/HBASE-24840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang resolved HBASE-24840. Fix Version/s: 3.0.0-alpha-1 Resolution: Fixed > Avoid shows closing region task when create table > - > > Key: HBASE-24840 > URL: https://issues.apache.org/jira/browse/HBASE-24840 > Project: HBase > Issue Type: Improvement > Components: monitoring >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-24840-afterpatch.png, HBASE-24840-beforepatch.png > > > Currently when we create region dir, we new a HRegion and then close it, this > step is before the real initialize, so there will exist a closing region task > at first, it is strange, and will confuse users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Joseph295 opened a new pull request #2568: HBASE-25205 Corrupted hfiles append timestamp every time the region i…
Joseph295 opened a new pull request #2568: URL: https://github.com/apache/hbase/pull/2568 …s trying to open This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25203) Change the reference url to flaky list in our jenkins jobs
[ https://issues.apache.org/jira/browse/HBASE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25203: -- Fix Version/s: 2.2.7 1.4.14 2.4.0 1.7.0 2.3.3 3.0.0-alpha-1 > Change the reference url to flaky list in our jenkins jobs > -- > > Key: HBASE-25203 > URL: https://issues.apache.org/jira/browse/HBASE-25203 > Project: HBase > Issue Type: Sub-task > Components: flakies, jenkins >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7 > > > Nightly > Flaky tests > PreCommit > personality -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25203) Change the reference url to flaky list in our jenkins jobs
[ https://issues.apache.org/jira/browse/HBASE-25203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25203. --- Hadoop Flags: Reviewed Resolution: Fixed Pushed to all active branches. Thanks [~zghao] for reviewing. > Change the reference url to flaky list in our jenkins jobs > -- > > Key: HBASE-25203 > URL: https://issues.apache.org/jira/browse/HBASE-25203 > Project: HBase > Issue Type: Sub-task > Components: flakies, jenkins >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 1.4.14, 2.2.7 > > > Nightly > Flaky tests > PreCommit > personality -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25204) Nightly job failed as the name of jdk and maven changed
[ https://issues.apache.org/jira/browse/HBASE-25204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-25204: -- Assignee: Guanghao Zhang > Nightly job failed as the name of jdk and maven changed > > > Key: HBASE-25204 > URL: https://issues.apache.org/jira/browse/HBASE-25204 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > > See > [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/85/console] > [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/103/console] > > org.codehaus.groovy.control.MultipleCompilationErrorsException: startup > failed: WorkflowScript: 508: Tool type "maven" does not have an install of > "Maven (latest)" configured - did you mean "maven_latest"? @ line 508, column > 19. maven 'Maven (latest)' ^ WorkflowScript: 510: Tool type "jdk" does not > have an install of "JDK 1.8 (latest)" configured - did you mean > "jdk_1.8_latest"? @ line 510, column 17. jdk "JDK 1.8 (latest)" > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #2566: HBASE-25203 Change the reference url to flaky list in our jenkins jobs
Apache9 merged pull request #2566: URL: https://github.com/apache/hbase/pull/2566 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2537: HBASE-24977 Meta table shouldn't be modified as read only
Apache-HBase commented on pull request #2537: URL: https://github.com/apache/hbase/pull/2537#issuecomment-712670788 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 13s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | checkstyle | 1m 15s | master passed | | +1 :green_heart: | spotbugs | 2m 20s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 34s | the patch passed | | +1 :green_heart: | checkstyle | 1m 1s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 37s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 40m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2537 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 60c001d3a2e8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7d00e43dce | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2537: HBASE-24977 Meta table shouldn't be modified as read only
Apache-HBase commented on pull request #2537: URL: https://github.com/apache/hbase/pull/2537#issuecomment-712671103 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 49s | master passed | | +1 :green_heart: | compile | 1m 12s | master passed | | +1 :green_heart: | shadedjars | 7m 25s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 35s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 7m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 53s | hbase-server in the patch failed. | | | | 41m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2537 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 91b98ab5a114 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7d00e43dce | | Default Java | 2020-01-14 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/3/testReport/ | | Max. process+thread count | 599 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2537/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2567: HBASE-25204 Nightly job failed as the name of jdk and maven changed
Apache9 commented on a change in pull request #2567:
URL: https://github.com/apache/hbase/pull/2567#discussion_r508266887
##
File path: dev-support/adhoc_run_tests/Jenkinsfile
##
@@ -51,10 +51,10 @@ pipeline {
stages {
stage ('run tests') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
Review comment:
Do we still use this file? Anyway, not a problem of the PR here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] pankaj72981 closed pull request #2537: HBASE-24977 Meta table shouldn't be modified as read only
pankaj72981 closed pull request #2537: URL: https://github.com/apache/hbase/pull/2537 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pankaj72981 opened a new pull request #2313: HBASE-24900 Make retain assignment configurable during SCP
pankaj72981 opened a new pull request #2313: URL: https://github.com/apache/hbase/pull/2313 Added "hbase.master.scp.retain.assignment" configuration parameter to contol the retain assignment during SCP. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pankaj72981 closed pull request #2313: HBASE-24900 Make retain assignment configurable during SCP
pankaj72981 closed pull request #2313: URL: https://github.com/apache/hbase/pull/2313 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] symat commented on pull request #2539: HBASE-25181 Configure hash algorithm in wrapped encryption keys
symat commented on pull request #2539: URL: https://github.com/apache/hbase/pull/2539#issuecomment-712646028 @busbey what do you think about the comment / question above? Do you think my approach (failing in case of unexpected hash algorithm) is OK, or do you think we should try to open / decode the HFile in this case using the original hash algorithm? I'm happy to change the logics, if you think this would be better This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
