[GitHub] [hbase] saintstack commented on a change in pull request #2543: HBASE-25184 Move RegionLocationFinder to hbase-balancer
saintstack commented on a change in pull request #2543:
URL: https://github.com/apache/hbase/pull/2543#discussion_r594871640
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
##
@@ -17,235 +17,187 @@
*/
package org.apache.hadoop.hbase.master.balancer;
-import java.io.FileNotFoundException;
+import com.google.errorprone.annotations.RestrictedApi;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
+import java.util.Collections;
import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
-import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.hbase.ClusterMetrics;
import org.apache.hadoop.hbase.HDFSBlocksDistribution;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.RegionInfo;
import org.apache.hadoop.hbase.client.TableDescriptor;
-import org.apache.hadoop.hbase.master.MasterServices;
-import org.apache.hadoop.hbase.master.assignment.AssignmentManager;
-import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+
import org.apache.hbase.thirdparty.com.google.common.cache.CacheBuilder;
import org.apache.hbase.thirdparty.com.google.common.cache.CacheLoader;
import org.apache.hbase.thirdparty.com.google.common.cache.LoadingCache;
-import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
import org.apache.hbase.thirdparty.com.google.common.util.concurrent.Futures;
import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ListenableFuture;
import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ListeningExecutorService;
import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.MoreExecutors;
import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
/**
- * This will find where data for a region is located in HDFS. It ranks
- * {@link ServerName}'s by the size of the store files they are holding for a
- * given region.
- *
+ * This will find where data for a region is located in HDFS. It ranks {@link
ServerName}'s by the
+ * size of the store files they are holding for a given region.
*/
@InterfaceAudience.Private
-class RegionLocationFinder {
+class RegionLocationFinder extends Configured {
Review comment:
I thought this class was about finding Region locations. Now I see it is
about HDFS block locations per Region... something else altogether. Seems like
the class is named incorrectly? Perhaps fix? RegionHDFSBlockLocationsFinder?
##
File path:
hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/BalancerService.java
##
@@ -0,0 +1,42 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.balancer;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HDFSBlocksDistribution;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * This is the cluster we want to balance. It provides methods to let us get
the information we
Review comment:
This comment is odd. Is BalanceService a good name for this? There is no
start/stop of this Service? There is not even a balance method in this
BalanceService?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] javierluca edited a comment on pull request #3053: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn and unload regions
javierluca edited a comment on pull request #3053: URL: https://github.com/apache/hbase/pull/3053#issuecomment-799974235 cc @saintstack First commit revives your previous changes. Second one addresses the commented issue with RegionMover. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca edited a comment on pull request #3053: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn and unload regions
javierluca edited a comment on pull request #3053: URL: https://github.com/apache/hbase/pull/3053#issuecomment-799974235 cc @saintstack First commit revives your previous changes. Second one is addressing the commented issue with RegionMover. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca edited a comment on pull request #3053: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn and unload regions
javierluca edited a comment on pull request #3053: URL: https://github.com/apache/hbase/pull/3053#issuecomment-799974235 cc @saintstack First commit is yours. Second one is addressing the commented issue with RegionMover. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca commented on pull request #3053: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn and unload regions
javierluca commented on pull request #3053: URL: https://github.com/apache/hbase/pull/3053#issuecomment-799974235 cc @saintstack This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca edited a comment on pull request #2973: HBASE-25594 graceful_stop.sh fails to unload regions when ran at localhost
javierluca edited a comment on pull request #2973: URL: https://github.com/apache/hbase/pull/2973#issuecomment-799973621 Actually changes made it to master here: https://github.com/apache/hbase/commit/59ec375d679dcbf1701a92d4206ecc676a8fb069 Trying further improvements at https://github.com/apache/hbase/pull/3053 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca commented on pull request #2973: HBASE-25594 graceful_stop.sh fails to unload regions when ran at localhost
javierluca commented on pull request #2973: URL: https://github.com/apache/hbase/pull/2973#issuecomment-799973621 Actually changes made it to master here: https://github.com/apache/hbase/commit/59ec375d679dcbf1701a92d4206ecc676a8fb069 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca edited a comment on pull request #3048: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn
javierluca edited a comment on pull request #3048: URL: https://github.com/apache/hbase/pull/3048#issuecomment-799958984 @saintstack In my opinion, your change was better than mine because we could use `"localhost"` as argument, and that will work very well with `regionservers.sh` which will bypass the ssh section when localhost, which does not work well with secured clusters: https://github.com/apache/hbase/blob/59ec375d679dcbf1701a92d4206ecc676a8fb069/bin/regionservers.sh#L61-L64 Just needed to address the `stripServer` issue. May be something like this will do it? ```shell unload_hostname="$hostname" if [[ "$hostname" -eq "localhost" ]]; then unload_hostname=`/bin/hostname -f` fi unload_args="--filename $filename --maxthreads $maxthreads $noack --operation unload \ --timeout $movetimeout --regionserverhost $unload_hostname" ``` If you don't mind, I would like to submit a patch including the changes of this PR + above fix for the RegionMover. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3053: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn and unload regions
Apache-HBase commented on pull request #3053: URL: https://github.com/apache/hbase/pull/3053#issuecomment-799971045 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | -0 :warning: | shellcheck | 0m 1s | The patch generated 10 new + 23 unchanged - 15 fixed = 33 total (was 38) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 2m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3053 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 1ae3ee9f020e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 59ec375d67 | | shellcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/1/artifact/yetus-general-check/output/diff-patch-shellcheck.txt | | Max. process+thread count | 51 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/1/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3053: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn and unload regions
Apache-HBase commented on pull request #3053: URL: https://github.com/apache/hbase/pull/3053#issuecomment-799970796 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 21s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 2m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3053 | | Optional Tests | | | uname | Linux 6a2a01ffe08e 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 59ec375d67 | | Max. process+thread count | 41 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3053: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn and unload regions
Apache-HBase commented on pull request #3053: URL: https://github.com/apache/hbase/pull/3053#issuecomment-799970407 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 1m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3053 | | Optional Tests | | | uname | Linux 97ff1b847090 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 59ec375d67 | | Max. process+thread count | 53 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3053/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca commented on pull request #3048: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn
javierluca commented on pull request #3048: URL: https://github.com/apache/hbase/pull/3048#issuecomment-799969668 @saintstack addressing it at https://github.com/apache/hbase/pull/3053. Please have a look This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca opened a new pull request #3053: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn and unload regions
javierluca opened a new pull request #3053: URL: https://github.com/apache/hbase/pull/3053 Previous: https://github.com/apache/hbase/pull/3048 Adding it again because it was revered: https://github.com/apache/hbase/commit/c36e40eb1c0e794358de3f0635df1e5848c7ac89 And dddressing this issue: https://github.com/apache/hbase/pull/3048#pullrequestreview-612818495 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bitterfox commented on a change in pull request #2988: HBASE-25608 Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
bitterfox commented on a change in pull request #2988:
URL: https://github.com/apache/hbase/pull/2988#discussion_r594873850
##
File path:
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##
@@ -657,6 +683,49 @@ public static void configureIncrementalLoadMap(Job job,
TableDescriptor tableDes
LOG.info("Incremental table " + tableDescriptor.getTableName() + " output
configured.");
}
+ /**
+ * Configure HBase cluster key to load region location for
locality-sensitive it's enabled.
+ * It's not necessary to call this method explicitly when the cluster key
for HBase cluster to be
+ * used to load region location is configured in the job configuration.
+ * Call this method when another HBase cluster key is configured in the job
configuration.
+ * For example, you should call when you load data from HBase cluster A using
+ * {@link TableInputFormat} and generate hfiles for HBase cluster B.
+ * Otherwise, HFileOutputFormat2 fetch location from cluster A and
locality-sensitive won't
+ * working correctly.
+ * {@link #configureIncrementalLoad(Job, Table, RegionLocator)} calls this
method using
+ * {@link Table#getConfiguration} as clusterConf.
+ * See HBASE-25608.
+ *
+ * @param job which has configuration to be updated
+ * @param clusterConf which contains cluster key of the HBase cluster to be
locality-sensitive
+ *
+ * @see #configureIncrementalLoad(Job, Table, RegionLocator)
+ * @see #LOCALITY_SENSITIVE_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_QUORUM_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_CLIENT_PORT_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_ZNODE_PARENT_CONF_KEY
+ */
+ public static void configureLocalitySensitiveCluster(Job job, Configuration
clusterConf) {
Review comment:
Could you review the proposal release note?
```
Added configurations to specify the ZK cluster key for the remote cluster in
HFileOutputFormat2.
These configurations are used to fetch region location for
locality-sensitive HFile generation.
Previously, it fetches from cluster which is configured in Job configuration,
but it does not work well when HBase clusters for input and output are
different.
HFileOutputFormat2#configureIncrementalLoad(Job, Table, RegionLocator)
configure them using Table's configuration.
You can also configure them by calling
HFileOutputFormat2#configureRemoteCluster explicitly.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2988: HBASE-25608 Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
Apache-HBase commented on pull request #2988: URL: https://github.com/apache/hbase/pull/2988#issuecomment-799963062 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 18s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 26s | master passed | | +1 :green_heart: | compile | 0m 35s | master passed | | +1 :green_heart: | shadedjars | 8m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 50s | the patch passed | | +1 :green_heart: | compile | 0m 38s | the patch passed | | +1 :green_heart: | javac | 0m 38s | the patch passed | | +1 :green_heart: | shadedjars | 8m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 13m 30s | hbase-mapreduce in the patch passed. | | | | 47m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2988/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2988 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f87f9cb90ee7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 59ec375d67 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2988/3/testReport/ | | Max. process+thread count | 2947 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2988/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25649) Complete the work on moving all the balancer related classes to hbase-balancer module
[ https://issues.apache.org/jira/browse/HBASE-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302228#comment-17302228 ] Michael Stack commented on HBASE-25649: --- Sorry. Late to design. Left a comment. > Complete the work on moving all the balancer related classes to > hbase-balancer module > - > > Key: HBASE-25649 > URL: https://issues.apache.org/jira/browse/HBASE-25649 > Project: HBase > Issue Type: Umbrella > Components: Balancer >Reporter: Duo Zhang >Priority: Major > > This is the follow up issue of HBASE-23933, where we set up the new > hbase-balancer module and moved several classes into it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3018: HBASE-25627: HBase replication should have a metric to represent if the source is stuck getting initialized
Apache-HBase commented on pull request #3018: URL: https://github.com/apache/hbase/pull/3018#issuecomment-799959223 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 45s | master passed | | +1 :green_heart: | compile | 1m 48s | master passed | | +1 :green_heart: | shadedjars | 8m 23s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 28s | the patch passed | | +1 :green_heart: | compile | 1m 23s | the patch passed | | +1 :green_heart: | javac | 1m 23s | the patch passed | | +1 :green_heart: | shadedjars | 8m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 54s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 232m 4s | hbase-server in the patch passed. | | | | 269m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3018/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3018 | | JIRA Issue | HBASE-25627 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux eae4b90be800 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5457554826 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3018/4/testReport/ | | Max. process+thread count | 2956 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3018/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca edited a comment on pull request #3048: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn
javierluca edited a comment on pull request #3048: URL: https://github.com/apache/hbase/pull/3048#issuecomment-799958984 @saintstack In my opinion, your change was better than mine because we could use `"localhost"` as argument, and that will work very well with `regionservers.sh` which will bypass the ssh section when localhost, which does not work well with secured clusters: https://github.com/apache/hbase/blob/59ec375d679dcbf1701a92d4206ecc676a8fb069/bin/regionservers.sh#L61-L64 Just needed to address the `stripServer` issue. May be something like this will do it? ```shell unload_hostname="$hostname" if [[ "$hostname" == "localhost" ]]; then unload_hostname=`/bin/hostname -f` fi unload_args="--filename $filename --maxthreads $maxthreads $noack --operation unload \ --timeout $movetimeout --regionserverhost $unload_hostname" ``` If you don't mind, I would like to submit a patch including the changes of this PR + above fix for the RegionMover. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca commented on pull request #3048: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn
javierluca commented on pull request #3048: URL: https://github.com/apache/hbase/pull/3048#issuecomment-799958984 @saintstack In my opinion, your change was better because we could use "localhost" as argument, and that will work very well with `regionservers.sh` which will bypass the ssh section that does not work well with secured clusters: https://github.com/apache/hbase/blob/59ec375d679dcbf1701a92d4206ecc676a8fb069/bin/regionservers.sh#L61-L64 Just needed to address the `stripServer` issue. May be something like this will do it? ```shell unload_hostname="$hostname" if [[ "$hostname" == "localhost" ]]; then unload_hostname=`/bin/hostname -f` fi unload_args="--filename $filename --maxthreads $maxthreads $noack --operation unload \ --timeout $movetimeout --regionserverhost $unload_hostname" ``` If you don't mind, I would like to submit a patch including the changes of this PR + above fix for the RegionMover. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2988: HBASE-25608 Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
Apache-HBase commented on pull request #2988: URL: https://github.com/apache/hbase/pull/2988#issuecomment-799958044 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | checkstyle | 0m 22s | master passed | | +1 :green_heart: | spotbugs | 0m 47s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 0m 47s | the patch passed | | +1 :green_heart: | javac | 0m 47s | the patch passed | | +1 :green_heart: | checkstyle | 0m 20s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 29s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 0m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 37m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2988/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2988 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7f3f5655199a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 59ec375d67 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2988/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2988: HBASE-25608 Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
Apache-HBase commented on pull request #2988: URL: https://github.com/apache/hbase/pull/2988#issuecomment-799956588 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 52s | master passed | | +1 :green_heart: | compile | 0m 26s | master passed | | +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 21s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 32s | the patch passed | | +1 :green_heart: | compile | 0m 26s | the patch passed | | +1 :green_heart: | javac | 0m 26s | the patch passed | | +1 :green_heart: | shadedjars | 6m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 57s | hbase-mapreduce in the patch passed. | | | | 33m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2988/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2988 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3b5e27a58ca6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 59ec375d67 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2988/3/testReport/ | | Max. process+thread count | 4414 (vs. ulimit of 3) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2988/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25184) Move RegionLocationFinder to hbase-balancer
[ https://issues.apache.org/jira/browse/HBASE-25184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302224#comment-17302224 ] Michael Stack commented on HBASE-25184: --- I see now. Its even package private to the balancer package. It just seemed like a generic functionality, too broad to be carried back to the balancer module. Thanks. > Move RegionLocationFinder to hbase-balancer > --- > > Key: HBASE-25184 > URL: https://issues.apache.org/jira/browse/HBASE-25184 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] xcangCRM commented on a change in pull request #3018: HBASE-25627: HBase replication should have a metric to represent if the source is stuck getting initialized
xcangCRM commented on a change in pull request #3018:
URL: https://github.com/apache/hbase/pull/3018#discussion_r594861844
##
File path:
hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics2/lib/DynamicMetricsRegistry.java
##
@@ -452,6 +450,41 @@ public MutableGaugeLong getGauge(String gaugeName, long
potentialStartingValue)
return (MutableGaugeLong) metric;
}
+ /**
+ * Get a MetricMutableGaugeInt from the storage. If it is not there
atomically put it.
+ *
+ * @param gaugeName name of the gauge to create or get.
+ * @param potentialStartingValue value of the new gauge if we have to create
it.
+ */
+ public MutableGaugeInt getGaugeInt(String gaugeName, int
potentialStartingValue) {
+//Try and get the guage.
+MutableMetric metric = metricsMap.get(gaugeName);
+
+//If it's not there then try and put a new one in the storage.
+if (metric == null) {
+
Review comment:
extra space line
##
File path:
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSourceImpl.java
##
@@ -49,6 +50,7 @@
private final MutableFastCounter completedRecoveryQueue;
private final MutableFastCounter failedRecoveryQueue;
private final MutableGaugeLong walReaderBufferUsageBytes;
+ private final MutableGaugeInt sourceInitializing;
Review comment:
never mind, I see you have comments below. :)
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSource.java
##
@@ -465,6 +465,22 @@ public synchronized UUID getPeerUUID() {
}
+ /**
+ * Bad Endpoint with failing connection to peer on demand.
+ */
+ public static class BadReplicationEndpoint extends
DoNothingReplicationEndpoint {
+static boolean failing = true;
+
+@Override
+public synchronized UUID getPeerUUID() {
+ if (failing) {
Review comment:
nit:
can we say
return failing ? null : super.getPeerUUID();
##
File path:
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSourceImpl.java
##
@@ -49,6 +50,7 @@
private final MutableFastCounter completedRecoveryQueue;
private final MutableFastCounter failedRecoveryQueue;
private final MutableGaugeLong walReaderBufferUsageBytes;
+ private final MutableGaugeInt sourceInitializing;
Review comment:
Nit:
so, this is the number of source initializing, right?
mind adding a minor comment for the metrics?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on pull request #3018: HBASE-25627: HBase replication should have a metric to represent if the source is stuck getting initialized
sandeepvinayak commented on pull request #3018: URL: https://github.com/apache/hbase/pull/3018#issuecomment-799947829 @wchevreuil do you mind reviewing this one? This is after your review [here](https://github.com/apache/hbase/pull/3009) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache-HBase commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-799947457 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 5s | master passed | | +1 :green_heart: | compile | 2m 48s | master passed | | +1 :green_heart: | shadedjars | 6m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 12m 49s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 7s | the patch passed | | +1 :green_heart: | compile | 2m 57s | the patch passed | | +1 :green_heart: | javac | 2m 57s | the patch passed | | +1 :green_heart: | shadedjars | 7m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 12m 52s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 163m 24s | root in the patch failed. | | | | 227m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/10/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1708 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 560b5d6269c7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5457554826 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/10/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/10/testReport/ | | Max. process+thread count | 5489 (vs. ulimit of 3) | | modules | C: hbase-logging hbase-common hbase-metrics-api hbase-metrics hbase-hadoop-compat hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded hbase-shaded/hbase-shaded-client hbase-shaded/hbase-shaded-client-byo-hadoop hbase-shaded/hbase-shaded-mapreduce hbase-hbtop hbase-assembly hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-testing-util-tester hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/10/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3049: HBASE-25643 The delayed FlushRegionEntry should be removed when we ne…
Apache-HBase commented on pull request #3049: URL: https://github.com/apache/hbase/pull/3049#issuecomment-799946861 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | compile | 1m 7s | master passed | | +1 :green_heart: | shadedjars | 6m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 8s | the patch passed | | +1 :green_heart: | compile | 1m 8s | the patch passed | | +1 :green_heart: | javac | 1m 8s | the patch passed | | +1 :green_heart: | shadedjars | 6m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 141m 32s | hbase-server in the patch passed. | | | | 170m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/10/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3049 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a4a918f6fdad 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5457554826 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/10/testReport/ | | Max. process+thread count | 3770 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/10/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3018: HBASE-25627: HBase replication should have a metric to represent if the source is stuck getting initialized
Apache-HBase commented on pull request #3018: URL: https://github.com/apache/hbase/pull/3018#issuecomment-799944591 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 36s | master passed | | +1 :green_heart: | compile | 1m 33s | master passed | | +1 :green_heart: | shadedjars | 7m 22s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 0s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 36s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed | | +1 :green_heart: | javac | 1m 36s | the patch passed | | +1 :green_heart: | shadedjars | 7m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 39s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 200m 45s | hbase-server in the patch passed. | | | | 234m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3018/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3018 | | JIRA Issue | HBASE-25627 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4bd345724180 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5457554826 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3018/4/testReport/ | | Max. process+thread count | 3619 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3018/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25594) graceful_stop.sh fails to unload regions when ran at localhost
[ https://issues.apache.org/jira/browse/HBASE-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25594. --- Fix Version/s: 2.4.3 2.5.0 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed I pushed your PR to 2.4+ [~akiraluca] (Added the doc changes from HBASE-25663 here when I pushed). Thanks for the PR. Shout if you need it to go back further. > graceful_stop.sh fails to unload regions when ran at localhost > -- > > Key: HBASE-25594 > URL: https://issues.apache.org/jira/browse/HBASE-25594 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 1.4.13 >Reporter: Javier Akira Luca de Tena >Assignee: Javier Akira Luca de Tena >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3 > > > We usually use graceful_stop.sh from the Master to restart RegionServers. > However, in some scenarios we may not have privileges to restart remote > RegionServers (it uses ssh). > But we can still use graceful_stop.sh on the same host we want to restart. > In order to detect the execution at localhost, graceful_stop.sh uses > /bin/hostname. > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/bin/graceful_stop.sh#L106-L110] > When RegionMover strips the host to not include it in the list of target > hosts, we filter it out by checking all RegionServer hosts in the cluster: > > [https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L382-L384] > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L692] > But the list of RegionServer hosts returned by Admin#getRegionServers are > FDQN, while the hostname provided from graceful_stop.sh is not FDQN, making > the comparison fail. > Same happens for branch-1 region_mover.rb, which is the place I reproduced in > my environment: > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L305] > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L175] > > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L186-L192] > > This can be fixed just by using "/bin/hostname -f" in the graceful_stop.sh > script. > Will provide patch soon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25663) Make graceful_stop localhostname compare match even if fqdn
[ https://issues.apache.org/jira/browse/HBASE-25663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack reopened HBASE-25663: --- Reopening to revert in favor of HBASE-25594 > Make graceful_stop localhostname compare match even if fqdn > --- > > Key: HBASE-25663 > URL: https://issues.apache.org/jira/browse/HBASE-25663 > Project: HBase > Issue Type: Improvement >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3 > > > The hostname to operate on will be fqdn usually. Internal to the > graceful_stop script it tries to check if we are running locally but the > compare is of a fqdn to what 'hostname' returns; the latter usually returns > name only. Allow name-only and fqdn match. While here cleanup doc around > 'hostname' passed to graceful_stop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25594) graceful_stop.sh fails to unload regions when ran at localhost
[ https://issues.apache.org/jira/browse/HBASE-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack reopened HBASE-25594: --- Reopening to apply the fix here instead of what was over in HBASE-25663. > graceful_stop.sh fails to unload regions when ran at localhost > -- > > Key: HBASE-25594 > URL: https://issues.apache.org/jira/browse/HBASE-25594 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 1.4.13 >Reporter: Javier Akira Luca de Tena >Assignee: Javier Akira Luca de Tena >Priority: Minor > > We usually use graceful_stop.sh from the Master to restart RegionServers. > However, in some scenarios we may not have privileges to restart remote > RegionServers (it uses ssh). > But we can still use graceful_stop.sh on the same host we want to restart. > In order to detect the execution at localhost, graceful_stop.sh uses > /bin/hostname. > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/bin/graceful_stop.sh#L106-L110] > When RegionMover strips the host to not include it in the list of target > hosts, we filter it out by checking all RegionServer hosts in the cluster: > > [https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L382-L384] > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L692] > But the list of RegionServer hosts returned by Admin#getRegionServers are > FDQN, while the hostname provided from graceful_stop.sh is not FDQN, making > the comparison fail. > Same happens for branch-1 region_mover.rb, which is the place I reproduced in > my environment: > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L305] > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L175] > > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L186-L192] > > This can be fixed just by using "/bin/hostname -f" in the graceful_stop.sh > script. > Will provide patch soon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25663) Make graceful_stop localhostname compare match even if fqdn
[ https://issues.apache.org/jira/browse/HBASE-25663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302208#comment-17302208 ] Michael Stack commented on HBASE-25663: --- Reverted from branch-2.4+ > Make graceful_stop localhostname compare match even if fqdn > --- > > Key: HBASE-25663 > URL: https://issues.apache.org/jira/browse/HBASE-25663 > Project: HBase > Issue Type: Improvement >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > > The hostname to operate on will be fqdn usually. Internal to the > graceful_stop script it tries to check if we are running locally but the > compare is of a fqdn to what 'hostname' returns; the latter usually returns > name only. Allow name-only and fqdn match. While here cleanup doc around > 'hostname' passed to graceful_stop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25663) Make graceful_stop localhostname compare match even if fqdn
[ https://issues.apache.org/jira/browse/HBASE-25663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25663: -- Fix Version/s: (was: 2.4.3) (was: 2.5.0) (was: 3.0.0-alpha-1) > Make graceful_stop localhostname compare match even if fqdn > --- > > Key: HBASE-25663 > URL: https://issues.apache.org/jira/browse/HBASE-25663 > Project: HBase > Issue Type: Improvement >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > > The hostname to operate on will be fqdn usually. Internal to the > graceful_stop script it tries to check if we are running locally but the > compare is of a fqdn to what 'hostname' returns; the latter usually returns > name only. Allow name-only and fqdn match. While here cleanup doc around > 'hostname' passed to graceful_stop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25374) Make REST Client connection and socket time out configurable
[ https://issues.apache.org/jira/browse/HBASE-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302206#comment-17302206 ] Pankaj Kumar commented on HBASE-25374: -- Thank you [~stack] Sir... > Make REST Client connection and socket time out configurable > > > Key: HBASE-25374 > URL: https://issues.apache.org/jira/browse/HBASE-25374 > Project: HBase > Issue Type: Improvement > Components: REST >Affects Versions: 3.0.0-alpha-1, 2.3.4, 2.5.0, 2.4.3 >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3 > > > Currently there is no way to configure connection and socket timeout for REST > client, > https://github.com/apache/hbase/blob/7d0a687e5798a2f4ca3190b409169f7e17a75b34/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/Client.java#L102 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on pull request #3048: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn
saintstack commented on pull request #3048: URL: https://github.com/apache/hbase/pull/3048#issuecomment-799936784 Or, let me just revert this and commit your patch; it works for you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bitterfox commented on a change in pull request #2988: HBASE-25608 Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
bitterfox commented on a change in pull request #2988:
URL: https://github.com/apache/hbase/pull/2988#discussion_r594846183
##
File path:
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##
@@ -657,6 +683,49 @@ public static void configureIncrementalLoadMap(Job job,
TableDescriptor tableDes
LOG.info("Incremental table " + tableDescriptor.getTableName() + " output
configured.");
}
+ /**
+ * Configure HBase cluster key to load region location for
locality-sensitive it's enabled.
+ * It's not necessary to call this method explicitly when the cluster key
for HBase cluster to be
+ * used to load region location is configured in the job configuration.
+ * Call this method when another HBase cluster key is configured in the job
configuration.
+ * For example, you should call when you load data from HBase cluster A using
+ * {@link TableInputFormat} and generate hfiles for HBase cluster B.
+ * Otherwise, HFileOutputFormat2 fetch location from cluster A and
locality-sensitive won't
+ * working correctly.
+ * {@link #configureIncrementalLoad(Job, Table, RegionLocator)} calls this
method using
+ * {@link Table#getConfiguration} as clusterConf.
+ * See HBASE-25608.
+ *
+ * @param job which has configuration to be updated
+ * @param clusterConf which contains cluster key of the HBase cluster to be
locality-sensitive
+ *
+ * @see #configureIncrementalLoad(Job, Table, RegionLocator)
+ * @see #LOCALITY_SENSITIVE_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_QUORUM_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_CLIENT_PORT_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_ZNODE_PARENT_CONF_KEY
+ */
+ public static void configureLocalitySensitiveCluster(Job job, Configuration
clusterConf) {
Review comment:
Thank you for your review! It seems to me 'remote cluster' is a better
name than what I had.
I'll update PR and also prepare a release note soon
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #3048: HBASE-25663 Make graceful_stop localhostname compare match even if fqdn
saintstack commented on pull request #3048: URL: https://github.com/apache/hbase/pull/3048#issuecomment-799927706 @javierluca Thank you for taking a look. Looks like the stripServer needs to handle 'localhost'. Let me make an addendum This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3046: HBASE-25653 Add units and round off region size to 2 digits after decimal
Apache-HBase commented on pull request #3046: URL: https://github.com/apache/hbase/pull/3046#issuecomment-799924438 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 24s | master passed | | +1 :green_heart: | compile | 0m 57s | master passed | | +1 :green_heart: | shadedjars | 6m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 35s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 142m 59s | hbase-server in the patch passed. | | | | 169m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3046/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3046 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2dbaf795c7cc 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5457554826 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3046/4/testReport/ | | Max. process+thread count | 5228 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3046/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25395) Introduce PersistedStoreEngine and PersistedStoreFileManager
[ https://issues.apache.org/jira/browse/HBASE-25395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-25395. -- Release Note: Added `hbase.storefile.tracking.persist.enabled` to enable `PersistedStoreEngine` and `PersistedStoreFileManager` that fork the file paths of in-memory object `storefile` in StoreFileManager into a system table `hbase:storefile`. This persisted paths will be reused when region (re)opens on existing store without scanning the file storage. Resolution: Fixed > Introduce PersistedStoreEngine and PersistedStoreFileManager > - > > Key: HBASE-25395 > URL: https://issues.apache.org/jira/browse/HBASE-25395 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-1 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > Create a new wrapper store engine on top of the default store engine, as well > as, introducing a new store file manager that keep tracking path of HFiles of > user tables into a hbase self-managed system table (further will have a > different tasks to merge into meta table and check throughput do not change > much) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3052: HBASE-25639 meta replica state is not respected during active master …
Apache-HBase commented on pull request #3052: URL: https://github.com/apache/hbase/pull/3052#issuecomment-799924066 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 50s | branch-2.3 passed | | +1 :green_heart: | compile | 1m 9s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 6m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 23s | the patch passed | | +1 :green_heart: | compile | 1m 8s | the patch passed | | +1 :green_heart: | javac | 1m 8s | the patch passed | | +1 :green_heart: | shadedjars | 6m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 210m 4s | hbase-server in the patch passed. | | | | 239m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3052/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3052 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b41945ddcd61 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 1a695e872b | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3052/1/testReport/ | | Max. process+thread count | 2798 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3052/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3046: HBASE-25653 Add units and round off region size to 2 digits after decimal
Apache-HBase commented on pull request #3046: URL: https://github.com/apache/hbase/pull/3046#issuecomment-799923132 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 34s | master passed | | +1 :green_heart: | compile | 1m 8s | master passed | | +1 :green_heart: | shadedjars | 7m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 6m 51s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 135m 30s | hbase-server in the patch passed. | | | | 164m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3046/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3046 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a0d5043b8ccc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5457554826 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3046/4/testReport/ | | Max. process+thread count | 4096 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3046/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25395) Introduce PersistedStoreEngine and PersistedStoreFileManager
[ https://issues.apache.org/jira/browse/HBASE-25395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu updated HBASE-25395: - Affects Version/s: 3.0.0-alpha-1 > Introduce PersistedStoreEngine and PersistedStoreFileManager > - > > Key: HBASE-25395 > URL: https://issues.apache.org/jira/browse/HBASE-25395 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-1 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > Create a new wrapper store engine on top of the default store engine, as well > as, introducing a new store file manager that keep tracking path of HFiles of > user tables into a hbase self-managed system table (further will have a > different tasks to merge into meta table and check throughput do not change > much) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25395) Introduce PersistedStoreEngine and PersistedStoreFileManager
[ https://issues.apache.org/jira/browse/HBASE-25395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302201#comment-17302201 ] Tak-Lon (Stephen) Wu commented on HBASE-25395: -- committed to feature branch [HBASE-24749|https://github.com/apache/hbase/tree/HBASE-24749] > Introduce PersistedStoreEngine and PersistedStoreFileManager > - > > Key: HBASE-25395 > URL: https://issues.apache.org/jira/browse/HBASE-25395 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > Create a new wrapper store engine on top of the default store engine, as well > as, introducing a new store file manager that keep tracking path of HFiles of > user tables into a hbase self-managed system table (further will have a > different tasks to merge into meta table and check throughput do not change > much) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] taklwu edited a comment on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
taklwu edited a comment on pull request #2931: URL: https://github.com/apache/hbase/pull/2931#issuecomment-799918388 fixed the commit from fd649ce544553a7b2a2cfe9df0ea8a9bc49fa571 to cca4cbf173f4320dcb9262e9a97deba27f7de15e with adding `Reviewed-by:` and `Signed-off-by:` to branch HBASE-24749 , and rebased on master 5457554826519232c7d81ab382a5f345da0c0da2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] taklwu commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
taklwu commented on pull request #2931: URL: https://github.com/apache/hbase/pull/2931#issuecomment-799918388 fixed the commit from [fd649ce544553a7b2a2cfe9df0ea8a9bc49fa571](https://github.com/apache/hbase/commit/fd649ce544553a7b2a2cfe9df0ea8a9bc49fa571) to [7b58ada2ff3e06b2a4550d69aa121e0c1c8f3c67](https://github.com/apache/hbase/commit/7b58ada2ff3e06b2a4550d69aa121e0c1c8f3c67) with adding `Reviewed-by:` and `Signed-off-by:` to branch HBASE-24749 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #3049: HBASE-25643 The delayed FlushRegionEntry should be removed when we ne…
bsglz commented on a change in pull request #3049:
URL: https://github.com/apache/hbase/pull/3049#discussion_r594837551
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
##
@@ -873,6 +885,13 @@ public boolean isMaximumWait(final long maximumWait) {
return (EnvironmentEdgeManager.currentTime() - this.createTime) >
maximumWait;
}
+/**
+ * @return True if the entry is a delay flush task
+ */
+public boolean isDelay() {
Review comment:
No, will fix, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] taklwu merged pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
taklwu merged pull request #2931: URL: https://github.com/apache/hbase/pull/2931 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] taklwu commented on pull request #2931: HBASE-25395 Introduce PersistedStoreEngine and PersistedStoreFileManager
taklwu commented on pull request #2931: URL: https://github.com/apache/hbase/pull/2931#issuecomment-799913582 > Minor comments, approved. Hey Zach @z-york since your comment are minor and I will implement them in the upcoming JIRA, I will push the commit AS IS to save another around of CI build In addition, @joshelser I didn't hear back from you so I think the reviewed by @anmolnar should be good enough, and if you want to come back and ask more change, we can have it in a addendum PR/JIRA. I'm pushing this to the feature branch HBASE-24749 and continues on other PRs this week. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-25594) graceful_stop.sh fails to unload regions when ran at localhost
[ https://issues.apache.org/jira/browse/HBASE-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302194#comment-17302194 ] Javier Akira Luca de Tena edited comment on HBASE-25594 at 3/16/21, 3:21 AM: - [~stack] I left a comment in your PR [https://github.com/apache/hbase/pull/3048#pullrequestreview-612818495] I would appreciate if you have a look at it since I am not 100% it works as expected. was (Author: akiraluca): [~stack] I left a comment in your PR [https://github.com/apache/hbase/pull/3048#pullrequestreview-612818495] I would appreciate if you have a look at it. > graceful_stop.sh fails to unload regions when ran at localhost > -- > > Key: HBASE-25594 > URL: https://issues.apache.org/jira/browse/HBASE-25594 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 1.4.13 >Reporter: Javier Akira Luca de Tena >Assignee: Javier Akira Luca de Tena >Priority: Minor > > We usually use graceful_stop.sh from the Master to restart RegionServers. > However, in some scenarios we may not have privileges to restart remote > RegionServers (it uses ssh). > But we can still use graceful_stop.sh on the same host we want to restart. > In order to detect the execution at localhost, graceful_stop.sh uses > /bin/hostname. > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/bin/graceful_stop.sh#L106-L110] > When RegionMover strips the host to not include it in the list of target > hosts, we filter it out by checking all RegionServer hosts in the cluster: > > [https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L382-L384] > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L692] > But the list of RegionServer hosts returned by Admin#getRegionServers are > FDQN, while the hostname provided from graceful_stop.sh is not FDQN, making > the comparison fail. > Same happens for branch-1 region_mover.rb, which is the place I reproduced in > my environment: > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L305] > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L175] > > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L186-L192] > > This can be fixed just by using "/bin/hostname -f" in the graceful_stop.sh > script. > Will provide patch soon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25594) graceful_stop.sh fails to unload regions when ran at localhost
[ https://issues.apache.org/jira/browse/HBASE-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302194#comment-17302194 ] Javier Akira Luca de Tena commented on HBASE-25594: --- [~stack] I left a comment in your PR [https://github.com/apache/hbase/pull/3048#pullrequestreview-612818495] I would appreciate if you have a look at it. > graceful_stop.sh fails to unload regions when ran at localhost > -- > > Key: HBASE-25594 > URL: https://issues.apache.org/jira/browse/HBASE-25594 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 1.4.13 >Reporter: Javier Akira Luca de Tena >Assignee: Javier Akira Luca de Tena >Priority: Minor > > We usually use graceful_stop.sh from the Master to restart RegionServers. > However, in some scenarios we may not have privileges to restart remote > RegionServers (it uses ssh). > But we can still use graceful_stop.sh on the same host we want to restart. > In order to detect the execution at localhost, graceful_stop.sh uses > /bin/hostname. > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/bin/graceful_stop.sh#L106-L110] > When RegionMover strips the host to not include it in the list of target > hosts, we filter it out by checking all RegionServer hosts in the cluster: > > [https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L382-L384] > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L692] > But the list of RegionServer hosts returned by Admin#getRegionServers are > FDQN, while the hostname provided from graceful_stop.sh is not FDQN, making > the comparison fail. > Same happens for branch-1 region_mover.rb, which is the place I reproduced in > my environment: > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L305] > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L175] > > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L186-L192] > > This can be fixed just by using "/bin/hostname -f" in the graceful_stop.sh > script. > Will provide patch soon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache-HBase commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-799912356 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | compile | 8m 35s | master passed | | +1 :green_heart: | checkstyle | 2m 19s | master passed | | +1 :green_heart: | spotbugs | 24m 57s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 59s | the patch passed | | +1 :green_heart: | compile | 8m 39s | the patch passed | | +1 :green_heart: | javac | 8m 39s | root generated 0 new + 1560 unchanged - 2 fixed = 1560 total (was 1562) | | +1 :green_heart: | checkstyle | 2m 19s | root: The patch generated 0 new + 163 unchanged - 3 fixed = 163 total (was 166) | | -0 :warning: | shellcheck | 0m 2s | The patch generated 1 new + 171 unchanged - 0 fixed = 172 total (was 171) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 49s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 19m 47s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 30m 4s | the patch passed | ||| _ Other Tests _ | | -1 :x: | asflicense | 6m 4s | The patch generated 1 ASF License warnings. | | | | 131m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1708 | | Optional Tests | dupname asflicense shellcheck shelldocs xml javac hadoopcheck compile spotbugs hbaseanti checkstyle | | uname | Linux d88809d8199d 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5457554826 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | shellcheck | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/10/artifact/yetus-general-check/output/diff-patch-shellcheck.txt | | asflicense | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/10/artifact/yetus-general-check/output/patch-asflicense-problems.txt | | Max. process+thread count | 126 (vs. ulimit of 3) | | modules | C: hbase-logging hbase-common hbase-metrics-api hbase-metrics hbase-hadoop-compat hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded hbase-shaded/hbase-shaded-client hbase-shaded/hbase-shaded-client-byo-hadoop hbase-shaded/hbase-shaded-mapreduce hbase-hbtop hbase-assembly hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-testing-util-tester hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/10/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3049: HBASE-25643 The delayed FlushRegionEntry should be removed when we ne…
Apache-HBase commented on pull request #3049: URL: https://github.com/apache/hbase/pull/3049#issuecomment-799905272 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 34s | master passed | | +1 :green_heart: | compile | 3m 43s | master passed | | +1 :green_heart: | checkstyle | 1m 21s | master passed | | +1 :green_heart: | spotbugs | 2m 34s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 14s | the patch passed | | +1 :green_heart: | compile | 3m 50s | the patch passed | | +1 :green_heart: | javac | 3m 50s | the patch passed | | +1 :green_heart: | checkstyle | 1m 22s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 20m 51s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 55m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3049 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux a2b125f6638c 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5457554826 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 86 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/10/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3052: HBASE-25639 meta replica state is not respected during active master …
Apache-HBase commented on pull request #3052: URL: https://github.com/apache/hbase/pull/3052#issuecomment-799896778 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | branch-2.3 passed | | +1 :green_heart: | compile | 0m 53s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 5m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 14s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 5m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 137m 26s | hbase-server in the patch passed. | | | | 160m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3052/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3052 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 598fa2238963 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 1a695e872b | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3052/1/testReport/ | | Max. process+thread count | 4217 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3052/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25597) Add row info in Exception when cell size exceeds maxCellSize
[
https://issues.apache.org/jira/browse/HBASE-25597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302166#comment-17302166
]
Baiqiang Zhao commented on HBASE-25597:
---
Thank you [~stack] !
> Add row info in Exception when cell size exceeds maxCellSize
>
>
> Key: HBASE-25597
> URL: https://issues.apache.org/jira/browse/HBASE-25597
> Project: HBase
> Issue Type: Improvement
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
>
> When cell size exceeds maxCellSize(default is 10M), client will get a
> DoNotRetryIOException, code as below:
> {code:java}
> private void checkCellSizeLimit(final HRegion r, final Mutation m) throws
> IOException {
> if (r.maxCellSize > 0) {
> CellScanner cells = m.cellScanner();
> while (cells.advance()) {
> int size = PrivateCellUtil.estimatedSerializedSizeOf(cells.current());
> if (size > r.maxCellSize) {
> String msg = "Cell with size " + size + " exceeds limit of " +
> r.maxCellSize + " bytes";
> LOG.debug(msg);
> throw new DoNotRetryIOException(msg);
> }
> }
> }
> }
> {code}
> There is no row related information, which makes troubleshooting difficult.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3043: HBASE-25568 Upgrade Thrift jar to fix CVE-2020-13949
Apache-HBase commented on pull request #3043: URL: https://github.com/apache/hbase/pull/3043#issuecomment-799889549 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 45s | master passed | | +1 :green_heart: | compile | 2m 25s | master passed | | +1 :green_heart: | shadedjars | 6m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 57s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 2m 24s | the patch passed | | +1 :green_heart: | javac | 2m 24s | the patch passed | | +1 :green_heart: | shadedjars | 6m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 2s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 272m 53s | root in the patch failed. | | | | 309m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3043/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3043 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d826902a27e9 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f4e1ab7b1d | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3043/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3043/3/testReport/ | | Max. process+thread count | 3695 (vs. ulimit of 3) | | modules | C: hbase-thrift hbase-examples . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3043/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #3049: HBASE-25643 The delayed FlushRegionEntry should be removed when we ne…
bsglz commented on a change in pull request #3049:
URL: https://github.com/apache/hbase/pull/3049#discussion_r594808025
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
##
@@ -127,20 +127,22 @@ public MemStoreFlusher(final Configuration conf,
this.blockingWaitTime = conf.getInt("hbase.hstore.blockingWaitTime",
9);
int handlerCount = conf.getInt("hbase.hstore.flusher.count", 2);
-if (handlerCount < 1) {
- LOG.warn("hbase.hstore.flusher.count was configed to {} which is less
than 1, corrected to 1",
- handlerCount);
- handlerCount = 1;
+if (server != null) {
+ if (handlerCount < 1) {
+LOG.warn("hbase.hstore.flusher.count was configed to {} which is less
than 1, "
++ "corrected to 1", handlerCount);
+handlerCount = 1;
+ }
+ LOG.info("globalMemStoreLimit="
+ + TraditionalBinaryPrefix
+
.long2String(this.server.getRegionServerAccounting().getGlobalMemStoreLimit(),
"", 1)
+ + ", globalMemStoreLimitLowMark="
+ + TraditionalBinaryPrefix.long2String(
+
this.server.getRegionServerAccounting().getGlobalMemStoreLimitLowMark(), "", 1)
+ + ", Offheap="
+ + (this.server.getRegionServerAccounting().isOffheap()));
}
this.flushHandlers = new FlushHandler[handlerCount];
Review comment:
Yeah, in TestMemStoreFlusher, we don't want the handler to run, so i set
the hbase.hstore.flusher.count to 0 in conf.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #3049: HBASE-25643 The delayed FlushRegionEntry should be removed when we ne…
bsglz commented on a change in pull request #3049:
URL: https://github.com/apache/hbase/pull/3049#discussion_r594806546
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -8416,6 +8416,10 @@ public void incrementFlushesQueuedCount() {
flushesQueued.increment();
}
+ public void decrementFlushesQueuedCount() {
Review comment:
Will fix, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3046: HBASE-25653 Add units and round off region size to 2 digits after decimal
Apache-HBase commented on pull request #3046: URL: https://github.com/apache/hbase/pull/3046#issuecomment-799881168 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 52s | master passed | | +1 :green_heart: | compile | 3m 7s | master passed | | +1 :green_heart: | checkstyle | 1m 5s | master passed | | +1 :green_heart: | spotbugs | 2m 9s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | +1 :green_heart: | compile | 3m 5s | the patch passed | | +1 :green_heart: | javac | 3m 5s | the patch passed | | +1 :green_heart: | checkstyle | 1m 4s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 24s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 45m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3046/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3046 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7d2a9dd5bfcf 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5457554826 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3046/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3018: HBASE-25627: HBase replication should have a metric to represent if the source is stuck getting initialized
Apache-HBase commented on pull request #3018: URL: https://github.com/apache/hbase/pull/3018#issuecomment-799880173 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 37s | master passed | | +1 :green_heart: | compile | 3m 39s | master passed | | +1 :green_heart: | checkstyle | 1m 17s | master passed | | +1 :green_heart: | spotbugs | 2m 47s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 3m 44s | the patch passed | | +1 :green_heart: | javac | 3m 44s | the patch passed | | +1 :green_heart: | checkstyle | 0m 13s | The patch passed checkstyle in hbase-hadoop-compat | | +1 :green_heart: | checkstyle | 1m 2s | hbase-server: The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 49s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 4m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 53m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3018/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3018 | | JIRA Issue | HBASE-25627 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux d03dfcc6e9d6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 5457554826 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3018/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-25594) graceful_stop.sh fails to unload regions when ran at localhost
[ https://issues.apache.org/jira/browse/HBASE-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302137#comment-17302137 ] Javier Akira Luca de Tena edited comment on HBASE-25594 at 3/16/21, 1:19 AM: - [~stack] I think that will perfectly work for what I was trying to achieve. Thank you. Hope you backport it to branch-2. was (Author: akiraluca): I think that will perfectly work for what I was trying to achieve. Thank you. > graceful_stop.sh fails to unload regions when ran at localhost > -- > > Key: HBASE-25594 > URL: https://issues.apache.org/jira/browse/HBASE-25594 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 1.4.13 >Reporter: Javier Akira Luca de Tena >Assignee: Javier Akira Luca de Tena >Priority: Minor > > We usually use graceful_stop.sh from the Master to restart RegionServers. > However, in some scenarios we may not have privileges to restart remote > RegionServers (it uses ssh). > But we can still use graceful_stop.sh on the same host we want to restart. > In order to detect the execution at localhost, graceful_stop.sh uses > /bin/hostname. > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/bin/graceful_stop.sh#L106-L110] > When RegionMover strips the host to not include it in the list of target > hosts, we filter it out by checking all RegionServer hosts in the cluster: > > [https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L382-L384] > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L692] > But the list of RegionServer hosts returned by Admin#getRegionServers are > FDQN, while the hostname provided from graceful_stop.sh is not FDQN, making > the comparison fail. > Same happens for branch-1 region_mover.rb, which is the place I reproduced in > my environment: > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L305] > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L175] > > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L186-L192] > > This can be fixed just by using "/bin/hostname -f" in the graceful_stop.sh > script. > Will provide patch soon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25594) graceful_stop.sh fails to unload regions when ran at localhost
[ https://issues.apache.org/jira/browse/HBASE-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Javier Akira Luca de Tena resolved HBASE-25594. --- Resolution: Duplicate > graceful_stop.sh fails to unload regions when ran at localhost > -- > > Key: HBASE-25594 > URL: https://issues.apache.org/jira/browse/HBASE-25594 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 1.4.13 >Reporter: Javier Akira Luca de Tena >Assignee: Javier Akira Luca de Tena >Priority: Minor > > We usually use graceful_stop.sh from the Master to restart RegionServers. > However, in some scenarios we may not have privileges to restart remote > RegionServers (it uses ssh). > But we can still use graceful_stop.sh on the same host we want to restart. > In order to detect the execution at localhost, graceful_stop.sh uses > /bin/hostname. > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/bin/graceful_stop.sh#L106-L110] > When RegionMover strips the host to not include it in the list of target > hosts, we filter it out by checking all RegionServer hosts in the cluster: > > [https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L382-L384] > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L692] > But the list of RegionServer hosts returned by Admin#getRegionServers are > FDQN, while the hostname provided from graceful_stop.sh is not FDQN, making > the comparison fail. > Same happens for branch-1 region_mover.rb, which is the place I reproduced in > my environment: > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L305] > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L175] > > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L186-L192] > > This can be fixed just by using "/bin/hostname -f" in the graceful_stop.sh > script. > Will provide patch soon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25594) graceful_stop.sh fails to unload regions when ran at localhost
[ https://issues.apache.org/jira/browse/HBASE-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302137#comment-17302137 ] Javier Akira Luca de Tena commented on HBASE-25594: --- I think that will perfectly work for what I was trying to achieve. Thank you. > graceful_stop.sh fails to unload regions when ran at localhost > -- > > Key: HBASE-25594 > URL: https://issues.apache.org/jira/browse/HBASE-25594 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 1.4.13 >Reporter: Javier Akira Luca de Tena >Assignee: Javier Akira Luca de Tena >Priority: Minor > > We usually use graceful_stop.sh from the Master to restart RegionServers. > However, in some scenarios we may not have privileges to restart remote > RegionServers (it uses ssh). > But we can still use graceful_stop.sh on the same host we want to restart. > In order to detect the execution at localhost, graceful_stop.sh uses > /bin/hostname. > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/bin/graceful_stop.sh#L106-L110] > When RegionMover strips the host to not include it in the list of target > hosts, we filter it out by checking all RegionServer hosts in the cluster: > > [https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L382-L384] > > [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L692] > But the list of RegionServer hosts returned by Admin#getRegionServers are > FDQN, while the hostname provided from graceful_stop.sh is not FDQN, making > the comparison fail. > Same happens for branch-1 region_mover.rb, which is the place I reproduced in > my environment: > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L305] > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L175] > > [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L186-L192] > > This can be fixed just by using "/bin/hostname -f" in the graceful_stop.sh > script. > Will provide patch soon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] javierluca commented on pull request #2973: HBASE-25594 graceful_stop.sh fails to unload regions when ran at localhost
javierluca commented on pull request #2973: URL: https://github.com/apache/hbase/pull/2973#issuecomment-799868302 Seems this was better solved at https://github.com/apache/hbase/pull/3048. So will close this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] javierluca closed pull request #2973: HBASE-25594 graceful_stop.sh fails to unload regions when ran at localhost
javierluca closed pull request #2973: URL: https://github.com/apache/hbase/pull/2973 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache9 commented on a change in pull request #1708:
URL: https://github.com/apache/hbase/pull/1708#discussion_r594792572
##
File path: bin/hbase
##
@@ -775,25 +783,31 @@ HBASE_OPTS="$HBASE_OPTS -Dhbase.log.dir=$HBASE_LOG_DIR"
HBASE_OPTS="$HBASE_OPTS -Dhbase.log.file=$HBASE_LOGFILE"
HBASE_OPTS="$HBASE_OPTS -Dhbase.home.dir=$HBASE_HOME"
HBASE_OPTS="$HBASE_OPTS -Dhbase.id.str=$HBASE_IDENT_STRING"
-HBASE_OPTS="$HBASE_OPTS -Dhbase.root.logger=${HBASE_ROOT_LOGGER:-INFO,console}"
+HBASE_OPTS="$HBASE_OPTS
-Dhbase.root.logger.level=${HBASE_ROOT_LOGGER_LEVEL:-INFO}"
+HBASE_OPTS="$HBASE_OPTS
-Dhbase.root.logger.appender=${HBASE_ROOT_LOGGER_APPENDER:-console}"
Review comment:
Seems log4j2 also supports properties files in newer versions now. Let
me see if we could use a property file so we could still use the old config
pattern.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] DivyeshChandra commented on a change in pull request #3046: HBASE-25653 Add units and round off region size to 2 digits after decimal
DivyeshChandra commented on a change in pull request #3046:
URL: https://github.com/apache/hbase/pull/3046#discussion_r594789222
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -284,7 +284,7 @@ private double getAverageRegionSizeMb(final
List tableRegions,
double avgRegionSize;
int targetRegionCount = tableDescriptor.getNormalizerTargetRegionCount();
long targetRegionSize = tableDescriptor.getNormalizerTargetRegionSize();
-LOG.debug("Table {} configured with target region count {}, target region
size {}", table,
+LOG.debug("Table {} configured with target region count {}, target region
size {} MB", table,
Review comment:
>For sure these are MB values? I tried looking at code but it doesn't
say explicitly...
Yeah, but there is a hint of it in the
[alter](https://github.com/apache/hbase/blob/5457554826519232c7d81ab382a5f345da0c0da2/hbase-shell/src/main/ruby/shell/commands/alter.rb#L51)
command of hbase shell.
Also, `avgRegionSize` is set to `targetRegionSize` if its set, else the
computed value for `avgRegionSize` is in MBs. So its an inference.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sandeepvinayak commented on a change in pull request #3018: HBASE-25627: HBase replication should have a metric to represent if the source is stuck getting initialized
sandeepvinayak commented on a change in pull request #3018: URL: https://github.com/apache/hbase/pull/3018#discussion_r594786984 ## File path: hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java ## @@ -126,6 +129,9 @@ public MetricsReplicationSourceSourceImpl(MetricsReplicationSourceImpl rms, Stri oldestWalAgeKey = this.keyPrefix + "oldestWalAge"; oldestWalAge = rms.getMetricsRegistry().getGauge(oldestWalAgeKey, 0L); + +sourceInitializingKey = this.keyPrefix + "numInitializing"; Review comment: yeah, I think for global level, it should be numInitializing and the source level should be isInitializing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache9 commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-799859021 > This looks great. > > What are the implications? Add them as release note. There is the xml format for configuring. What else? Call out your exclude of log4j1. It looks like UI loglevel still works. That's sweet. I was hoping we could back port this to branch-2 as log4j1 has been dead for a long time... We could start a discussion thread on de mailing list to see others opinion? And I think we could also ask the hadoop community to upgrade to log4j2 too, and also, remove the dependencies on log4j or log4j2 when publishing libraries which will be depended by others. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache9 commented on a change in pull request #1708:
URL: https://github.com/apache/hbase/pull/1708#discussion_r594784683
##
File path: bin/hbase
##
@@ -775,25 +783,31 @@ HBASE_OPTS="$HBASE_OPTS -Dhbase.log.dir=$HBASE_LOG_DIR"
HBASE_OPTS="$HBASE_OPTS -Dhbase.log.file=$HBASE_LOGFILE"
HBASE_OPTS="$HBASE_OPTS -Dhbase.home.dir=$HBASE_HOME"
HBASE_OPTS="$HBASE_OPTS -Dhbase.id.str=$HBASE_IDENT_STRING"
-HBASE_OPTS="$HBASE_OPTS -Dhbase.root.logger=${HBASE_ROOT_LOGGER:-INFO,console}"
+HBASE_OPTS="$HBASE_OPTS
-Dhbase.root.logger.level=${HBASE_ROOT_LOGGER_LEVEL:-INFO}"
+HBASE_OPTS="$HBASE_OPTS
-Dhbase.root.logger.appender=${HBASE_ROOT_LOGGER_APPENDER:-console}"
Review comment:
Let me try...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache9 commented on a change in pull request #1708:
URL: https://github.com/apache/hbase/pull/1708#discussion_r594784640
##
File path: bin/hbase
##
@@ -658,7 +663,10 @@ elif [ "$COMMAND" = "mapredcp" ] ; then
for f in "${HBASE_HOME}"/lib/client-facing-thirdparty/*.jar; do
if [[ ! "${f}" =~ ^.*/htrace-core-3.*\.jar$ ]] && \
[ "${f}" != "htrace-core.jar$" ] && \
- [[ ! "${f}" =~ ^.*/slf4j-log4j.*$ ]]; then
+ [[ ! "${f}" =~ ^.*/log4j.*$ ]] && \
+ [[ ! "${f}" =~ ^.*/slf4j.*$ ]] && \
+ [[ ! "${f}" =~ ^.*/jcl-over-slf4j.*$ ]] && \
+ [[ ! "${f}" =~ ^.*/jul-to-slf4j.*$ ]]; then
Review comment:
Let me check this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3052: HBASE-25639 meta replica state is not respected during active master …
Apache-HBase commented on pull request #3052: URL: https://github.com/apache/hbase/pull/3052#issuecomment-799856806 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 16s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | branch-2.3 passed | | +1 :green_heart: | compile | 3m 11s | branch-2.3 passed | | +1 :green_heart: | checkstyle | 1m 22s | branch-2.3 passed | | +1 :green_heart: | spotbugs | 2m 8s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 3m 14s | the patch passed | | +1 :green_heart: | javac | 3m 14s | the patch passed | | -0 :warning: | checkstyle | 1m 14s | hbase-server: The patch generated 3 new + 103 unchanged - 0 fixed = 106 total (was 103) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 46s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 49m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3052/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3052 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 411f2b6848ec 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 1a695e872b | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3052/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 86 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3052/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25184) Move RegionLocationFinder to hbase-balancer
[ https://issues.apache.org/jira/browse/HBASE-25184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302132#comment-17302132 ] Duo Zhang commented on HBASE-25184: --- RegionLocationFinder is only used by balancer related classes. It will calculate the size of storefiles located locally for a region, and we will use this as a rank when computing region plans. Thanks. > Move RegionLocationFinder to hbase-balancer > --- > > Key: HBASE-25184 > URL: https://issues.apache.org/jira/browse/HBASE-25184 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3043: HBASE-25568 Upgrade Thrift jar to fix CVE-2020-13949
Apache-HBase commented on pull request #3043: URL: https://github.com/apache/hbase/pull/3043#issuecomment-799854911 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 49s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | compile | 2m 50s | master passed | | +1 :green_heart: | shadedjars | 6m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 2m 56s | the patch passed | | +1 :green_heart: | javac | 2m 56s | the patch passed | | +1 :green_heart: | shadedjars | 6m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 171m 26s | root in the patch passed. | | | | 211m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3043/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3043 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c42deb5cef4d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f4e1ab7b1d | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3043/3/testReport/ | | Max. process+thread count | 7156 (vs. ulimit of 3) | | modules | C: hbase-thrift hbase-examples . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3043/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3050: HBASE-25662 Fix spotbugs warning in RoundRobinTableInputFormat
Apache9 commented on pull request #3050: URL: https://github.com/apache/hbase/pull/3050#issuecomment-799854070 IIRC in the current java world, you should never make use of LinkedList. It will be slower than ArrayList for most cases. And if taking from the first element does matter, we could make use of ArrayQueue or ArrayDeque. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2352: HBASE-24925 SCP reduce unnecessary get requests
saintstack commented on a change in pull request #2352:
URL: https://github.com/apache/hbase/pull/2352#discussion_r594777019
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
##
@@ -318,4 +326,39 @@ protected void deleteZooKeeper(TableName tableName) {
LOG.warn("Failed deleting table state from zookeeper", e);
}
}
+
+ public void asyncLoadTableState() {
+new Thread(() -> {
+ LOG.info("load table state...");
+ ThreadFactoryBuilder builder = new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("TableStateFetcher" +
"-%1$d");
+ ExecutorService
+executorService =
Executors.newFixedThreadPool(master.getConfiguration().getInt("hbase.table.state.fetcher.threads",10),
builder.build());
+
+ try {
+List results =
MetaTableAccessor.fullScan(master.getConnection(),
ClientMetaTableAccessor.QueryType.TABLE);
+List> futures = new ArrayList<>();
+results.forEach(r -> futures.add(executorService.submit(new
Callable() {
Review comment:
How much of a diff does using the executor make?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -1556,6 +1556,7 @@ public RegionInfo loadRegionFromMeta(String
regionEncodedName) throws UnknownReg
}
private void loadMeta() throws IOException {
+master.getTableStateManager().asyncLoadTableState();
Review comment:
The 'next work' will be a new PR or another version of this patch?
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache-HBase commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-799848896 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 51s | master passed | | +1 :green_heart: | compile | 2m 36s | master passed | | +1 :green_heart: | shadedjars | 7m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 10m 19s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 57s | the patch passed | | +1 :green_heart: | compile | 2m 42s | the patch passed | | +1 :green_heart: | javac | 2m 42s | the patch passed | | +1 :green_heart: | shadedjars | 8m 22s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 10m 28s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 358m 50s | root in the patch failed. | | | | 416m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1708 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2bf8fc2059e6 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8337fb2a72 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/9/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/9/testReport/ | | Max. process+thread count | 3016 (vs. ulimit of 3) | | modules | C: hbase-logging hbase-common hbase-metrics-api hbase-metrics hbase-hadoop-compat hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded hbase-shaded/hbase-shaded-client hbase-shaded/hbase-shaded-client-byo-hadoop hbase-shaded/hbase-shaded-mapreduce hbase-hbtop hbase-assembly hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-testing-util-tester hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/9/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
saintstack commented on pull request #2445: URL: https://github.com/apache/hbase/pull/2445#issuecomment-799847038 Closing out draft. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack closed pull request #2445: HBASE-11288 Splittable Meta (WIP) Draft
saintstack closed pull request #2445: URL: https://github.com/apache/hbase/pull/2445 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2469: HBASE-25112 Add new procedure metrics for master UI only
saintstack commented on pull request #2469: URL: https://github.com/apache/hbase/pull/2469#issuecomment-799846799 Looks good. Whats it add to the UI @Joseph295 ? Put an image up on the JIRA? What failed here? Want me to rerun the PR so you can see what failed (not there any more it seems). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2562: HBASE-25199 Remove deprecated HStore#getStoreHomedir methods
saintstack commented on pull request #2562: URL: https://github.com/apache/hbase/pull/2562#issuecomment-799844904 Any update here @HorizonNet ? Thanks boss. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2626: HBASE-25029 Resolve the TODO of AssignmentManager's loadMeta() method.
saintstack commented on pull request #2626: URL: https://github.com/apache/hbase/pull/2626#issuecomment-799843476 Any update here @sanjeetnishad95 ? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2663: HBASE-24637 - Reseek regression related to filter SKIP hinting
saintstack commented on pull request #2663: URL: https://github.com/apache/hbase/pull/2663#issuecomment-799840766 @ramkrish86 Any update here sir? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24395) ServerName#getHostname() is case sensitive
[ https://issues.apache.org/jira/browse/HBASE-24395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-24395: -- Status: In Progress (was: Patch Available) > ServerName#getHostname() is case sensitive > -- > > Key: HBASE-24395 > URL: https://issues.apache.org/jira/browse/HBASE-24395 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Affects Versions: 1.3.1 >Reporter: Bo Cui >Assignee: Bo Cui >Priority: Major > Attachments: HBase-24395.patch, image-2020-05-18-17-42-57-119.png > > > ServerName calss，the getServerName(String hostName, int port, long > startcode),equals and compareTo are case insensitive, but getHostname() is > case sensitive. > if hostName is HOSTNAME1, ServerName is hostname1,1,1589615319931, and > getHostname() returns HOSTNAME1. > and then BaseLoadBalancer#retainAssignment() uses ServerName#getHostname(), > all keys of serversByHostname are > upperCase(HOSTNAME1,HOSTNAME2,HOSTNAME3,HOSTNAME4...) from > ServerManager#createDestinationServersList, but oldServerName.getHostname() > is lowerCase(hostname1,hostname2,hostname3...) from walLog dir. > !image-2020-05-18-17-42-57-119.png! > and finally...all region of old ServerName will be assigned to random hosts -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24395) ServerName#getHostname() is case sensitive
[ https://issues.apache.org/jira/browse/HBASE-24395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-24395. --- Fix Version/s: 1.7.0 Hadoop Flags: Reviewed Resolution: Fixed Merged to branch-1. Thanks for patch [~Bo Cui] and review [~pankajkumar] > ServerName#getHostname() is case sensitive > -- > > Key: HBASE-24395 > URL: https://issues.apache.org/jira/browse/HBASE-24395 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Affects Versions: 1.3.1 >Reporter: Bo Cui >Assignee: Bo Cui >Priority: Major > Fix For: 1.7.0 > > Attachments: HBase-24395.patch, image-2020-05-18-17-42-57-119.png > > > ServerName calss，the getServerName(String hostName, int port, long > startcode),equals and compareTo are case insensitive, but getHostname() is > case sensitive. > if hostName is HOSTNAME1, ServerName is hostname1,1,1589615319931, and > getHostname() returns HOSTNAME1. > and then BaseLoadBalancer#retainAssignment() uses ServerName#getHostname(), > all keys of serversByHostname are > upperCase(HOSTNAME1,HOSTNAME2,HOSTNAME3,HOSTNAME4...) from > ServerManager#createDestinationServersList, but oldServerName.getHostname() > is lowerCase(hostname1,hostname2,hostname3...) from walLog dir. > !image-2020-05-18-17-42-57-119.png! > and finally...all region of old ServerName will be assigned to random hosts -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #2690: HBASE-24395 Backport HBASE-20589 to branch-1
saintstack merged pull request #2690: URL: https://github.com/apache/hbase/pull/2690 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack closed pull request #2729: Remove the use of term segregate
saintstack closed pull request #2729: URL: https://github.com/apache/hbase/pull/2729 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2729: Remove the use of term segregate
saintstack commented on pull request #2729: URL: https://github.com/apache/hbase/pull/2729#issuecomment-799838050 Closing incomplete (if a good idea) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack closed pull request #2747: HBASE-24850 CellComparator perf improvement
saintstack closed pull request #2747: URL: https://github.com/apache/hbase/pull/2747 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2747: HBASE-24850 CellComparator perf improvement
saintstack commented on pull request #2747: URL: https://github.com/apache/hbase/pull/2747#issuecomment-799837711 Closing. You can read the comments here even if closed @ramkrish86 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25374) Make REST Client connection and socket time out configurable
[ https://issues.apache.org/jira/browse/HBASE-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302103#comment-17302103 ] Michael Stack commented on HBASE-25374: --- Merged to branch-2.3+ for [~pankajkumar] > Make REST Client connection and socket time out configurable > > > Key: HBASE-25374 > URL: https://issues.apache.org/jira/browse/HBASE-25374 > Project: HBase > Issue Type: Improvement > Components: REST >Affects Versions: 3.0.0-alpha-1, 2.3.4, 2.5.0, 2.4.3 >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3 > > > Currently there is no way to configure connection and socket timeout for REST > client, > https://github.com/apache/hbase/blob/7d0a687e5798a2f4ca3190b409169f7e17a75b34/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/Client.java#L102 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] huaxiangsun opened a new pull request #3052: HBASE-25639 meta replica state is not respected during active master …
huaxiangsun opened a new pull request #3052: URL: https://github.com/apache/hbase/pull/3052 …switch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25374) Make REST Client connection and socket time out configurable
[ https://issues.apache.org/jira/browse/HBASE-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25374. --- Fix Version/s: 2.4.3 2.3.5 2.5.0 Hadoop Flags: Reviewed Release Note: Configuration parameter to set rest client connection timeout "hbase.rest.client.conn.timeout" Default is 2 * 1000 "hbase.rest.client.socket.timeout" Default of 30 * 1000 Resolution: Fixed > Make REST Client connection and socket time out configurable > > > Key: HBASE-25374 > URL: https://issues.apache.org/jira/browse/HBASE-25374 > Project: HBase > Issue Type: Improvement > Components: REST >Affects Versions: 3.0.0-alpha-1, 2.3.4, 2.5.0, 2.4.3 >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3 > > > Currently there is no way to configure connection and socket timeout for REST > client, > https://github.com/apache/hbase/blob/7d0a687e5798a2f4ca3190b409169f7e17a75b34/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/Client.java#L102 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25374) Make REST Client connection and socket time out configurable
[ https://issues.apache.org/jira/browse/HBASE-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25374: -- Affects Version/s: 2.4.3 2.5.0 3.0.0-alpha-1 2.3.4 Status: In Progress (was: Patch Available) > Make REST Client connection and socket time out configurable > > > Key: HBASE-25374 > URL: https://issues.apache.org/jira/browse/HBASE-25374 > Project: HBase > Issue Type: Improvement > Components: REST >Affects Versions: 2.3.4, 3.0.0-alpha-1, 2.5.0, 2.4.3 >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 3.0.0-alpha-1 > > > Currently there is no way to configure connection and socket timeout for REST > client, > https://github.com/apache/hbase/blob/7d0a687e5798a2f4ca3190b409169f7e17a75b34/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/Client.java#L102 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #2752: HBASE-25374 Make REST Client connection and socket time out configurable
saintstack merged pull request #2752: URL: https://github.com/apache/hbase/pull/2752 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] mnpoonia commented on pull request #2991: HBASE-25593 Backport changes from HBASE-24418 to branch-1
mnpoonia commented on pull request #2991: URL: https://github.com/apache/hbase/pull/2991#issuecomment-799834899 @virajjasani Can you help me review this in your free cycles This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2769: HBASE-25032 Wait for region server to become online before adding it to online servers in Master
saintstack commented on pull request #2769: URL: https://github.com/apache/hbase/pull/2769#issuecomment-799834561 Thank you @caroliney14 . This is a nice patch. Would be great to land it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2780: HBASE-25301 NPE while running balance_rsgroup if any split region is …
saintstack commented on a change in pull request #2780:
URL: https://github.com/apache/hbase/pull/2780#discussion_r594762484
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
##
@@ -1127,7 +1128,9 @@ private boolean isTableInGroup(TableName tableName,
String groupName,
.isTableState(tn, TableState.State.DISABLED,
TableState.State.DISABLING)) {
continue;
}
-if (region.isSplitParent()) {
+if (
+
regionStates.getOrCreateRegionStateNode(region).getState().equals(RegionState.State.SPLIT)
Review comment:
getOrCreateRegionStateNode could have a side-effect when there is no
RegionStateNode for 'region'. In this case, we'll add to RegionState a new RSN
for 'region' if none present Do we want to do this?
You make the above call because you don't have access to the internal
createRegionStateNode(regionInfo ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #3001: HBASE-25518 Support separate child regions to different region servers
saintstack commented on a change in pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#discussion_r594754063
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManagerUtil.java
##
@@ -189,6 +190,67 @@ private static void unlock(List
regionNodes) {
return ArrayUtils.addAll(primaryRegionProcs, replicaRegionAssignProcs);
}
+ /**
+ * Create round robin assign procedures for the give regions,
Review comment:
s/give/given/
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionSplit.java
##
@@ -146,8 +152,70 @@ public void testSplitTableRegion() throws Exception {
UTIL.getAdmin().enableTable(tableName);
Thread.sleep(500);
-assertEquals("Table region not correct.", 2,
-UTIL.getHBaseCluster().getRegions(tableName).size());
+List tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+assertEquals("Table region not correct.", 2, tableRegions.size());
+Map regionInfoMap =
UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+assertEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
+ regionInfoMap.get(tableRegions.get(1).getRegionInfo()));
+ }
+
+ @Test
+ public void testSplitTableRegionAndSeparateChildRegions() throws Exception {
+cleanupTest();
+//restart master
+
UTIL.getConfiguration().setBoolean(HConstants.HBASE_ENABLE_SEPARATE_CHILD_REGIONS,
true);
+setupCluster();
+
+final TableName tableName = TableName.valueOf(name.getMethodName());
+final ProcedureExecutor procExec =
getMasterProcedureExecutor();
+
+RegionInfo[] regions =
+ MasterProcedureTestingUtility.createTable(procExec, tableName, null,
ColumnFamilyName);
+insertData(tableName);
+int splitRowNum = startRowNum + rowCount / 2;
+byte[] splitKey = Bytes.toBytes("" + splitRowNum);
+
+assertTrue("not able to find a splittable region", regions != null);
+assertTrue("not able to find a splittable region", regions.length == 1);
+
+// Split region of the table
+long procId = procExec.submitProcedure(
+ new SplitTableRegionProcedure(procExec.getEnvironment(), regions[0],
splitKey));
+// Wait the completion
+ProcedureTestingUtility.waitProcedure(procExec, procId);
+ProcedureTestingUtility.assertProcNotFailed(procExec, procId);
+
+assertTrue("not able to split table",
UTIL.getHBaseCluster().getRegions(tableName).size() == 2);
+
+//disable table
+UTIL.getAdmin().disableTable(tableName);
+Thread.sleep(500);
+
+//stop master
+UTIL.getHBaseCluster().stopMaster(0);
+UTIL.getHBaseCluster().waitOnMaster(0);
+Thread.sleep(500);
+
+//restart master
+JVMClusterUtil.MasterThread t = UTIL.getHBaseCluster().startMaster();
+Thread.sleep(500);
+
+UTIL.invalidateConnection();
+// enable table
+UTIL.getAdmin().enableTable(tableName);
+Thread.sleep(500);
+
+List tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+assertEquals("Table region not correct.", 2, tableRegions.size());
+Map regionInfoMap =
UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+assertNotEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
Review comment:
How can you be sure the roundrobin will not assign both daughters to
same server?
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionSplit.java
##
@@ -146,8 +152,70 @@ public void testSplitTableRegion() throws Exception {
UTIL.getAdmin().enableTable(tableName);
Thread.sleep(500);
-assertEquals("Table region not correct.", 2,
-UTIL.getHBaseCluster().getRegions(tableName).size());
+List tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+assertEquals("Table region not correct.", 2, tableRegions.size());
+Map regionInfoMap =
UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+assertEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
+ regionInfoMap.get(tableRegions.get(1).getRegionInfo()));
+ }
+
+ @Test
+ public void testSplitTableRegionAndSeparateChildRegions() throws Exception {
+cleanupTest();
Review comment:
You should make a new test suite with this test in it rather than do
this cleanupTest and setupCluster... These methods are for before the test
class and after... not for calling in the test as here.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManagerUtil.java
##
@@ -189,6 +190,67 @@ private static void unlock(List
regionNodes) {
return ArrayUtils.addAll(primaryRegionProcs, replicaRegionAssignProcs);
}
+ /**
+ * Create round robin assign procedures for the give
[jira] [Commented] (HBASE-25184) Move RegionLocationFinder to hbase-balancer
[ https://issues.apache.org/jira/browse/HBASE-25184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302084#comment-17302084 ] Michael Stack commented on HBASE-25184: --- hbase-balancer doesn't seem like right place for RegionLocationFinder? What you thinking [~zhangduo] > Move RegionLocationFinder to hbase-balancer > --- > > Key: HBASE-25184 > URL: https://issues.apache.org/jira/browse/HBASE-25184 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25597) Add row info in Exception when cell size exceeds maxCellSize
[
https://issues.apache.org/jira/browse/HBASE-25597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-25597.
---
Fix Version/s: 2.4.3
2.3.5
2.5.0
3.0.0-alpha-1
Hadoop Flags: Reviewed
Resolution: Fixed
Helpful. Thanks for the PR [~DeanZ]. Merged to 2.3+
> Add row info in Exception when cell size exceeds maxCellSize
>
>
> Key: HBASE-25597
> URL: https://issues.apache.org/jira/browse/HBASE-25597
> Project: HBase
> Issue Type: Improvement
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
>
> When cell size exceeds maxCellSize(default is 10M), client will get a
> DoNotRetryIOException, code as below:
> {code:java}
> private void checkCellSizeLimit(final HRegion r, final Mutation m) throws
> IOException {
> if (r.maxCellSize > 0) {
> CellScanner cells = m.cellScanner();
> while (cells.advance()) {
> int size = PrivateCellUtil.estimatedSerializedSizeOf(cells.current());
> if (size > r.maxCellSize) {
> String msg = "Cell with size " + size + " exceeds limit of " +
> r.maxCellSize + " bytes";
> LOG.debug(msg);
> throw new DoNotRetryIOException(msg);
> }
> }
> }
> }
> {code}
> There is no row related information, which makes troubleshooting difficult.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #2976: HBASE-25597 Add row info in Exception when cell size exceeds maxCellSize
saintstack merged pull request #2976: URL: https://github.com/apache/hbase/pull/2976 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
