[GitHub] [hbase] bsglz commented on a change in pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
bsglz commented on a change in pull request #2254:
URL: https://github.com/apache/hbase/pull/2254#discussion_r470424739
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
##
@@ -508,35 +508,57 @@ default void postAssign(final
ObserverContext ctx,
* Called prior to unassigning a given region.
* @param ctx the environment to interact with the framework and master
* @param regionInfo
- * @param force whether to force unassignment or not
*/
default void preUnassign(final ObserverContext
ctx,
- final RegionInfo regionInfo, final boolean force) throws IOException {}
+final RegionInfo regionInfo) throws IOException {}
Review comment:
Will just remove them, since it is IA.LimitedPrivate. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] infraio merged pull request #2257: HBASE-24879 Fix the shellcheck download url
infraio merged pull request #2257: URL: https://github.com/apache/hbase/pull/2257 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
bsglz commented on a change in pull request #2254:
URL: https://github.com/apache/hbase/pull/2254#discussion_r470423500
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
##
@@ -508,35 +508,57 @@ default void postAssign(final
ObserverContext ctx,
* Called prior to unassigning a given region.
* @param ctx the environment to interact with the framework and master
* @param regionInfo
- * @param force whether to force unassignment or not
*/
default void preUnassign(final ObserverContext
ctx,
- final RegionInfo regionInfo, final boolean force) throws IOException {}
+final RegionInfo regionInfo) throws IOException {}
/**
* Called after the region unassignment has been requested.
* @param ctx the environment to interact with the framework and master
* @param regionInfo
+ */
+ default void postUnassign(final
ObserverContext ctx,
+final RegionInfo regionInfo) throws IOException {}
+
+ /**
+ * Called prior to unassigning a given region.
+ * @param ctx the environment to interact with the framework and master
+ * @param regionInfo region info
+ * @param force whether to force unassignment or not
+ * @deprecated since 2.4.0 and will be removed in 4.0.0.
+ * Use {@link #preUnassign(ObserverContext, RegionInfo)} instead.
+ */
+ @Deprecated
+ default void preUnassign(final ObserverContext
ctx,
+final RegionInfo regionInfo, final boolean force) throws IOException {}
+
+ /**
+ * Called after the region unassignment has been requested.
+ * @param ctx the environment to interact with the framework and master
+ * @param regionInfo region info
* @param force whether to force unassignment or not
+ * @deprecated since 2.4.0 and will be removed in 4.0.0.
Review comment:
Make sense, let's remove it. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
bsglz commented on a change in pull request #2254:
URL: https://github.com/apache/hbase/pull/2254#discussion_r470423388
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
##
@@ -508,35 +508,57 @@ default void postAssign(final
ObserverContext ctx,
* Called prior to unassigning a given region.
* @param ctx the environment to interact with the framework and master
* @param regionInfo
- * @param force whether to force unassignment or not
*/
default void preUnassign(final ObserverContext
ctx,
- final RegionInfo regionInfo, final boolean force) throws IOException {}
+final RegionInfo regionInfo) throws IOException {}
/**
* Called after the region unassignment has been requested.
* @param ctx the environment to interact with the framework and master
* @param regionInfo
+ */
+ default void postUnassign(final
ObserverContext ctx,
+final RegionInfo regionInfo) throws IOException {}
+
+ /**
+ * Called prior to unassigning a given region.
+ * @param ctx the environment to interact with the framework and master
+ * @param regionInfo region info
+ * @param force whether to force unassignment or not
+ * @deprecated since 2.4.0 and will be removed in 4.0.0.
Review comment:
Make sense, let's remove it. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
bsglz commented on a change in pull request #2254:
URL: https://github.com/apache/hbase/pull/2254#discussion_r470422725
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
##
@@ -591,7 +596,11 @@
* @param forcible If true, force unassign (Will remove region from
regions-in-transition too if
* present. If results in double assignment use hbck -fix to
resolve. To be used by
* experts).
+ * @deprecated since 2.4.0 and will be removed in 4.0.0. Use {@link
#unassign(byte[])}
+ * instead.
+ * @see https://issues.apache.org/jira/browse/HBASE-24875;>HBASE-24875
*/
+ @Deprecated
CompletableFuture unassign(byte[] regionName, boolean forcible);
Review comment:
Will fix later. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
bsglz commented on a change in pull request #2254:
URL: https://github.com/apache/hbase/pull/2254#discussion_r470422377
##
File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
##
@@ -784,9 +791,12 @@ default void move(byte[] encodedRegionName, byte[]
destServerName) throws IOExce
* @param force If true, force unassign (Will remove region
from regions-in-transition too if
* present. If results in double assignment use hbck -fix to resolve. To be
used by experts).
* @throws IOException if a remote or network exception occurs
+ * @deprecated since 2.4.0 and will be removed in 4.0.0. Use {@link
#unassign(byte[])}
+ * instead.
+ * @see https://issues.apache.org/jira/browse/HBASE-24875;>HBASE-24875
*/
- void unassign(byte[] regionName, boolean force)
- throws IOException;
+ @Deprecated
+ void unassign(byte[] regionName, boolean force) throws IOException;
Review comment:
Will fix later. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
Apache9 commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673896406 The inconsistency is between the data on filesystem and zookeeper. In general, after the meta table has been initialized, it should have a directory on the file system, and also a znode to indicate its state on zookeeper. The problem here is that, in your scenario, the data on zookeeper are all gone. This is what I called 'inconsistency'. And I've also mentioned several times, it works for several versions does not mean it will work for all the versions. They are just different implementation choices, and you can not force the developpers to not use the data on zookeeper to determine what to do next. So I do not think it worth to ask on the dev list about 'whether' the InitMetaProcedure should delete meta without checking 'partial' or not. The problem here is you delete the state znode for meta region on zookeeper, not a problem for InitMetaProcedure. Either you add it back through HBCK2, or you should have a overall design on how we could fix the problem, without using HBCK2. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] taklwu edited a comment on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
taklwu edited a comment on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673880783 > how we could start a cluster with no data on zookeeper? IMO the title of the google design may be going on the cloud use cases that has been restarting on just HFiles without WAL and without Zookeeper (but all the user tables are flushed and disabled before terminating it). I knew that may not be mentioned in the [book tutorial](https://hbase.apache.org/book.html), and it may be a good time to clarify how that cases are actually working and some users has been using in HBase-1.4.x and HBase maybe before 2.1.7. Then we can see what the gaps maybe now in [updated]branch-2.4+ (we don't need that in 2.3.0 and 2.2.5 becase `InitMetaProcedure` does not have `INIT_META_WRITE_FS_LAYOUT`) to support it back (basically, that's the intention of having this PR and [PR#2113](https://github.com/apache/hbase/pull/2113) ) > As you want to start the HMaster and recover from the inconsistency what does `inconsistency` mean here? I see your point that using `InitMetaProcedure#INIT_META_WRITE_FS_LAYOUT` to `indicate` inconsistency, but if we don't delete meta and just starting the cluster, IMO HBCK `-detail` will show a clean result without any inconsistency? (correct me if I'm wrong) . For this topic, I may just start a email on if `InitMetaProcedure` should delete meta without checking `partial` and `consistency`, please bear with me, and this may be the only thing I want a quick discussion instead of a long design doc on the cloud use cases. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2257: HBASE-24879 Fix the shellcheck download url
Apache-HBase commented on pull request #2257: URL: https://github.com/apache/hbase/pull/2257#issuecomment-673892836 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 54s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | hadolint | 0m 0s | hadolint was not available. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.2 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 4m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2257/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2257 | | Optional Tests | dupname asflicense hadolint shellcheck shelldocs | | uname | Linux b678f0ad0811 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2257/out/precommit/personality/provided.sh | | git revision | branch-2.2 / 882d2ae861 | | Max. process+thread count | 46 (vs. ulimit of 1) | | modules | C: U: | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2257/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2257: HBASE-24879 Fix the shellcheck download url
Apache-HBase commented on pull request #2257: URL: https://github.com/apache/hbase/pull/2257#issuecomment-673891881 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2257/1/console in case of problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
Apache-HBase commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673890844 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 6s | master passed | | +1 :green_heart: | compile | 2m 30s | master passed | | +1 :green_heart: | shadedjars | 5m 45s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 26s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. | | -0 :warning: | javadoc | 0m 51s | hbase-thrift in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 2m 30s | the patch passed | | +1 :green_heart: | javac | 2m 30s | the patch passed | | +1 :green_heart: | shadedjars | 5m 41s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 24s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 50s | hbase-thrift in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 132m 34s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 4m 25s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 23s | hbase-shell in the patch passed. | | | | 178m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2254 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9b69beb999c3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-thrift.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-thrift.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/testReport/ | | Max. process+thread count | 3760 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] taklwu edited a comment on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
taklwu edited a comment on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673880783 > how we could start a cluster with no data on zookeeper? IMO the title of the google design may be going on the cloud use cases that has been restarting on just HFiles without WAL and without Zookeeper (but all the user tables are flushed and disabled before terminating it). I knew that may not be mentioned in the [book tutorial](https://hbase.apache.org/book.html), and it may be a good time to clarify how that cases are actually working and some users has been using in HBase-1.4.x and HBase maybe before 2.1.7. Then we can see what the gaps maybe now in branch-2.2+ to support it back (basically, that's the intention of having this PR and [PR#2113](https://github.com/apache/hbase/pull/2113) ) > As you want to start the HMaster and recover from the inconsistency what does `inconsistency` mean here? I see your point that using `InitMetaProcedure#INIT_META_WRITE_FS_LAYOUT` to `indicate` inconsistency, but if we don't delete meta and just starting the cluster, IMO HBCK `-detail` will show a clean result without any inconsistency? (correct me if I'm wrong) . For this topic, I may just start a email on if `InitMetaProcedure` should delete meta without checking `partial` and `consistency`, please bear with me, and this may be the only thing I want a quick discussion instead of a long design doc on the cloud use cases. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24880) Remove ReplicationPeerConfigUpgrader
[ https://issues.apache.org/jira/browse/HBASE-24880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177482#comment-17177482 ] niuyulin commented on HBASE-24880: -- let me remove it > Remove ReplicationPeerConfigUpgrader > > > Key: HBASE-24880 > URL: https://issues.apache.org/jira/browse/HBASE-24880 > Project: HBase > Issue Type: Task > Components: Replication >Reporter: Duo Zhang >Priority: Major > > The comment says it will be removed in 3.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] busbey commented on a change in pull request #2215: HBASE-24627 Normalize one table at a time
busbey commented on a change in pull request #2215:
URL: https://github.com/apache/hbase/pull/2215#discussion_r470408344
##
File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
##
@@ -856,11 +856,23 @@ void unassign(byte[] regionName, boolean force)
* the request was submitted successfully. We need to check logs for the
details of which regions
* were split/merged.
*
- * @return true if region normalizer ran, false
otherwise.
+ * @return {@code true} if region normalizer ran, {@code false} otherwise.
* @throws IOException if a remote or network exception occurs
*/
boolean normalize() throws IOException;
Review comment:
nit: could we add a default method here that calls
`normalize(NormalizeTableFilterParams)` with an empty set of NTFparams? seems
like we could clean up some stuff in the various *Admin implementations and
`RequestConverter`
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
##
@@ -1281,6 +1281,14 @@
*/
CompletableFuture normalize();
Review comment:
nit: could we add a default method here that calls
`normalize(NormalizeTableFilterParams)` with an empty set of NTFparams? seems
like we could clean up some stuff in the various *Admin implementations and
`RequestConverter`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #2215: HBASE-24627 Normalize one table at a time
busbey commented on a change in pull request #2215:
URL: https://github.com/apache/hbase/pull/2215#discussion_r470408396
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
##
@@ -1281,6 +1281,14 @@
*/
CompletableFuture normalize();
Review comment:
nit: could we add a default method here that calls
`normalize(NormalizeTableFilterParams)` with a default one? seems like we could
clean up some stuff in the various *Admin implementations and `RequestConverter`
##
File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
##
@@ -856,11 +856,23 @@ void unassign(byte[] regionName, boolean force)
* the request was submitted successfully. We need to check logs for the
details of which regions
* were split/merged.
*
- * @return true if region normalizer ran, false
otherwise.
+ * @return {@code true} if region normalizer ran, {@code false} otherwise.
* @throws IOException if a remote or network exception occurs
*/
boolean normalize() throws IOException;
Review comment:
nit: could we add a default method here that calls
`normalize(NormalizeTableFilterParams)` with a default one? seems like we could
clean up some stuff in the various *Admin implementations and `RequestConverter`
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/NormalizeTableFilterParams.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import java.util.List;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.yetus.audience.InterfaceStability;
+
+/**
+ * A collection of criteria used for table selection. The logic of table
selection is as follows:
+ *
+ *
+ * When no parameter values are provided, an unfiltered list of all user
tables is returned.
+ *
+ *
+ * When a list of {@link TableName TableNames} are provided, the filter
starts with any of
+ * these tables that exist.
+ *
+ *
+ * When a {@code namespace} name is provided, the filter starts with all
the tables present in
+ * that namespace.
+ *
+ *
+ * If both a list of {@link TableName TableNames} and a {@code namespace}
name are provided,
+ * the {@link TableName} list is honored and the {@code namespace} name is
ignored.
+ *
+ *
+ * If a {@code regex} is provided, this subset of {@link TableName
TableNames} is further
+ * reduced to those that match the provided regular expression.
+ *
+ *
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
Review comment:
nit: our [API surface
docs](http://hbase.apache.org/book.html#hbase.client.api.surface) say that
anything labeled IA.Public is inherently IS.Stable and shouldn't carry a label.
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/NormalizeTableFilterParams.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import java.util.List;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.yetus.audience.InterfaceStability;
+
+/**
+ * A collection of criteria used for table selection. The logic of table
selection is as follows:
+ *
+ *
+ * When no
[GitHub] [hbase] taklwu commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
taklwu commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673880783 > how we could start a cluster with no data on zookeeper? IMO the title of the google design may be going on the cloud use cases that has been restarting on just HFiles without WAL and without Zookeeper (but all the user tables are flushed and disabled before terminating it). I knew that may not be mentioned in the [book tutorial](https://hbase.apache.org/book.html), and it may be a good time to clarify how that cases are actually working and some users has been using in HBase-1.4.x and HBase maybe before 2.1.7. Then we can see what the gaps maybe now in branch-2.2+ to support it back (basically, that's the intention of having this PR and [PR#2113](https://github.com/apache/hbase/pull/2113) ) > As you want to start the HMaster and recover from the inconsistency what does `inconsistency` mean here? I see your point that using `InitMetaProcedure#INIT_META_WRITE_FS_LAYOUT` to `indicate` inconsistency, but if we don't delete meta and just starting the cluster, IMO HBCK `-detail` will show a clean result without any inconsistency? we may not hit any `inconsistency` when getting into `InitMetaProcedure`. For this topic, I may just start a email on if `InitMetaProcedure` should delete meta without checking `partial` and `consistency`, please bear with me, and this may be the only thing I want a quick discussion instead of a long design doc on the cloud use cases. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
busbey commented on a change in pull request #2254:
URL: https://github.com/apache/hbase/pull/2254#discussion_r470395358
##
File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
##
@@ -784,9 +791,12 @@ default void move(byte[] encodedRegionName, byte[]
destServerName) throws IOExce
* @param force If true, force unassign (Will remove region
from regions-in-transition too if
* present. If results in double assignment use hbck -fix to resolve. To be
used by experts).
* @throws IOException if a remote or network exception occurs
+ * @deprecated since 2.4.0 and will be removed in 4.0.0. Use {@link
#unassign(byte[])}
+ * instead.
+ * @see https://issues.apache.org/jira/browse/HBASE-24875;>HBASE-24875
*/
- void unassign(byte[] regionName, boolean force)
- throws IOException;
+ @Deprecated
+ void unassign(byte[] regionName, boolean force) throws IOException;
Review comment:
use a default method here to call `unassign(b[])` so that our various
implementations can skip including the same.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
##
@@ -508,35 +508,57 @@ default void postAssign(final
ObserverContext ctx,
* Called prior to unassigning a given region.
* @param ctx the environment to interact with the framework and master
* @param regionInfo
- * @param force whether to force unassignment or not
*/
default void preUnassign(final ObserverContext
ctx,
- final RegionInfo regionInfo, final boolean force) throws IOException {}
+final RegionInfo regionInfo) throws IOException {}
/**
* Called after the region unassignment has been requested.
* @param ctx the environment to interact with the framework and master
* @param regionInfo
+ */
+ default void postUnassign(final
ObserverContext ctx,
+final RegionInfo regionInfo) throws IOException {}
+
+ /**
+ * Called prior to unassigning a given region.
+ * @param ctx the environment to interact with the framework and master
+ * @param regionInfo region info
+ * @param force whether to force unassignment or not
+ * @deprecated since 2.4.0 and will be removed in 4.0.0.
Review comment:
This class is IA.LimitedPrivate, so we could choose to break compat for
2.4.0 and just remove this method for coprocessors.
If we are keeping it, when should a coprocessor expect this specific version
of `preUnassign` to get called? would it be called instead of the version
without the force parameter or in addition to? These details should be incldued
in the javadoc.
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java
##
@@ -591,7 +596,11 @@
* @param forcible If true, force unassign (Will remove region from
regions-in-transition too if
* present. If results in double assignment use hbck -fix to
resolve. To be used by
* experts).
+ * @deprecated since 2.4.0 and will be removed in 4.0.0. Use {@link
#unassign(byte[])}
+ * instead.
+ * @see https://issues.apache.org/jira/browse/HBASE-24875;>HBASE-24875
*/
+ @Deprecated
CompletableFuture unassign(byte[] regionName, boolean forcible);
Review comment:
use a default method here to call unassign(b[]) so that our various
implementations can skip including the same.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java
##
@@ -80,9 +78,6 @@ protected void serializeStateData(ProcedureStateSerializer
serializer) throws IO
if (this.destinationServer != null) {
state.setDestinationServer(ProtobufUtil.toServerName(destinationServer));
}
-if (force) {
- state.setForce(true);
Review comment:
we should add a comment to the protobuf definition for
`UnassignRegionStateData` in `MasterProcedure` that the field for force is
ignored.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
##
@@ -1697,7 +1697,6 @@ public UnassignRegionResponse
unassignRegion(RpcController controller,
try {
final byte [] regionName = req.getRegion().getValue().toByteArray();
RegionSpecifierType type = req.getRegion().getType();
- final boolean force = req.getForce();
Review comment:
we should add a comment in the protobuf defition for
`UnassignRegionRequest` in `Master` that the field for force is ignored.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
##
@@ -508,35 +508,57 @@ default void postAssign(final
ObserverContext ctx,
* Called prior to unassigning a given region.
* @param ctx the environment to interact with the framework and master
* @param regionInfo
- * @param force whether to force unassignment or not
*/
default void preUnassign(final
[jira] [Created] (HBASE-24880) Remove ReplicationPeerConfigUpgrader
Duo Zhang created HBASE-24880: - Summary: Remove ReplicationPeerConfigUpgrader Key: HBASE-24880 URL: https://issues.apache.org/jira/browse/HBASE-24880 Project: HBase Issue Type: Task Components: Replication Reporter: Duo Zhang The comment says it will be removed in 3.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24869) migrate website generation to new asf jenkins
[ https://issues.apache.org/jira/browse/HBASE-24869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-24869: Fix Version/s: 3.0.0-alpha-1 Resolution: Fixed Status: Resolved (was: Patch Available) > migrate website generation to new asf jenkins > - > > Key: HBASE-24869 > URL: https://issues.apache.org/jira/browse/HBASE-24869 > Project: HBase > Issue Type: Task > Components: build, website >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Update our website generation so we can use it on the new jenkins ci server > * needs a job name that has no spaces (or fix the script to handle paths with > spaces) > * needs to only run on nodes labeled git-websites (so it will have the creds > to push updates) > * needs to set editable email notification on failure (details in comment) > Also we will need to converte to a pipeline DSL > * define tools, namely maven (alternative get [Tool Environment > Plugin|https://plugins.jenkins.io/toolenv/]) > * set timeout for 4 hours (alternative get [build timeout > plugin|https://plugins.jenkins.io/build-timeout/]) > * needs to clean worksapce when done (haven't found an alternatiave, maybe > it's a default for non-pipeline jobs now?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24869) migrate website generation to new asf jenkins
[ https://issues.apache.org/jira/browse/HBASE-24869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177454#comment-17177454 ] Sean Busbey commented on HBASE-24869: - I checked on the job on ci-hadoop today and things appear to be working properly. closing this out. > migrate website generation to new asf jenkins > - > > Key: HBASE-24869 > URL: https://issues.apache.org/jira/browse/HBASE-24869 > Project: HBase > Issue Type: Task > Components: build, website >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > Update our website generation so we can use it on the new jenkins ci server > * needs a job name that has no spaces (or fix the script to handle paths with > spaces) > * needs to only run on nodes labeled git-websites (so it will have the creds > to push updates) > * needs to set editable email notification on failure (details in comment) > Also we will need to converte to a pipeline DSL > * define tools, namely maven (alternative get [Tool Environment > Plugin|https://plugins.jenkins.io/toolenv/]) > * set timeout for 4 hours (alternative get [build timeout > plugin|https://plugins.jenkins.io/build-timeout/]) > * needs to clean worksapce when done (haven't found an alternatiave, maybe > it's a default for non-pipeline jobs now?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] infraio opened a new pull request #2257: HBASE-24879 Fix the shellcheck download url
infraio opened a new pull request #2257: URL: https://github.com/apache/hbase/pull/2257 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24879) Fix the shellcheck download url
[ https://issues.apache.org/jira/browse/HBASE-24879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-24879: --- Description: *10:43:08* You are downloading ShellCheck from an outdated URL! *10:43:08* *10:43:08* Please update to the new URL: *10:43:08* [https://github.com/koalaman/shellcheck/releases/download/stable/shellcheck-stable.linux.x86_64.tar.xz] *10:43:08* *10:43:08* For more information, see: *10:43:08* [https://github.com/koalaman/shellcheck/issues/1871] *10:43:08* *10:43:08* PS: Sorry for breaking your build :( *10:43:08* You are downloading ShellCheck from an outdated URL! > Fix the shellcheck download url > --- > > Key: HBASE-24879 > URL: https://issues.apache.org/jira/browse/HBASE-24879 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Priority: Major > > *10:43:08* You are downloading ShellCheck from an outdated URL! *10:43:08* > *10:43:08* Please update to the new URL: *10:43:08* > [https://github.com/koalaman/shellcheck/releases/download/stable/shellcheck-stable.linux.x86_64.tar.xz] > *10:43:08* *10:43:08* For more information, see: *10:43:08* > [https://github.com/koalaman/shellcheck/issues/1871] *10:43:08* *10:43:08* > PS: Sorry for breaking your build :( *10:43:08* You are downloading > ShellCheck from an outdated URL! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24879) Fix the shellcheck download url
Guanghao Zhang created HBASE-24879: -- Summary: Fix the shellcheck download url Key: HBASE-24879 URL: https://issues.apache.org/jira/browse/HBASE-24879 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
Apache-HBase commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673858160 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 35s | master passed | | +1 :green_heart: | checkstyle | 2m 30s | master passed | | +1 :green_heart: | spotbugs | 4m 40s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | checkstyle | 0m 29s | hbase-client: The patch generated 0 new + 106 unchanged - 1 fixed = 106 total (was 107) | | +1 :green_heart: | checkstyle | 1m 2s | The patch passed checkstyle in hbase-server | | +1 :green_heart: | checkstyle | 0m 40s | The patch passed checkstyle in hbase-thrift | | +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-shell | | +1 :green_heart: | rubocop | 0m 10s | There were no new rubocop issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 24s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 44s | The patch does not generate ASF License warnings. | | | | 43m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2254 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle rubocop | | uname | Linux 77bdef923b34 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2191: HBASE-24813 ReplicationSource should clear buffer usage on Replicatio…
Apache9 commented on a change in pull request #2191:
URL: https://github.com/apache/hbase/pull/2191#discussion_r470374275
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -650,6 +650,10 @@ public void terminate(String reason, Exception cause,
boolean clearMetrics, bool
if (worker.entryReader.isAlive()) {
worker.entryReader.interrupt();
}
+ } else {
+//If worker is already stopped but there was still entries batched,
+//we need to clear buffer used for non processed entries
+worker.clearWALEntryBatch();
Review comment:
We will not do this for the above condition branch?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2215: HBASE-24627 Normalize one table at a time
Apache-HBase commented on pull request #2215: URL: https://github.com/apache/hbase/pull/2215#issuecomment-673841254 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 41s | master passed | | +1 :green_heart: | compile | 3m 38s | master passed | | +1 :green_heart: | shadedjars | 6m 25s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | | -0 :warning: | javadoc | 0m 58s | hbase-thrift in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 32s | the patch passed | | +1 :green_heart: | compile | 3m 37s | the patch passed | | +1 :green_heart: | javac | 3m 37s | the patch passed | | +1 :green_heart: | shadedjars | 6m 27s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 58s | hbase-thrift in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 2s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 16s | hbase-client in the patch passed. | | -1 :x: | unit | 191m 23s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 5m 26s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 1s | hbase-shell in the patch passed. | | | | 244m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2215 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 86643aa1a219 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-thrift.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-thrift.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/testReport/ | | Max. process+thread count | 3414 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
[GitHub] [hbase] Apache9 commented on pull request #2130: HBASE-24765: Dynamic master discovery
Apache9 commented on pull request #2130: URL: https://github.com/apache/hbase/pull/2130#issuecomment-673840254 Oh shit, I forgot to hit the submit review button... Sorry @bharathv , I've commented long ago but forgot to submit them... Hope you still have patience to fix the review comments... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2130: HBASE-24765: Dynamic master discovery
Apache9 commented on a change in pull request #2130:
URL: https://github.com/apache/hbase/pull/2130#discussion_r459835061
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
##
@@ -170,6 +214,11 @@ public static String getMasterAddr(Configuration conf)
throws UnknownHostExcepti
callable.call(controller, stub, resp -> {
if (controller.failed()) {
future.completeExceptionally(controller.getFailed());
+// RPC has failed, trigger a refresh of master end points. We can have
some spurious
+// refreshes, but that is okay since the RPC is not expensive and not
in a hot path.
+synchronized (refreshMasters) {
Review comment:
Usually a notify without any flag will have strange race problem...
At least let's have a 'triggerRefresh' flag to guard redundant notification?
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
##
@@ -226,17 +276,19 @@ private IOException badResponse(String debug) {
private CompletableFuture call(Callable callable,
Predicate isValidResp, String debug) {
+Set masterServers = masterAddr2Stub.keySet();
Review comment:
I think here we should assign masterAddr2Stub to a local variable before
using it?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
##
@@ -2931,6 +2935,27 @@ public GetActiveMasterResponse
getActiveMaster(RpcController rpcController,
return resp.build();
}
+ @Override
+ public GetMastersResponse getMasters(RpcController rpcController,
GetMastersRequest request)
+ throws ServiceException {
+GetMastersResponse.Builder resp = GetMastersResponse.newBuilder();
+// Active master
+Optional serverName = master.getActiveMaster();
+serverName.ifPresent(name ->
resp.addMasterServers(GetMastersResponseEntry.newBuilder()
+
.setServerName(ProtobufUtil.toServerName(name)).setIsActive(true).build()));
+// Backup masters
+try {
+ // TODO: Cache the backup masters to avoid a ZK RPC for each
getMasters() call.
Review comment:
I think this should be done in this patch, as now this method will
replace the old getActiveMaster method, which makes it not only be used in our
internal refresh but also be used by end users, we should not let users still
have the ability to harmmer zookeeper...
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
##
@@ -259,17 +311,40 @@ private RegionLocations
transformMetaRegionLocations(GetMetaRegionLocationsRespo
.thenApply(GetClusterIdResponse::getClusterId);
}
- private ServerName transformServerName(GetActiveMasterResponse resp) {
-return ProtobufUtil.toServerName(resp.getServerName());
+ private static boolean hasActiveMaster(GetMastersResponse resp) {
+List activeMasters =
+
resp.getMasterServersList().stream().filter(GetMastersResponseEntry::getIsActive).collect(
+Collectors.toList());
+return activeMasters.size() == 1;
+ }
+
+ private static ServerName filterActiveMaster(GetMastersResponse resp) {
+List activeMasters =
+
resp.getMasterServersList().stream().filter(GetMastersResponseEntry::getIsActive).collect(
+Collectors.toList());
+Preconditions.checkState(activeMasters.size() == 1);
Review comment:
This is a behavior change? I do not think we will throw
IllegalStateException when there is no active master before this patch?
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
##
@@ -89,11 +97,17 @@
private final int hedgedReadFanOut;
// Configured list of masters to probe the meta information from.
- private final ImmutableMap
masterAddr2Stub;
+ private volatile ImmutableMap
masterAddr2Stub;
// RPC client used to talk to the masters.
private final RpcClient rpcClient;
private final RpcControllerFactory rpcControllerFactory;
+ private final int rpcTimeoutMs;
+ // For synchronizing on refreshing the master end-points
+ private final Object refreshMasters = new Object();
+ // Refreshed every WAIT_TIME_OUT_MS or unless explicitly invoked.
+ private static final int WAIT_TIME_OUT_MS = 5 * 60 * 1000; // 5 mins
Review comment:
Do we want to make it configurable?
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java
##
@@ -259,17 +311,40 @@ private RegionLocations
transformMetaRegionLocations(GetMetaRegionLocationsRespo
.thenApply(GetClusterIdResponse::getClusterId);
}
- private ServerName transformServerName(GetActiveMasterResponse resp) {
-return ProtobufUtil.toServerName(resp.getServerName());
+ private static boolean hasActiveMaster(GetMastersResponse resp) {
+List activeMasters =
+
[GitHub] [hbase] Apache9 commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
Apache9 commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673836007 I think we'd better start from a high level design, on how we could start a cluster with no data on zookeeper. My proposal is always, introducing a new option in HBCK2 to reconstruct the data on zookeeper, but you actually start the cluster. If you guys think it is possible to do this in normal code path, please give an overall design. Open a google doc? So others could comments on it? For deleting meta, I agree that we should not delete it if it is not 'partial'. But as I said above many times, if there is no inconsitency, scheduling an InitMetaProcedure implies that we do not have a meta table yet, so in InitMetaProcedure it is always safe to delete the meta directory. So even if we add a check in InitMetaProcedure to determine whether the meta table is 'real' partial, the result should be an ERROR log to tell users something crtical happens, please fix the inconsistency before moving forward. I do not think this is what you guys want here? As you want to start the HMaster and recover from the inconsistency? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2249: HBASE-24871 Replication may loss data when refresh recovered replicat…
infraio commented on a change in pull request #2249:
URL: https://github.com/apache/hbase/pull/2249#discussion_r470366397
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
##
@@ -516,7 +516,7 @@ public void refreshSources(String peerId) throws
IOException {
ReplicationSourceInterface replicationSource = createSource(queueId,
peer);
this.oldsources.add(replicationSource);
for (SortedSet walsByGroup :
walsByIdRecoveredQueues.get(queueId).values()) {
- walsByGroup.forEach(wal -> src.enqueueLog(new Path(wal)));
+ walsByGroup.forEach(wal -> replicationSource.enqueueLog(new
Path(wal)));
Review comment:
> So the problem is that logs from recovered queues were getting added
to the new "normal" queue? And these logs are never read, then?
Yes. These log will be added to normal source. And will read by normal
source. So this should not loss data but only affect the replicate order.
@ddupg I thought the issue title need to be changed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] infraio opened a new pull request #2256: HBASE-24878 Backport part of HBASE-24079
infraio opened a new pull request #2256: URL: https://github.com/apache/hbase/pull/2256 hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java Be explicit about timestamping to avoid concurrent edit landing server-side and messing up test expectation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio merged pull request #2248: HBASE-24870 Ignore TestAsyncTableRSCrashPublish
infraio merged pull request #2248: URL: https://github.com/apache/hbase/pull/2248 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio merged pull request #2247: HBASE-24847 Backport HBASE-23956: Use less resources running tests to…
infraio merged pull request #2247: URL: https://github.com/apache/hbase/pull/2247 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24878) Backport part of HBASE-24079
Guanghao Zhang created HBASE-24878: -- Summary: Backport part of HBASE-24079 Key: HBASE-24878 URL: https://issues.apache.org/jira/browse/HBASE-24878 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2215: HBASE-24627 Normalize one table at a time
Apache-HBase commented on pull request #2215: URL: https://github.com/apache/hbase/pull/2215#issuecomment-673783465 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 41s | master passed | | +1 :green_heart: | compile | 3m 2s | master passed | | +1 :green_heart: | shadedjars | 5m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 58s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 3m 3s | the patch passed | | +1 :green_heart: | javac | 3m 3s | the patch passed | | +1 :green_heart: | shadedjars | 5m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 45s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 6s | hbase-client in the patch passed. | | -1 :x: | unit | 139m 33s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 4m 41s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 25s | hbase-shell in the patch passed. | | | | 190m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2215 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2cc118e1b8db 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/testReport/ | | Max. process+thread count | 4471 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24079) [Flakey Tests] Misc fixes and debug; fix BindException in Thrift tests; add waits on quota table to come online; etc.
[
https://issues.apache.org/jira/browse/HBASE-24079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang resolved HBASE-24079.
Resolution: Fixed
Cannot cherry-pick to branch-2.2 directly. Will open new issue to backport some
of them.
> [Flakey Tests] Misc fixes and debug; fix BindException in Thrift tests; add
> waits on quota table to come online; etc.
> -
>
> Key: HBASE-24079
> URL: https://issues.apache.org/jira/browse/HBASE-24079
> Project: HBase
> Issue Type: Test
> Components: flakies
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> Some fixes that have accumulated over last few days.
> Fixes for weird NPE on construction of ClientAsyncPrefetchScanner, catches
> for unexpected, test-killing exceptions in RSProcedureDispatcher and in
> CompactSplit; add waits on quota table to come on line and regionservers to
> go down before proceeding; add retry if BindException in thrift tests;
> disabled a few flakies; etc. Add some debug too.
> {code}
>
> hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientAsyncPrefetchScanner.java
> Refactor to avoid NPE timing issue referencing lock during Construction.
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
> Comment
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> Refactor. Catch NPE during startup and return it instead as failed
> initialization.
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplit.java
> Catch IndexOutOfBounds exception and convert to non-split request.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/TestCachedClusterId.java
> Make less furious. Make it less flakie.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java
> Debug. Catch exception to log, then rethrow.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
> Guess that waiting longer on compaction to succeed may help make this
> less flakey.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
> Be explicit about timestamping to avoid concurrent edit landing
> server-side and messing up test expectation.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMasterRegistry.java
> Add wait on meta before proceeding w/ test.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java
> Be explicit that edits are distinct.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCacheRefCnt.java
> Add @Ignore on RAM test... Fails sporadically.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionMoveAndAbandon.java
> Add wait for all RegionServers going down before proceeding; was
> messing up RS accounting.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
> Make balancer test sloppier; less restrictive; would fail on occasion
> by being just outside test limits.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
> Add wait on quota table coming up; helps make this less flakie.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Be explicity about timestamps; see if helps w/ flakie failure.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
> Catch and ignore if issue in shutdown; don't care if after test.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java
> Comment.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
> Add retry to see if helps w/ odd failure; grant hasn't propagated?
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCellACLWithMultipleVersions.java
> Explicit w/ timestamps so no accidental overlap of puts.
>
> hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
>
> hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
> Hack to deal w/ BindException on startup.
>
> hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThrift2ServerCmdLine.java
> Use loopback.
>
> hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java
>
[jira] [Reopened] (HBASE-24079) [Flakey Tests] Misc fixes and debug; fix BindException in Thrift tests; add waits on quota table to come online; etc.
[
https://issues.apache.org/jira/browse/HBASE-24079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reopened HBASE-24079:
Reopen for cherry-pick to branch-2.2.
> [Flakey Tests] Misc fixes and debug; fix BindException in Thrift tests; add
> waits on quota table to come online; etc.
> -
>
> Key: HBASE-24079
> URL: https://issues.apache.org/jira/browse/HBASE-24079
> Project: HBase
> Issue Type: Test
> Components: flakies
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> Some fixes that have accumulated over last few days.
> Fixes for weird NPE on construction of ClientAsyncPrefetchScanner, catches
> for unexpected, test-killing exceptions in RSProcedureDispatcher and in
> CompactSplit; add waits on quota table to come on line and regionservers to
> go down before proceeding; add retry if BindException in thrift tests;
> disabled a few flakies; etc. Add some debug too.
> {code}
>
> hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientAsyncPrefetchScanner.java
> Refactor to avoid NPE timing issue referencing lock during Construction.
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
> Comment
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
> Refactor. Catch NPE during startup and return it instead as failed
> initialization.
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplit.java
> Catch IndexOutOfBounds exception and convert to non-split request.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/TestCachedClusterId.java
> Make less furious. Make it less flakie.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java
> Debug. Catch exception to log, then rethrow.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
> Guess that waiting longer on compaction to succeed may help make this
> less flakey.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
> Be explicit about timestamping to avoid concurrent edit landing
> server-side and messing up test expectation.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMasterRegistry.java
> Add wait on meta before proceeding w/ test.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java
> Be explicit that edits are distinct.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCacheRefCnt.java
> Add @Ignore on RAM test... Fails sporadically.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionMoveAndAbandon.java
> Add wait for all RegionServers going down before proceeding; was
> messing up RS accounting.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
> Make balancer test sloppier; less restrictive; would fail on occasion
> by being just outside test limits.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
> Add wait on quota table coming up; helps make this less flakie.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Be explicity about timestamps; see if helps w/ flakie failure.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
> Catch and ignore if issue in shutdown; don't care if after test.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java
> Comment.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
> Add retry to see if helps w/ odd failure; grant hasn't propagated?
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCellACLWithMultipleVersions.java
> Explicit w/ timestamps so no accidental overlap of puts.
>
> hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
>
> hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
> Hack to deal w/ BindException on startup.
>
> hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThrift2ServerCmdLine.java
> Use loopback.
>
> hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java
> Disable flakie test.
> {code}
--
This message was sent by Atlassian
[GitHub] [hbase] taklwu commented on a change in pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
taklwu commented on a change in pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#discussion_r470312667
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/InitMetaProcedure.java
##
@@ -71,7 +71,11 @@ private static void writeFsLayout(Path rootDir,
Configuration conf) throws IOExc
LOG.info("BOOTSTRAP: creating hbase:meta region");
FileSystem fs = rootDir.getFileSystem(conf);
Path tableDir = CommonFSUtils.getTableDir(rootDir,
TableName.META_TABLE_NAME);
-if (fs.exists(tableDir) && !fs.delete(tableDir, true)) {
+boolean removeMeta = conf.getBoolean(HConstants.REMOVE_META_ON_RESTART,
Review comment:
@josthe new JIRA is
[HBASE-24833](https://issues.apache.org/jira/browse/HBASE-24833) and the
discussion are mainly on the new
[PR#2237](https://github.com/apache/hbase/pull/2237). I may need to send email
to dev@ list for a boarder discussion if we should not depend on the data on
zookeeper (that will help us to prevent deleting the meta directory)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
Apache-HBase commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673765633 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 2m 12s | master passed | | +1 :green_heart: | shadedjars | 6m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 44s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | compile | 2m 12s | the patch passed | | +1 :green_heart: | javac | 2m 12s | the patch passed | | +1 :green_heart: | shadedjars | 6m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 6s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 199m 59s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 5m 19s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 8m 15s | hbase-shell in the patch passed. | | | | 246m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2254 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 05487a793338 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/testReport/ | | Max. process+thread count | 3498 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] taklwu edited a comment on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
taklwu edited a comment on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673751605 > should not depend on the data on zookeeper. I agreed with you that we may not be ready to totally skip relying on the data stored on zookeeper, that's definitely a boarder discussion on what HBase currently depends on Zookeeper (branch-2 and master), especially if data on Zookeeper could be ephemeral or removed. (I thought we're in the progress of moving data into ROOT region, aren't we ? e.g. [Proc-v2](https://issues.apache.org/jira/browse/HBASE-20610) Also, my initial goal is that the meta data/directory should not be deleted if possible, and we're trying to provide a persisted condition not to always delete meta if it's not `partial` (protected by the ZK data). sorry that I may be newbie on the proc-v2 and zk data, should we start a thread on the dev@ list to discuss about the following ? (my goal is to find a consensus how we can move this PR further) 1. should we delete meta directory when HM starts ? 2. after 2.2+, should not depend on the data on zookeeper and have more of the info into proc-v2 in the master region? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] taklwu commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
taklwu commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673751605 > should not depend on the data on zookeeper. I agreed with you that we may not be ready to totally skip relying on the data stored on zookeeper, that's definitely a boarder discussion on what HBase currently depends on Zookeeper (branch-2 and master), especially if data on Zookeeper could be ephemeral or removed. (I thought we're in the progress of moving data into ROOT region, aren't we ? e.g. [Proc-v2](https://issues.apache.org/jira/browse/HBASE-20610) Also, my initial goal is that the meta data/directory should not be deleted if possible, and we're trying to provide a persisted condition not to always delete meta if it's not `partial` (protected by the ZK data). sorry that I may be newbie on the proc-v2 and zk data, should we start a thread on the dev@ list to discuss about the following ? (my goal is to find a consensus how we can move this PR to either completes it or not fixed) 1. should we delete meta directory when HM starts ? 2. after 2.2+, should not depend on the data on zookeeper and have more of the info into proc-v2 in the master region? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2215: HBASE-24627 Normalize one table at a time
Apache-HBase commented on pull request #2215: URL: https://github.com/apache/hbase/pull/2215#issuecomment-673750541 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 44s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 38s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 39s | master passed | | +1 :green_heart: | checkstyle | 2m 41s | master passed | | +1 :green_heart: | spotbugs | 7m 46s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 24s | the patch passed | | +1 :green_heart: | checkstyle | 0m 10s | The patch passed checkstyle in hbase-protocol-shaded | | +1 :green_heart: | checkstyle | 0m 32s | The patch passed checkstyle in hbase-client | | +1 :green_heart: | checkstyle | 1m 9s | hbase-server: The patch generated 0 new + 105 unchanged - 2 fixed = 105 total (was 107) | | +1 :green_heart: | checkstyle | 0m 41s | The patch passed checkstyle in hbase-thrift | | +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-shell | | -0 :warning: | rubocop | 0m 12s | The patch generated 7 new + 352 unchanged - 0 fixed = 359 total (was 352) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 5s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 3m 26s | the patch passed | | +1 :green_heart: | spotbugs | 10m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. | | | | 63m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2215 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool rubocop | | uname | Linux bed34c38dc6b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | rubocop | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 95 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
Apache-HBase commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673745776 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 2m 28s | master passed | | +1 :green_heart: | shadedjars | 5m 42s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 27s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | | -0 :warning: | javadoc | 0m 50s | hbase-thrift in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 57s | the patch passed | | +1 :green_heart: | compile | 2m 30s | the patch passed | | +1 :green_heart: | javac | 2m 30s | the patch passed | | +1 :green_heart: | shadedjars | 5m 43s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 50s | hbase-thrift in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 5s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 127m 40s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 4m 45s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 5s | hbase-shell in the patch passed. | | | | 174m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2254 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3888d4acd833 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-thrift.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-thrift.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/testReport/ | | Max. process+thread count | 4385 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] taklwu commented on a change in pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
taklwu commented on a change in pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#discussion_r470286861
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/InitMetaProcedure.java
##
@@ -71,7 +71,11 @@ private static void writeFsLayout(Path rootDir,
Configuration conf) throws IOExc
LOG.info("BOOTSTRAP: creating hbase:meta region");
FileSystem fs = rootDir.getFileSystem(conf);
Path tableDir = CommonFSUtils.getTableDir(rootDir,
TableName.META_TABLE_NAME);
-if (fs.exists(tableDir) && !fs.delete(tableDir, true)) {
+boolean removeMeta = conf.getBoolean(HConstants.REMOVE_META_ON_RESTART,
Review comment:
@joshelser the new JIRA is
[HBASE-24833](https://issues.apache.org/jira/browse/HBASE-24833) and the
discussion are mainly on the new
[PR#2237](https://github.com/apache/hbase/pull/2237). I may need to send email
to dev@ list for a boarder discussion if we should not depend on the data on
zookeeper (that will help us to prevent deleting the meta directory)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2130: HBASE-24765: Dynamic master discovery
bharathv commented on a change in pull request #2130:
URL: https://github.com/apache/hbase/pull/2130#discussion_r470268400
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterAddressRefresher.java
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.ClientMetaService;
+
+/**
+ * Thread safe utility that keeps master end points used by {@link
MasterRegistry} up to date. This
+ * uses the RPC {@link ClientMetaService#getMasters} to fetch the latest list
of registered masters.
+ * By default the refresh happens periodically (configured via
+ * {@link #PERIODIC_REFRESH_INTERVAL_SECS}). The refresh can also be triggered
on demand via
+ * {@link #refreshNow()}. To prevent a flood of on-demand refreshes we expect
that any attempts two
+ * should be spaced at least {@link #MIN_SECS_BETWEEN_REFRESHES} seconds apart.
+ */
+@InterfaceAudience.Private
+public class MasterAddressRefresher implements Closeable {
+ private static final Logger LOG =
LoggerFactory.getLogger(MasterAddressRefresher.class);
+ public static final String PERIODIC_REFRESH_INTERVAL_SECS =
+ "hbase.client.master_registry.refresh_interval_secs";
+ private static final int PERIODIC_REFRESH_INTERVAL_SECS_DEFAULT = 300;
+ public static final String MIN_SECS_BETWEEN_REFRESHES =
+ "hbase.client.master_registry.min_secs_between_refreshes";
+ private static final long MIN_SECS_BETWEEN_REFRESHES_DEFAULT = 60;
+
+ private final ExecutorService pool;
+ private final MasterRegistry registry;
+ private final long periodicRefreshMs;
+ private final long timeBetweenRefreshesMs;
+ private final Object refreshMasters = new Object();
+
+ @Override
+ public void close() {
+pool.shutdownNow();
+ }
+
+ /**
+ * Thread that refreshes the master end points until it is interrupted via
{@link #close()}.
+ * Multiple callers attempting to refresh at the same time synchronize on
{@link #refreshMasters}.
+ */
+ private class RefreshThread implements Runnable {
+@Override
+public void run() {
+ long lastRpcTs = 0;
+ while (!Thread.interrupted()) {
+try {
+ // Spurious wake ups are okay, worst case we make an extra RPC call
to refresh. We won't
+ // have duplicate refreshes because once the thread is past the
wait(), notify()s are
+ // ignored until the thread is back to the waiting state.
+ synchronized (refreshMasters) {
+refreshMasters.wait(periodicRefreshMs);
+ }
+ long currentTs = EnvironmentEdgeManager.currentTime();
+ if (lastRpcTs != 0 && currentTs - lastRpcTs <=
timeBetweenRefreshesMs) {
+continue;
+ }
+ lastRpcTs = currentTs;
+ LOG.debug("Attempting to refresh master address end points.");
+ Set newMasters = new
HashSet<>(registry.getMasters().get());
+ registry.populateMasterStubs(newMasters);
+ LOG.debug("Finished refreshing master end points. {}", newMasters);
+} catch (InterruptedException e) {
+ LOG.debug("Interrupted during wait, aborting
refresh-masters-thread.", e);
+ break;
+} catch (ExecutionException | IOException e) {
+ LOG.debug("Error populating latest list of masters.", e);
+}
+ }
+}
+ }
+
+ MasterAddressRefresher(Configuration conf, MasterRegistry registry) {
+pool =
[jira] [Updated] (HBASE-23976) [flakey test] TestVerifyBucketCacheFile
[
https://issues.apache.org/jira/browse/HBASE-23976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23976:
-
Attachment:
TEST-org.apache.hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile.xml
> [flakey test] TestVerifyBucketCacheFile
> ---
>
> Key: HBASE-23976
> URL: https://issues.apache.org/jira/browse/HBASE-23976
> Project: HBase
> Issue Type: Test
> Components: regionserver, test
>Affects Versions: 3.0.0-alpha-1
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments:
> TEST-org.apache.hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile.xml
>
>
> I see sporadic failures in this test class. Sometimes a failure on
> {{assertTrue(file.delete())}}, an inconsistent annoyance. However, this one
> looks more sinister.
> {noformat}
> 2020-03-12 12:11:35,059 ERROR [Time-limited test] bucket.BucketCache(312):
> Can't restore from
> file[/Users/ndimiduk/repos/apache/hbase/hbase-server/target/test-data/5e5c5f5f-d5c2-94b2-8ce9-cf561f4f19f7/bucket.persistence]
> because of
> java.io.IOException: Mismatch of checksum! The persistent checksum is
> ���#20;B#16;k�#3;��2�Ӏk, but the calculate checksum is
> �o���r��w��c��4
> at
> org.apache.hadoop.hbase.io.hfile.bucket.PersistentIOEngine.verifyFileIntegrity(PersistentIOEngine.java:55)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.parsePB(BucketCache.java:1158)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.retrieveFromFile(BucketCache.java:1106)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:310)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:258)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile.testRetrieveFromFile(TestVerifyBucketCacheFile.java:116)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23976) [flakey test] TestVerifyBucketCacheFile
[
https://issues.apache.org/jira/browse/HBASE-23976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177333#comment-17177333
]
Nick Dimiduk commented on HBASE-23976:
--
Hit this one again on
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2215/2/
{noformat}
2020-08-12 00:17:40,613 ERROR [Time-limited test] bucket.BucketCache(315):
Can't restore from
file[/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2215/yetus-jdk11-hadoop3-check/src/hbase-server/target/test-data/1e54fb4f-2756-7c15-ee69-37886735e13a/bucket.persistence]
because of
java.io.IOException: Mismatch of checksum! The persistent checksum is
�Y#24;�tC�G6x��*�`�, but the calculate checksum is
#4;ɇu���<#12;��@U��$
at
org.apache.hadoop.hbase.io.hfile.bucket.PersistentIOEngine.verifyFileIntegrity(PersistentIOEngine.java:55)
at
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.parsePB(BucketCache.java:1161)
at
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.retrieveFromFile(BucketCache.java:1109)
at
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:313)
at
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:261)
at
org.apache.hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile.testRetrieveFromFile(TestVerifyBucketCacheFile.java:119)
{noformat}
> [flakey test] TestVerifyBucketCacheFile
> ---
>
> Key: HBASE-23976
> URL: https://issues.apache.org/jira/browse/HBASE-23976
> Project: HBase
> Issue Type: Test
> Components: regionserver, test
>Affects Versions: 3.0.0-alpha-1
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> I see sporadic failures in this test class. Sometimes a failure on
> {{assertTrue(file.delete())}}, an inconsistent annoyance. However, this one
> looks more sinister.
> {noformat}
> 2020-03-12 12:11:35,059 ERROR [Time-limited test] bucket.BucketCache(312):
> Can't restore from
> file[/Users/ndimiduk/repos/apache/hbase/hbase-server/target/test-data/5e5c5f5f-d5c2-94b2-8ce9-cf561f4f19f7/bucket.persistence]
> because of
> java.io.IOException: Mismatch of checksum! The persistent checksum is
> ���#20;B#16;k�#3;��2�Ӏk, but the calculate checksum is
> �o���r��w��c��4
> at
> org.apache.hadoop.hbase.io.hfile.bucket.PersistentIOEngine.verifyFileIntegrity(PersistentIOEngine.java:55)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.parsePB(BucketCache.java:1158)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.retrieveFromFile(BucketCache.java:1106)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:310)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:258)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.TestVerifyBucketCacheFile.testRetrieveFromFile(TestVerifyBucketCacheFile.java:116)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2247: HBASE-24847 Backport HBASE-23956: Use less resources running tests to…
Apache-HBase commented on pull request #2247: URL: https://github.com/apache/hbase/pull/2247#issuecomment-673720378 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 26 new or modified test files. | ||| _ branch-2.2 Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 8s | branch-2.2 passed | | +1 :green_heart: | compile | 2m 50s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 6m 11s | branch-2.2 passed | | +1 :green_heart: | shadedjars | 4m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 6m 8s | branch-2.2 passed | | +0 :ok: | spotbugs | 1m 22s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 21m 36s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 43s | the patch passed | | +1 :green_heart: | compile | 2m 49s | the patch passed | | +1 :green_heart: | javac | 2m 49s | the patch passed | | +1 :green_heart: | checkstyle | 0m 34s | hbase-client: The patch generated 0 new + 4 unchanged - 6 fixed = 4 total (was 10) | | +1 :green_heart: | checkstyle | 0m 15s | The patch passed checkstyle in hbase-procedure | | +1 :green_heart: | checkstyle | 1m 24s | hbase-server: The patch generated 0 new + 279 unchanged - 8 fixed = 279 total (was 287) | | +1 :green_heart: | checkstyle | 0m 20s | The patch passed checkstyle in hbase-mapreduce | | +1 :green_heart: | checkstyle | 0m 39s | The patch passed checkstyle in hbase-thrift | | +1 :green_heart: | checkstyle | 0m 13s | The patch passed checkstyle in hbase-rsgroup | | +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-shell | | +1 :green_heart: | checkstyle | 0m 13s | The patch passed checkstyle in hbase-endpoint | | +1 :green_heart: | checkstyle | 0m 17s | The patch passed checkstyle in hbase-rest | | +1 :green_heart: | checkstyle | 0m 14s | The patch passed checkstyle in hbase-examples | | +1 :green_heart: | checkstyle | 2m 1s | root: The patch generated 0 new + 283 unchanged - 14 fixed = 283 total (was 297) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 27s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedjars | 4m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 25m 8s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | javadoc | 6m 21s | the patch passed | | +1 :green_heart: | findbugs | 23m 0s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 178m 27s | root in the patch failed. | | +1 :green_heart: | asflicense | 6m 16s | The patch does not generate ASF License warnings. | | | | 313m 44s | | | Reason | Tests | |---:|:--| | Failed junit tests | TEST-classes | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2247 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux 47f0fbe44e4d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2247/out/precommit/personality/provided.sh | | git revision | branch-2.2 / 058569c621 | | Default Java | 1.8.0_181 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/5/artifact/out/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/5/testReport/ | | Max. process+thread count | 3560 (vs. ulimit of 1) | | modules | C: hbase-client
[GitHub] [hbase] ndimiduk commented on a change in pull request #2130: HBASE-24765: Dynamic master discovery
ndimiduk commented on a change in pull request #2130:
URL: https://github.com/apache/hbase/pull/2130#discussion_r470252040
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterAddressRefresher.java
##
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.ClientMetaService;
+
+/**
+ * Thread safe utility that keeps master end points used by {@link
MasterRegistry} up to date. This
+ * uses the RPC {@link ClientMetaService#getMasters} to fetch the latest list
of registered masters.
+ * By default the refresh happens periodically (configured via
+ * {@link #PERIODIC_REFRESH_INTERVAL_SECS}). The refresh can also be triggered
on demand via
+ * {@link #refreshNow()}. To prevent a flood of on-demand refreshes we expect
that any attempts two
+ * should be spaced at least {@link #MIN_SECS_BETWEEN_REFRESHES} seconds apart.
+ */
+@InterfaceAudience.Private
+public class MasterAddressRefresher implements Closeable {
+ private static final Logger LOG =
LoggerFactory.getLogger(MasterAddressRefresher.class);
+ public static final String PERIODIC_REFRESH_INTERVAL_SECS =
+ "hbase.client.master_registry.refresh_interval_secs";
+ private static final int PERIODIC_REFRESH_INTERVAL_SECS_DEFAULT = 300;
+ public static final String MIN_SECS_BETWEEN_REFRESHES =
+ "hbase.client.master_registry.min_secs_between_refreshes";
+ private static final long MIN_SECS_BETWEEN_REFRESHES_DEFAULT = 60;
+
+ private final ExecutorService pool;
+ private final MasterRegistry registry;
+ private final long periodicRefreshMs;
+ private final long timeBetweenRefreshesMs;
+ private final Object refreshMasters = new Object();
+
+ @Override
+ public void close() {
+pool.shutdownNow();
+ }
+
+ /**
+ * Thread that refreshes the master end points until it is interrupted via
{@link #close()}.
+ * Multiple callers attempting to refresh at the same time synchronize on
{@link #refreshMasters}.
+ */
+ private class RefreshThread implements Runnable {
+@Override
+public void run() {
+ long lastRpcTs = 0;
+ while (!Thread.interrupted()) {
+try {
+ // Spurious wake ups are okay, worst case we make an extra RPC call
to refresh. We won't
+ // have duplicate refreshes because once the thread is past the
wait(), notify()s are
+ // ignored until the thread is back to the waiting state.
+ synchronized (refreshMasters) {
+refreshMasters.wait(periodicRefreshMs);
+ }
+ long currentTs = EnvironmentEdgeManager.currentTime();
+ if (lastRpcTs != 0 && currentTs - lastRpcTs <=
timeBetweenRefreshesMs) {
+continue;
+ }
+ lastRpcTs = currentTs;
+ LOG.debug("Attempting to refresh master address end points.");
+ Set newMasters = new
HashSet<>(registry.getMasters().get());
+ registry.populateMasterStubs(newMasters);
+ LOG.debug("Finished refreshing master end points. {}", newMasters);
+} catch (InterruptedException e) {
+ LOG.debug("Interrupted during wait, aborting
refresh-masters-thread.", e);
+ break;
+} catch (ExecutionException | IOException e) {
+ LOG.debug("Error populating latest list of masters.", e);
+}
+ }
+}
+ }
+
+ MasterAddressRefresher(Configuration conf, MasterRegistry registry) {
+pool =
[jira] [Commented] (HBASE-24876) Fix the flaky job url in hbase-personality.sh
[
https://issues.apache.org/jira/browse/HBASE-24876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177326#comment-17177326
]
Hudson commented on HBASE-24876:
Results for branch branch-1.4
[build #3 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/3/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/3//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/3//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1.4/3//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Fix the flaky job url in hbase-personality.sh
> -
>
> Key: HBASE-24876
> URL: https://issues.apache.org/jira/browse/HBASE-24876
> Project: HBase
> Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.6, 2.3.2
>
>
>
> I found that the precommit job of branch-2.2 still used the wrong url. See
> [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/1/console]
> {code:java}
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: Personality: patch unit
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: EXCLUDE_TESTS_URL=
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: INCLUDE_TESTS_URL=
> 16:56:14 --2020-08-12 08:56:14--
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/excludes/
> 16:56:14 Resolving builds.apache.org (builds.apache.org)... 195.201.213.130,
> 2a01:4f8:c0:2cc9::2
> 16:56:14 Connecting to builds.apache.org
> (builds.apache.org)|195.201.213.130|:443... connected.
> 16:56:15 HTTP request sent, awaiting response... 404
> 16:56:15 2020-08-12 08:56:15 ERROR 404: (no description).
> 16:56:15
> 16:56:15 Wget error 8 in fetching excludes file from url
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/excludes/.
> Ignoring and proceeding.{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
Apache-HBase commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673697315 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 15s | master passed | | +1 :green_heart: | checkstyle | 2m 40s | master passed | | +1 :green_heart: | spotbugs | 5m 13s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | checkstyle | 0m 30s | hbase-client: The patch generated 0 new + 106 unchanged - 1 fixed = 106 total (was 107) | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 4 new + 39 unchanged - 0 fixed = 43 total (was 39) | | +1 :green_heart: | checkstyle | 0m 43s | The patch passed checkstyle in hbase-thrift | | +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-shell | | +1 :green_heart: | rubocop | 0m 16s | There were no new rubocop issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 7s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 45s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 49s | The patch does not generate ASF License warnings. | | | | 46m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2254 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle rubocop | | uname | Linux 43a1cbf28d1d 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 95 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #2130: HBASE-24765: Dynamic master discovery
virajjasani commented on pull request #2130: URL: https://github.com/apache/hbase/pull/2130#issuecomment-673624360 Ah, somehow missed this today. Will take a look tomorrow This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on pull request #2130: HBASE-24765: Dynamic master discovery
bharathv commented on pull request #2130: URL: https://github.com/apache/hbase/pull/2130#issuecomment-673604488 any more thoughts on this? i think I addressed the review comments, let me know if I missed something. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24844) Exception on standalone (master) shutdown
[
https://issues.apache.org/jira/browse/HBASE-24844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177183#comment-17177183
]
Nick Dimiduk commented on HBASE-24844:
--
Thanks for the quick turn-around [~wenfeiyi666] and [~vjasani] !
> Exception on standalone (master) shutdown
> -
>
> Key: HBASE-24844
> URL: https://issues.apache.org/jira/browse/HBASE-24844
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 3.0.0-alpha-1
>Reporter: Nick Dimiduk
>Assignee: wenfeiyi666
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7
>
>
> Running HBase ({{master}} branch) in standalone mode, terminating the process
> results in the following stack traces logged at error. It appears we shutdown
> the zookeeper client out-of-order with {{shutdown}} of the thread pools.
> {noformat}
> 2020-08-10 14:21:46,777 INFO [RS:0;localhost:16020] zookeeper.ZooKeeper:
> Session: 0x100111361f20001 closed
> 2020-08-10 14:21:46,778 INFO [RS:0;localhost:16020]
> regionserver.HRegionServer: Exiting; stopping=localhost,16020,1597094491257;
> zookeeper connection closed.
> 2020-08-10 14:21:46,778 ERROR [main-EventThread] zookeeper.ClientCnxn: Error
> while calling watcher
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@6e61af4b rejected from
> java.util.concurrent.ThreadPoolExecutor@6a5e365f[Terminated, pool size = 0,
> active threads = 0, queued tasks = 0, completed tasks = 4]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> at
> java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:559)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> 2020-08-10 14:21:46,778 INFO [shutdown-hook-0] regionserver.ShutdownHook:
> Starting fs shutdown hook thread.
> 2020-08-10 14:21:46,779 ERROR [main-EventThread] zookeeper.ClientCnxn: Error
> while calling watcher
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@7d41da91 rejected from
> java.util.concurrent.ThreadPoolExecutor@6a5e365f[Terminated, pool size = 0,
> active threads = 0, queued tasks = 0, completed tasks = 4]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> at
> java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:559)
> at
> org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> 2020-08-10 14:21:46,780 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down for session: 0x100111361f20001
> 2020-08-10 14:21:46,780 INFO [shutdown-hook-0] regionserver.ShutdownHook:
> Shutdown hook finished.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24876) Fix the flaky job url in hbase-personality.sh
[
https://issues.apache.org/jira/browse/HBASE-24876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177180#comment-17177180
]
Hudson commented on HBASE-24876:
Results for branch branch-1
[build #6 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/6/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/6//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/6//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/6//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Fix the flaky job url in hbase-personality.sh
> -
>
> Key: HBASE-24876
> URL: https://issues.apache.org/jira/browse/HBASE-24876
> Project: HBase
> Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.6, 2.3.2
>
>
>
> I found that the precommit job of branch-2.2 still used the wrong url. See
> [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/1/console]
> {code:java}
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: Personality: patch unit
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: EXCLUDE_TESTS_URL=
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: INCLUDE_TESTS_URL=
> 16:56:14 --2020-08-12 08:56:14--
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/excludes/
> 16:56:14 Resolving builds.apache.org (builds.apache.org)... 195.201.213.130,
> 2a01:4f8:c0:2cc9::2
> 16:56:14 Connecting to builds.apache.org
> (builds.apache.org)|195.201.213.130|:443... connected.
> 16:56:15 HTTP request sent, awaiting response... 404
> 16:56:15 2020-08-12 08:56:15 ERROR 404: (no description).
> 16:56:15
> 16:56:15 Wget error 8 in fetching excludes file from url
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/excludes/.
> Ignoring and proceeding.{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
Apache-HBase commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673558052 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 54s | master passed | | +1 :green_heart: | compile | 2m 29s | master passed | | +1 :green_heart: | shadedjars | 6m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 2m 14s | the patch passed | | +1 :green_heart: | javac | 2m 14s | the patch passed | | +1 :green_heart: | shadedjars | 5m 57s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 45s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 2s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 137m 28s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 4m 7s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 14s | hbase-shell in the patch passed. | | | | 183m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2254 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9c8dd840b851 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/2/testReport/ | | Max. process+thread count | 5043 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24876) Fix the flaky job url in hbase-personality.sh
[
https://issues.apache.org/jira/browse/HBASE-24876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177107#comment-17177107
]
Hudson commented on HBASE-24876:
Results for branch branch-2.2
[build #6 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/6/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/6//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/6//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/6//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix the flaky job url in hbase-personality.sh
> -
>
> Key: HBASE-24876
> URL: https://issues.apache.org/jira/browse/HBASE-24876
> Project: HBase
> Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.6, 2.3.2
>
>
>
> I found that the precommit job of branch-2.2 still used the wrong url. See
> [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/1/console]
> {code:java}
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: Personality: patch unit
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: EXCLUDE_TESTS_URL=
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: INCLUDE_TESTS_URL=
> 16:56:14 --2020-08-12 08:56:14--
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/excludes/
> 16:56:14 Resolving builds.apache.org (builds.apache.org)... 195.201.213.130,
> 2a01:4f8:c0:2cc9::2
> 16:56:14 Connecting to builds.apache.org
> (builds.apache.org)|195.201.213.130|:443... connected.
> 16:56:15 HTTP request sent, awaiting response... 404
> 16:56:15 2020-08-12 08:56:15 ERROR 404: (no description).
> 16:56:15
> 16:56:15 Wget error 8 in fetching excludes file from url
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/excludes/.
> Ignoring and proceeding.{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] bsglz commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
bsglz commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673549359 No prob. @virajjasani This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2247: HBASE-24847 Backport HBASE-23956: Use less resources running tests to…
Apache-HBase commented on pull request #2247: URL: https://github.com/apache/hbase/pull/2247#issuecomment-673501324 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 26 new or modified test files. | ||| _ branch-2.2 Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 46s | branch-2.2 passed | | +1 :green_heart: | compile | 2m 57s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 6m 17s | branch-2.2 passed | | +1 :green_heart: | shadedjars | 4m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 6m 10s | branch-2.2 passed | | +0 :ok: | spotbugs | 1m 18s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 21m 16s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 43s | the patch passed | | +1 :green_heart: | compile | 2m 57s | the patch passed | | +1 :green_heart: | javac | 2m 57s | the patch passed | | +1 :green_heart: | checkstyle | 0m 36s | hbase-client: The patch generated 0 new + 4 unchanged - 6 fixed = 4 total (was 10) | | +1 :green_heart: | checkstyle | 0m 14s | The patch passed checkstyle in hbase-procedure | | +1 :green_heart: | checkstyle | 1m 24s | hbase-server: The patch generated 0 new + 279 unchanged - 8 fixed = 279 total (was 287) | | +1 :green_heart: | checkstyle | 0m 20s | The patch passed checkstyle in hbase-mapreduce | | +1 :green_heart: | checkstyle | 0m 38s | The patch passed checkstyle in hbase-thrift | | +1 :green_heart: | checkstyle | 0m 14s | The patch passed checkstyle in hbase-rsgroup | | +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-shell | | +1 :green_heart: | checkstyle | 0m 14s | The patch passed checkstyle in hbase-endpoint | | +1 :green_heart: | checkstyle | 0m 18s | The patch passed checkstyle in hbase-rest | | +1 :green_heart: | checkstyle | 0m 14s | The patch passed checkstyle in hbase-examples | | +1 :green_heart: | checkstyle | 1m 55s | root: The patch generated 0 new + 283 unchanged - 14 fixed = 283 total (was 297) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 26s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedjars | 4m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 25m 46s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | javadoc | 6m 22s | the patch passed | | +1 :green_heart: | findbugs | 24m 40s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 184m 29s | root in the patch failed. | | +1 :green_heart: | asflicense | 5m 37s | The patch does not generate ASF License warnings. | | | | 323m 19s | | | Reason | Tests | |---:|:--| | Failed junit tests | TEST-classes | | | hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2247 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux 6e6fff42bcdc 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2247/out/precommit/personality/provided.sh | | git revision | branch-2.2 / a838d727fa | | Default Java | 1.8.0_181 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/4/artifact/out/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/4/testReport/ | | Max. process+thread
[GitHub] [hbase] Apache-HBase commented on pull request #2191: HBASE-24813 ReplicationSource should clear buffer usage on Replicatio…
Apache-HBase commented on pull request #2191: URL: https://github.com/apache/hbase/pull/2191#issuecomment-673491296 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 0m 59s | master passed | | +1 :green_heart: | shadedjars | 6m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | | -0 :warning: | patch | 6m 55s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 6m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 213m 47s | hbase-server in the patch failed. | | | | 239m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2191 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 21c4556c297b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d2eb69df77 | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/testReport/ | | Max. process+thread count | 2967 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2248: HBASE-24870 Ignore TestAsyncTableRSCrashPublish
Apache-HBase commented on pull request #2248: URL: https://github.com/apache/hbase/pull/2248#issuecomment-673487926 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 56s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-2.2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 36s | branch-2.2 passed | | +1 :green_heart: | compile | 0m 58s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 1m 30s | branch-2.2 passed | | +1 :green_heart: | shadedjars | 4m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | branch-2.2 passed | | +0 :ok: | spotbugs | 3m 27s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 26s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 14s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | checkstyle | 1m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 25s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 27m 42s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | | +1 :green_heart: | findbugs | 3m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 227m 33s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 299m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2248/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2248 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7698c4fa7649 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2248/out/precommit/personality/provided.sh | | git revision | branch-2.2 / a838d727fa | | Default Java | 1.8.0_181 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2248/2/testReport/ | | Max. process+thread count | 3996 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2248/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
virajjasani commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673486059 > > Is this driven by https://issues.apache.org/jira/browse/HBASE-20881 ? > > Seems earlier, by https://issues.apache.org/jira/browse/HBASE-14614(AMv2), though the UnassignProcedure.java has force variate, but does not take effect. Okk sounds good, in this case, you can also evaluate removing `force` from UnassignProcedure class with this PR :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #2255: HBASE-24877 Add option to avoid aborting RS process upon uncaught exc…
virajjasani commented on a change in pull request #2255:
URL: https://github.com/apache/hbase/pull/2255#discussion_r469904081
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -583,16 +614,27 @@ private void initialize() {
PriorityBlockingQueue queue = entry.getValue();
tryStartNewShipper(walGroupId, queue);
}
+this.startupOngoing.set(false);
}
@Override
public void startup() {
// mark we are running now
this.sourceRunning = true;
-initThread = new Thread(this::initialize);
-Threads.setDaemonThreadRunning(initThread,
- Thread.currentThread().getName() + ".replicationSource," + this.queueId,
- this::uncaughtException);
+this.retryStartup.set(true);
+do {
+ if(retryStartup.get()) {
+retryStartup.set(false);
+startupOngoing.set(true);
+initThread = new Thread(this::initialize);
+Threads.setDaemonThreadRunning(initThread,
+ Thread.currentThread().getName() + ".replicationSource," +
this.queueId,
+ (t,e) -> {
+ uncaughtException(t, e);
+ retryStartup.set(true);
Review comment:
I hope if we encounter uncaughtException here, we want to retry the loop
again.
If so, shall we also add `startupOngoing.set(true);` here explicitely? Just
in case if it is set to false in `initialize()`?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -35,6 +35,8 @@
import java.util.concurrent.PriorityBlockingQueue;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
Review comment:
nit: redundant
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
##
@@ -120,6 +122,13 @@
// ReplicationEndpoint which will handle the actual replication
private volatile ReplicationEndpoint replicationEndpoint;
+ private AtomicBoolean retryStartup = new AtomicBoolean(false);
+
+ private AtomicBoolean startupOngoing = new AtomicBoolean(false);
Review comment:
nit: `final` for both?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
Apache-HBase commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673480875 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 43s | master passed | | +1 :green_heart: | checkstyle | 2m 35s | master passed | | +1 :green_heart: | spotbugs | 4m 32s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | +1 :green_heart: | checkstyle | 0m 29s | hbase-client: The patch generated 0 new + 106 unchanged - 1 fixed = 106 total (was 107) | | -0 :warning: | checkstyle | 1m 11s | hbase-server: The patch generated 4 new + 39 unchanged - 0 fixed = 43 total (was 39) | | +1 :green_heart: | checkstyle | 0m 44s | The patch passed checkstyle in hbase-thrift | | +1 :green_heart: | checkstyle | 0m 10s | The patch passed checkstyle in hbase-shell | | +1 :green_heart: | rubocop | 0m 12s | There were no new rubocop issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 9s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 40s | The patch does not generate ASF License warnings. | | | | 44m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2254 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle rubocop | | uname | Linux 909014dd7a02 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2191: HBASE-24813 ReplicationSource should clear buffer usage on Replicatio…
Apache-HBase commented on pull request #2191: URL: https://github.com/apache/hbase/pull/2191#issuecomment-673458194 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | master passed | | +1 :green_heart: | compile | 1m 8s | master passed | | +1 :green_heart: | shadedjars | 6m 20s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | | -0 :warning: | patch | 7m 13s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 6m 25s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 46s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 149m 55s | hbase-server in the patch failed. | | | | 178m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2191 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b8c3e7b2881c 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d2eb69df77 | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/testReport/ | | Max. process+thread count | 4384 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
Apache-HBase commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673456581 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 5s | master passed | | +1 :green_heart: | compile | 2m 47s | master passed | | +1 :green_heart: | shadedjars | 6m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 41s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | compile | 2m 12s | the patch passed | | +1 :green_heart: | javac | 2m 12s | the patch passed | | +1 :green_heart: | shadedjars | 6m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 11s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 206m 9s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 5m 17s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 59s | hbase-shell in the patch passed. | | | | 254m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2254 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ab7a20de30f4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b8fd621201 | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/testReport/ | | Max. process+thread count | 3141 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24876) Fix the flaky job url in hbase-personality.sh
[
https://issues.apache.org/jira/browse/HBASE-24876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176974#comment-17176974
]
Guanghao Zhang commented on HBASE-24876:
This fix worked. See
[https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/4/console]
{code:java}
18:59:09 --2020-08-13 10:59:06--
https://ci-hadoop.apache.org/job/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/excludes/
18:59:09 Resolving ci-hadoop.apache.org (ci-hadoop.apache.org)... 3.231.27.78
18:59:09 Connecting to ci-hadoop.apache.org
(ci-hadoop.apache.org)|3.231.27.78|:443... connected.
18:59:09 HTTP request sent, awaiting response... 200
18:59:09 Length: 1542 (1.5K) [application/octet-stream]
18:59:09 Saving to: âexcludesâ
18:59:09
18:59:09 0K . 100%
33.3M=0s
18:59:09
18:59:09 2020-08-13 10:59:06 (33.3 MB/s) - âexcludesâ saved [1542/1542]
18:59:09
18:59:15 cd
/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2247/src
18:59:15 /usr/share/maven/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2247/yetus-m2/hbase-branch-2.2-patch-1
-DHBasePatchProcess -PrunAllTests
-Dtest.exclude.pattern=**/replication.multiwal.TestReplicationEndpointWithMultipleWAL.java,**/replication.multiwal.TestReplicationSyncUpToolWithMultipleWAL.java,**/client.TestSnapshotDFSTemporaryDirectory.java,**/client.replication.TestReplicationAdminWithClusters.java,**/replication.multiwal.TestReplicationEndpointWithMultipleAsyncWAL.java,**/snapshot.TestExportSnapshot.java,**/master.TestAssignmentManagerMetrics.java,**/replication.TestReplicationSyncUpTool.java,**/TestJMXConnectorServer.java,**/regionserver.TestRegionReplicaFailover.java,**/client.TestFromClientSide3.java,**/client.TestAsyncTableAdminApi.java,**/snapshot.TestMobSecureExportSnapshot.java,**/TestAcidGuaranteesWithEagerPolicy.java,**/rsgroup.TestRSGroupsBasics.java,**/replication.TestReplicationSmallTests.java,**/client.TestSnapshotTemporaryDirectory.java,**/client.TestAdmin1.java,**/replication.TestReplicationKillSlaveRS.java,**/replication.TestReplicationDroppedTables.java,**/replication.TestReplicationEndpoint.java,**/client.TestSnapshotTemporaryDirectoryWithRegionReplicas.java,**/client.TestRestoreSnapshotFromClientSimple.java,**/TestAcidGuaranteesWithNoInMemCompaction.java,**/replication.regionserver.TestRegionReplicaReplicationEndpoint.java,**/replication.TestAddToSerialReplicationPeer.java,**/util.TestFromClientSide3WoUnsafe.java,**/client.TestCloneSnapshotFromClientNormal.java,**/replication.TestReplicationKillSlaveRSWithSeparateOldWALs.java,**/snapshot.TestMobExportSnapshot.java,**/replication.regionserver.TestReplicator.java,**/client.TestAsyncRegionAdminApi.java
clean test -fae >
/home/jenkins/jenkins-home/workspace/Base-PreCommit-GitHub-PR_PR-2247/out/patch-unit-root.txt
2>&1
{code}
> Fix the flaky job url in hbase-personality.sh
> -
>
> Key: HBASE-24876
> URL: https://issues.apache.org/jira/browse/HBASE-24876
> Project: HBase
> Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 1.4.14, 2.2.6, 2.3.2
>
>
>
> I found that the precommit job of branch-2.2 still used the wrong url. See
> [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2247/1/console]
> {code:java}
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: Personality: patch unit
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: EXCLUDE_TESTS_URL=
> 16:56:14 [Wed Aug 12 08:56:14 UTC 2020 INFO]: INCLUDE_TESTS_URL=
> 16:56:14 --2020-08-12 08:56:14--
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/excludes/
> 16:56:14 Resolving builds.apache.org (builds.apache.org)... 195.201.213.130,
> 2a01:4f8:c0:2cc9::2
> 16:56:14 Connecting to builds.apache.org
> (builds.apache.org)|195.201.213.130|:443... connected.
> 16:56:15 HTTP request sent, awaiting response... 404
> 16:56:15 2020-08-12 08:56:15 ERROR 404: (no description).
> 16:56:15
> 16:56:15 Wget error 8 in fetching excludes file from url
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/excludes/.
> Ignoring and proceeding.{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24869) migrate website generation to new asf jenkins
[
https://issues.apache.org/jira/browse/HBASE-24869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176963#comment-17176963
]
Hudson commented on HBASE-24869:
Results for branch master
[build #7 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> migrate website generation to new asf jenkins
> -
>
> Key: HBASE-24869
> URL: https://issues.apache.org/jira/browse/HBASE-24869
> Project: HBase
> Issue Type: Task
> Components: build, website
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> Update our website generation so we can use it on the new jenkins ci server
> * needs a job name that has no spaces (or fix the script to handle paths with
> spaces)
> * needs to only run on nodes labeled git-websites (so it will have the creds
> to push updates)
> * needs to set editable email notification on failure (details in comment)
> Also we will need to converte to a pipeline DSL
> * define tools, namely maven (alternative get [Tool Environment
> Plugin|https://plugins.jenkins.io/toolenv/])
> * set timeout for 4 hours (alternative get [build timeout
> plugin|https://plugins.jenkins.io/build-timeout/])
> * needs to clean worksapce when done (haven't found an alternatiave, maybe
> it's a default for non-pipeline jobs now?)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24583) Normalizer can't actually merge empty regions when neighbor is larger than average size
[
https://issues.apache.org/jira/browse/HBASE-24583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176964#comment-17176964
]
Hudson commented on HBASE-24583:
Results for branch master
[build #7 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Normalizer can't actually merge empty regions when neighbor is larger than
> average size
> ---
>
> Key: HBASE-24583
> URL: https://issues.apache.org/jira/browse/HBASE-24583
> Project: HBase
> Issue Type: Bug
> Components: master, Normalizer
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> There are plenty of cases where empty regions can accumulate -- incorrect
> guessing at split points, old data is automatically expiring off, The
> normalizer stubbornly refuses to handle this case, despite this being an
> original feature it was intended to support (HBASE-6613).
> Earlier discussion had concerns for a user pre-splitting a table and then the
> normalizer coming along and merging those splits away before they could be
> populated. Thus, right now, the default behavior via
> {{hbase.normalizer.merge.min_region_size.mb=1}} is to not split any region
> that's so small. Later, we added
> {{hbase.normalizer.merge.min_region_age.days=3}}, which prevents us from
> merging any region too young. So there's plenty of nobs for an operator to
> customize their behavior.
> But when I set {{hbase.normalizer.merge.min_region_size.mb=0}}, I still end
> up with stubborn regions that won't merge away. Looks like a large neighbor
> will prevent a merge from going through.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24568) do-release need not wait for tag
[
https://issues.apache.org/jira/browse/HBASE-24568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176965#comment-17176965
]
Hudson commented on HBASE-24568:
Results for branch master
[build #7 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> do-release need not wait for tag
>
>
> Key: HBASE-24568
> URL: https://issues.apache.org/jira/browse/HBASE-24568
> Project: HBase
> Issue Type: Bug
> Components: build, community
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Making release failed waiting for tag to propagate to GitHub. On inspection,
> it seems the GitHub url is missing host information.
> {noformat}
> Waiting up to 30 seconds for tag to propagate to github mirror...
> + sleep 30
> + max_propagation_time=0
> + check_for_tag 2.3.0RC0
> + curl -s --head --fail /releases/tag/2.3.0RC0
> + (( max_propagation_time <= 0 ))
> + echo 'ERROR: Taking more than 5 minutes to propagate Release Tag 2.3.0RC0
> to github mirror.'
> ERROR: Taking more than 5 minutes to propagate Release Tag 2.3.0RC0 to github
> mirror.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24844) Exception on standalone (master) shutdown
[
https://issues.apache.org/jira/browse/HBASE-24844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176962#comment-17176962
]
Hudson commented on HBASE-24844:
Results for branch master
[build #7 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/7//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Exception on standalone (master) shutdown
> -
>
> Key: HBASE-24844
> URL: https://issues.apache.org/jira/browse/HBASE-24844
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 3.0.0-alpha-1
>Reporter: Nick Dimiduk
>Assignee: wenfeiyi666
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7
>
>
> Running HBase ({{master}} branch) in standalone mode, terminating the process
> results in the following stack traces logged at error. It appears we shutdown
> the zookeeper client out-of-order with {{shutdown}} of the thread pools.
> {noformat}
> 2020-08-10 14:21:46,777 INFO [RS:0;localhost:16020] zookeeper.ZooKeeper:
> Session: 0x100111361f20001 closed
> 2020-08-10 14:21:46,778 INFO [RS:0;localhost:16020]
> regionserver.HRegionServer: Exiting; stopping=localhost,16020,1597094491257;
> zookeeper connection closed.
> 2020-08-10 14:21:46,778 ERROR [main-EventThread] zookeeper.ClientCnxn: Error
> while calling watcher
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@6e61af4b rejected from
> java.util.concurrent.ThreadPoolExecutor@6a5e365f[Terminated, pool size = 0,
> active threads = 0, queued tasks = 0, completed tasks = 4]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> at
> java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:559)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> 2020-08-10 14:21:46,778 INFO [shutdown-hook-0] regionserver.ShutdownHook:
> Starting fs shutdown hook thread.
> 2020-08-10 14:21:46,779 ERROR [main-EventThread] zookeeper.ClientCnxn: Error
> while calling watcher
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@7d41da91 rejected from
> java.util.concurrent.ThreadPoolExecutor@6a5e365f[Terminated, pool size = 0,
> active threads = 0, queued tasks = 0, completed tasks = 4]
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> at
> java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
> at
> org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:559)
> at
> org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> 2020-08-10 14:21:46,780 INFO [main-EventThread]
[GitHub] [hbase] Apache-HBase commented on pull request #2255: HBASE-24877 Add option to avoid aborting RS process upon uncaught exc…
Apache-HBase commented on pull request #2255: URL: https://github.com/apache/hbase/pull/2255#issuecomment-673445925 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 57s | master passed | | +1 :green_heart: | checkstyle | 1m 20s | master passed | | +1 :green_heart: | spotbugs | 2m 28s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 19s | the patch passed | | -0 :warning: | checkstyle | 1m 19s | hbase-server: The patch generated 18 new + 2 unchanged - 0 fixed = 20 total (was 2) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 14m 25s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 47s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 42m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2255/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2255 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 8e2d0f0fe0ac 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2255/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2255/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
bsglz commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673444023 > Is this driven by https://issues.apache.org/jira/browse/HBASE-20881 ? Seems earlier, by https://issues.apache.org/jira/browse/HBASE-14614(AMv2), though the UnassignProcedure.java has force variate, but does not take effect. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2255: HBASE-24877 Add option to avoid aborting RS process upon uncaught exc…
Apache-HBase commented on pull request #2255: URL: https://github.com/apache/hbase/pull/2255#issuecomment-673440803 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 0m 56s | master passed | | +1 :green_heart: | shadedjars | 5m 59s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 32s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | shadedjars | 5m 35s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 6m 48s | hbase-server in the patch failed. | | | | 31m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2255/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2255 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2f5e5acb36da 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c81ef7368e | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2255/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2255/1/testReport/ | | Max. process+thread count | 771 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2255/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
virajjasani commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673429097 Is this driven by https://issues.apache.org/jira/browse/HBASE-20881 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2254: HBASE-24875 Remove the force param for unassign since it dose not take effect any more
Apache-HBase commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673426778 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 1s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 1s | master passed | | +1 :green_heart: | compile | 2m 48s | master passed | | +1 :green_heart: | shadedjars | 7m 30s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 32s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 50s | hbase-server in master failed. | | -0 :warning: | javadoc | 1m 8s | hbase-thrift in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 18s | the patch passed | | +1 :green_heart: | compile | 3m 4s | the patch passed | | +1 :green_heart: | javac | 3m 4s | the patch passed | | +1 :green_heart: | shadedjars | 7m 25s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 30s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 50s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 1m 0s | hbase-thrift in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 16s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 134m 7s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 4m 22s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 20s | hbase-shell in the patch passed. | | | | 187m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2254 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 280906dc808b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / b8fd621201 | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-thrift.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-thrift.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/testReport/ | | Max. process+thread count | 4059 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server hbase-thrift hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2254/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil opened a new pull request #2255: HBASE-24877 Add option to avoid aborting RS process upon uncaught exc…
wchevreuil opened a new pull request #2255: URL: https://github.com/apache/hbase/pull/2255 …eptions happen on replication source This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24877) Add option to avoid aborting RS process upon uncaught exceptions happen on replication source
Wellington Chevreuil created HBASE-24877: Summary: Add option to avoid aborting RS process upon uncaught exceptions happen on replication source Key: HBASE-24877 URL: https://issues.apache.org/jira/browse/HBASE-24877 Project: HBase Issue Type: Improvement Components: Replication Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil Currently, we abort entire RS process if any uncaught exceptions happens on ReplicationSource initialization. This may be too extreme on certain deployments, where custom replication endpoint implementations may choose to do so when remote peers are unavailable, but source cluster shouldn't be brought down entirely. Similarly, source reader and shipper threads would cause RS to abort on any runtime exception occurrence while running. This patch adds configuration option (false by default, to keep the original behaviour), to avoid aborting entire RS processes under these conditions. Instead, if ReplicationSource initialization fails with a RuntimeException, it keeps retrying the source startup. In the case of readers/shippers runtime errors, it refreshes the replication source, terminating current source and its readers/shippers and creating new ones. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bsglz commented on pull request #2254: HBASE-24875 The describe and param for unassign not correct since the…
bsglz commented on pull request #2254: URL: https://github.com/apache/hbase/pull/2254#issuecomment-673416359 > Bit confused by the original description here, it suggests this was only a doc change, but there are new methods definitions and deprecations in the API. > > Can you explain better the motivation behind this change? Yeah, it indeed confusion, updated yet. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24875) Remove the force param for unassign since it dose not take effect any more
[ https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-24875: --- Description: Currently unassign region in fact only close it, so not need force param any more.(was: The semantic of unassign not correct and the force param does't take effct since the implementation changed at server side) > Remove the force param for unassign since it dose not take effect any more > -- > > Key: HBASE-24875 > URL: https://issues.apache.org/jira/browse/HBASE-24875 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > Currently unassign region in fact only close it, so not need force param any > more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24875) Remove the force param for unassign since it dose not take effect any more
[ https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-24875: --- Summary: Remove the force param for unassign since it dose not take effect any more (was: The semantic of unassign changed and the force param does't take effect) > Remove the force param for unassign since it dose not take effect any more > -- > > Key: HBASE-24875 > URL: https://issues.apache.org/jira/browse/HBASE-24875 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > The semantic of unassign not correct and the force param does't take effct > since the implementation changed at server side -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24875) The semantic of unassign changed and the force param does't take effect
[ https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-24875: --- Summary: The semantic of unassign changed and the force param does't take effect (was: The semantic of unassign changed and the force param does't take effct) > The semantic of unassign changed and the force param does't take effect > --- > > Key: HBASE-24875 > URL: https://issues.apache.org/jira/browse/HBASE-24875 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > The semantic of unassign not correct and the force param does't take effct > since the implementation changed at server side -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24875) The semantic of unassign changed and the force param does't take effct
[ https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-24875: --- Summary: The semantic of unassign changed and the force param does't take effct (was: The semantic of unassign not correct and the force param does't take effct since the implementation changed at server side) > The semantic of unassign changed and the force param does't take effct > -- > > Key: HBASE-24875 > URL: https://issues.apache.org/jira/browse/HBASE-24875 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > The semantic of unassign not correct and the force param does't take effct > since the implementation changed at server side -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24875) The semantic of unassign not correct and the force param does't take effct since the implementation changed at server side
[
https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Wang updated HBASE-24875:
---
Summary: The semantic of unassign not correct and the force param does't
take effct since the implementation changed at server side (was: The describe
and param for unassign not correct since the implementation changed at server
side)
> The semantic of unassign not correct and the force param does't take effct
> since the implementation changed at server side
> --
>
> Key: HBASE-24875
> URL: https://issues.apache.org/jira/browse/HBASE-24875
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The description of unassign in Admin.java shows below, it is not true, in
> fact, we just close the region now, also we do not need the force param any
> more.
> {code:java}
> /**
> * Unassign a region from current hosting regionserver. Region will then be
> assigned to a
> * regionserver chosen at random. Region could be reassigned back to the
> same server. Use {@link
> * #move(byte[], ServerName)} if you want to control the region movement.
> *
> * @param regionName Region to unassign. Will clear any existing RegionPlan
> if one found.
> * @param force If true, force unassign (Will remove region from
> regions-in-transition too if
> * present. If results in double assignment use hbck -fix to resolve. To be
> used by experts).
> * @throws IOException if a remote or network exception occurs
> */
> void unassign(byte[] regionName, boolean force)
> throws IOException;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24875) The semantic of unassign not correct and the force param does't take effct since the implementation changed at server side
[
https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Wang updated HBASE-24875:
---
Description: The semantic of unassign not correct and the force param
does't take effct since the implementation changed at server side (was: The
description of unassign in Admin.java shows below, it is not true, in fact, we
just close the region now, also we do not need the force param any more.
{code:java}
/**
* Unassign a region from current hosting regionserver. Region will then be
assigned to a
* regionserver chosen at random. Region could be reassigned back to the same
server. Use {@link
* #move(byte[], ServerName)} if you want to control the region movement.
*
* @param regionName Region to unassign. Will clear any existing RegionPlan if
one found.
* @param force If true, force unassign (Will remove region from
regions-in-transition too if
* present. If results in double assignment use hbck -fix to resolve. To be
used by experts).
* @throws IOException if a remote or network exception occurs
*/
void unassign(byte[] regionName, boolean force)
throws IOException;
{code})
> The semantic of unassign not correct and the force param does't take effct
> since the implementation changed at server side
> --
>
> Key: HBASE-24875
> URL: https://issues.apache.org/jira/browse/HBASE-24875
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
>
> The semantic of unassign not correct and the force param does't take effct
> since the implementation changed at server side
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24854) Correct the help content of assign and unassign commands in hbase shell
[
https://issues.apache.org/jira/browse/HBASE-24854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176923#comment-17176923
]
Zheng Wang commented on HBASE-24854:
Thanks for all the comments.
> Correct the help content of assign and unassign commands in hbase shell
> ---
>
> Key: HBASE-24854
> URL: https://issues.apache.org/jira/browse/HBASE-24854
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.2
>
>
> Help content in assign.rb:
> Assign a region. Use with caution. *If region already assigned,*
> *this command will do a force reassign.* For experts only.
>
> Help content in unassign.rb:
> Unassign a region. *Unassign will close region in current location and then*
> *reopen it again.* Pass 'true' to force the unassignment ('force' will clear
> all in-memory state in master before the reassign. If results in
> double assignment use hbck -fix to resolve. To be used by experts).
>
> The bold part are not right now, so update them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24873) Not able to access Flink 1.7.2 with HBase 2.0.0 included in HDP cluster 3.0.1
[
https://issues.apache.org/jira/browse/HBASE-24873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani resolved HBASE-24873.
--
Hadoop Flags: (was: Incompatible change)
Resolution: Invalid
> Not able to access Flink 1.7.2 with HBase 2.0.0 included in HDP cluster 3.0.1
> -
>
> Key: HBASE-24873
> URL: https://issues.apache.org/jira/browse/HBASE-24873
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
> Environment: * I am using Ambari Server 2.7.1 with HDP cluster 3.0.1
> with YARN 3.1.1 and HBase 2.0.0
> * Using hbase-client-2.0.0.jar along with Flink 1.7.2
> *
> [flink-1.7.2-bin-hadoop27-scala_2.11.tgz|https://archive.apache.org/dist/flink/flink-1.7.2/flink-1.7.2-bin-hadoop27-scala_2.11.tgz]
> using this one for trying.
>Reporter: Pasha Shaik
>Priority: Blocker
> Attachments: 0874A89C-597E-451A-8986-E619A0E8237B.jpeg,
> 0ECDB56A-3A76-424F-8926-A9FEB1BD96BB.jpeg,
> 4237EAEA-D3CD-4791-8322-49E2F9FA8666.png,
> 667CC1DB-CF0E-44E9-B9E8-1B44151FC00E.jpeg,
> 87939306-6571-4886-A23B-4780897B88D4.jpeg
>
>
> * I am not able to access Flink 1.7.2 with HDP 3.0.1
> * The YARN version is 3.1.1 and HBASE is 2.0.0
> * Flink is successfully getting mounted on Yarn and showing it as RUNNING.
> * But in actual when I try to test my code, it is showing below error.
> * The .tgz which I used is
> [flink-1.7.2-bin-hadoop27-scala_2.11.tgz|https://archive.apache.org/dist/flink/flink-1.7.2/flink-1.7.2-bin-hadoop27-scala_2.11.tgz]
> * The reason for the failure is w.r.t HDP 3.0.1, the associated HBase-Client
> ("org.apache.hbase:hbase-client:2.0.0") is still not in sync with
> Flink-Hbase_2.11-1.7.2 as the HTable.class constructor is completely removed
> in this version as those respective classes still uses that and throws below
> I/O exception.
> * Please find the logs and screenshots for more info.
>
>
> -**d
> HERE ARE THE LOGS BELOW
> *org.apache.flink.runtime.client.JobExecutionException: Failed to submit job
> cbb64a9b4e2e3ad0167eb4ceeb53ac87 (Flink Java Job at Tue Aug 11 10:10:47 CEST
> 2020) at*
> org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1325)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:447)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> ~[scala-library-2.11.11.jar:?] at
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> ~[scala-library-2.11.11.jar:?] at
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> ~[scala-library-2.11.11.jar:?] at
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:122)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> akka.actor.ActorCell.invoke(ActorCell.scala:495)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[akka-actor_2.11-2.4.20.jar:?]
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> ~[scala-library-2.11.11.jar:?] at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> ~[scala-library-2.11.11.jar:?] at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> ~[scala-library-2.11.11.jar:?] at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> ~
>
> *[scala-library-2.11.11.jar:?]Caused by:
> org.apache.flink.runtime.JobException: Creating the input splits caused an
> error: connection is closed at
>
[jira] [Commented] (HBASE-24873) Not able to access Flink 1.7.2 with HBase 2.0.0 included in HDP cluster 3.0.1
[
https://issues.apache.org/jira/browse/HBASE-24873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176910#comment-17176910
]
Viraj Jasani commented on HBASE-24873:
--
Closing this Jira as it is not related to OpenSource HBase project.
> Not able to access Flink 1.7.2 with HBase 2.0.0 included in HDP cluster 3.0.1
> -
>
> Key: HBASE-24873
> URL: https://issues.apache.org/jira/browse/HBASE-24873
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
> Environment: * I am using Ambari Server 2.7.1 with HDP cluster 3.0.1
> with YARN 3.1.1 and HBase 2.0.0
> * Using hbase-client-2.0.0.jar along with Flink 1.7.2
> *
> [flink-1.7.2-bin-hadoop27-scala_2.11.tgz|https://archive.apache.org/dist/flink/flink-1.7.2/flink-1.7.2-bin-hadoop27-scala_2.11.tgz]
> using this one for trying.
>Reporter: Pasha Shaik
>Priority: Blocker
> Attachments: 0874A89C-597E-451A-8986-E619A0E8237B.jpeg,
> 0ECDB56A-3A76-424F-8926-A9FEB1BD96BB.jpeg,
> 4237EAEA-D3CD-4791-8322-49E2F9FA8666.png,
> 667CC1DB-CF0E-44E9-B9E8-1B44151FC00E.jpeg,
> 87939306-6571-4886-A23B-4780897B88D4.jpeg
>
>
> * I am not able to access Flink 1.7.2 with HDP 3.0.1
> * The YARN version is 3.1.1 and HBASE is 2.0.0
> * Flink is successfully getting mounted on Yarn and showing it as RUNNING.
> * But in actual when I try to test my code, it is showing below error.
> * The .tgz which I used is
> [flink-1.7.2-bin-hadoop27-scala_2.11.tgz|https://archive.apache.org/dist/flink/flink-1.7.2/flink-1.7.2-bin-hadoop27-scala_2.11.tgz]
> * The reason for the failure is w.r.t HDP 3.0.1, the associated HBase-Client
> ("org.apache.hbase:hbase-client:2.0.0") is still not in sync with
> Flink-Hbase_2.11-1.7.2 as the HTable.class constructor is completely removed
> in this version as those respective classes still uses that and throws below
> I/O exception.
> * Please find the logs and screenshots for more info.
>
>
> -**d
> HERE ARE THE LOGS BELOW
> *org.apache.flink.runtime.client.JobExecutionException: Failed to submit job
> cbb64a9b4e2e3ad0167eb4ceeb53ac87 (Flink Java Job at Tue Aug 11 10:10:47 CEST
> 2020) at*
> org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1325)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:447)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> ~[scala-library-2.11.11.jar:?] at
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> ~[scala-library-2.11.11.jar:?] at
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> ~[scala-library-2.11.11.jar:?] at
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:122)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> akka.actor.ActorCell.invoke(ActorCell.scala:495)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[akka-actor_2.11-2.4.20.jar:?]
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> ~[scala-library-2.11.11.jar:?] at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> ~[scala-library-2.11.11.jar:?] at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> ~[scala-library-2.11.11.jar:?] at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> ~
>
> *[scala-library-2.11.11.jar:?]Caused by:
> org.apache.flink.runtime.JobException: Creating the input splits caused an
> error: connection is closed at
>
[jira] [Commented] (HBASE-24873) Not able to access Flink 1.7.2 with HBase 2.0.0 included in HDP cluster 3.0.1
[
https://issues.apache.org/jira/browse/HBASE-24873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176909#comment-17176909
]
Viraj Jasani commented on HBASE-24873:
--
HBase Opensource community has EOL'ed 2.0 release line and also, the community
does not own HDP packages. Could you please reach out to Hortonworks support to
get more help on this issue as this involves Flink and Ambari also with HDP
3.0.1?
Thanks
> Not able to access Flink 1.7.2 with HBase 2.0.0 included in HDP cluster 3.0.1
> -
>
> Key: HBASE-24873
> URL: https://issues.apache.org/jira/browse/HBASE-24873
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
> Environment: * I am using Ambari Server 2.7.1 with HDP cluster 3.0.1
> with YARN 3.1.1 and HBase 2.0.0
> * Using hbase-client-2.0.0.jar along with Flink 1.7.2
> *
> [flink-1.7.2-bin-hadoop27-scala_2.11.tgz|https://archive.apache.org/dist/flink/flink-1.7.2/flink-1.7.2-bin-hadoop27-scala_2.11.tgz]
> using this one for trying.
>Reporter: Pasha Shaik
>Priority: Blocker
> Attachments: 0874A89C-597E-451A-8986-E619A0E8237B.jpeg,
> 0ECDB56A-3A76-424F-8926-A9FEB1BD96BB.jpeg,
> 4237EAEA-D3CD-4791-8322-49E2F9FA8666.png,
> 667CC1DB-CF0E-44E9-B9E8-1B44151FC00E.jpeg,
> 87939306-6571-4886-A23B-4780897B88D4.jpeg
>
>
> * I am not able to access Flink 1.7.2 with HDP 3.0.1
> * The YARN version is 3.1.1 and HBASE is 2.0.0
> * Flink is successfully getting mounted on Yarn and showing it as RUNNING.
> * But in actual when I try to test my code, it is showing below error.
> * The .tgz which I used is
> [flink-1.7.2-bin-hadoop27-scala_2.11.tgz|https://archive.apache.org/dist/flink/flink-1.7.2/flink-1.7.2-bin-hadoop27-scala_2.11.tgz]
> * The reason for the failure is w.r.t HDP 3.0.1, the associated HBase-Client
> ("org.apache.hbase:hbase-client:2.0.0") is still not in sync with
> Flink-Hbase_2.11-1.7.2 as the HTable.class constructor is completely removed
> in this version as those respective classes still uses that and throws below
> I/O exception.
> * Please find the logs and screenshots for more info.
>
>
> -**d
> HERE ARE THE LOGS BELOW
> *org.apache.flink.runtime.client.JobExecutionException: Failed to submit job
> cbb64a9b4e2e3ad0167eb4ceeb53ac87 (Flink Java Job at Tue Aug 11 10:10:47 CEST
> 2020) at*
> org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1325)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:447)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> ~[scala-library-2.11.11.jar:?] at
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> ~[scala-library-2.11.11.jar:?] at
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> ~[scala-library-2.11.11.jar:?] at
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:122)
> ~[flink-runtime_2.11-1.4.2.jar:1.4.2] at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> akka.actor.ActorCell.invoke(ActorCell.scala:495)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[akka-actor_2.11-2.4.20.jar:?]
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> ~[akka-actor_2.11-2.4.20.jar:?] at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> ~[scala-library-2.11.11.jar:?] at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> ~[scala-library-2.11.11.jar:?] at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> ~[scala-library-2.11.11.jar:?] at
>
[GitHub] [hbase] Apache-HBase commented on pull request #2191: HBASE-24813 ReplicationSource should clear buffer usage on Replicatio…
Apache-HBase commented on pull request #2191: URL: https://github.com/apache/hbase/pull/2191#issuecomment-673400498 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 0s | master passed | | +1 :green_heart: | checkstyle | 1m 29s | master passed | | +1 :green_heart: | spotbugs | 2m 50s | master passed | | -0 :warning: | patch | 3m 0s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 30s | the patch passed | | +1 :green_heart: | checkstyle | 1m 26s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 14m 44s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 43m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2191 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 7e2fddf0496d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d2eb69df77 | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2191/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
Apache9 commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673400372 Anyway, let's get back to the initial goal? I think it is to support restart a cluster with nothing on zookeeper? Let's have an overall design first? As I said above, it is not easy to force all developpers to follow this rule. If we want to do this, we need to find a suitable way to force the later developpers know, they should not depend on the data on zookeeper... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24854) Correct the help content of assign and unassign commands in hbase shell
[
https://issues.apache.org/jira/browse/HBASE-24854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani resolved HBASE-24854.
--
Fix Version/s: 2.3.2
2.2.7
2.4.0
3.0.0-alpha-1
Hadoop Flags: Reviewed
Resolution: Fixed
> Correct the help content of assign and unassign commands in hbase shell
> ---
>
> Key: HBASE-24854
> URL: https://issues.apache.org/jira/browse/HBASE-24854
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.2
>
>
> Help content in assign.rb:
> Assign a region. Use with caution. *If region already assigned,*
> *this command will do a force reassign.* For experts only.
>
> Help content in unassign.rb:
> Unassign a region. *Unassign will close region in current location and then*
> *reopen it again.* Pass 'true' to force the unassignment ('force' will clear
> all in-memory state in master before the reassign. If results in
> double assignment use hbck -fix to resolve. To be used by experts).
>
> The bold part are not right now, so update them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24854) Correct the help content of assign and unassign commands in hbase shell
[
https://issues.apache.org/jira/browse/HBASE-24854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176896#comment-17176896
]
Viraj Jasani commented on HBASE-24854:
--
Thanks [~filtertip] for the contribution. Applied changes to master, branch-2,
2.3, 2.2.
Since HBCK2 is mentioned in the updated command description, we don't need to
merge this in branch-1.
> Correct the help content of assign and unassign commands in hbase shell
> ---
>
> Key: HBASE-24854
> URL: https://issues.apache.org/jira/browse/HBASE-24854
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Minor
>
> Help content in assign.rb:
> Assign a region. Use with caution. *If region already assigned,*
> *this command will do a force reassign.* For experts only.
>
> Help content in unassign.rb:
> Unassign a region. *Unassign will close region in current location and then*
> *reopen it again.* Pass 'true' to force the unassignment ('force' will clear
> all in-memory state in master before the reassign. If results in
> double assignment use hbck -fix to resolve. To be used by experts).
>
> The bold part are not right now, so update them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 edited a comment on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
Apache9 edited a comment on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673393155 I do not see why we need to change this. Consider the usage of InitMetaProcedure, if there is a meta directory there, it must be 'partial'. And even it is not partial, we are still safe to delete it, as there should be no data in it. If this is not the case, we must have inconsistency in our cluster. For safety, I think we could skip the deletion of the meta directory, but we should still fail the procedure and thus fail the initialization of master. Users should fix the inconsistency before starting again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
Apache9 commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673393155 I do not see why we need to change this. Consider the usage of InitMetaProcedure, if there is a meta directory there, it must be 'partial'. And even it is not partial, we are still safe to delete it, as there should be no data in it. If this is not the case, we should have inconsistency in our cluster. For safety, I think we could skip the deletion of the meta directory, but we should still fail the procedure and thus fail the initialization of master. Users should fix the inconsistency before starting again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
anoopsjohn commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673390864 InitMetaProcedure was there before also.. Only in 2.1.7, this delete of the META dir added. Am asking abt that only @Apache9 . That is the major concern. The init proc was before doing FS init if the meta dir is not yet there. And then init the assign of the META in the cluster. Now it will do both what so ever, the cur status of the meta dir. The req is a change for that. Is that some thing u veto? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani closed pull request #2241: HBASE-24854 Correct the help content of assign and unassign commands …
virajjasani closed pull request #2241: URL: https://github.com/apache/hbase/pull/2241 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24871) Replication may loss data when refresh recovered replication sources
[
https://issues.apache.org/jira/browse/HBASE-24871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-24871:
-
Component/s: Replication
> Replication may loss data when refresh recovered replication sources
>
>
> Key: HBASE-24871
> URL: https://issues.apache.org/jira/browse/HBASE-24871
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.2.5
>Reporter: Guanghao Zhang
>Assignee: Sun Xin
>Priority: Major
>
> See code in
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java#L519]
>
> The WAL enqueued to wrong replication source.
> {code:java}
> for (String queueId : previousQueueIds) {
> ReplicationSourceInterface replicationSource = createSource(queueId,
> peer);
> this.oldsources.add(replicationSource);
> this.queueStorage.getWALsInQueue(this.server.getServerName(), queueId)
> .forEach(wal -> src.enqueueLog(new Path(wal)));
> toStartup.add(replicationSource);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
Apache9 commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673387968 And for the problem here, it is why we introduce the InitMetaProcedure. For InitMetaProcedure itself, the assumption is that it will do the initialization work, it will be very strange that, we schedule a 'InitMeta' procedure, but we already have a meta in place? Why not just do not schedule the InitMetaProcedure at the first place? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2237: HBASE-24833: Bootstrap should not delete the META table directory if …
Apache9 commented on pull request #2237: URL: https://github.com/apache/hbase/pull/2237#issuecomment-673386671 For 2.2+ we have a completely different way to do master initialization. This is exactly what we I said above, it is not easy to force all the developpers to follow the rule, that the data on zookeeper are not stable. Maybe it can work for now, but maybe it will be broken in the future... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
