[jira] [Commented] (HBASE-23830) TestReplicationEndpoint appears to fail a lot in my attempts for a clean test run locally.
[
https://issues.apache.org/jira/browse/HBASE-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035088#comment-17035088
]
Michael Stack commented on HBASE-23830:
---
This is an ugly test. Replicating between two regions. It is really old too w/
loads of configuration. Need to dig in on why we stop replicating. Thanks for
logs.
> TestReplicationEndpoint appears to fail a lot in my attempts for a clean test
> run locally.
> --
>
> Key: HBASE-23830
> URL: https://issues.apache.org/jira/browse/HBASE-23830
> Project: HBase
> Issue Type: Test
>Affects Versions: master
>Reporter: Mark Robert Miller
>Priority: Major
> Attachments: test_fails.tar.xz
>
>
> This test is failing for me like 30-40% of the time. Fail seems to usually be
> as below. I've tried increasing the wait timeout but that does not seem to
> help at all.
> {code}
> [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.145 s <<< FAILURE! - in
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint[ERROR] Tests run:
> 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 105.145 s <<< FAILURE! -
> in org.apache.hadoop.hbase.replication.TestReplicationEndpoint[ERROR]
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testInterClusterReplication
> Time elapsed: 38.725 s <<< FAILURE!java.lang.AssertionError: Waiting timed
> out after [30,000] msec Failed to replicate all edits, expected = 2500
> replicated = 2476 at org.junit.Assert.fail(Assert.java:89) at
> org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:203) at
> org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:137) at
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testInterClusterReplication(TestReplicationEndpoint.java:235){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r378058210
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/SlowLogPayload.java
##
@@ -0,0 +1,318 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.client;
+
+import org.apache.hadoop.hbase.util.GsonUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.gson.Gson;
+import org.apache.hbase.thirdparty.com.google.gson.JsonObject;
+import org.apache.hbase.thirdparty.com.google.gson.JsonSerializer;
+
+/**
+ * SlowLog payload for client
+ */
+@InterfaceAudience.Private
+final public class SlowLogPayload {
+
+ private static final Gson GSON = GsonUtil.createGson()
+.setPrettyPrinting()
+.registerTypeAdapter(SlowLogPayload.class, (JsonSerializer)
+ (slowLogPayload, type, jsonSerializationContext) -> {
+Gson gson = new Gson();
+JsonObject jsonObj = (JsonObject) gson.toJsonTree(slowLogPayload);
+if (slowLogPayload.getMultiGetsCount() == 0) {
+ jsonObj.remove("multiGetsCount");
+}
+if (slowLogPayload.getMultiMutationsCount() == 0) {
+ jsonObj.remove("multiMutationsCount");
+}
+if (slowLogPayload.getMultiServiceCalls() == 0) {
+ jsonObj.remove("multiServiceCalls");
+}
+return jsonObj;
+ }).create();
+
+ private long startTime;
+ private int processingTime;
+ private int queueTime;
+ private long responseSize;
+ private String clientAddress;
+ private String serverClass;
+ private String methodName;
+ private String callDetails;
+ private String param;
+ private transient String regionName;
+ private String userName;
+ private int multiGetsCount;
+ private int multiMutationsCount;
+ private int multiServiceCalls;
+
+ public long getStartTime() {
+return startTime;
+ }
+
+ public int getProcessingTime() {
+return processingTime;
+ }
+
+ public int getQueueTime() {
+return queueTime;
+ }
+
+ public long getResponseSize() {
+return responseSize;
+ }
+
+ public String getClientAddress() {
+return clientAddress;
+ }
+
+ public String getServerClass() {
+return serverClass;
+ }
+
+ public String getMethodName() {
+return methodName;
+ }
+
+ public String getCallDetails() {
+return callDetails;
+ }
+
+ public String getParam() {
+return param;
+ }
+
+ public String getRegionName() {
+return regionName;
+ }
+
+ public String getUserName() {
+return userName;
+ }
+
+ public int getMultiGetsCount() {
+return multiGetsCount;
+ }
+
+ public int getMultiMutationsCount() {
+return multiMutationsCount;
+ }
+
+ public int getMultiServiceCalls() {
+return multiServiceCalls;
+ }
+
+ private SlowLogPayload(final long startTime, final int processingTime, final
int queueTime,
+ final long responseSize, final String clientAddress, final String
serverClass,
+ final String methodName, final String callDetails, final String param,
+ final String regionName, final String userName, final int multiGetsCount,
+ final int multiMutationsCount, final int multiServiceCalls) {
+this.startTime = startTime;
+this.processingTime = processingTime;
+this.queueTime = queueTime;
+this.responseSize = responseSize;
+this.clientAddress = clientAddress;
+this.serverClass = serverClass;
+this.methodName = methodName;
+this.callDetails = callDetails;
+this.param = param;
+this.regionName = regionName;
+this.userName = userName;
+this.multiGetsCount = multiGetsCount;
+this.multiMutationsCount = multiMutationsCount;
+this.multiServiceCalls = multiServiceCalls;
+ }
+
+ public static class SlowLogPayloadBuilder {
+private long startTime;
+private int processingTime;
+private int queueTime;
+private long responseSize;
+private String clientAddress;
+private String serverClass;
+private String methodName;
+private String callDetails;
+private String param;
+private String regionName;
+p
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r378055692
##
File path: hbase-shell/src/main/ruby/hbase/admin.rb
##
@@ -1453,6 +1453,97 @@ def list_decommissioned_regionservers
@admin.listDecommissionedRegionServers
end
+
#--
+# Retrieve SlowLog Responses from RegionServers
+def get_slowlog_responses(server_names, args)
+ unless server_names.is_a?(Array) || server_names.is_a?(String)
+raise(ArgumentError,
+ "#{server_names.class} of #{server_names.inspect} is not of
Array/String type")
+ end
+ if server_names == '*'
+server_names = getServerNames([], true)
+ else
+server_names_list = get_server_names(server_names)
+server_names = getServerNames(server_names_list, false)
+ end
+ slow_log_responses = @admin.getSlowLogResponses(server_names)
+ filtered_log_responses = filter_slow_responses(args,
+ slow_log_responses)
+ puts 'Retrieved SlowLog Responses from RegionServers.'
+ if args.empty?
+puts slow_log_responses.to_s
+ else
+puts filtered_log_responses.to_s
+ end
+end
+
+def filter_slow_responses(args, slow_log_responses)
+ filtered_log_responses = java.util.ArrayList.new
Review comment:
Just re-iterating the comment on Jira to better align it here:
```
Copying Andrew's earlier comment:
"Filtering or sorting on other attributes would be done locally in the
client. I think it best to let the client index the list of ResponseDetail
however it likes.
The shell commands are one client of the admin APIs. This seems a good place
to put additional convenience filtering."
Although I agree with scalability concern but I believe filter done by
client(shell) might not be too much of an overhead right?
```
Even for quite hugh cluster, shell should be able to process it without
trouble.
Please suggest.
FYI @apurtell
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r378057276
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/slowlog/SlowLogEventHandler.java
##
@@ -0,0 +1,137 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver.slowlog;
+
+import com.lmax.disruptor.EventHandler;
+import com.lmax.disruptor.RingBuffer;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.TooSlowLog.SlowLogPayload;
+
+/**
+ * Event Handler run by disruptor ringbuffer consumer
+ */
+@InterfaceAudience.Private
+class SlowLogEventHandler implements EventHandler {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SlowLogEventHandler.class);
+
+ private final SlowLogPayload[] slowLogPayloads;
+
+ private AtomicInteger slowLogPayloadIdx = new AtomicInteger();
+
+ private static final Lock LOCK = new ReentrantLock();
+
+ private static final int WRITE_LOCK_WAIT = 1;
+ private static final int READ_LOCK_WAIT = 1;
+
+ SlowLogEventHandler(int eventCount) {
+this.slowLogPayloads = new SlowLogPayload[eventCount];
+ }
+
+ /**
+ * Called when a publisher has published an event to the {@link RingBuffer}
+ *
+ * @param event published to the {@link RingBuffer}
+ * @param sequence of the event being processed
+ * @param endOfBatch flag to indicate if this is the last event in a batch
from
+ * the {@link RingBuffer}
+ * @throws Exception if the EventHandler would like the exception handled
further up the chain
+ */
+ @Override
+ public void onEvent(RingBufferTruck event, long sequence, boolean
endOfBatch) throws Exception {
+SlowLogPayload slowLogPayload = event.getPayload();
+if (LOG.isTraceEnabled()) {
+ LOG.trace("Received Slow Log Event. RingBuffer sequence: {},
isEndOfBatch: {}, " +
+ "Event Call: {}", sequence, endOfBatch,
+slowLogPayload.getCallDetails());
+}
+try {
+ if (LOCK.tryLock(WRITE_LOCK_WAIT, TimeUnit.SECONDS)) {
+int index = slowLogPayloadIdx.getAndSet(
+ (slowLogPayloadIdx.get() + 1) % slowLogPayloads.length);
+slowLogPayloads[index] = slowLogPayload;
+ }
Review comment:
Based on the nature of ring buffer, we anyways are going to lose some imp
debug log which happened long time back hence we are not much concerned about
logging every single detail here, even if we miss a couple of them, it should
not be end of the world IMHO.
What we do want to log is complete info of ongoing RPC calls and most
importantly identify one of those regular slow log events based on which we can
make further decisions. With ring buffer, anyways we might lose one-off cases.
Hence, as part of sub-task, we are going to provide an option to user for
saving all slow logs with complete info in HDFS files.
cc: @apurtell
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1162: HBASE-23816 [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS!
Apache-HBase commented on issue #1162: HBASE-23816 [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS! URL: https://github.com/apache/hbase/pull/1162#issuecomment-585048298 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 27s | branch-2 passed | | +1 :green_heart: | compile | 0m 29s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 21s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | branch-2 passed | | +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 50s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 49s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 15m 32s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 20s | the patch passed | | +1 :green_heart: | findbugs | 0m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 10m 52s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 56m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1162/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1162 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux f0a3d60c92e0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1162/out/precommit/personality/provided.sh | | git revision | branch-2 / 50eaa0293e | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1162/4/testReport/ | | Max. process+thread count | 7704 (vs. ulimit of 1) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1162/4/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095
saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164#discussion_r378054663 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. +* If there is any issue with ZK (like connection overload / timeouts), the entire HBase service appears frozen and there is little visibility into it. +* Exposing zookeeper to all the clients can be risky since it can potentially be abused to DDOS. +* Embedded ZK client is bundled with hbase client jar as a dependency (with it’s log spew :-]). + +== Goal + +We would like to remove this ZK dependency in the HBase client and instead have the clients query a preconfigured list of active and standby master host:port addresses. This brings all the client interactions with HBase under the same RPC framework that is holistically controlled by a set of hbase client configuration parameters. This also alleviates the pressure on ZK cluster which is critical from an operational standpoint as some core processes like replication, log splitting, master election etc. depend on it. The next section describes the kind of changes needed on both server and client side to support this behavior. + +== Proposed design + +As mentioned above, clients now get a pre configured list active and standby master addresses that they can query to fetch the meta information needed for connection setup. Something like, + +[source, xml] +- + + hbase.masters + master1:16000,master2:16001,master3:16000 + +- + +Clients should be robust enough to handle configuration changes to this parameter since master hosts can change (added/removed) over time and not every client can afford a restart. + +One thing to note here is that having masters in the init/read/write path for clients means that + +* At Least one active/standby master is now needed for connection creation. Earlier this was not a requirement because the clients looked up the cluster ID from the relevant znode and init successfully. So, technically a master need not be around to create a connection to the cluster. +* Masters are now active part of read write path in client life cycle under certain scenarios. If the client cache of meta locations/active master is purged/stale, at least one master (active/stand-by) serving the latest information should exist. Earlier this information was served by ZK and clients look up the latest cluster ID/active master/meta locations from the relevant znodes and get going. +* There is a higher connection load on the masters than before. +* More state synchronization traffic (see below) + +End users should factor these requirements into their cluster deployment if they intend to use this feature. + +=== Server side changes + +Now that the master end points are considered as source of truth for clients, they should track the latest meta information for clusterID, active master and meta table locations. Since the clients can connect to any master end point
[GitHub] [hbase] saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095
saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164#discussion_r378053858 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. +* If there is any issue with ZK (like connection overload / timeouts), the entire HBase service appears frozen and there is little visibility into it. +* Exposing zookeeper to all the clients can be risky since it can potentially be abused to DDOS. +* Embedded ZK client is bundled with hbase client jar as a dependency (with it’s log spew :-]). + +== Goal Review comment: s/Goal/Description/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095
saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164#discussion_r378054583 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. +* If there is any issue with ZK (like connection overload / timeouts), the entire HBase service appears frozen and there is little visibility into it. +* Exposing zookeeper to all the clients can be risky since it can potentially be abused to DDOS. +* Embedded ZK client is bundled with hbase client jar as a dependency (with it’s log spew :-]). + +== Goal + +We would like to remove this ZK dependency in the HBase client and instead have the clients query a preconfigured list of active and standby master host:port addresses. This brings all the client interactions with HBase under the same RPC framework that is holistically controlled by a set of hbase client configuration parameters. This also alleviates the pressure on ZK cluster which is critical from an operational standpoint as some core processes like replication, log splitting, master election etc. depend on it. The next section describes the kind of changes needed on both server and client side to support this behavior. + +== Proposed design + +As mentioned above, clients now get a pre configured list active and standby master addresses that they can query to fetch the meta information needed for connection setup. Something like, + +[source, xml] +- + + hbase.masters + master1:16000,master2:16001,master3:16000 + +- + +Clients should be robust enough to handle configuration changes to this parameter since master hosts can change (added/removed) over time and not every client can afford a restart. + +One thing to note here is that having masters in the init/read/write path for clients means that + +* At Least one active/standby master is now needed for connection creation. Earlier this was not a requirement because the clients looked up the cluster ID from the relevant znode and init successfully. So, technically a master need not be around to create a connection to the cluster. +* Masters are now active part of read write path in client life cycle under certain scenarios. If the client cache of meta locations/active master is purged/stale, at least one master (active/stand-by) serving the latest information should exist. Earlier this information was served by ZK and clients look up the latest cluster ID/active master/meta locations from the relevant znodes and get going. +* There is a higher connection load on the masters than before. +* More state synchronization traffic (see below) + +End users should factor these requirements into their cluster deployment if they intend to use this feature. + +=== Server side changes + +Now that the master end points are considered as source of truth for clients, they should track the latest meta information for clusterID, active master and meta table locations. Since the clients can connect to any master end point
[GitHub] [hbase] saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095
saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164#discussion_r378054145 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. +* If there is any issue with ZK (like connection overload / timeouts), the entire HBase service appears frozen and there is little visibility into it. +* Exposing zookeeper to all the clients can be risky since it can potentially be abused to DDOS. +* Embedded ZK client is bundled with hbase client jar as a dependency (with it’s log spew :-]). + +== Goal + +We would like to remove this ZK dependency in the HBase client and instead have the clients query a preconfigured list of active and standby master host:port addresses. This brings all the client interactions with HBase under the same RPC framework that is holistically controlled by a set of hbase client configuration parameters. This also alleviates the pressure on ZK cluster which is critical from an operational standpoint as some core processes like replication, log splitting, master election etc. depend on it. The next section describes the kind of changes needed on both server and client side to support this behavior. + +== Proposed design Review comment: This is the refguide. Audience is operators. Also feature is 'done' so s/Proposed// This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095
saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164#discussion_r378052401 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. Review comment: s/source of truth/store/ ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095
saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164#discussion_r378052549 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. Review comment: Good This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095
saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164#discussion_r378054311 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. +* If there is any issue with ZK (like connection overload / timeouts), the entire HBase service appears frozen and there is little visibility into it. +* Exposing zookeeper to all the clients can be risky since it can potentially be abused to DDOS. +* Embedded ZK client is bundled with hbase client jar as a dependency (with it’s log spew :-]). + +== Goal + +We would like to remove this ZK dependency in the HBase client and instead have the clients query a preconfigured list of active and standby master host:port addresses. This brings all the client interactions with HBase under the same RPC framework that is holistically controlled by a set of hbase client configuration parameters. This also alleviates the pressure on ZK cluster which is critical from an operational standpoint as some core processes like replication, log splitting, master election etc. depend on it. The next section describes the kind of changes needed on both server and client side to support this behavior. + +== Proposed design + +As mentioned above, clients now get a pre configured list active and standby master addresses that they can query to fetch the meta information needed for connection setup. Something like, + +[source, xml] +- + + hbase.masters + master1:16000,master2:16001,master3:16000 + +- + +Clients should be robust enough to handle configuration changes to this parameter since master hosts can change (added/removed) over time and not every client can afford a restart. + +One thing to note here is that having masters in the init/read/write path for clients means that + +* At Least one active/standby master is now needed for connection creation. Earlier this was not a requirement because the clients looked up the cluster ID from the relevant znode and init successfully. So, technically a master need not be around to create a connection to the cluster. +* Masters are now active part of read write path in client life cycle under certain scenarios. If the client cache of meta locations/active master is purged/stale, at least one master (active/stand-by) serving the latest information should exist. Earlier this information was served by ZK and clients look up the latest cluster ID/active master/meta locations from the relevant znodes and get going. +* There is a higher connection load on the masters than before. +* More state synchronization traffic (see below) Review comment: Good This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Servi
[GitHub] [hbase] saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095
saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164#discussion_r378054046 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. +* If there is any issue with ZK (like connection overload / timeouts), the entire HBase service appears frozen and there is little visibility into it. +* Exposing zookeeper to all the clients can be risky since it can potentially be abused to DDOS. +* Embedded ZK client is bundled with hbase client jar as a dependency (with it’s log spew :-]). + +== Goal + +We would like to remove this ZK dependency in the HBase client and instead have the clients query a preconfigured list of active and standby master host:port addresses. This brings all the client interactions with HBase under the same RPC framework that is holistically controlled by a set of hbase client configuration parameters. This also alleviates the pressure on ZK cluster which is critical from an operational standpoint as some core processes like replication, log splitting, master election etc. depend on it. The next section describes the kind of changes needed on both server and client side to support this behavior. Review comment: s/We would like to remove this/This feature removes the/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095
saintstack commented on a change in pull request #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164#discussion_r378054451 ## File path: dev-support/design-docs/HBASE-18095-Zookeeper-less-client-connection-design.adoc ## @@ -0,0 +1,112 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + += HBASE-18095: Zookeeper-less client connection + + +== Context +Currently, Zookeeper (ZK) lies in the critical code path of connection init. To set up a connection to a given HBase cluster, client relies on the zookeeper quorum configured in the client hbase-site.xml and attempts to fetch the following information. + +* ClusterID +* Active HMaster server name +* Meta table region locations + +ZK is deemed the source of truth since other processes that maintain the cluster state persist the changes to this data into ZK. So it is an obvious place to look at for clients to fetch the latest cluster state. However this comes with it’s own set of problems, some of them are below. + +* Timeouts and retry logic for ZK clients are managed separately from HBase configuration. This is more administration overhead for end users (example: multiple timeouts are to be configured for different types of RPCs. client->master, client->ZK etc.). This prevents HBase from having a single holistic timeout configuration that applies to any RPCs. +* If there is any issue with ZK (like connection overload / timeouts), the entire HBase service appears frozen and there is little visibility into it. +* Exposing zookeeper to all the clients can be risky since it can potentially be abused to DDOS. +* Embedded ZK client is bundled with hbase client jar as a dependency (with it’s log spew :-]). + +== Goal + +We would like to remove this ZK dependency in the HBase client and instead have the clients query a preconfigured list of active and standby master host:port addresses. This brings all the client interactions with HBase under the same RPC framework that is holistically controlled by a set of hbase client configuration parameters. This also alleviates the pressure on ZK cluster which is critical from an operational standpoint as some core processes like replication, log splitting, master election etc. depend on it. The next section describes the kind of changes needed on both server and client side to support this behavior. + +== Proposed design + +As mentioned above, clients now get a pre configured list active and standby master addresses that they can query to fetch the meta information needed for connection setup. Something like, + +[source, xml] +- + + hbase.masters + master1:16000,master2:16001,master3:16000 + +- + +Clients should be robust enough to handle configuration changes to this parameter since master hosts can change (added/removed) over time and not every client can afford a restart. + +One thing to note here is that having masters in the init/read/write path for clients means that + +* At Least one active/standby master is now needed for connection creation. Earlier this was not a requirement because the clients looked up the cluster ID from the relevant znode and init successfully. So, technically a master need not be around to create a connection to the cluster. +* Masters are now active part of read write path in client life cycle under certain scenarios. If the client cache of meta locations/active master is purged/stale, at least one master (active/stand-by) serving the latest information should exist. Earlier this information was served by ZK and clients look up the latest cluster ID/active master/meta locations from the relevant znodes and get going. +* There is a higher connection load on the masters than before. +* More state synchronization traffic (see below) + +End users should factor these requirements into their cluster deployment if they intend to use this feature. + +=== Server side changes + +Now that the master end points are considered as source of truth for clients, they should track the latest meta information for clusterID, active master and meta table locations. Since the clients can connect to any master end point
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r378054411
##
File path: hbase-shell/src/main/ruby/hbase/admin.rb
##
@@ -1453,6 +1453,97 @@ def list_decommissioned_regionservers
@admin.listDecommissionedRegionServers
end
+
#--
+# Retrieve SlowLog Responses from RegionServers
+def get_slowlog_responses(server_names, args)
+ unless server_names.is_a?(Array) || server_names.is_a?(String)
+raise(ArgumentError,
+ "#{server_names.class} of #{server_names.inspect} is not of
Array/String type")
+ end
+ if server_names == '*'
+server_names = getServerNames([], true)
+ else
+server_names_list = get_server_names(server_names)
+server_names = getServerNames(server_names_list, false)
+ end
+ slow_log_responses = @admin.getSlowLogResponses(server_names)
+ filtered_log_responses = filter_slow_responses(args,
+ slow_log_responses)
+ puts 'Retrieved SlowLog Responses from RegionServers.'
+ if args.empty?
+puts slow_log_responses.to_s
+ else
+puts filtered_log_responses.to_s
+ end
+end
+
+def filter_slow_responses(args, slow_log_responses)
+ filtered_log_responses = java.util.ArrayList.new
+ slow_log_responses.each do |slow_log|
+if args.key? 'REGION_NAME'
+ region_name = args['REGION_NAME']
+ unless region_name.nil?
+if region_name.eql?(slow_log.getRegionName)
+ filtered_log_responses.add(slow_log)
+ next
+end
+ end
+end
+if args.key? 'TABLE_NAME'
+ table_name = args['TABLE_NAME']
+ unless table_name.nil?
+if slow_log.getRegionName.start_with?(table_name)
+ filtered_log_responses.add(slow_log)
+ next
+end
+ end
+end
+if args.key? 'CLIENT_IP'
+ client_ip = args['CLIENT_IP']
+ unless client_ip.nil?
+if client_ip.eql?(slow_log.getClientAddress)
+ filtered_log_responses.add(slow_log)
+ next
+end
+ end
+end
+if args.key? 'USER'
+ user = args['USER']
+ unless user.nil?
+if user.eql?(slow_log.getUserName)
+ filtered_log_responses.add(slow_log)
+ next
+end
+ end
+end
+ end
+ filtered_log_responses
+end
+
+
#--
+# Clears SlowLog Responses from RegionServers
+def clear_slowlog_responses(server_names)
+ unless server_names.nil? || server_names.is_a?(Array) ||
server_names.is_a?(String)
+raise(ArgumentError,
+ "#{server_names.class} of #{server_names.inspect} is not of
correct type")
+ end
+ if server_names.nil?
+server_names = getServerNames([], true)
+ else
+server_names_list = get_server_names(server_names)
+server_names = getServerNames(server_names_list, false)
+ end
+ clear_log_responses = @admin.clearSlowLogResponses(server_names)
+ clear_log_success_count = 0
+ clear_log_responses.each do |response|
+if response
+ clear_log_success_count += 1
+end
+ end
+ puts 'Cleared Slowlog responses from ' \
+ "#{clear_log_success_count}/#{clear_log_responses.size}
RegionServers"
Review comment:
Good question, this is bit tricky. If clear log responses didn't go through,
may be client can retry but if it again keeps failing for some RS, it would be
nice to display server names where it failed but again that is difficult to
implement. Might fit well for a sub-task.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[ https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035068#comment-17035068 ] HBase QA commented on HBASE-23823: -- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HBASE-Build/1127/console in case of problems. > Run maven with more than default single thread (--threads=0.5C) > --- > > Key: HBASE-23823 > URL: https://issues.apache.org/jira/browse/HBASE-23823 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > > > See tail of parent task where we made commits to hbase-personality trying to > make maven run more furiously on nightly builds by upping the maven thread > count (the --threads flag ... see > https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3). > The effort was abandoned because it unsettled the build. This issue > separates messing with the maven threads flag from the broader topic the > parent covers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[ https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035067#comment-17035067 ] Michael Stack commented on HBASE-23823: --- The failure in unit tests was this in hbase-common: "The forked VM terminated without properly saying goodbye. VM crash or System.exit called?" > Run maven with more than default single thread (--threads=0.5C) > --- > > Key: HBASE-23823 > URL: https://issues.apache.org/jira/browse/HBASE-23823 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > > > See tail of parent task where we made commits to hbase-personality trying to > make maven run more furiously on nightly builds by upping the maven thread > count (the --threads flag ... see > https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3). > The effort was abandoned because it unsettled the build. This issue > separates messing with the maven threads flag from the broader topic the > parent covers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[ https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23823: -- Attachment: 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > Run maven with more than default single thread (--threads=0.5C) > --- > > Key: HBASE-23823 > URL: https://issues.apache.org/jira/browse/HBASE-23823 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > > > See tail of parent task where we made commits to hbase-personality trying to > make maven run more furiously on nightly builds by upping the maven thread > count (the --threads flag ... see > https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3). > The effort was abandoned because it unsettled the build. This issue > separates messing with the maven threads flag from the broader topic the > parent covers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[
https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035061#comment-17035061
]
HBase QA commented on HBASE-23823:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
7s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
2s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
31s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 15s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 4s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/1126/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-23823 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12993219/0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch
|
| Optional Tests | dupname asflicense shellcheck shelldocs javac javadoc unit
shadedjars hadoopcheck xml compile |
| uname | Linux 96fdf2b4296c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 12f98bc50e |
| Default Java | 1.8.0_181 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/1126/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/1126/testReport/ |
| Max. process+thread count | 397 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1126/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.11.1 https://yet
[jira] [Commented] (HBASE-23825) Increment proto conversion is broken
[ https://issues.apache.org/jira/browse/HBASE-23825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035056#comment-17035056 ] Sean Busbey commented on HBASE-23825: - bq. Do you need someone to take over HBASE-23220 Sean Busbey? Happy to do it after this lands. ah yes, good catch. please do. > Increment proto conversion is broken > > > Key: HBASE-23825 > URL: https://issues.apache.org/jira/browse/HBASE-23825 > Project: HBase > Issue Type: Bug > Components: Increment >Affects Versions: 1.4.0, 1.2.6, 1.3.2, 1.4.1, 1.5.0, 1.1.11, 1.3.3, 1.4.2, > 1.4.3, 1.4.4, 1.4.5, 1.3.2.1, 1.4.6, 1.4.8, 1.4.7, 1.4.9, 1.4.10, 1.3.4, > 1.3.5, 1.3.6, 1.4.11, 1.4.12 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Blocker > Fix For: 1.6.0, 1.3.7, 1.4.13 > > > While converting the request back to Increment using ProtobufUtil.toIncrement > we incorrectly use the optimization to avoid copying the byte > array(HBaseZeroCopyByteString#zeroCopyGetBytes) on a BoundedByteString. The > optimization was only meant for LiteralByteString where it is safe to use the > backing byte array, however it ends up being used to BoundedByteString which > is a subclass of LiteralByteString. This essentially breaks increments since > we end up creating wrong cells on the server side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on issue #1162: HBASE-23816 [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS!
saintstack commented on issue #1162: HBASE-23816 [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS! URL: https://github.com/apache/hbase/pull/1162#issuecomment-585033263 Fix checkstyle complaints This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
saintstack commented on a change in pull request #1141: HBASE-23808 [Flakey
Test] TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r378035129
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -2804,9 +2804,15 @@ public MemoryBoundedLogMessageBuffer
getRegionServerFatalLogBuffer() {
* Master runs a coordinated stop of all RegionServers and then itself.
*/
public void shutdown() throws IOException {
+if (!isInitialized()) {
+ LOG.info("Shutdown requested but we're not the active master. Proceeding
as a stop.");
Review comment:
Yeah, this stuff is tricky and there be dragons if you disturb the current
order.
The standbys don't go down currently?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
saintstack commented on a change in pull request #1141: HBASE-23808 [Flakey
Test] TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r378036278
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -66,43 +78,45 @@ public void testMasterShutdown() throws Exception {
Configuration conf = HBaseConfiguration.create();
// Start the cluster
-HBaseTestingUtility htu = new HBaseTestingUtility(conf);
-StartMiniClusterOption option = StartMiniClusterOption.builder()
-
.numMasters(NUM_MASTERS).numRegionServers(NUM_RS).numDataNodes(NUM_RS).build();
-htu.startMiniCluster(option);
-MiniHBaseCluster cluster = htu.getHBaseCluster();
-
-// get all the master threads
-List masterThreads = cluster.getMasterThreads();
-
-// wait for each to come online
-for (MasterThread mt : masterThreads) {
- assertTrue(mt.isAlive());
-}
-
-// find the active master
-HMaster active = null;
-for (int i = 0; i < masterThreads.size(); i++) {
- if (masterThreads.get(i).getMaster().isActiveMaster()) {
-active = masterThreads.get(i).getMaster();
-break;
+try {
+ htu = new HBaseTestingUtility(conf);
Review comment:
On hbase.ipc.client.failed.servers.expiry being aggressive... remove the
config?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
saintstack commented on a change in pull request #1141: HBASE-23808 [Flakey
Test] TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r378035422
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -30,26 +34,38 @@
import org.apache.hadoop.hbase.LocalHBaseCluster;
import org.apache.hadoop.hbase.MiniHBaseCluster;
import org.apache.hadoop.hbase.StartMiniClusterOption;
-import org.apache.hadoop.hbase.client.Admin;
-import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.Waiter;
+import org.apache.hadoop.hbase.client.AsyncConnection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.testclassification.LargeTests;
import org.apache.hadoop.hbase.testclassification.MasterTests;
import org.apache.hadoop.hbase.util.JVMClusterUtil.MasterThread;
+import org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient;
+import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
@Category({MasterTests.class, LargeTests.class})
public class TestMasterShutdown {
+ private static final Logger LOG =
LoggerFactory.getLogger(TestMasterShutdown.class);
@ClassRule
public static final HBaseClassTestRule CLASS_RULE =
HBaseClassTestRule.forClass(TestMasterShutdown.class);
- private static final Logger LOG =
LoggerFactory.getLogger(TestMasterShutdown.class);
+ private HBaseTestingUtility htu;
+
+ @Before
+ public void shutdownCluster() throws IOException {
+if (htu != null) {
Review comment:
We can skip an @After ? i.e. should the shutdown of cluster be in @After if
not there already.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
saintstack commented on a change in pull request #1141: HBASE-23808 [Flakey
Test] TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r378036496
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -66,103 +82,172 @@ public void testMasterShutdown() throws Exception {
Configuration conf = HBaseConfiguration.create();
// Start the cluster
-HBaseTestingUtility htu = new HBaseTestingUtility(conf);
-StartMiniClusterOption option = StartMiniClusterOption.builder()
-
.numMasters(NUM_MASTERS).numRegionServers(NUM_RS).numDataNodes(NUM_RS).build();
-htu.startMiniCluster(option);
-MiniHBaseCluster cluster = htu.getHBaseCluster();
-
-// get all the master threads
-List masterThreads = cluster.getMasterThreads();
-
-// wait for each to come online
-for (MasterThread mt : masterThreads) {
- assertTrue(mt.isAlive());
-}
-
-// find the active master
-HMaster active = null;
-for (int i = 0; i < masterThreads.size(); i++) {
- if (masterThreads.get(i).getMaster().isActiveMaster()) {
-active = masterThreads.get(i).getMaster();
-break;
+try {
+ htu = new HBaseTestingUtility(conf);
+ StartMiniClusterOption option = StartMiniClusterOption.builder()
+.numMasters(NUM_MASTERS)
+.numRegionServers(NUM_RS)
+.numDataNodes(NUM_RS)
+.build();
+ final MiniHBaseCluster cluster = htu.startMiniCluster(option);
+
+ // wait for all master thread to spawn and start their run loop.
+ final long thirtySeconds = TimeUnit.SECONDS.toMillis(30);
+ final long oneSecond = TimeUnit.SECONDS.toMillis(1);
+ assertNotEquals(-1, htu.waitFor(thirtySeconds, oneSecond, () -> {
+final List masterThreads = cluster.getMasterThreads();
+return masterThreads != null
+ && masterThreads.size() >= 3
+ && masterThreads.stream().allMatch(Thread::isAlive);
+ }));
+
+ // find the active master
+ final HMaster active = cluster.getMaster();
+ assertNotNull(active);
+
+ // make sure the other two are backup masters
+ ClusterMetrics status = active.getClusterMetrics();
+ assertEquals(2, status.getBackupMasterNames().size());
+
+ // tell the active master to shutdown the cluster
+ active.shutdown();
+ assertNotEquals(-1, htu.waitFor(thirtySeconds, oneSecond,
+() -> CollectionUtils.isEmpty(cluster.getLiveMasterThreads(;
+ assertNotEquals(-1, htu.waitFor(thirtySeconds, oneSecond,
+() -> CollectionUtils.isEmpty(cluster.getLiveRegionServerThreads(;
+} finally {
+ if (htu != null) {
Review comment:
Somehow we are skipping this? Odd. Move to an @After?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[ https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035041#comment-17035041 ] HBase QA commented on HBASE-23823: -- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HBASE-Build/1126/console in case of problems. > Run maven with more than default single thread (--threads=0.5C) > --- > > Key: HBASE-23823 > URL: https://issues.apache.org/jira/browse/HBASE-23823 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > > > See tail of parent task where we made commits to hbase-personality trying to > make maven run more furiously on nightly builds by upping the maven thread > count (the --threads flag ... see > https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3). > The effort was abandoned because it unsettled the build. This issue > separates messing with the maven threads flag from the broader topic the > parent covers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[ https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23823: -- Attachment: 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > Run maven with more than default single thread (--threads=0.5C) > --- > > Key: HBASE-23823 > URL: https://issues.apache.org/jira/browse/HBASE-23823 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch, > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > > > See tail of parent task where we made commits to hbase-personality trying to > make maven run more furiously on nightly builds by upping the maven thread > count (the --threads flag ... see > https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3). > The effort was abandoned because it unsettled the build. This issue > separates messing with the maven threads flag from the broader topic the > parent covers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[ https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035031#comment-17035031 ] Michael Stack commented on HBASE-23823: --- This took the --threads arg. File count rose to almost 7674. Let me retry. > Run maven with more than default single thread (--threads=0.5C) > --- > > Key: HBASE-23823 > URL: https://issues.apache.org/jira/browse/HBASE-23823 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > > > See tail of parent task where we made commits to hbase-personality trying to > make maven run more furiously on nightly builds by upping the maven thread > count (the --threads flag ... see > https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3). > The effort was abandoned because it unsettled the build. This issue > separates messing with the maven threads flag from the broader topic the > parent covers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bharathv commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
bharathv commented on a change in pull request #1141: HBASE-23808 [Flakey Test]
TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r378018435
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -2804,9 +2804,15 @@ public MemoryBoundedLogMessageBuffer
getRegionServerFatalLogBuffer() {
* Master runs a coordinated stop of all RegionServers and then itself.
*/
public void shutdown() throws IOException {
+if (!isInitialized()) {
+ LOG.info("Shutdown requested but we're not the active master. Proceeding
as a stop.");
Review comment:
Well, I'm not entirely sure if this is the intended way to stop stand-by
masters. Reading the code.. it looks like the active master coordinates a
proper cluster shutdown. In that process, it removes the /hbase/running znode
that the standb-bys keep a watch on. See the following code in
ActiveMasterManager..
```
@Override
public void nodeDeleted(String path) {
// We need to keep track of the cluster's shutdown status while
// we wait on the current master. We consider that, if the cluster
// was already in a "shutdown" state when we started, that this master
// is part of a new cluster that was started shortly after the old
cluster
// shut down, so that state is now irrelevant. This means that the
shutdown
// state must be set while we wait on the active master in order
// to shutdown this master. See HBASE-8519.
if(path.equals(watcher.getZNodePaths().clusterStateZNode) &&
!master.isStopped()) {
clusterShutDown.set(true);
}
```
Ideally they should shut themselves down if the ZK event notifications
happen as expected. Is that not the case?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1164: HBASE-23331: Documentation for HBASE-18095
Apache-HBase commented on issue #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164#issuecomment-584998819 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-18095/client-locate-meta-no-zookeeper Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 32s | HBASE-18095/client-locate-meta-no-zookeeper passed | | +0 :ok: | refguide | 4m 46s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 4s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 4m 50s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 19s | The patch does not generate ASF License warnings. | | | | 22m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1164/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1164 | | JIRA Issue | HBASE-23331 | | Optional Tests | dupname asflicense refguide | | uname | Linux 40c0e195bda9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1164/out/precommit/personality/provided.sh | | git revision | HBASE-18095/client-locate-meta-no-zookeeper / 2eead6c9ff | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1164/1/artifact/out/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1164/1/artifact/out/patch-site/book.html | | Max. process+thread count | 96 (vs. ulimit of 1) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1164/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23331) Document HBASE-18095
[
https://issues.apache.org/jira/browse/HBASE-23331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035006#comment-17035006
]
HBase QA commented on HBASE-23331:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} HBASE-18095/client-locate-meta-no-zookeeper Compile
Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
32s{color} | {color:green} HBASE-18095/client-locate-meta-no-zookeeper passed
{color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
46s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
50s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1164/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hbase/pull/1164 |
| JIRA Issue | HBASE-23331 |
| Optional Tests | dupname asflicense refguide |
| uname | Linux 40c0e195bda9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1164/out/precommit/personality/provided.sh
|
| git revision | HBASE-18095/client-locate-meta-no-zookeeper / 2eead6c9ff |
| refguide |
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1164/1/artifact/out/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1164/1/artifact/out/patch-site/book.html
|
| Max. process+thread count | 96 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1164/1/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Document HBASE-18095
>
>
> Key: HBASE-23331
> URL: https://issues.apache.org/jira/browse/HBASE-23331
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Major
>
> Just a placeholder for documenting the parent jira. We should talk about the
> new configurations added and how to use them and intention behind the design.
> Documentation should include compatibility matrix between old and new client
> and server along with (rolling) upgrade procedure if any.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 opened a new pull request #1165: HBASE-22514 Move rsgroup feature into core of HBase
Apache9 opened a new pull request #1165: HBASE-22514 Move rsgroup feature into core of HBase URL: https://github.com/apache/hbase/pull/1165 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23827) [HBCK2] Add mergeRegions method to hbck service
[ https://issues.apache.org/jira/browse/HBASE-23827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-23827: -- Fix Version/s: (was: 2.1.9) 2.1.10 > [HBCK2] Add mergeRegions method to hbck service > --- > > Key: HBASE-23827 > URL: https://issues.apache.org/jira/browse/HBASE-23827 > Project: HBase > Issue Type: Task >Reporter: Sakthi >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.4, 2.1.10 > > > Let's add the mergeRegions method to the hbck service so that hbck2 can use > it from the client side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
Apache-HBase commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore… URL: https://github.com/apache/hbase/pull/1141#issuecomment-584993304 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 48s | master passed | | +1 :green_heart: | compile | 0m 58s | master passed | | +1 :green_heart: | checkstyle | 1m 23s | master passed | | +1 :green_heart: | shadedjars | 5m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | | +0 :ok: | spotbugs | 4m 38s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 36s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 16s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | checkstyle | 1m 17s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 16m 42s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | | +1 :green_heart: | findbugs | 4m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 58m 39s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 118m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1141 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 32ae11ce03fa 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1141/out/precommit/personality/provided.sh | | git revision | master / 12f98bc50e | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/3/testReport/ | | Max. process+thread count | 7309 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/3/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22514) Move rsgroup feature into core of HBase
[
https://issues.apache.org/jira/browse/HBASE-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034995#comment-17034995
]
Hudson commented on HBASE-22514:
Results for branch HBASE-22514
[build #270 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/270/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/270//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/270//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22514/270//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Move rsgroup feature into core of HBase
> ---
>
> Key: HBASE-22514
> URL: https://issues.apache.org/jira/browse/HBASE-22514
> Project: HBase
> Issue Type: Umbrella
> Components: Admin, Client, rsgroup
>Reporter: Yechao Chen
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-22514.master.001.patch,
> image-2019-05-31-18-25-38-217.png
>
>
> The class RSGroupAdminClient is not public
> we need to use java api RSGroupAdminClient to manager RSG
> so RSGroupAdminClient should be public
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23331) Document HBASE-18095
[ https://issues.apache.org/jira/browse/HBASE-23331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-23331: - Status: Patch Available (was: Open) > Document HBASE-18095 > > > Key: HBASE-23331 > URL: https://issues.apache.org/jira/browse/HBASE-23331 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Major > > Just a placeholder for documenting the parent jira. We should talk about the > new configurations added and how to use them and intention behind the design. > Documentation should include compatibility matrix between old and new client > and server along with (rolling) upgrade procedure if any. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23331) Document HBASE-18095
[ https://issues.apache.org/jira/browse/HBASE-23331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034994#comment-17034994 ] Bharath Vissapragada commented on HBASE-23331: -- [~busbey] The PR for the docs is up. You mentioned this in the dev discuss thread. So just FYI. > Document HBASE-18095 > > > Key: HBASE-23331 > URL: https://issues.apache.org/jira/browse/HBASE-23331 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Major > > Just a placeholder for documenting the parent jira. We should talk about the > new configurations added and how to use them and intention behind the design. > Documentation should include compatibility matrix between old and new client > and server along with (rolling) upgrade procedure if any. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bharathv opened a new pull request #1164: HBASE-23331: Documentation for HBASE-18095
bharathv opened a new pull request #1164: HBASE-23331: Documentation for HBASE-18095 URL: https://github.com/apache/hbase/pull/1164 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-23818) Cleanup the remaining RSGroupInfo.getTables call in the code base
[ https://issues.apache.org/jira/browse/HBASE-23818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-23818. --- Fix Version/s: HBASE-22514 Hadoop Flags: Reviewed Resolution: Fixed Merged to branch HBASE-22514. Thanks [~stack] and [~vjasani] for reviewing. > Cleanup the remaining RSGroupInfo.getTables call in the code base > - > > Key: HBASE-23818 > URL: https://issues.apache.org/jira/browse/HBASE-23818 > Project: HBase > Issue Type: Sub-task > Components: rsgroup >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-22514 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #1152: HBASE-23818 Cleanup the remaining RSGroupInfo.getTables call in the c…
Apache9 merged pull request #1152: HBASE-23818 Cleanup the remaining RSGroupInfo.getTables call in the c… URL: https://github.com/apache/hbase/pull/1152 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23825) Increment proto conversion is broken
[ https://issues.apache.org/jira/browse/HBASE-23825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034962#comment-17034962 ] Hudson commented on HBASE-23825: FAILURE: Integrated in Jenkins build HBase-1.3-IT #676 (See [https://builds.apache.org/job/HBase-1.3-IT/676/]) HBASE-23825 Increment proto conversion is broken (achouhan: rev 666c78d5ed03d30493d75c7f322d28e1a18601aa) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestProtobufUtil.java > Increment proto conversion is broken > > > Key: HBASE-23825 > URL: https://issues.apache.org/jira/browse/HBASE-23825 > Project: HBase > Issue Type: Bug > Components: Increment >Affects Versions: 1.4.0, 1.2.6, 1.3.2, 1.4.1, 1.5.0, 1.1.11, 1.3.3, 1.4.2, > 1.4.3, 1.4.4, 1.4.5, 1.3.2.1, 1.4.6, 1.4.8, 1.4.7, 1.4.9, 1.4.10, 1.3.4, > 1.3.5, 1.3.6, 1.4.11, 1.4.12 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Blocker > Fix For: 1.6.0, 1.3.7, 1.4.13 > > > While converting the request back to Increment using ProtobufUtil.toIncrement > we incorrectly use the optimization to avoid copying the byte > array(HBaseZeroCopyByteString#zeroCopyGetBytes) on a BoundedByteString. The > optimization was only meant for LiteralByteString where it is safe to use the > backing byte array, however it ends up being used to BoundedByteString which > is a subclass of LiteralByteString. This essentially breaks increments since > we end up creating wrong cells on the server side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test]
TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r377993610
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -2804,9 +2804,15 @@ public MemoryBoundedLogMessageBuffer
getRegionServerFatalLogBuffer() {
* Master runs a coordinated stop of all RegionServers and then itself.
*/
public void shutdown() throws IOException {
+if (!isInitialized()) {
+ LOG.info("Shutdown requested but we're not the active master. Proceeding
as a stop.");
Review comment:
I think another reasonable alternative here is to reject the RPC back to the
client with some form of `DoNotRetryIOException` and a message about cannot
shutdown from an inactive master.
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (HBASE-23825) Increment proto conversion is broken
[ https://issues.apache.org/jira/browse/HBASE-23825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan resolved HBASE-23825. Hadoop Flags: Reviewed Resolution: Fixed > Increment proto conversion is broken > > > Key: HBASE-23825 > URL: https://issues.apache.org/jira/browse/HBASE-23825 > Project: HBase > Issue Type: Bug > Components: Increment >Affects Versions: 1.4.0, 1.2.6, 1.3.2, 1.4.1, 1.5.0, 1.1.11, 1.3.3, 1.4.2, > 1.4.3, 1.4.4, 1.4.5, 1.3.2.1, 1.4.6, 1.4.8, 1.4.7, 1.4.9, 1.4.10, 1.3.4, > 1.3.5, 1.3.6, 1.4.11, 1.4.12 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Blocker > Fix For: 1.6.0, 1.3.7, 1.4.13 > > > While converting the request back to Increment using ProtobufUtil.toIncrement > we incorrectly use the optimization to avoid copying the byte > array(HBaseZeroCopyByteString#zeroCopyGetBytes) on a BoundedByteString. The > optimization was only meant for LiteralByteString where it is safe to use the > backing byte array, however it ends up being used to BoundedByteString which > is a subclass of LiteralByteString. This essentially breaks increments since > we end up creating wrong cells on the server side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23825) Increment proto conversion is broken
[ https://issues.apache.org/jira/browse/HBASE-23825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034960#comment-17034960 ] Abhishek Singh Chouhan commented on HBASE-23825: I've pushed to branch-1, 1.4, 1.3. Thanks for the reviews [~apurtell] [~anoop.hbase] [~sakthi] > Increment proto conversion is broken > > > Key: HBASE-23825 > URL: https://issues.apache.org/jira/browse/HBASE-23825 > Project: HBase > Issue Type: Bug > Components: Increment >Affects Versions: 1.4.0, 1.2.6, 1.3.2, 1.4.1, 1.5.0, 1.1.11, 1.3.3, 1.4.2, > 1.4.3, 1.4.4, 1.4.5, 1.3.2.1, 1.4.6, 1.4.8, 1.4.7, 1.4.9, 1.4.10, 1.3.4, > 1.3.5, 1.3.6, 1.4.11, 1.4.12 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Blocker > Fix For: 1.6.0, 1.3.7, 1.4.13 > > > While converting the request back to Increment using ProtobufUtil.toIncrement > we incorrectly use the optimization to avoid copying the byte > array(HBaseZeroCopyByteString#zeroCopyGetBytes) on a BoundedByteString. The > optimization was only meant for LiteralByteString where it is safe to use the > backing byte array, however it ends up being used to BoundedByteString which > is a subclass of LiteralByteString. This essentially breaks increments since > we end up creating wrong cells on the server side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] abhishek-chouhan merged pull request #1161: HBASE-23825 Increment proto conversion is broken
abhishek-chouhan merged pull request #1161: HBASE-23825 Increment proto conversion is broken URL: https://github.com/apache/hbase/pull/1161 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test]
TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r377988314
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -66,103 +82,172 @@ public void testMasterShutdown() throws Exception {
Configuration conf = HBaseConfiguration.create();
// Start the cluster
-HBaseTestingUtility htu = new HBaseTestingUtility(conf);
-StartMiniClusterOption option = StartMiniClusterOption.builder()
-
.numMasters(NUM_MASTERS).numRegionServers(NUM_RS).numDataNodes(NUM_RS).build();
-htu.startMiniCluster(option);
-MiniHBaseCluster cluster = htu.getHBaseCluster();
-
-// get all the master threads
-List masterThreads = cluster.getMasterThreads();
-
-// wait for each to come online
-for (MasterThread mt : masterThreads) {
- assertTrue(mt.isAlive());
-}
-
-// find the active master
-HMaster active = null;
-for (int i = 0; i < masterThreads.size(); i++) {
- if (masterThreads.get(i).getMaster().isActiveMaster()) {
-active = masterThreads.get(i).getMaster();
-break;
+try {
+ htu = new HBaseTestingUtility(conf);
+ StartMiniClusterOption option = StartMiniClusterOption.builder()
+.numMasters(NUM_MASTERS)
+.numRegionServers(NUM_RS)
+.numDataNodes(NUM_RS)
+.build();
+ final MiniHBaseCluster cluster = htu.startMiniCluster(option);
+
+ // wait for all master thread to spawn and start their run loop.
+ final long thirtySeconds = TimeUnit.SECONDS.toMillis(30);
+ final long oneSecond = TimeUnit.SECONDS.toMillis(1);
+ assertNotEquals(-1, htu.waitFor(thirtySeconds, oneSecond, () -> {
+final List masterThreads = cluster.getMasterThreads();
+return masterThreads != null
+ && masterThreads.size() >= 3
+ && masterThreads.stream().allMatch(Thread::isAlive);
+ }));
+
+ // find the active master
+ final HMaster active = cluster.getMaster();
+ assertNotNull(active);
+
+ // make sure the other two are backup masters
+ ClusterMetrics status = active.getClusterMetrics();
+ assertEquals(2, status.getBackupMasterNames().size());
+
+ // tell the active master to shutdown the cluster
+ active.shutdown();
+ assertNotEquals(-1, htu.waitFor(thirtySeconds, oneSecond,
+() -> CollectionUtils.isEmpty(cluster.getLiveMasterThreads(;
+ assertNotEquals(-1, htu.waitFor(thirtySeconds, oneSecond,
+() -> CollectionUtils.isEmpty(cluster.getLiveRegionServerThreads(;
+} finally {
+ if (htu != null) {
+htu.shutdownMiniCluster();
+htu = null;
}
}
-assertNotNull(active);
-// make sure the other two are backup masters
-ClusterMetrics status = active.getClusterMetrics();
-assertEquals(2, status.getBackupMasterNames().size());
-
-// tell the active master to shutdown the cluster
-active.shutdown();
-
-for (int i = NUM_MASTERS - 1; i >= 0 ;--i) {
- cluster.waitOnMaster(i);
-}
-// make sure all the masters properly shutdown
-assertEquals(0, masterThreads.size());
-
-htu.shutdownMiniCluster();
}
- private Connection createConnection(HBaseTestingUtility util) throws
InterruptedException {
-// the cluster may have not been initialized yet which means we can not
get the cluster id thus
-// an exception will be thrown. So here we need to retry.
-for (;;) {
- try {
-return ConnectionFactory.createConnection(util.getConfiguration());
- } catch (Exception e) {
-Thread.sleep(10);
+ /**
+ * This test appears to be an intentional race between a thread that issues
a shutdown RPC to the
+ * master, while the master is concurrently realizing it cannot initialize
because there are no
+ * region servers available to it. The expected behavior is that master
initialization is
+ * interruptable via said shutdown RPC.
+ */
+ @Test
+ public void testMasterShutdownBeforeStartingAnyRegionServer() throws
Exception {
+LocalHBaseCluster hbaseCluster = null;
+try {
+ htu = new HBaseTestingUtility(
+createMasterShutdownBeforeStartingAnyRegionServerConfiguration());
+
+ // configure a cluster with
+ final StartMiniClusterOption options = StartMiniClusterOption.builder()
+.numDataNodes(1)
+.numMasters(1)
+.numRegionServers(0)
+.masterClass(HMaster.class)
+.rsClass(MiniHBaseCluster.MiniHBaseClusterRegionServer.class)
+.createRootDir(true)
+.build();
+
+ // Can't simply `htu.startMiniCluster(options)` because that method
waits for the master to
+ // start completely. However, this test's premise is that a partially
started master should
+ // still respond to a shutdown RPC. So instead, we manag
[jira] [Created] (HBASE-23831) TestChoreService is very sensitive to resources.
Mark Robert Miller created HBASE-23831: -- Summary: TestChoreService is very sensitive to resources. Key: HBASE-23831 URL: https://issues.apache.org/jira/browse/HBASE-23831 Project: HBase Issue Type: Test Affects Versions: master Reporter: Mark Robert Miller More details following. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore… URL: https://github.com/apache/hbase/pull/1141#issuecomment-584956495 > > More digging, will be back. > > Is there more to it than fixing the ServerManager initialization? Meaning, does the test still fail after my waitFor() fix? That fixes the test, but I think that's exactly the kind of gap that HBASE-8422 was trying to close. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1162: HBASE-23816 [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS!
Apache-HBase commented on issue #1162: HBASE-23816 [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS! URL: https://github.com/apache/hbase/pull/1162#issuecomment-584917069 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 1s | Shelldocs was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 7m 57s | branch-2 passed | | +1 :green_heart: | compile | 4m 8s | branch-2 passed | | +1 :green_heart: | checkstyle | 3m 19s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 4m 18s | branch-2 passed | | +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 17m 47s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 56s | the patch passed | | +1 :green_heart: | compile | 3m 21s | the patch passed | | +1 :green_heart: | javac | 3m 21s | the patch passed | | -1 :x: | checkstyle | 0m 19s | hbase-mapreduce: The patch generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) | | -1 :x: | checkstyle | 2m 35s | root: The patch generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) | | +1 :green_heart: | shellcheck | 0m 4s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedjars | 4m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 21m 16s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 4m 6s | the patch passed | | +1 :green_heart: | findbugs | 17m 58s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 1m 46s | root in the patch failed. | | +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | | | | 120m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1162/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1162 | | Optional Tests | dupname asflicense shellcheck shelldocs javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux f6208474a52a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1162/out/precommit/personality/provided.sh | | git revision | branch-2 / 50eaa0293e | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1162/3/artifact/out/diff-checkstyle-hbase-mapreduce.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1162/3/artifact/out/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1162/3/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1162/3/testReport/ | | Max. process+thread count | 376 (vs. ulimit of 1) | | modules | C: hbase-mapreduce . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1162/3/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) shellcheck=0.7.0 findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
[jira] [Commented] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[
https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034930#comment-17034930
]
HBase QA commented on HBASE-23823:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
6s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
2s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 3s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 14s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.access.TestAccessController3 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/1125/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-23823 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12993101/0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch
|
| Optional Tests | dupname asflicense shellcheck shelldocs javac javadoc unit
shadedjars hadoopcheck xml compile |
| uname | Linux 48d7c463fe58 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 12f98bc50e |
| Default Java | 1.8.0_181 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/1125/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/1125/testReport/ |
| Max. process+thread count | 7674 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1125/console |
| versions |
[GitHub] [hbase] abhishek-chouhan commented on issue #1161: HBASE-23825 Increment proto conversion is broken
abhishek-chouhan commented on issue #1161: HBASE-23825 Increment proto conversion is broken URL: https://github.com/apache/hbase/pull/1161#issuecomment-584903990 As @apurtell pointed out, test failure is unrelated and due to a timeout. Will commit this later today, unless objections. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (HBASE-23743) Release 1.4.13
[ https://issues.apache.org/jira/browse/HBASE-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi reassigned HBASE-23743: -- Assignee: Sakthi (was: Sean Busbey) > Release 1.4.13 > -- > > Key: HBASE-23743 > URL: https://issues.apache.org/jira/browse/HBASE-23743 > Project: HBase > Issue Type: Task > Components: build, community >Affects Versions: 1.4.13 >Reporter: Sean Busbey >Assignee: Sakthi >Priority: Major > Fix For: 1.4.13 > > > over due for 1.4.13. I'll try to use the release manager scripts this time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23825) Increment proto conversion is broken
[ https://issues.apache.org/jira/browse/HBASE-23825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034925#comment-17034925 ] Sakthi commented on HBASE-23825: Yup, let me assign that one to myself [~apurtell] > Increment proto conversion is broken > > > Key: HBASE-23825 > URL: https://issues.apache.org/jira/browse/HBASE-23825 > Project: HBase > Issue Type: Bug > Components: Increment >Affects Versions: 1.4.0, 1.2.6, 1.3.2, 1.4.1, 1.5.0, 1.1.11, 1.3.3, 1.4.2, > 1.4.3, 1.4.4, 1.4.5, 1.3.2.1, 1.4.6, 1.4.8, 1.4.7, 1.4.9, 1.4.10, 1.3.4, > 1.3.5, 1.3.6, 1.4.11, 1.4.12 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Blocker > Fix For: 1.6.0, 1.3.7, 1.4.13 > > > While converting the request back to Increment using ProtobufUtil.toIncrement > we incorrectly use the optimization to avoid copying the byte > array(HBaseZeroCopyByteString#zeroCopyGetBytes) on a BoundedByteString. The > optimization was only meant for LiteralByteString where it is safe to use the > backing byte array, however it ends up being used to BoundedByteString which > is a subclass of LiteralByteString. This essentially breaks increments since > we end up creating wrong cells on the server side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] jatsakthi commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API
jatsakthi commented on issue #1138: HBASE-22827 Expose multi-region merge in shell and Admin API URL: https://github.com/apache/hbase/pull/1138#issuecomment-584902672 @saintstack mind give this one a quick review, please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23825) Increment proto conversion is broken
[ https://issues.apache.org/jira/browse/HBASE-23825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034922#comment-17034922 ] Andrew Kyle Purtell commented on HBASE-23825: - Do you need someone to take over HBASE-23220 [~busbey]? Happy to do it after this lands. Sounds like Sakthi will do HBASE-23743. > Increment proto conversion is broken > > > Key: HBASE-23825 > URL: https://issues.apache.org/jira/browse/HBASE-23825 > Project: HBase > Issue Type: Bug > Components: Increment >Affects Versions: 1.4.0, 1.2.6, 1.3.2, 1.4.1, 1.5.0, 1.1.11, 1.3.3, 1.4.2, > 1.4.3, 1.4.4, 1.4.5, 1.3.2.1, 1.4.6, 1.4.8, 1.4.7, 1.4.9, 1.4.10, 1.3.4, > 1.3.5, 1.3.6, 1.4.11, 1.4.12 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Blocker > Fix For: 1.6.0, 1.3.7, 1.4.13 > > > While converting the request back to Increment using ProtobufUtil.toIncrement > we incorrectly use the optimization to avoid copying the byte > array(HBaseZeroCopyByteString#zeroCopyGetBytes) on a BoundedByteString. The > optimization was only meant for LiteralByteString where it is safe to use the > backing byte array, however it ends up being used to BoundedByteString which > is a subclass of LiteralByteString. This essentially breaks increments since > we end up creating wrong cells on the server side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23825) Increment proto conversion is broken
[ https://issues.apache.org/jira/browse/HBASE-23825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034920#comment-17034920 ] Andrew Kyle Purtell commented on HBASE-23825: - Linking HBASE-23743 (release 1.4.13) and HBASE-23220 (release 1.6.0). > Increment proto conversion is broken > > > Key: HBASE-23825 > URL: https://issues.apache.org/jira/browse/HBASE-23825 > Project: HBase > Issue Type: Bug > Components: Increment >Affects Versions: 1.4.0, 1.2.6, 1.3.2, 1.4.1, 1.5.0, 1.1.11, 1.3.3, 1.4.2, > 1.4.3, 1.4.4, 1.4.5, 1.3.2.1, 1.4.6, 1.4.8, 1.4.7, 1.4.9, 1.4.10, 1.3.4, > 1.3.5, 1.3.6, 1.4.11, 1.4.12 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Blocker > Fix For: 1.6.0, 1.3.7, 1.4.13 > > > While converting the request back to Increment using ProtobufUtil.toIncrement > we incorrectly use the optimization to avoid copying the byte > array(HBaseZeroCopyByteString#zeroCopyGetBytes) on a BoundedByteString. The > optimization was only meant for LiteralByteString where it is safe to use the > backing byte array, however it ends up being used to BoundedByteString which > is a subclass of LiteralByteString. This essentially breaks increments since > we end up creating wrong cells on the server side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] apurtell commented on issue #1161: HBASE-23825 Increment proto conversion is broken
apurtell commented on issue #1161: HBASE-23825 Increment proto conversion is broken URL: https://github.com/apache/hbase/pull/1161#issuecomment-584897666 Better result, and TestRegionReplicaReplicationEndpoint seems unrelated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #754: HBASE-22978 : Online slow response log
Apache-HBase commented on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-584893797 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 1s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 38s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 53s | master passed | | +1 :green_heart: | compile | 3m 28s | master passed | | +1 :green_heart: | checkstyle | 2m 40s | master passed | | +0 :ok: | refguide | 6m 1s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 5m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 6m 0s | master passed | | +0 :ok: | spotbugs | 1m 28s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 24m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 19s | the patch passed | | +1 :green_heart: | compile | 3m 7s | the patch passed | | +1 :green_heart: | cc | 3m 7s | the patch passed | | +1 :green_heart: | javac | 3m 7s | the patch passed | | -1 :x: | checkstyle | 2m 22s | root: The patch generated 1 new + 448 unchanged - 2 fixed = 449 total (was 450) | | +1 :green_heart: | rubocop | 0m 10s | There were no new rubocop issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 5m 1s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 4m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 16m 58s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | hbaseprotoc | 8m 54s | the patch passed | | +1 :green_heart: | javadoc | 5m 28s | the patch passed | | +1 :green_heart: | findbugs | 26m 54s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 67m 1s | root in the patch failed. | | +1 :green_heart: | asflicense | 2m 50s | The patch does not generate ASF License warnings. | | | | 213m 59s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestMobRestoreSnapshotFromClientAfterSplittingRegions | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/10/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/754 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide xml cc hbaseprotoc prototool rubocop | | uname | Linux 671e23b0a005 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-754/out/precommit/personality/provided.sh | | git revision | master / 12f98bc50e | | Default Java | 1.8.0_181 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/10/artifact/out/branch-site/book.html | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/10/artifact/out/diff-checkstyle-root.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/10/artifact/out/patch-site/book.html | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/10/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-754/10/testReport/ | | Max. process+thread count | 7407 (vs. ulimit of 1) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-serve
[GitHub] [hbase] Apache-HBase commented on issue #1161: HBASE-23825 Increment proto conversion is broken
Apache-HBase commented on issue #1161: HBASE-23825 Increment proto conversion is broken URL: https://github.com/apache/hbase/pull/1161#issuecomment-584883392 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 1m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 22s | branch-1 passed | | +1 :green_heart: | compile | 1m 40s | branch-1 passed with JDK v1.8.0_242 | | +1 :green_heart: | compile | 1m 50s | branch-1 passed with JDK v1.7.0_252 | | +1 :green_heart: | checkstyle | 6m 33s | branch-1 passed | | +1 :green_heart: | shadedjars | 4m 29s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 11s | branch-1 passed with JDK v1.8.0_242 | | +1 :green_heart: | javadoc | 1m 17s | branch-1 passed with JDK v1.7.0_252 | | +0 :ok: | spotbugs | 4m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 39s | branch-1 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 1s | the patch passed | | +1 :green_heart: | compile | 1m 44s | the patch passed with JDK v1.8.0_242 | | +1 :green_heart: | javac | 1m 44s | the patch passed | | +1 :green_heart: | compile | 1m 38s | the patch passed with JDK v1.7.0_252 | | +1 :green_heart: | javac | 1m 38s | the patch passed | | +1 :green_heart: | checkstyle | 3m 42s | The patch passed checkstyle in hbase-protocol | | +1 :green_heart: | checkstyle | 2m 14s | hbase-server: The patch generated 0 new + 7 unchanged - 2 fixed = 7 total (was 9) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 6m 57s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed with JDK v1.8.0_242 | | +1 :green_heart: | javadoc | 1m 13s | the patch passed with JDK v1.7.0_252 | | +1 :green_heart: | findbugs | 6m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 33s | hbase-protocol in the patch passed. | | -1 :x: | unit | 160m 8s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 44s | The patch does not generate ASF License warnings. | | | | 236m 53s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1161/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1161 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux e89ef3877e8c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1161/out/precommit/personality/provided.sh | | git revision | branch-1 / 942bb77 | | Default Java | 1.7.0_252 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_242 /usr/lib/jvm/zulu-7-amd64:1.7.0_252 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1161/4/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1161/4/testReport/ | | Max. process+thread count | 4290 (vs. ulimit of 1) | | modules | C: hbase-protocol hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1161/4/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. --
[jira] [Commented] (HBASE-23830) TestReplicationEndpoint appears to fail a lot in my attempts for a clean test run locally.
[
https://issues.apache.org/jira/browse/HBASE-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034883#comment-17034883
]
Mark Robert Miller commented on HBASE-23830:
I've attached the 13 fail logs that I got out of the last 30 runs on master.
> TestReplicationEndpoint appears to fail a lot in my attempts for a clean test
> run locally.
> --
>
> Key: HBASE-23830
> URL: https://issues.apache.org/jira/browse/HBASE-23830
> Project: HBase
> Issue Type: Test
>Affects Versions: master
>Reporter: Mark Robert Miller
>Priority: Major
> Attachments: test_fails.tar.xz
>
>
> This test is failing for me like 30-40% of the time. Fail seems to usually be
> as below. I've tried increasing the wait timeout but that does not seem to
> help at all.
> {code}
> [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.145 s <<< FAILURE! - in
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint[ERROR] Tests run:
> 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 105.145 s <<< FAILURE! -
> in org.apache.hadoop.hbase.replication.TestReplicationEndpoint[ERROR]
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testInterClusterReplication
> Time elapsed: 38.725 s <<< FAILURE!java.lang.AssertionError: Waiting timed
> out after [30,000] msec Failed to replicate all edits, expected = 2500
> replicated = 2476 at org.junit.Assert.fail(Assert.java:89) at
> org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:203) at
> org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:137) at
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testInterClusterReplication(TestReplicationEndpoint.java:235){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23830) TestReplicationEndpoint appears to fail a lot in my attempts for a clean test run locally.
[
https://issues.apache.org/jira/browse/HBASE-23830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Robert Miller updated HBASE-23830:
---
Attachment: test_fails.tar.xz
> TestReplicationEndpoint appears to fail a lot in my attempts for a clean test
> run locally.
> --
>
> Key: HBASE-23830
> URL: https://issues.apache.org/jira/browse/HBASE-23830
> Project: HBase
> Issue Type: Test
>Affects Versions: master
>Reporter: Mark Robert Miller
>Priority: Major
> Attachments: test_fails.tar.xz
>
>
> This test is failing for me like 30-40% of the time. Fail seems to usually be
> as below. I've tried increasing the wait timeout but that does not seem to
> help at all.
> {code}
> [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.145 s <<< FAILURE! - in
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint[ERROR] Tests run:
> 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 105.145 s <<< FAILURE! -
> in org.apache.hadoop.hbase.replication.TestReplicationEndpoint[ERROR]
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testInterClusterReplication
> Time elapsed: 38.725 s <<< FAILURE!java.lang.AssertionError: Waiting timed
> out after [30,000] msec Failed to replicate all edits, expected = 2500
> replicated = 2476 at org.junit.Assert.fail(Assert.java:89) at
> org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:203) at
> org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:137) at
> org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testInterClusterReplication(TestReplicationEndpoint.java:235){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23830) TestReplicationEndpoint appears to fail a lot in my attempts for a clean test run locally.
Mark Robert Miller created HBASE-23830:
--
Summary: TestReplicationEndpoint appears to fail a lot in my
attempts for a clean test run locally.
Key: HBASE-23830
URL: https://issues.apache.org/jira/browse/HBASE-23830
Project: HBase
Issue Type: Test
Affects Versions: master
Reporter: Mark Robert Miller
This test is failing for me like 30-40% of the time. Fail seems to usually be
as below. I've tried increasing the wait timeout but that does not seem to help
at all.
{code}
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 105.145
s <<< FAILURE! - in
org.apache.hadoop.hbase.replication.TestReplicationEndpoint[ERROR] Tests run:
7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 105.145 s <<< FAILURE! -
in org.apache.hadoop.hbase.replication.TestReplicationEndpoint[ERROR]
org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testInterClusterReplication
Time elapsed: 38.725 s <<< FAILURE!java.lang.AssertionError: Waiting timed
out after [30,000] msec Failed to replicate all edits, expected = 2500
replicated = 2476 at org.junit.Assert.fail(Assert.java:89) at
org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:203) at
org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:137) at
org.apache.hadoop.hbase.replication.TestReplicationEndpoint.testInterClusterReplication(TestReplicationEndpoint.java:235){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[ https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034873#comment-17034873 ] HBase QA commented on HBASE-23823: -- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HBASE-Build/1125/console in case of problems. > Run maven with more than default single thread (--threads=0.5C) > --- > > Key: HBASE-23823 > URL: https://issues.apache.org/jira/browse/HBASE-23823 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > > > See tail of parent task where we made commits to hbase-personality trying to > make maven run more furiously on nightly builds by upping the maven thread > count (the --threads flag ... see > https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3). > The effort was abandoned because it unsettled the build. This issue > separates messing with the maven threads flag from the broader topic the > parent covers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23823) Run maven with more than default single thread (--threads=0.5C)
[ https://issues.apache.org/jira/browse/HBASE-23823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-23823: -- Status: Patch Available (was: Open) > Run maven with more than default single thread (--threads=0.5C) > --- > > Key: HBASE-23823 > URL: https://issues.apache.org/jira/browse/HBASE-23823 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > Attachments: > 0001-HBASE-23823-Run-maven-with-more-than-default-single-.patch > > > See tail of parent task where we made commits to hbase-personality trying to > make maven run more furiously on nightly builds by upping the maven thread > count (the --threads flag ... see > https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3). > The effort was abandoned because it unsettled the build. This issue > separates messing with the maven threads flag from the broader topic the > parent covers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[
https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034780#comment-17034780
]
Hudson commented on HBASE-18095:
Results for branch HBASE-18095/client-locate-meta-no-zookeeper
[build #72 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/72/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/72//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/72//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/72//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Provide an option for clients to find the server hosting META that does not
> involve the ZooKeeper client
>
>
> Key: HBASE-18095
> URL: https://issues.apache.org/jira/browse/HBASE-18095
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Reporter: Andrew Kyle Purtell
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch
>
>
> Clients are required to connect to ZooKeeper to find the location of the
> regionserver hosting the meta table region. Site configuration provides the
> client a list of ZK quorum peers and the client uses an embedded ZK client to
> query meta location. Timeouts and retry behavior of this embedded ZK client
> are managed orthogonally to HBase layer settings and in some cases the ZK
> cannot manage what in theory the HBase client can, i.e. fail fast upon outage
> or network partition.
> We should consider new configuration settings that provide a list of
> well-known master and backup master locations, and with this information the
> client can contact any of the master processes directly. Any master in either
> active or passive state will track meta location and respond to requests for
> it with its cached last known location. If this location is stale, the client
> can ask again with a flag set that requests the master refresh its location
> cache and return the up-to-date location. Every client interaction with the
> cluster thus uses only HBase RPC as transport, with appropriate settings
> applied to the connection. The configuration toggle that enables this
> alternative meta location lookup should be false by default.
> This removes the requirement that HBase clients embed the ZK client and
> contact the ZK service directly at the beginning of the connection lifecycle.
> This has several benefits. ZK service need not be exposed to clients, and
> their potential abuse, yet no benefit ZK provides the HBase server cluster is
> compromised. Normalizing HBase client and ZK client timeout settings and
> retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer
> necessary.
> And, from [~ghelmling]: There is an additional complication here for
> token-based authentication. When a delegation token is used for SASL
> authentication, the client uses the cluster ID obtained from Zookeeper to
> select the token identifier to use. So there would also need to be some
> Zookeeper-less, unauthenticated way to obtain the cluster ID as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
Apache-HBase commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore… URL: https://github.com/apache/hbase/pull/1141#issuecomment-584820517 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 53s | master passed | | +1 :green_heart: | compile | 1m 1s | master passed | | +1 :green_heart: | checkstyle | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 5m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | | +0 :ok: | spotbugs | 4m 42s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 40s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 25s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | checkstyle | 1m 12s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 11s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | | +1 :green_heart: | findbugs | 5m 13s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 91m 9s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 153m 13s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.master.TestMasterShutdown | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1141 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux c49df62887e3 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1141/out/precommit/personality/provided.sh | | git revision | master / 12f98bc50e | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/2/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/2/testReport/ | | Max. process+thread count | 5642 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1141/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23828) Remove unused hadoop.guava.version from pom.xml
[ https://issues.apache.org/jira/browse/HBASE-23828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034779#comment-17034779 ] Norbert Kalmár commented on HBASE-23828: As to why it is a bad thing having guava 11.0.2 on the classpath: [CVE-2018-10237|https://nvd.nist.gov/vuln/detail/CVE-2018-10237] "Unbounded memory allocation in Google Guava 11.0 through 24.x before 24.1.1 allows remote attackers to conduct denial of service attacks against servers that depend on this library and [...]" > Remove unused hadoop.guava.version from pom.xml > --- > > Key: HBASE-23828 > URL: https://issues.apache.org/jira/browse/HBASE-23828 > Project: HBase > Issue Type: Improvement >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > > 11.0.2 > is still used in hbase-backup, I missed it at first... > So, this should be either updated or removed. > Checking which is feasible... > Update: > So even if I remove hadoop.guava.version, with hadoop-2 profile, 11.0.2 will > be used during the build, and it will be on the classpath. > Since hadoop only upgraded to guava 27.0 in hadoop-3, I'm not sure what we > can do here. hadoop-2 is incompatible with guava 2x.x versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23828) Remove unused hadoop.guava.version from pom.xml
[ https://issues.apache.org/jira/browse/HBASE-23828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Kalmár updated HBASE-23828: --- Description: 11.0.2 is still used in hbase-backup, I missed it at first... So, this should be either updated or removed. Checking which is feasible... Update: So even if I remove hadoop.guava.version, with hadoop-2 profile, 11.0.2 will be used during the build, and it will be on the classpath. Since hadoop only upgraded to guava 27.0 in hadoop-3, I'm not sure what we can do here. hadoop-2 is incompatible with guava 2x.x versions. was: 11.0.2 is still used in hbase-backup, I missed it at first... So, this should be either updated or removed. Checking which is feasible... So even if I remove hadoop.guava.version, with hadoop-2 profile, 11.0.2 will be used during the build, and it will be on the classpath. Since hadoop only upgraded to guava 27.0 in hadoop-3, I'm not sure what we can do here. hadoop-2 is incompatible with guava 2x.x versions. > Remove unused hadoop.guava.version from pom.xml > --- > > Key: HBASE-23828 > URL: https://issues.apache.org/jira/browse/HBASE-23828 > Project: HBase > Issue Type: Improvement >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > > 11.0.2 > is still used in hbase-backup, I missed it at first... > So, this should be either updated or removed. > Checking which is feasible... > Update: > So even if I remove hadoop.guava.version, with hadoop-2 profile, 11.0.2 will > be used during the build, and it will be on the classpath. > Since hadoop only upgraded to guava 27.0 in hadoop-3, I'm not sure what we > can do here. hadoop-2 is incompatible with guava 2x.x versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23828) Remove unused hadoop.guava.version from pom.xml
[ https://issues.apache.org/jira/browse/HBASE-23828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034774#comment-17034774 ] Norbert Kalmár commented on HBASE-23828: Great, thanks [~busbey]! > Remove unused hadoop.guava.version from pom.xml > --- > > Key: HBASE-23828 > URL: https://issues.apache.org/jira/browse/HBASE-23828 > Project: HBase > Issue Type: Improvement >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > > 11.0.2 > is still used in hbase-backup, I missed it at first... > So, this should be either updated or removed. > Checking which is feasible... > So even if I remove hadoop.guava.version, with hadoop-2 profile, 11.0.2 will > be used during the build, and it will be on the classpath. > Since hadoop only upgraded to guava 27.0 in hadoop-3, I'm not sure what we > can do here. hadoop-2 is incompatible with guava 2x.x versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23828) Remove unused hadoop.guava.version from pom.xml
[ https://issues.apache.org/jira/browse/HBASE-23828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034719#comment-17034719 ] Sean Busbey commented on HBASE-23828: - this is a very good reason for me to push on my desire for hbase 3 to drop Hadoop 2 support entirely. let me get my DISCUSS thread going on dev. maybe by tomorrow? > Remove unused hadoop.guava.version from pom.xml > --- > > Key: HBASE-23828 > URL: https://issues.apache.org/jira/browse/HBASE-23828 > Project: HBase > Issue Type: Improvement >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > > 11.0.2 > is still used in hbase-backup, I missed it at first... > So, this should be either updated or removed. > Checking which is feasible... > So even if I remove hadoop.guava.version, with hadoop-2 profile, 11.0.2 will > be used during the build, and it will be on the classpath. > Since hadoop only upgraded to guava 27.0 in hadoop-3, I'm not sure what we > can do here. hadoop-2 is incompatible with guava 2x.x versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] VladRodionov commented on a change in pull request #921: HBASE-22749: Distributed MOB compactions
VladRodionov commented on a change in pull request #921: HBASE-22749:
Distributed MOB compactions
URL: https://github.com/apache/hbase/pull/921#discussion_r377829446
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCompactionChore.java
##
@@ -0,0 +1,253 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mob;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.CompactionState;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+
+/**
+ * Periodic MOB compaction chore.
+ * It runs MOB compaction on region servers in parallel, thus
+ * utilizing distributed cluster resources. To avoid possible major
+ * compaction storms, one can specify maximum number regions to be compacted
+ * in parallel by setting configuration parameter:
+ * 'hbase.mob.major.compaction.region.batch.size', which by default is 0
(unlimited).
+ *
+ */
+@InterfaceAudience.Private
+public class MobFileCompactionChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCompactionChore.class);
+ private Configuration conf;
+ private HMaster master;
+ private int regionBatchSize = 0;// not set - compact all
+
+ public MobFileCompactionChore(HMaster master) {
+super(master.getServerName() + "-MobFileCompactionChore", master,
+
master.getConfiguration().getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+ MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD),
+
master.getConfiguration().getInt(MobConstants.MOB_COMPACTION_CHORE_PERIOD,
+ MobConstants.DEFAULT_MOB_COMPACTION_CHORE_PERIOD),
+TimeUnit.SECONDS);
+this.master = master;
+this.conf = master.getConfiguration();
+this.regionBatchSize =
+
master.getConfiguration().getInt(MobConstants.MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE,
+ MobConstants.DEFAULT_MOB_MAJOR_COMPACTION_REGION_BATCH_SIZE);
+
+ }
+
+ @VisibleForTesting
+ public MobFileCompactionChore(Configuration conf, int batchSize) {
+this.conf = conf;
+this.regionBatchSize = batchSize;
+ }
+
+ @Override
+ protected void chore() {
+
+boolean reported = false;
+
+try (Connection conn = ConnectionFactory.createConnection(conf);
+Admin admin = conn.getAdmin();) {
+
+ TableDescriptors htds = master.getTableDescriptors();
+ Map map = htds.getAll();
+ for (TableDescriptor htd : map.values()) {
+if (!master.getTableStateManager().isTableState(htd.getTableName(),
+ TableState.State.ENABLED)) {
+ LOG.info("Skipping MOB compaction on table {} because it is not
ENABLED",
+htd.getTableName());
+ continue;
+} else {
+ LOG.info("Starting MOB compaction on table {}, checking {} column
families",
+htd.getTableName(), htd.getColumnFamilyCount());
+}
+for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
+ try {
+if (hcd.isMobEnabled()) {
+ if (!reported) {
+master.reportMobCompactionStart(htd.getTableName());
+reported = true;
+ }
+ LOG.info("Major MO
[GitHub] [hbase] VladRodionov commented on a change in pull request #921: HBASE-22749: Distributed MOB compactions
VladRodionov commented on a change in pull request #921: HBASE-22749:
Distributed MOB compactions
URL: https://github.com/apache/hbase/pull/921#discussion_r377828808
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java
##
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.mob;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.backup.HFileArchiver;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.master.HMaster;
+import org.apache.hadoop.hbase.regionserver.BloomType;
+import org.apache.hadoop.hbase.regionserver.HStoreFile;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * The class MobFileCleanerChore for running cleaner regularly to remove the
expired
+ * and obsolete (files which have no active references to) mob files.
+ */
+@InterfaceAudience.Private
+public class MobFileCleanerChore extends ScheduledChore {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MobFileCleanerChore.class);
+ private final HMaster master;
+ private ExpiredMobFileCleaner cleaner;
+
+ static {
+Configuration.addDeprecation(MobConstants.DEPRECATED_MOB_CLEANER_PERIOD,
+ MobConstants.MOB_CLEANER_PERIOD);
+ }
+
+ public MobFileCleanerChore(HMaster master) {
+super(master.getServerName() + "-ExpiredMobFileCleanerChore", master,
+master.getConfiguration().getInt(MobConstants.MOB_CLEANER_PERIOD,
+ MobConstants.DEFAULT_MOB_CLEANER_PERIOD),
+master.getConfiguration().getInt(MobConstants.MOB_CLEANER_PERIOD,
+ MobConstants.DEFAULT_MOB_CLEANER_PERIOD),
+TimeUnit.SECONDS);
+this.master = master;
+cleaner = new ExpiredMobFileCleaner();
+cleaner.setConf(master.getConfiguration());
+checkObsoleteConfigurations();
+ }
+
+ private void checkObsoleteConfigurations() {
+Configuration conf = master.getConfiguration();
+
+if (conf.get("hbase.mob.compaction.mergeable.threshold") != null) {
+ LOG.warn("'hbase.mob.compaction.mergeable.threshold' is obsolete and not
used anymore.");
+}
+if (conf.get("hbase.mob.delfile.max.count") != null) {
+ LOG.warn("'hbase.mob.delfile.max.count' is obsolete and not used
anymore.");
+}
+if (conf.get("hbase.mob.compaction.threads.max") != null) {
+ LOG.warn("'hbase.mob.compaction.threads.max' is obsolete and not used
anymore.");
+}
+if (conf.get("hbase.mob.compaction.batch.size") != null) {
+ LOG.warn("'hbase.mob.compaction.batch.size' is obsolete and not used
anymore.");
+}
+ }
+
+ @VisibleForTesting
+ public MobFileCleanerChore() {
+this.master = null;
+ }
+
+ @Override
+ @edu.umd.cs.findbugs.annotations.SuppressWarnings(value =
"REC_CATCH_EXCEPTION",
+ justification = "Intentional")
+
+ protected void chore() {
+TableDescriptors htds = master.getTableDescriptors();
+
+Map map = null;
+try {
+ map = htds.getAll();
+} catch (IOException e) {
+ LOG.e
[GitHub] [hbase] virajjasani commented on issue #754: HBASE-22978 : Online slow response log
virajjasani commented on issue #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#issuecomment-584788210 > > > 1. Since the detailed info is going through this ring buffer subsystem, why do we continue to log at all? Why not replace the logging with a metric? > > > > > > > > 1. Ring Buffer is not a replacement of slow logs that we write to file, it's just a complete representation of slow logs, whereas in log files we might have trimmed logs. We also have plan to write all slowlogs to HDFS (optionally) and provide HDFS file path to user optionally, at that time we can think of removing logging in log files. However, I am not sure about metric. Which metric are you suggesting to expose? You mean slow logs with some parameters? > > I'm suggesting we do away with file logging and instead expose the presence of slow activity as a (new?) metric. > > By the time this patch lands, we'll have the logged information in two places, neither of which, on their own, allow an operator to easily see how prevalent the problem is. I'm suggesting that we could instead provide a metric that says "this region server is experiencing N slow request per time interval T", as well as providing detailed incident fidelity via this new mechanism. Thanks @ndimiduk Would you be fine with exposing metric as a sub task? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r377801205
##
File path:
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/client/ThriftAdmin.java
##
@@ -1157,6 +1158,18 @@ public boolean isSnapshotCleanupEnabled() {
throw new NotImplementedException("isSnapshotCleanupEnabled not supported
in ThriftAdmin");
}
+ @Override
+ public List getSlowLogResponses(final List
serverNames)
+ throws IOException {
+throw new NotImplementedException("getSlowLogResponses not supported in
ThriftAdmin");
Review comment:
Should we have this support in Thrift? If you are in favor of it, let me add
it in separate patch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log URL: https://github.com/apache/hbase/pull/754#discussion_r377736971 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java ## @@ -1519,6 +1519,15 @@ "hbase.master.executor.logreplayops.threads"; public static final int MASTER_LOG_REPLAY_OPS_THREADS_DEFAULT = 10; + public static final String SLOW_LOG_RING_BUFFER_SIZE = Review comment: It is also being used in `TestOnlineSlowLogProvider` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r377664251
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/SlowLogPayload.java
##
@@ -0,0 +1,318 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.client;
+
+import org.apache.hadoop.hbase.util.GsonUtil;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.apache.hbase.thirdparty.com.google.gson.Gson;
+import org.apache.hbase.thirdparty.com.google.gson.JsonObject;
+import org.apache.hbase.thirdparty.com.google.gson.JsonSerializer;
+
+/**
+ * SlowLog payload for client
+ */
+@InterfaceAudience.Private
+final public class SlowLogPayload {
+
+ private static final Gson GSON = GsonUtil.createGson()
+.setPrettyPrinting()
+.registerTypeAdapter(SlowLogPayload.class, (JsonSerializer)
+ (slowLogPayload, type, jsonSerializationContext) -> {
Review comment:
I agree with you, Jackson does have quite good annotations to deal with such
scenarios, but Gson doesn't and we have moved from Jackson to shaded Gson so I
also had to search for this specific way of dealing with serialization filter
(I was also not much aware of it)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r377743288
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
##
@@ -496,11 +518,51 @@ void logResponse(Message param, String methodName,
String call, String tag,
}
}
}
- responseInfo.put("multi.gets", numGets);
- responseInfo.put("multi.mutations", numMutations);
- responseInfo.put("multi.servicecalls", numServiceCalls);
+ responseInfo.put(MULTI_GETS, numGets);
+ responseInfo.put(MULTI_MUTATIONS, numMutations);
+ responseInfo.put(MULTI_SERVICE_CALLS, numServiceCalls);
}
+final String tag = tooLarge ? "TooLarge" : "TooSlow";
Review comment:
I agree that it should not be necessary to make this mutually exclusive but
this is the logic we have already:
```
logResponse(param,
md.getName(), md.getName() + "(" + param.getClass().getName() +
")",
(tooLarge ? "TooLarge" : "TooSlow"),
status.getClient(), startTime, processingTime, qTime,
responseSize);
```
Hence, I thought of keeping it same(mutually exclusive)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #754: HBASE-22978 : Online slow response log
virajjasani commented on a change in pull request #754: HBASE-22978 : Online
slow response log
URL: https://github.com/apache/hbase/pull/754#discussion_r377739714
##
File path: hbase-protocol-shaded/src/main/protobuf/TooSlowLog.proto
##
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto2";
+
+// This file contains protocol buffers that are used for Online TooSlowLogs
+// To be used as Ring Buffer payload
+package hbase.pb;
+
+option java_package = "org.apache.hadoop.hbase.shaded.protobuf.generated";
+option java_outer_classname = "TooSlowLog";
+option java_generate_equals_and_hash = true;
+option optimize_for = SPEED;
+
+message SlowLogPayload {
+required int64 start_time = 1;
+required int32 processing_time = 2;
+required int32 queue_time = 3;
+required int64 response_size = 4;
+required string client_address = 5;
+required string server_class = 6;
+required string method_name = 7;
+required string call_details = 8;
+required string param = 9;
+required string user_name = 10;
Review comment:
We use this for storing only. For querying, we are converting this POJO to
`org.apache.hadoop.hbase.client.SlowLogPayload`, it's just that we need
userName here so that while converting to queryable POJO, we can copy the value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (HBASE-23829) Get `-PrunSmallTests` passing on JDK11
Nick Dimiduk created HBASE-23829:
Summary: Get `-PrunSmallTests` passing on JDK11
Key: HBASE-23829
URL: https://issues.apache.org/jira/browse/HBASE-23829
Project: HBase
Issue Type: Sub-task
Components: test
Reporter: Nick Dimiduk
Start with the small tests, shaking out issues identified by the harness. So
far it seems like {{-Dhadoop.profile=3.0}} and
{{-Dhadoop-three.version=3.3.0-SNAPSHOT}} maybe be required.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23816) [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS!
[
https://issues.apache.org/jira/browse/HBASE-23816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034704#comment-17034704
]
Michael Stack commented on HBASE-23816:
---
Yeah, its one of those fun ones
> [Flakey Test]
> TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91)
> Wrong FS!
>
>
> Key: HBASE-23816
> URL: https://issues.apache.org/jira/browse/HBASE-23816
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: Screen Shot 2020-02-11 at 9.00.06 AM.png
>
>
> Can't repro locally w/o running with lots of parallelism. Follow-on from
> parent issue.
> Here is error:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 115.741 s <<< FAILURE! - in
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster
> [ERROR]
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState
> Time elapsed: 74.459 s <<< ERROR!
> java.lang.IllegalArgumentException: Wrong FS:
> file:/Users/stack/checkouts/hbase.apache.git/hbase-mapreduce/target/test-data/fad01338-65ef-e973-2412-05de114016fe/.hbase-snapshot/tableWithRefsV2,
> expected: hdfs://localhost:58051
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:107)
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91)
> {code}
> With v1 had just run find which does same thing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] abhishek-chouhan commented on issue #1161: HBASE-23825 Increment proto conversion is broken
abhishek-chouhan commented on issue #1161: HBASE-23825 Increment proto conversion is broken URL: https://github.com/apache/hbase/pull/1161#issuecomment-584778803 Bunch of tests timed out(which pass locally). Getting one more QA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23816) [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS!
[
https://issues.apache.org/jira/browse/HBASE-23816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034642#comment-17034642
]
Nick Dimiduk commented on HBASE-23816:
--
I ran this one in a loop while doing other things yesterday, still couldn't get
a repro.
> [Flakey Test]
> TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91)
> Wrong FS!
>
>
> Key: HBASE-23816
> URL: https://issues.apache.org/jira/browse/HBASE-23816
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: Screen Shot 2020-02-11 at 9.00.06 AM.png
>
>
> Can't repro locally w/o running with lots of parallelism. Follow-on from
> parent issue.
> Here is error:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 115.741 s <<< FAILURE! - in
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster
> [ERROR]
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState
> Time elapsed: 74.459 s <<< ERROR!
> java.lang.IllegalArgumentException: Wrong FS:
> file:/Users/stack/checkouts/hbase.apache.git/hbase-mapreduce/target/test-data/fad01338-65ef-e973-2412-05de114016fe/.hbase-snapshot/tableWithRefsV2,
> expected: hdfs://localhost:58051
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:107)
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91)
> {code}
> With v1 had just run find which does same thing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on issue #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore… URL: https://github.com/apache/hbase/pull/1141#issuecomment-584748786 Updated per PR feedback. Also some further restructuring of the test, including some learnings from HBASE-23647. I think there's a real bug here. I can issue a `shutdown` RPC that is acknowledged by the master and commences shutdown. However, at least one subsystem is not getting the message and continues on. This seems to leave the master in a quasi-initialized zombie state. (I believe this is what @bharathv was seeing when he added the `waitFor` on the `ServerManager` instance) Eventually the 200sec master initialization timeout trips and the test fails. More digging, will be back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test]
TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r377391768
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -33,23 +34,34 @@
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
-import org.apache.hadoop.hbase.testclassification.LargeTests;
import org.apache.hadoop.hbase.testclassification.MasterTests;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
import org.apache.hadoop.hbase.util.JVMClusterUtil.MasterThread;
+import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
-@Category({MasterTests.class, LargeTests.class})
+@Category({MasterTests.class, MediumTests.class})
Review comment:
Seems to be snappy for me... What's the idea behind these annotations?
They're supposed to indicate the local dev experience, or the overworked
Jenkins experience?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test]
TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r377394544
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -128,41 +142,50 @@ public void
testMasterShutdownBeforeStartingAnyRegionServer() throws Exception {
conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, 1);
// Start the cluster
-final HBaseTestingUtility util = new HBaseTestingUtility(conf);
-util.startMiniDFSCluster(3);
-util.startMiniZKCluster();
-util.createRootDir();
-final LocalHBaseCluster cluster =
-new LocalHBaseCluster(conf, NUM_MASTERS, NUM_RS, HMaster.class,
-MiniHBaseCluster.MiniHBaseClusterRegionServer.class);
-final int MASTER_INDEX = 0;
-final MasterThread master = cluster.getMasters().get(MASTER_INDEX);
-master.start();
-LOG.info("Called master start on " + master.getName());
-Thread shutdownThread = new Thread("Shutdown-Thread") {
- @Override
- public void run() {
-LOG.info("Before call to shutdown master");
-try (Connection connection = createConnection(util); Admin admin =
connection.getAdmin()) {
- admin.shutdown();
-} catch (Exception e) {
- LOG.info("Error while calling Admin.shutdown, which is expected: " +
e.getMessage());
+LocalHBaseCluster cluster = null;
+try {
+ htu = new HBaseTestingUtility(conf);
+ htu.startMiniDFSCluster(3);
+ htu.startMiniZKCluster();
+ htu.createRootDir();
+ cluster = new LocalHBaseCluster(conf, NUM_MASTERS, NUM_RS, HMaster.class,
+MiniHBaseCluster.MiniHBaseClusterRegionServer.class);
+ final int MASTER_INDEX = 0;
+ final MasterThread master = cluster.getMasters().get(MASTER_INDEX);
+ master.start();
+ LOG.info("Called master start on " + master.getName());
+ final LocalHBaseCluster finalCluster = cluster;
+ Thread shutdownThread = new Thread("Shutdown-Thread") {
+@Override
+public void run() {
+ LOG.info("Before call to shutdown master");
+ try (Connection connection = createConnection(htu); Admin admin =
connection.getAdmin()) {
+admin.shutdown();
Review comment:
I re-read your changes on the branch. Let me try to merge those with the
general cleanup I'm attempting here. Hopefully when it's done, you can just
drop the change to this test from the feature branch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test]
TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r38169
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -128,41 +142,50 @@ public void
testMasterShutdownBeforeStartingAnyRegionServer() throws Exception {
conf.setInt(ServerManager.WAIT_ON_REGIONSERVERS_MINTOSTART, 1);
// Start the cluster
-final HBaseTestingUtility util = new HBaseTestingUtility(conf);
-util.startMiniDFSCluster(3);
-util.startMiniZKCluster();
-util.createRootDir();
-final LocalHBaseCluster cluster =
-new LocalHBaseCluster(conf, NUM_MASTERS, NUM_RS, HMaster.class,
-MiniHBaseCluster.MiniHBaseClusterRegionServer.class);
-final int MASTER_INDEX = 0;
-final MasterThread master = cluster.getMasters().get(MASTER_INDEX);
-master.start();
-LOG.info("Called master start on " + master.getName());
-Thread shutdownThread = new Thread("Shutdown-Thread") {
- @Override
- public void run() {
-LOG.info("Before call to shutdown master");
-try (Connection connection = createConnection(util); Admin admin =
connection.getAdmin()) {
- admin.shutdown();
-} catch (Exception e) {
- LOG.info("Error while calling Admin.shutdown, which is expected: " +
e.getMessage());
+LocalHBaseCluster cluster = null;
+try {
+ htu = new HBaseTestingUtility(conf);
+ htu.startMiniDFSCluster(3);
+ htu.startMiniZKCluster();
+ htu.createRootDir();
+ cluster = new LocalHBaseCluster(conf, NUM_MASTERS, NUM_RS, HMaster.class,
+MiniHBaseCluster.MiniHBaseClusterRegionServer.class);
+ final int MASTER_INDEX = 0;
+ final MasterThread master = cluster.getMasters().get(MASTER_INDEX);
+ master.start();
+ LOG.info("Called master start on " + master.getName());
+ final LocalHBaseCluster finalCluster = cluster;
+ Thread shutdownThread = new Thread("Shutdown-Thread") {
+@Override
+public void run() {
+ LOG.info("Before call to shutdown master");
+ try (Connection connection = createConnection(htu); Admin admin =
connection.getAdmin()) {
+admin.shutdown();
+ } catch (Exception e) {
+LOG.info("Error while calling Admin.shutdown, which is expected: "
+ e.getMessage());
+ }
+ LOG.info("After call to shutdown master");
+ finalCluster.waitOnMaster(MASTER_INDEX);
}
-LOG.info("After call to shutdown master");
-cluster.waitOnMaster(MASTER_INDEX);
+ };
+ shutdownThread.start();
+ LOG.info("Called master join on " + master.getName());
+ master.join();
+ shutdownThread.join();
+
+ List masterThreads = cluster.getMasters();
+ // make sure all the masters properly shutdown
+ assertEquals(0, masterThreads.size());
+} finally {
+ if (cluster != null) {
+cluster.shutdown();
+ }
+ if (htu != null) {
+htu.shutdownMiniZKCluster();
Review comment:
_nod_
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore…
ndimiduk commented on a change in pull request #1141: HBASE-23808 [Flakey Test]
TestMasterShutdown#testMasterShutdownBefore…
URL: https://github.com/apache/hbase/pull/1141#discussion_r377391906
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##
@@ -66,43 +78,45 @@ public void testMasterShutdown() throws Exception {
Configuration conf = HBaseConfiguration.create();
// Start the cluster
-HBaseTestingUtility htu = new HBaseTestingUtility(conf);
-StartMiniClusterOption option = StartMiniClusterOption.builder()
-
.numMasters(NUM_MASTERS).numRegionServers(NUM_RS).numDataNodes(NUM_RS).build();
-htu.startMiniCluster(option);
-MiniHBaseCluster cluster = htu.getHBaseCluster();
-
-// get all the master threads
-List masterThreads = cluster.getMasterThreads();
-
-// wait for each to come online
-for (MasterThread mt : masterThreads) {
- assertTrue(mt.isAlive());
-}
-
-// find the active master
-HMaster active = null;
-for (int i = 0; i < masterThreads.size(); i++) {
- if (masterThreads.get(i).getMaster().isActiveMaster()) {
-active = masterThreads.get(i).getMaster();
-break;
+try {
+ htu = new HBaseTestingUtility(conf);
Review comment:
This is a new conf, created above... Let me move them in here after htu is
created but before cluster start. I'm not clear on why
`hbase.ipc.client.failed.servers.expiry` is set so aggressively.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-23816) [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS!
[
https://issues.apache.org/jira/browse/HBASE-23816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034604#comment-17034604
]
Michael Stack commented on HBASE-23816:
---
This is interesting from last nights nightly. Both v1 and v2 failed complaining
about wrong fs. Seems to argue that my attempt at separating the two tests out
will not address failure. Need to dig in on why this happens sporadically.
!Screen Shot 2020-02-11 at 9.00.06 AM.png!
> [Flakey Test]
> TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91)
> Wrong FS!
>
>
> Key: HBASE-23816
> URL: https://issues.apache.org/jira/browse/HBASE-23816
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: Screen Shot 2020-02-11 at 9.00.06 AM.png
>
>
> Can't repro locally w/o running with lots of parallelism. Follow-on from
> parent issue.
> Here is error:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 115.741 s <<< FAILURE! - in
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster
> [ERROR]
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState
> Time elapsed: 74.459 s <<< ERROR!
> java.lang.IllegalArgumentException: Wrong FS:
> file:/Users/stack/checkouts/hbase.apache.git/hbase-mapreduce/target/test-data/fad01338-65ef-e973-2412-05de114016fe/.hbase-snapshot/tableWithRefsV2,
> expected: hdfs://localhost:58051
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:107)
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91)
> {code}
> With v1 had just run find which does same thing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23816) [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS!
[
https://issues.apache.org/jira/browse/HBASE-23816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-23816:
--
Attachment: Screen Shot 2020-02-11 at 9.00.06 AM.png
> [Flakey Test]
> TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91)
> Wrong FS!
>
>
> Key: HBASE-23816
> URL: https://issues.apache.org/jira/browse/HBASE-23816
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: Screen Shot 2020-02-11 at 9.00.06 AM.png
>
>
> Can't repro locally w/o running with lots of parallelism. Follow-on from
> parent issue.
> Here is error:
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 115.741 s <<< FAILURE! - in
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster
> [ERROR]
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState
> Time elapsed: 74.459 s <<< ERROR!
> java.lang.IllegalArgumentException: Wrong FS:
> file:/Users/stack/checkouts/hbase.apache.git/hbase-mapreduce/target/test-data/fad01338-65ef-e973-2412-05de114016fe/.hbase-snapshot/tableWithRefsV2,
> expected: hdfs://localhost:58051
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:107)
>at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91)
> {code}
> With v1 had just run find which does same thing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23693) Split failure may cause region hole and data loss when use zk assign
[
https://issues.apache.org/jira/browse/HBASE-23693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-23693:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for confirming [~tangtianhang]! Marking as resolved.
> Split failure may cause region hole and data loss when use zk assign
>
>
> Key: HBASE-23693
> URL: https://issues.apache.org/jira/browse/HBASE-23693
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.4.8
>Reporter: tianhang tang
>Assignee: tianhang tang
>Priority: Critical
> Fix For: 1.5.1
>
> Attachments: HBASE-23693.branch-1.001.patch
>
>
> to mock this case, I add a sleep code in SplitTransactionImpl.excute after
> the PONR and before openDaughters:
> {code:java}
> public PairOfSameType execute(final Server server,
> final RegionServerServices services, User user) throws IOException {
> this.server = server;
> this.rsServices = services;
> useZKForAssignment = server == null ? true :
> ConfigUtil.useZKForAssignment(server.getConfiguration());
> if (useCoordinatedStateManager(server)) {
> std =
> ((BaseCoordinatedStateManager) server.getCoordinatedStateManager())
> .getSplitTransactionCoordination().getDefaultDetails();
> }
> PairOfSameType regions = createDaughters(server, services, user);
> if (this.parent.getCoprocessorHost() != null) {
> if (user == null) {
> parent.getCoprocessorHost().preSplitAfterPONR();
> } else {
> try {
> user.getUGI().doAs(new PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> parent.getCoprocessorHost().preSplitAfterPONR();
> return null;
> }
> });
> } catch (InterruptedException ie) {
> InterruptedIOException iioe = new InterruptedIOException();
> iioe.initCause(ie);
> throw iioe;
> }
> }
> }
>
> //sleep here!!!
> try {
> Thread.sleep(1000 * 60 * 60);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> regions = stepsAfterPONR(server, services, regions, user);
> transition(SplitTransactionPhase.COMPLETED);
> return regions;
> }
> {code}
> so the split transaction will hang.
> then i try to reproduce this problem:
> 1.Create a test table and move it into a test rsgroup, there is only 1 RS in
> the test group
> 2.Trigger a region split
> 3.The split transaction step after the PONR and sleep, regioninfo in meta has
> been updated
> 4.Kill the RS process to mock machine crash
> 5.ServerCrashProcedure cleanup SPLITING_NEW region, the daughter regions will
> be deleted
> 6.ServerCrashProcedure try to assign the parent region, because RS is down
> and assign fails, the region status is set to FAILED_OPEN and put back into
> regionsInTransition. But at this time, due to RS crash, the node of the
> region under ZK region-in-transition no longer exist
> 7.CatalogJanitor thread is blocked due to RIT
> 8.Switch active master
> 9.The CatalogJanitor thread on the new master executes normally and the
> parent region is cleaned up because split = true && offline = true in the
> meta table
> 10.We have a hole in the test table and loss data.
>
> I modified the code when ServerCrashProcedure cleans up the child regions, it
> will update the parent regioninfo in the meta table, and this problem is no
> longer reproduced.
> I will upload the patch later.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #1152: HBASE-23818 Cleanup the remaining RSGroupInfo.getTables call in the c…
Apache9 commented on a change in pull request #1152: HBASE-23818 Cleanup the remaining RSGroupInfo.getTables call in the c… URL: https://github.com/apache/hbase/pull/1152#discussion_r377651652 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupUtil.java ## @@ -31,9 +34,33 @@ private static final Logger LOG = LoggerFactory.getLogger(RSGroupUtil.class); + public static final String RS_GROUP_ENABLED = "hbase.balancer.rsgroup.enabled"; Review comment: You can disable the feature, then there will only be single default rs group. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23828) Remove unused hadoop.guava.version from pom.xml
[ https://issues.apache.org/jira/browse/HBASE-23828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Kalmár updated HBASE-23828: --- Description: 11.0.2 is still used in hbase-backup, I missed it at first... So, this should be either updated or removed. Checking which is feasible... So even if I remove hadoop.guava.version, with hadoop-2 profile, 11.0.2 will be used during the build, and it will be on the classpath. Since hadoop only upgraded to guava 27.0 in hadoop-3, I'm not sure what we can do here. hadoop-2 is incompatible with guava 2x.x versions. was: 11.0.2 is still used in hbase-backup, I missed it at first... So, this should be either updated or removed. Checking which is feasible... > Remove unused hadoop.guava.version from pom.xml > --- > > Key: HBASE-23828 > URL: https://issues.apache.org/jira/browse/HBASE-23828 > Project: HBase > Issue Type: Improvement >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > > 11.0.2 > is still used in hbase-backup, I missed it at first... > So, this should be either updated or removed. > Checking which is feasible... > So even if I remove hadoop.guava.version, with hadoop-2 profile, 11.0.2 will > be used during the build, and it will be on the classpath. > Since hadoop only upgraded to guava 27.0 in hadoop-3, I'm not sure what we > can do here. hadoop-2 is incompatible with guava 2x.x versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23828) Remove unused hadoop.guava.version from pom.xml
[ https://issues.apache.org/jira/browse/HBASE-23828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Kalmár updated HBASE-23828: --- Description: 11.0.2 is still used in hbase-backup, I missed it at first... So, this should be either updated or removed. Checking which is feasible... was: 11.0.2 is not used anywhere. It was added in https://github.com/apache/hbase/commit/0e95a8a0ae24b0d19b391d49794d6716a8e86bcd , when hbase-backup was using it. When it got removed in hbase-backup, this was forgotten. Static analyzers is now flagging hbase as there is a CVE for guava older versions. Fix: just remove it from pom.xml > Remove unused hadoop.guava.version from pom.xml > --- > > Key: HBASE-23828 > URL: https://issues.apache.org/jira/browse/HBASE-23828 > Project: HBase > Issue Type: Improvement >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > > 11.0.2 > is still used in hbase-backup, I missed it at first... > So, this should be either updated or removed. > Checking which is feasible... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] nkalmar closed pull request #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml
nkalmar closed pull request #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml URL: https://github.com/apache/hbase/pull/1163 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] nkalmar edited a comment on issue #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml
nkalmar edited a comment on issue #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml URL: https://github.com/apache/hbase/pull/1163#issuecomment-584630106 Yes, that's a big miss on my part, sorry, I'm looking into it... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] nkalmar commented on issue #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml
nkalmar commented on issue #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml URL: https://github.com/apache/hbase/pull/1163#issuecomment-584630106 Yes, that's a big mis on my part, sorry, I'm looking into it... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] HorizonNet commented on issue #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml
HorizonNet commented on issue #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml URL: https://github.com/apache/hbase/pull/1163#issuecomment-584626626 It seems that this property is still used in the `hadoop-2.0` profile of hbase-backup. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml
Apache-HBase commented on issue #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml URL: https://github.com/apache/hbase/pull/1163#issuecomment-584625550 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 2s | master passed | | +1 :green_heart: | compile | 3m 20s | master passed | | +1 :green_heart: | shadedjars | 5m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 58s | master passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 0m 8s | root in the patch failed. | | -1 :x: | compile | 0m 7s | root in the patch failed. | | -1 :x: | javac | 0m 7s | root in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | -1 :x: | shadedjars | 0m 5s | patch has 12 errors when building our shaded downstream artifacts. | | -1 :x: | hadoopcheck | 0m 5s | The patch causes 12 errors with Hadoop v2.8.5. | | -1 :x: | hadoopcheck | 0m 11s | The patch causes 12 errors with Hadoop v2.9.2. | | -1 :x: | javadoc | 0m 10s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 10s | root in the patch failed. | | +0 :ok: | asflicense | 0m 11s | ASF License check generated no output? | | | | 26m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1163 | | Optional Tests | dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile | | uname | Linux 9aacac1e2af9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1163/out/precommit/personality/provided.sh | | git revision | master / 12f98bc50e | | Default Java | 1.8.0_181 | | mvninstall | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/artifact/out/patch-mvninstall-root.txt | | compile | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/artifact/out/patch-compile-root.txt | | javac | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/artifact/out/patch-compile-root.txt | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/artifact/out/patch-shadedjars.txt | | hadoopcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/artifact/out/patch-javac-2.8.5.txt | | hadoopcheck | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/artifact/out/patch-javac-2.9.2.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/artifact/out/patch-javadoc-root.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/testReport/ | | Max. process+thread count | 86 (vs. ulimit of 1) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1163/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] nkalmar opened a new pull request #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml
nkalmar opened a new pull request #1163: HBASE-23828 - Remove unused hadoop.guava.version from pom.xml URL: https://github.com/apache/hbase/pull/1163 11.0.2 is not used anywhere. It was added in https://github.com/apache/hbase/commit/0e95a8a0ae24b0d19b391d49794d6716a8e86bcd , when hbase-backup was using it. When it got removed in hbase-backup, this was forgotten. Static analyzers is now flagging hbase as there is a CVE for guava older versions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work started] (HBASE-23828) Remove unused hadoop.guava.version from pom.xml
[ https://issues.apache.org/jira/browse/HBASE-23828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23828 started by Norbert Kalmár. -- > Remove unused hadoop.guava.version from pom.xml > --- > > Key: HBASE-23828 > URL: https://issues.apache.org/jira/browse/HBASE-23828 > Project: HBase > Issue Type: Improvement >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > > 11.0.2 is not used anywhere. It > was added in > https://github.com/apache/hbase/commit/0e95a8a0ae24b0d19b391d49794d6716a8e86bcd > , when hbase-backup was using it. When it got removed in hbase-backup, this > was forgotten. Static analyzers is now flagging hbase as there is a CVE for > guava older versions. > Fix: just remove it from pom.xml -- This message was sent by Atlassian Jira (v8.3.4#803005)
