[GitHub] [hbase] Apache-HBase commented on pull request #2148: HBASE-24752 NPE/500 accessing webui on master startup
Apache-HBase commented on pull request #2148: URL: https://github.com/apache/hbase/pull/2148#issuecomment-664264169 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 33s | branch-2.3 passed | | +1 :green_heart: | javadoc | 0m 37s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 137m 40s | hbase-server in the patch passed. | | | | 148m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2148/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2148 | | Optional Tests | javac javadoc unit | | uname | Linux 93b6e53face4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / ba70566ab9 | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2148/1/testReport/ | | Max. process+thread count | 4217 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2148/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn merged pull request #2150: HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
anoopsjohn merged pull request #2150: URL: https://github.com/apache/hbase/pull/2150 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2148: HBASE-24752 NPE/500 accessing webui on master startup
Apache-HBase commented on pull request #2148: URL: https://github.com/apache/hbase/pull/2148#issuecomment-664273629 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 2s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 20s | branch-2.3 passed | | -0 :warning: | javadoc | 0m 52s | hbase-server in branch-2.3 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 14s | the patch passed | | -0 :warning: | javadoc | 0m 47s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 135m 23s | hbase-server in the patch passed. | | | | 151m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2148/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2148 | | Optional Tests | javac javadoc unit | | uname | Linux 02bc98df29a3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / ba70566ab9 | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2148/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2148/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2148/1/testReport/ | | Max. process+thread count | 3771 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2148/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-20598) Upgrade to JRuby 9.2
[ https://issues.apache.org/jira/browse/HBASE-20598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165663#comment-17165663 ] Tamas Penzes commented on HBASE-20598: -- [~jackbearden] do you have a plan to finish this task? > Upgrade to JRuby 9.2 > > > Key: HBASE-20598 > URL: https://issues.apache.org/jira/browse/HBASE-20598 > Project: HBase > Issue Type: Bug > Components: dependencies, shell >Reporter: Josh Elser >Assignee: Jack Bearden >Priority: Major > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-20598.001.patch, HBASE-20598.002.patch > > > [~mdrob] pointed out that there's a JRuby 9.2 release. We should see if we > can get ourselves onto that from our current 9.1 release line. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24758) Avoid flooding replication source RSes logs when no sinks are available
[
https://issues.apache.org/jira/browse/HBASE-24758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-24758:
-
Component/s: Replication
> Avoid flooding replication source RSes logs when no sinks are available
>
>
> Key: HBASE-24758
> URL: https://issues.apache.org/jira/browse/HBASE-24758
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
>
> On HBaseInterClusterReplicationEndpoint.replicate, if no sinks are returned
> by ReplicationSinkManager, say remote peer is not available, we log message
> below and return false to source shipper thread, which then keeps retrying,
> flooding source RS log with the below messages:
> {noformat}
> WARN
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint:
> No replication sinks found, returning without replicating. The source should
> retry with the same set of edits.
> {noformat}
> This condition could also cause ReplicationSinkManager.chooseSinks to blow an
> NPE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2149: HBASE-24760 Allow system tables fallback to any rs groups
Apache-HBase commented on pull request #2149: URL: https://github.com/apache/hbase/pull/2149#issuecomment-664247124 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 12s | master passed | | +1 :green_heart: | compile | 1m 25s | master passed | | +1 :green_heart: | shadedjars | 6m 48s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 43s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 6m 18s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 44s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 9m 58s | hbase-server in the patch failed. | | | | 38m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2149 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1a31b462ed80 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8c0d7fa5b8 | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/testReport/ | | Max. process+thread count | 1578 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24665) MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
[ https://issues.apache.org/jira/browse/HBASE-24665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-24665: --- Fix Version/s: (was: 1.4.14) > MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll > -- > > Key: HBASE-24665 > URL: https://issues.apache.org/jira/browse/HBASE-24665 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.3.0, master, 2.1.10, 1.4.14, 2.2.6 >Reporter: wenfeiyi666 >Assignee: wenfeiyi666 >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7 > > > when use multiwal, any a wal request roll, all wal will be together roll. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] utf7 closed pull request #2153: HBASE-24777 InfoServer support ipv6 host and port
utf7 closed pull request #2153: URL: https://github.com/apache/hbase/pull/2153 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] utf7 opened a new pull request #2153: HBASE-24777 InfoServer support ipv6 host and port
utf7 opened a new pull request #2153: URL: https://github.com/apache/hbase/pull/2153 HBASE-24777 InfoServer support ipv6 host and port This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on pull request #2150: HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
anoopsjohn commented on pull request #2150: URL: https://github.com/apache/hbase/pull/2150#issuecomment-664337766 This is applied to branch-2 but not able to cherry pick to branch-2.3. There are conflicts. Can u pls raise PR for 2.3 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #2147: HBASE-24777 InfoServer support ipv6 host and port
virajjasani commented on pull request #2147: URL: https://github.com/apache/hbase/pull/2147#issuecomment-664252063 Even if we have some way to mock, it would be fine, but anyways, can be taken as a separate task. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2150: HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
Apache-HBase commented on pull request #2150: URL: https://github.com/apache/hbase/pull/2150#issuecomment-664274027 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 8s | branch-2 passed | | +1 :green_heart: | spotbugs | 1m 58s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 12s | the patch passed | | -0 :warning: | checkstyle | 1m 6s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 37s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 33m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2150/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2150 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux d525cf4e8f29 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 6cb51cc0f0 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2150/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2150/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] utf7 commented on pull request #2147: HBASE-24777 InfoServer support ipv6 host and port
utf7 commented on pull request #2147: URL: https://github.com/apache/hbase/pull/2147#issuecomment-664344675 thanks for review @virajjasani @Apache9 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-operator-tools] ZhaoBQ opened a new pull request #71: HBASE-24778 [hbase-operator-tools] Merging regions failed when the table is not default namespace
ZhaoBQ opened a new pull request #71: URL: https://github.com/apache/hbase-operator-tools/pull/71 [HBASE-24778](https://issues.apache.org/jira/browse/HBASE-24778) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd commented on a change in pull request #2138: HBASE-20717 Implement CCSMap - a better concurrent map with compacted data structure
Reidd commented on a change in pull request #2138:
URL: https://github.com/apache/hbase/pull/2138#discussion_r460777550
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/ccsmap/TestCompactedConcurrentSkipList.java
##
@@ -0,0 +1,1842 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hbase.ccsmap;
+
+import java.security.InvalidParameterException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.TreeMap;
+import java.util.Map.Entry;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentNavigableMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import
org.apache.hadoop.hbase.ccsmap.TestCompactedConcurrentSkipList.BehaviorCollections.RandomBehavior;
+import org.apache.hadoop.hbase.testclassification.SmallTests;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category(SmallTests.class)
+public class TestCompactedConcurrentSkipList {
+ static int BIG_KV_THRESHOLD = 1024;
+ static int DATA_PAGE_SIZE = 16 * 1024;
+ static int DEFAULT_MIN_GROUP = 5;
+ static int DEFAULT_MAX_GROUP = 10;
+
+ static class TypeHelper implements CompactedTypeHelper {
+
+@Override
+public int getCompactedSize(Integer key, String value) {
+ return 4 + value.length();
+}
+
+private void copyIntToArray(int val, byte[] data, int offset) {
+ for (int pos = offset; pos < offset + 4; ++pos) {
+data[pos] = (byte) (val & BYTE);
+val = val >> 8;
+ }
+}
+
+static int BYTE = 0xFF;
+
+private int getIntFromArray(byte[] data, int offset) {
+ int ret = 0;
+ for (int pos = offset + 3; pos >= offset; pos--) {
+int b = data[pos];
+ret = (ret << 8) | (b & BYTE);
+ }
+ return ret;
+}
+
+@Override
+public void compact(Integer key, String value, byte[] data, int offset,
+ int len) {
+ Assert.assertEquals(4 + value.length(), len);
+ copyIntToArray(key, data, offset);
+ byte[] src = value.getBytes();
+ Assert.assertEquals(value.length(), src.length);
+ System.arraycopy(src, 0, data, offset + 4, src.length);
+}
+
+@Override
+public KVPair decomposite(byte[] data, int offset, int
len) {
+ Assert.assertTrue(len > 4);
+ int intkey = getIntFromArray(data, offset);
+ String value = new String(data, offset + 4, len - 4);
+ return new KVPair<>(intkey, value);
+}
+
+private int compare(int key1, int key2) {
+ return Integer.compare(key1, key2);
+}
+
+@Override
+public int compare(byte[] data1, int offset1, int len1, byte[] data2,
+ int offset2, int len2) {
+ Assert.assertTrue(len1 > 4);
+ Assert.assertTrue(len2 > 4);
+ int key1 = getIntFromArray(data1, offset1);
+ int key2 = getIntFromArray(data2, offset2);
+ return compare(key1, key2);
+}
+
+@Override
+public int compare(Integer key, byte[] data, int offset, int len) {
+ Assert.assertTrue(len > 4);
+ return compare(key.intValue(), getIntFromArray(data, offset));
+}
+
+@Override
+public int compare(Integer key1, Integer key2) {
+ return compare(key1.intValue(), key2.intValue());
+}
+
+ }
+
+ interface KVGenerator {
+KVPair[] getNextGroup(Random random);
+
+KVPair getNextKV(Random random);
+
+void validate(int key, String value);
+ }
+
+ static class KVGeneratorFactory {
+static abstract class KVGeneratorBase implements KVGenerator {
+ private boolean single;
+ private boolean group;
+
+ public KVGeneratorBase(boolean supportSingle, boolean supportGroup) {
+this.single = supportSingle;
+this.group = supportGroup;
+ }
+
+ @Override
+ public KVPair[] getNextGroup(Random random) {
+throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public
[GitHub] [hbase] WenFeiYi opened a new pull request #2152: HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
WenFeiYi opened a new pull request #2152: URL: https://github.com/apache/hbase/pull/2152 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2149: HBASE-24760 Allow system tables fallback to any rs groups
Apache-HBase commented on pull request #2149: URL: https://github.com/apache/hbase/pull/2149#issuecomment-664328533 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +1 :green_heart: | checkstyle | 1m 12s | master passed | | +1 :green_heart: | spotbugs | 2m 1s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 37s | the patch passed | | -0 :warning: | checkstyle | 1m 6s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 49s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 34m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2149 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 5e4e571c121b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8c0d7fa5b8 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165575#comment-17165575
]
Danil Lipovoy commented on HBASE-23887:
---
[~bharathv]
Looks like there is some obstacle, could you please explain, what is a concern?
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] tedyu commented on pull request #2138: HBASE-20717 Implement CCSMap - a better concurrent map with compacted data structure
tedyu commented on pull request #2138: URL: https://github.com/apache/hbase/pull/2138#issuecomment-664289303 It would be good to involve one of the original developers so that we get their buy-in for the changes. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24758) Avoid flooding replication source RSes logs when no sinks are available
[
https://issues.apache.org/jira/browse/HBASE-24758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil resolved HBASE-24758.
--
Resolution: Fixed
Pushed to master, branch-2 and branch-2.3.
> Avoid flooding replication source RSes logs when no sinks are available
>
>
> Key: HBASE-24758
> URL: https://issues.apache.org/jira/browse/HBASE-24758
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.5
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> On HBaseInterClusterReplicationEndpoint.replicate, if no sinks are returned
> by ReplicationSinkManager, say remote peer is not available, we log message
> below and return false to source shipper thread, which then keeps retrying,
> flooding source RS log with the below messages:
> {noformat}
> WARN
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint:
> No replication sinks found, returning without replicating. The source should
> retry with the same set of edits.
> {noformat}
> This condition could also cause ReplicationSinkManager.chooseSinks to blow an
> NPE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24758) Avoid flooding replication source RSes logs when no sinks are available
[
https://issues.apache.org/jira/browse/HBASE-24758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-24758:
-
Fix Version/s: 2.4.0
2.3.1
3.0.0-alpha-1
> Avoid flooding replication source RSes logs when no sinks are available
>
>
> Key: HBASE-24758
> URL: https://issues.apache.org/jira/browse/HBASE-24758
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.5
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> On HBaseInterClusterReplicationEndpoint.replicate, if no sinks are returned
> by ReplicationSinkManager, say remote peer is not available, we log message
> below and return false to source shipper thread, which then keeps retrying,
> flooding source RS log with the below messages:
> {noformat}
> WARN
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint:
> No replication sinks found, returning without replicating. The source should
> retry with the same set of edits.
> {noformat}
> This condition could also cause ReplicationSinkManager.chooseSinks to blow an
> NPE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24758) Avoid flooding replication source RSes logs when no sinks are available
[
https://issues.apache.org/jira/browse/HBASE-24758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-24758:
-
Affects Version/s: 2.4.0
2.3.1
3.0.0-alpha-1
2.2.5
> Avoid flooding replication source RSes logs when no sinks are available
>
>
> Key: HBASE-24758
> URL: https://issues.apache.org/jira/browse/HBASE-24758
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.5
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
>
> On HBaseInterClusterReplicationEndpoint.replicate, if no sinks are returned
> by ReplicationSinkManager, say remote peer is not available, we log message
> below and return false to source shipper thread, which then keeps retrying,
> flooding source RS log with the below messages:
> {noformat}
> WARN
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint:
> No replication sinks found, returning without replicating. The source should
> retry with the same set of edits.
> {noformat}
> This condition could also cause ReplicationSinkManager.chooseSinks to blow an
> NPE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] anoopsjohn commented on pull request #2150: HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
anoopsjohn commented on pull request #2150: URL: https://github.com/apache/hbase/pull/2150#issuecomment-664188239 +1. Will merge once QA is good This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #2148: HBASE-24752 NPE/500 accessing webui on master startup
virajjasani commented on a change in pull request #2148:
URL: https://github.com/apache/hbase/pull/2148#discussion_r460747418
##
File path:
hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
##
@@ -224,6 +225,11 @@ AssignmentManager assignmentManager =
master.getAssignmentManager();
<& AssignmentManagerStatusTmpl;
assignmentManager=master.getAssignmentManager()&>
<%if !master.isInMaintenanceMode() %>
+ <%java>
+ while (master.getMasterCoprocessorHost() == null) {
+Threads.sleep(50);
Review comment:
Let's also print some message which we can removed after getting out of
this `while` loop?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] utf7 commented on pull request #2147: HBASE-24777 InfoServer support ipv6 host and port
utf7 commented on pull request #2147: URL: https://github.com/apache/hbase/pull/2147#issuecomment-664182562 @Apache9 @infraio @anoopsjohn mind have a look ,small patch ,don't know why yetus failed and the patch doesn't apply branch-2.2/branch-2.1, what should I do ? make another push , like : push origin origin branch-2.1:HBASE-214777 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165495#comment-17165495 ] Sean Busbey commented on HBASE-22114: - IIRC we need the ability for precommit to only run some modules if the using jdk8. I think I had a WIP branch somewhere that had this working. If the precommit builds are working over on the new CI set up I could try digging it up and getting it to work? > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.7.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2147: HBASE-24777 InfoServer support ipv6 host and port
Apache-HBase commented on pull request #2147: URL: https://github.com/apache/hbase/pull/2147#issuecomment-664167209 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 52s | master passed | | +1 :green_heart: | checkstyle | 0m 16s | master passed | | +1 :green_heart: | spotbugs | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | checkstyle | 0m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 43s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 0m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 30m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2147/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2147 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux cfce498787f3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 975cdf7b88 | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-http U: hbase-http | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2147/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2148: HBASE-24752 NPE/500 accessing webui on master startup
Apache-HBase commented on pull request #2148: URL: https://github.com/apache/hbase/pull/2148#issuecomment-664180045 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.3 Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 2m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2148/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2148 | | Optional Tests | dupname asflicense | | uname | Linux 3ad36e31a95d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / ba70566ab9 | | Max. process+thread count | 51 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2148/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2147: HBASE-24777 InfoServer support ipv6 host and port
Apache9 commented on pull request #2147: URL: https://github.com/apache/hbase/pull/2147#issuecomment-664223063 And for the UT, I think it could be another issue. I do not know if we have any IPv6 related UTs in our code base, as it has special requirements on the env? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] WenFeiYi commented on pull request #2150: HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
WenFeiYi commented on pull request #2150: URL: https://github.com/apache/hbase/pull/2150#issuecomment-664223292 also applies to 2.3 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2149: HBASE-24760 Allow system tables fallback to any rs groups
Apache-HBase commented on pull request #2149: URL: https://github.com/apache/hbase/pull/2149#issuecomment-664233711 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 28s | master passed | | +1 :green_heart: | compile | 0m 56s | master passed | | +1 :green_heart: | shadedjars | 5m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 6m 16s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 21s | hbase-server in the patch failed. | | | | 33m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2149 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f65a776bc1d7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 975cdf7b88 | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/testReport/ | | Max. process+thread count | 1590 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24778) [hbase-operator-tools] Merging regions failed when the table is not default namespace
[
https://issues.apache.org/jira/browse/HBASE-24778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Baiqiang Zhao updated HBASE-24778:
--
Description:
Suppose there is a table "test:test_table", if you want to merge regions and
run the command:
{code:java}
// /opt/hbase/bin/hbase org.apache.hbase.RegionsMerger test:test_table 10
{code}
Then you will get error:
{code:java}
// [main] ERROR org.apache.hbase.RegionsMerger - Merging regions failed:[main]
ERROR org.apache.hbase.RegionsMerger - Merging regions
failed:java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI: test:test_table
at org.apache.hadoop.fs.Path.initialize(Path.java:259)
~[hadoop-common-3.2.0.jar:?]
at org.apache.hadoop.fs.Path.(Path.java:217)
~[hadoop-common-3.2.0.jar:?]
at org.apache.hadoop.fs.Path.(Path.java:125)
~[hadoop-common-3.2.0.jar:?]
at org.apache.hbase.RegionsMerger.getTablePath(RegionsMerger.java:93)
~[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.hbase.RegionsMerger.mergeRegions(RegionsMerger.java:163)
~[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.hbase.RegionsMerger.run(RegionsMerger.java:237)
[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
[hadoop-common-3.2.0.jar:?]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
[hadoop-common-3.2.0.jar:?]
at org.apache.hbase.RegionsMerger.main(RegionsMerger.java:247)
[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]Caused by:
java.net.URISyntaxException: Relative path in absolute URI: test:test_table
at java.net.URI.checkPath(URI.java:1823) ~[?:1.8.0_121]
at java.net.URI.(URI.java:745) ~[?:1.8.0_121]
at org.apache.hadoop.fs.Path.initialize(Path.java:256)
~[hadoop-common-3.2.0.jar:?]
... 8 more
{code}
was:
Suppose there is a table "test:test_table", if you want to merge regions and
run the command:
{code:java}
// /opt/hbase/bin/hbase org.apache.hbase.RegionsMerger test:test_table 10
{code}
Then you will get error:
{code:java}
// [main] ERROR org.apache.hbase.RegionsMerger - Merging regions failed:[main]
ERROR org.apache.hbase.RegionsMerger - Merging regions
failed:java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI: test:test_table at
org.apache.hadoop.fs.Path.initialize(Path.java:259)
~[hadoop-common-3.2.0.jar:?] at org.apache.hadoop.fs.Path.(Path.java:217)
~[hadoop-common-3.2.0.jar:?] at org.apache.hadoop.fs.Path.(Path.java:125)
~[hadoop-common-3.2.0.jar:?] at
org.apache.hbase.RegionsMerger.getTablePath(RegionsMerger.java:93)
~[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT] at
org.apache.hbase.RegionsMerger.mergeRegions(RegionsMerger.java:163)
~[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT] at
org.apache.hbase.RegionsMerger.run(RegionsMerger.java:237)
[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT] at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
[hadoop-common-3.2.0.jar:?] at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
[hadoop-common-3.2.0.jar:?] at
org.apache.hbase.RegionsMerger.main(RegionsMerger.java:247)
[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]Caused by:
java.net.URISyntaxException: Relative path in absolute URI: test:test_table at
java.net.URI.checkPath(URI.java:1823) ~[?:1.8.0_121] at
java.net.URI.(URI.java:745) ~[?:1.8.0_121] at
org.apache.hadoop.fs.Path.initialize(Path.java:256)
~[hadoop-common-3.2.0.jar:?] ... 8 more
{code}
> [hbase-operator-tools] Merging regions failed when the table is not default
> namespace
> -
>
> Key: HBASE-24778
> URL: https://issues.apache.org/jira/browse/HBASE-24778
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Affects Versions: hbase-operator-tools-1.1.0
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
>
> Suppose there is a table "test:test_table", if you want to merge regions and
> run the command:
> {code:java}
> // /opt/hbase/bin/hbase org.apache.hbase.RegionsMerger test:test_table 10
> {code}
> Then you will get error:
> {code:java}
> // [main] ERROR org.apache.hbase.RegionsMerger - Merging regions
> failed:[main] ERROR org.apache.hbase.RegionsMerger - Merging regions
> failed:java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Relative path in absolute URI: test:test_table
> at org.apache.hadoop.fs.Path.initialize(Path.java:259)
> ~[hadoop-common-3.2.0.jar:?]
> at org.apache.hadoop.fs.Path.(Path.java:217)
> ~[hadoop-common-3.2.0.jar:?]
> at org.apache.hadoop.fs.Path.(Path.java:125)
> ~[hadoop-common-3.2.0.jar:?]
> at org.apache.hbase.RegionsMerger.getTablePath(RegionsMerger.java:93)
>
[GitHub] [hbase] WenFeiYi opened a new pull request #2148: HBASE-24752 NPE/500 accessing webui on master startup
WenFeiYi opened a new pull request #2148: URL: https://github.com/apache/hbase/pull/2148 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2147: HBASE-24777 InfoServer support ipv6 host and port
Apache-HBase commented on pull request #2147: URL: https://github.com/apache/hbase/pull/2147#issuecomment-664178699 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | compile | 0m 24s | master passed | | +1 :green_heart: | shadedjars | 7m 7s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 20s | hbase-http in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 33s | the patch passed | | +1 :green_heart: | compile | 0m 23s | the patch passed | | +1 :green_heart: | javac | 0m 23s | the patch passed | | +1 :green_heart: | shadedjars | 6m 12s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 20s | hbase-http in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 49s | hbase-http in the patch passed. | | | | 25m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2147/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2147 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bc161cd4f3e0 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 975cdf7b88 | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2147/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-http.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2147/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-http.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2147/2/testReport/ | | Max. process+thread count | 494 (vs. ulimit of 12500) | | modules | C: hbase-http U: hbase-http | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2147/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] WenFeiYi commented on pull request #2151: HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
WenFeiYi commented on pull request #2151: URL: https://github.com/apache/hbase/pull/2151#issuecomment-664226904 also applies to 2.1 and 2.2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] WenFeiYi opened a new pull request #2151: HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
WenFeiYi opened a new pull request #2151: URL: https://github.com/apache/hbase/pull/2151 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24778) [hbase-operator-tools] Merging regions failed when the table is not default namespace
[
https://issues.apache.org/jira/browse/HBASE-24778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Baiqiang Zhao updated HBASE-24778:
--
Description:
Suppose there is a table "test:test_table", if you want to merge regions and
run the command:
{code:java}
/opt/hbase/bin/hbase org.apache.hbase.RegionsMerger test:test_table 10
{code}
Then you will get error:
{code:java}
[main] ERROR org.apache.hbase.RegionsMerger - Merging regions failed:[main]
ERROR org.apache.hbase.RegionsMerger - Merging regions
failed:java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI: test:test_table
at org.apache.hadoop.fs.Path.initialize(Path.java:259)
~[hadoop-common-3.2.0.jar:?]
at org.apache.hadoop.fs.Path.(Path.java:217)
~[hadoop-common-3.2.0.jar:?]
at org.apache.hadoop.fs.Path.(Path.java:125)
~[hadoop-common-3.2.0.jar:?]
at org.apache.hbase.RegionsMerger.getTablePath(RegionsMerger.java:93)
~[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.hbase.RegionsMerger.mergeRegions(RegionsMerger.java:163)
~[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.hbase.RegionsMerger.run(RegionsMerger.java:237)
[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
[hadoop-common-3.2.0.jar:?]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
[hadoop-common-3.2.0.jar:?]
at org.apache.hbase.RegionsMerger.main(RegionsMerger.java:247)
[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]Caused by:
java.net.URISyntaxException: Relative path in absolute URI: test:test_table
at java.net.URI.checkPath(URI.java:1823) ~[?:1.8.0_121]
at java.net.URI.(URI.java:745) ~[?:1.8.0_121]
at org.apache.hadoop.fs.Path.initialize(Path.java:256)
~[hadoop-common-3.2.0.jar:?]
... 8 more
{code}
was:
Suppose there is a table "test:test_table", if you want to merge regions and
run the command:
{code:java}
// /opt/hbase/bin/hbase org.apache.hbase.RegionsMerger test:test_table 10
{code}
Then you will get error:
{code:java}
// [main] ERROR org.apache.hbase.RegionsMerger - Merging regions failed:[main]
ERROR org.apache.hbase.RegionsMerger - Merging regions
failed:java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI: test:test_table
at org.apache.hadoop.fs.Path.initialize(Path.java:259)
~[hadoop-common-3.2.0.jar:?]
at org.apache.hadoop.fs.Path.(Path.java:217)
~[hadoop-common-3.2.0.jar:?]
at org.apache.hadoop.fs.Path.(Path.java:125)
~[hadoop-common-3.2.0.jar:?]
at org.apache.hbase.RegionsMerger.getTablePath(RegionsMerger.java:93)
~[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.hbase.RegionsMerger.mergeRegions(RegionsMerger.java:163)
~[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.hbase.RegionsMerger.run(RegionsMerger.java:237)
[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
[hadoop-common-3.2.0.jar:?]
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
[hadoop-common-3.2.0.jar:?]
at org.apache.hbase.RegionsMerger.main(RegionsMerger.java:247)
[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]Caused by:
java.net.URISyntaxException: Relative path in absolute URI: test:test_table
at java.net.URI.checkPath(URI.java:1823) ~[?:1.8.0_121]
at java.net.URI.(URI.java:745) ~[?:1.8.0_121]
at org.apache.hadoop.fs.Path.initialize(Path.java:256)
~[hadoop-common-3.2.0.jar:?]
... 8 more
{code}
> [hbase-operator-tools] Merging regions failed when the table is not default
> namespace
> -
>
> Key: HBASE-24778
> URL: https://issues.apache.org/jira/browse/HBASE-24778
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Affects Versions: hbase-operator-tools-1.1.0
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
>
> Suppose there is a table "test:test_table", if you want to merge regions and
> run the command:
> {code:java}
> /opt/hbase/bin/hbase org.apache.hbase.RegionsMerger test:test_table 10
> {code}
> Then you will get error:
> {code:java}
> [main] ERROR org.apache.hbase.RegionsMerger - Merging regions failed:[main]
> ERROR org.apache.hbase.RegionsMerger - Merging regions
> failed:java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Relative path in absolute URI: test:test_table
> at org.apache.hadoop.fs.Path.initialize(Path.java:259)
> ~[hadoop-common-3.2.0.jar:?]
> at org.apache.hadoop.fs.Path.(Path.java:217)
> ~[hadoop-common-3.2.0.jar:?]
> at org.apache.hadoop.fs.Path.(Path.java:125)
> ~[hadoop-common-3.2.0.jar:?]
> at
[jira] [Updated] (HBASE-24754) Bulk load performance is degraded in HBase 2
[ https://issues.apache.org/jira/browse/HBASE-24754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai updated HBASE-24754: -- Description: in our Test,It is observed that Bulk load performance is degraded in HBase 2 . Test Input: 1: Table with 500 region(300 column family) 2: data =2 TB Data Sample 186000120150205100068110,1860001,20150205,5,404,735412,2938,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,1 3: Cluster: 7 node(2 master+5 Region Server) 4: No of Container Launched are same in both case HBase 2 took 10% more time then HBase 1.3 where test input is same for both cluster |Feature|HBase 2.2.3 Time(Sec)|HBase 1.3.1 Time(Sec)|Diff%|Snappy lib: | |BulkLoad|21837|19686.16|-10.93|Snappy lib: HBase 2.2.3: 1.4 HBase 1.3.1: 1.4| was: in our Test,It is observed that Bulk load performance is degraded in HBase 2 . Test Input: 1: Table with 500 region 2: data =2 TB 3: Cluster: 7 node(2 master+5 Region Server) 4: No of Container Launched are same in both case HBase 2 took 10% more time then HBase 1.3 where test input is same for both cluster |Feature|HBase 2.2.3 Time(Sec)|HBase 1.3.1 Time(Sec)|Diff%|Snappy lib: | |BulkLoad|21837|19686.16|-10.93|Snappy lib: HBase 2.2.3: 1.4 HBase 1.3.1: 1.4| > Bulk load performance is degraded in HBase 2 > - > > Key: HBASE-24754 > URL: https://issues.apache.org/jira/browse/HBASE-24754 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Priority: Major > > in our Test,It is observed that Bulk load performance is degraded in HBase 2 . > Test Input: > 1: Table with 500 region(300 column family) > 2: data =2 TB > Data Sample > 186000120150205100068110,1860001,20150205,5,404,735412,2938,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,1 > 3: Cluster: 7 node(2 master+5 Region Server) > 4: No of Container Launched are same in both case > HBase 2 took 10% more time then HBase 1.3 where test input is same for both > cluster > > |Feature|HBase 2.2.3 > Time(Sec)|HBase 1.3.1 > Time(Sec)|Diff%|Snappy lib: > | > |BulkLoad|21837|19686.16|-10.93|Snappy lib: > HBase 2.2.3: 1.4 > HBase 1.3.1: 1.4| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2147: HBASE-24777 InfoServer support ipv6 host and port
Apache-HBase commented on pull request #2147: URL: https://github.com/apache/hbase/pull/2147#issuecomment-664172019 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 2s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 48s | master passed | | +1 :green_heart: | compile | 0m 20s | master passed | | +1 :green_heart: | shadedjars | 5m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 0m 20s | the patch passed | | +1 :green_heart: | javac | 0m 20s | the patch passed | | +1 :green_heart: | shadedjars | 5m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 40s | hbase-http in the patch passed. | | | | 22m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2147/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2147 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a93ee111f8c7 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 975cdf7b88 | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2147/2/testReport/ | | Max. process+thread count | 469 (vs. ulimit of 12500) | | modules | C: hbase-http U: hbase-http | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2147/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Reidddddd commented on pull request #2138: HBASE-20717 Implement CCSMap - a better concurrent map with compacted data structure
Reidd commented on pull request #2138: URL: https://github.com/apache/hbase/pull/2138#issuecomment-664180870 Hi @tedyu (waving), long time no see! I addressed parts of your reviews, some of them I'm not sure as well. Most of codes can't explain itself, and as we get deep, I believe weird codes may appear more. And I will address them after we have some discussions, like the method decomposite()... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
anoopsjohn commented on a change in pull request #2113:
URL: https://github.com/apache/hbase/pull/2113#discussion_r460664425
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/InitMetaProcedure.java
##
@@ -71,7 +71,11 @@ private static void writeFsLayout(Path rootDir,
Configuration conf) throws IOExc
LOG.info("BOOTSTRAP: creating hbase:meta region");
FileSystem fs = rootDir.getFileSystem(conf);
Path tableDir = CommonFSUtils.getTableDir(rootDir,
TableName.META_TABLE_NAME);
-if (fs.exists(tableDir) && !fs.delete(tableDir, true)) {
+boolean removeMeta = conf.getBoolean(HConstants.REMOVE_META_ON_RESTART,
Review comment:
// Checking if meta needs initializing.
status.setStatus("Initializing meta table if this is a new deploy");
InitMetaProcedure initMetaProc = null;
// Print out state of hbase:meta on startup; helps debugging.
RegionState rs = this.assignmentManager.getRegionStates().
getRegionState(RegionInfoBuilder.FIRST_META_REGIONINFO);
LOG.info("hbase:meta {}", rs);
if (rs != null && rs.isOffline()) {
Optional optProc =
procedureExecutor.getProcedures().stream()
.filter(p -> p instanceof InitMetaProcedure).map(o ->
(InitMetaProcedure) o).findAny();
initMetaProc = optProc.orElseGet(() -> {
// schedule an init meta procedure if meta has not been deployed yet
InitMetaProcedure temp = new InitMetaProcedure();
procedureExecutor.submitProcedure(temp);
return temp;
});
}
So the checks we do see that META location is not there in zk and so it
thinks its new deploy. So here is what we need to tackle.
In cloud redeploy case we will see a pattern where we will have a clusterId
in the FS and not in zk. This can be used as an indicator? IMO we should find
it (using this way or other) that its a redeploy on an existing datset and all
these places, we need to consider that also to decide we need such bootstrap
steps.
We should not be doing that with a config way IMO. Because then in cloud
based deploy, what if the 1st time start fail and there is a need for this
bootstrap cleaning of META FS dir?
Even the other unknown server case also. Lets identify clearly this
redeploy case and act then only.
Can we pls ave that discuss and conclude on a solution for that and then
move forward?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2147: HBASE-24777 InfoServer support ipv6 host and port
Apache9 commented on pull request #2147: URL: https://github.com/apache/hbase/pull/2147#issuecomment-664219932 > @Apache9 @infraio @anoopsjohn mind have a look ,small patch ,don't know why yetus failed > > and the patch doesn't apply branch-2.2/branch-2.1, what should I do ? > > make another push , like : > > push origin origin branch-2.1:HBASE-214777 branch-2.1 is dead, so just ignore it. Please open another PR for branch-2.2. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil merged pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
wchevreuil merged pull request #2118: URL: https://github.com/apache/hbase/pull/2118 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
wchevreuil commented on pull request #2118: URL: https://github.com/apache/hbase/pull/2118#issuecomment-664218857 Latest UT failures seems unrelated, had the same passing locally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24778) [hbase-operator-tools] Merging regions failed when the table is not default namespace
Baiqiang Zhao created HBASE-24778:
-
Summary: [hbase-operator-tools] Merging regions failed when the
table is not default namespace
Key: HBASE-24778
URL: https://issues.apache.org/jira/browse/HBASE-24778
Project: HBase
Issue Type: Bug
Components: hbase-operator-tools
Affects Versions: hbase-operator-tools-1.1.0
Reporter: Baiqiang Zhao
Assignee: Baiqiang Zhao
Suppose there is a table "test:test_table", if you want to merge regions and
run the command:
{code:java}
// /opt/hbase/bin/hbase org.apache.hbase.RegionsMerger test:test_table 10
{code}
Then you will get error:
{code:java}
// [main] ERROR org.apache.hbase.RegionsMerger - Merging regions failed:[main]
ERROR org.apache.hbase.RegionsMerger - Merging regions
failed:java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI: test:test_table at
org.apache.hadoop.fs.Path.initialize(Path.java:259)
~[hadoop-common-3.2.0.jar:?] at org.apache.hadoop.fs.Path.(Path.java:217)
~[hadoop-common-3.2.0.jar:?] at org.apache.hadoop.fs.Path.(Path.java:125)
~[hadoop-common-3.2.0.jar:?] at
org.apache.hbase.RegionsMerger.getTablePath(RegionsMerger.java:93)
~[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT] at
org.apache.hbase.RegionsMerger.mergeRegions(RegionsMerger.java:163)
~[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT] at
org.apache.hbase.RegionsMerger.run(RegionsMerger.java:237)
[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT] at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
[hadoop-common-3.2.0.jar:?] at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
[hadoop-common-3.2.0.jar:?] at
org.apache.hbase.RegionsMerger.main(RegionsMerger.java:247)
[hbase-tools-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]Caused by:
java.net.URISyntaxException: Relative path in absolute URI: test:test_table at
java.net.URI.checkPath(URI.java:1823) ~[?:1.8.0_121] at
java.net.URI.(URI.java:745) ~[?:1.8.0_121] at
org.apache.hadoop.fs.Path.initialize(Path.java:256)
~[hadoop-common-3.2.0.jar:?] ... 8 more
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ddupg opened a new pull request #2149: HBASE-24760 Allow system tables fallback to any rs groups
ddupg opened a new pull request #2149: URL: https://github.com/apache/hbase/pull/2149 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] WenFeiYi opened a new pull request #2150: HBASE-24665 MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
WenFeiYi opened a new pull request #2150: URL: https://github.com/apache/hbase/pull/2150 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2149: HBASE-24760 Allow system tables fallback to any rs groups
Apache-HBase commented on pull request #2149: URL: https://github.com/apache/hbase/pull/2149#issuecomment-664209491 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | checkstyle | 1m 7s | master passed | | +1 :green_heart: | spotbugs | 2m 8s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | -0 :warning: | checkstyle | 1m 6s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 41s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 33m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2149 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 3123e67f5f0c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 975cdf7b88 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2149/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24779) Improve insight into replication WAL readers hung on checkQuota
Josh Elser created HBASE-24779:
--
Summary: Improve insight into replication WAL readers hung on
checkQuota
Key: HBASE-24779
URL: https://issues.apache.org/jira/browse/HBASE-24779
Project: HBase
Issue Type: Task
Reporter: Josh Elser
Assignee: Josh Elser
Helped a customer this past weekend who, with a large number of RegionServers,
has some RegionServers which replicated data to a peer without issues while
other RegionServers did not.
The number of queue logs varied over the past 24hrs in the same manner. Some
spikes in queued logs into 100's of logs, but other times, only 1's-10's of
logs were queued.
We were able to validate that there were "good" and "bad" RegionServers by
creating a test table, assigning it to a regionserver, enabling replication on
that table, and validating if the local puts were replicated to a peer. On a
good RS, data was replicated immediately. On a bad RS, data was never
replicated (at least, on the order of 10's of minutes which we waited).
On the "bad RS", we were able to observe that the \{{wal-reader}} thread(s) on
that RS were spending time in a Thread.sleep() in a different location than the
other. Specifically it was sitting in the
{{ReplicationSourceWALReader#checkQuota()}}'s sleep call, _not_ the
{{handleEmptyWALBatch()}} method on the same class.
My only assumption is that, somehow, these RegionServers got into a situation
where they "allocated" memory from the quota but never freed it. Then, because
the WAL reader thinks it has no free memory, it blocks indefinitely and there
are no pending edits to ship and (thus) free that memory. A cursory glance at
the code gives me a _lot_ of anxiety around places where we don't properly
clean it up (e.g. batches that fail to ship, dropping a peer). As a first stab,
let me add some more debugging so we can actually track this state properly for
the operators and their sanity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24779) Improve insight into replication WAL readers hung on checkQuota
[
https://issues.apache.org/jira/browse/HBASE-24779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165815#comment-17165815
]
Josh Elser commented on HBASE-24779:
{noformat}
"regionserver/my-regionserver:16020.replicationSource.my-regionserver.domain.com%2C16020%2C1595762757628,.replicationSource.wal-reader.my-regionserver.domain.com%2C16020%2C1595762757628,"
#531 daemon prio=5 os_prio=0 tid=0x7f0680d21000 nid=0x2c0eb
sleeping[0x7ef3bf827000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:148)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.checkQuota(ReplicationSourceWALReader.java:227)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:130)
Locked ownable synchronizers:
- None
{noformat}
For example, this is one of the {{wal-reader}} threads which is "hung" in a
loop, never actually reading entries off of the WALs. You may see one to many
of these (many, if this RS is also reading recovered queues in addition to its
active WAL)
> Improve insight into replication WAL readers hung on checkQuota
> ---
>
> Key: HBASE-24779
> URL: https://issues.apache.org/jira/browse/HBASE-24779
> Project: HBase
> Issue Type: Task
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
>
> Helped a customer this past weekend who, with a large number of
> RegionServers, has some RegionServers which replicated data to a peer without
> issues while other RegionServers did not.
> The number of queue logs varied over the past 24hrs in the same manner. Some
> spikes in queued logs into 100's of logs, but other times, only 1's-10's of
> logs were queued.
> We were able to validate that there were "good" and "bad" RegionServers by
> creating a test table, assigning it to a regionserver, enabling replication
> on that table, and validating if the local puts were replicated to a peer. On
> a good RS, data was replicated immediately. On a bad RS, data was never
> replicated (at least, on the order of 10's of minutes which we waited).
> On the "bad RS", we were able to observe that the \{{wal-reader}} thread(s)
> on that RS were spending time in a Thread.sleep() in a different location
> than the other. Specifically it was sitting in the
> {{ReplicationSourceWALReader#checkQuota()}}'s sleep call, _not_ the
> {{handleEmptyWALBatch()}} method on the same class.
> My only assumption is that, somehow, these RegionServers got into a situation
> where they "allocated" memory from the quota but never freed it. Then,
> because the WAL reader thinks it has no free memory, it blocks indefinitely
> and there are no pending edits to ship and (thus) free that memory. A cursory
> glance at the code gives me a _lot_ of anxiety around places where we don't
> properly clean it up (e.g. batches that fail to ship, dropping a peer). As a
> first stab, let me add some more debugging so we can actually track this
> state properly for the operators and their sanity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24777) InfoServer support ipv6 host and port
[ https://issues.apache.org/jira/browse/HBASE-24777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-24777: - Fix Version/s: 2.2.7 2.4.0 2.3.1 3.0.0-alpha-1 > InfoServer support ipv6 host and port > - > > Key: HBASE-24777 > URL: https://issues.apache.org/jira/browse/HBASE-24777 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 2.3.0, 2.1.9, 2.2.5 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.7 > > > ipv6 host and port is > [ a:::b:c:d]:16010 > ipv4 host and port is > a.b.c.d:16010 > we can use HostAndPort to handle it -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24777) InfoServer support ipv6 host and port
[ https://issues.apache.org/jira/browse/HBASE-24777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165791#comment-17165791 ] Viraj Jasani commented on HBASE-24777: -- [~chenyechao] Thanks for the nice patch. Could you please also raise a PR for branch-1? > InfoServer support ipv6 host and port > - > > Key: HBASE-24777 > URL: https://issues.apache.org/jira/browse/HBASE-24777 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 2.3.0, 2.1.9, 2.2.5 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.7 > > > ipv6 host and port is > [ a:::b:c:d]:16010 > ipv4 host and port is > a.b.c.d:16010 > we can use HostAndPort to handle it -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24757) ReplicationSink should limit the batch size for batch mutations based on hbase.rpc.rows.warning.threshold
[ https://issues.apache.org/jira/browse/HBASE-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-24757: - Fix Version/s: 2.2.7 2.4.0 1.7.0 2.3.1 3.0.0-alpha-1 > ReplicationSink should limit the batch size for batch mutations based on > hbase.rpc.rows.warning.threshold > - > > Key: HBASE-24757 > URL: https://issues.apache.org/jira/browse/HBASE-24757 > Project: HBase > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7 > > > At times there are quite a large no of WAL Edits to ship as part of > Replication and sometimes replication queues accumulate huge list of Edits to > process. ReplicationSink at the sink server usually goes through all Edits > and creates map of table -> list of rows grouped by clusterIds, and performs > batch mutation of all rows per table level. However, there is no limit to no > of Rows that are sent as part of batch mutate call. If no of rows > limit > threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn > "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is > turned on, RS will reject the whole batch without processing. > We should let Replication Sink honour this threshold value and accordingly > keep the size lower per batch mutation call. > Replication triggered batch mutations should always be consumed but keeping > limit of mutation low enough will let the system function at the same pace > and without triggering redundant warnings. This will also restrict > exploitation of cpu cycles at the destination server. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24632) Enable procedure-based log splitting as default in hbase3
[ https://issues.apache.org/jira/browse/HBASE-24632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165818#comment-17165818 ] Michael Stack commented on HBASE-24632: --- Merged to branch-2. Added new PR for master (there is something up w/ the hadoop3 profile) > Enable procedure-based log splitting as default in hbase3 > - > > Key: HBASE-24632 > URL: https://issues.apache.org/jira/browse/HBASE-24632 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Means changing this value in HConstants to false: >public static final boolean DEFAULT_HBASE_SPLIT_COORDINATED_BY_ZK = true; > Should probably also deprecate the current zk distributed split too so we can > clear out those classes to. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HBASE-24777) InfoServer support ipv6 host and port
[ https://issues.apache.org/jira/browse/HBASE-24777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-24777: - Comment: was deleted (was: [~chenyechao] Thanks for the nice patch. Could you please also raise a PR for branch-1?) > InfoServer support ipv6 host and port > - > > Key: HBASE-24777 > URL: https://issues.apache.org/jira/browse/HBASE-24777 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 2.3.0, 2.1.9, 2.2.5 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.7 > > > ipv6 host and port is > [ a:::b:c:d]:16010 > ipv4 host and port is > a.b.c.d:16010 > we can use HostAndPort to handle it -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24777) InfoServer support ipv6 host and port
[ https://issues.apache.org/jira/browse/HBASE-24777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-24777. -- Fix Version/s: 1.7.0 Hadoop Flags: Reviewed Resolution: Fixed > InfoServer support ipv6 host and port > - > > Key: HBASE-24777 > URL: https://issues.apache.org/jira/browse/HBASE-24777 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 2.3.0, 2.1.9, 2.2.5 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7 > > > ipv6 host and port is > [ a:::b:c:d]:16010 > ipv4 host and port is > a.b.c.d:16010 > we can use HostAndPort to handle it -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24779) Improve insight into replication WAL readers hung on checkQuota
[
https://issues.apache.org/jira/browse/HBASE-24779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165883#comment-17165883
]
Sean Busbey commented on HBASE-24779:
-
{quote}
We were able to validate that there were "good" and "bad" RegionServers by
creating a test table, assigning it to a regionserver, enabling replication on
that table, and validating if the local puts were replicated to a peer. On a
good RS, data was replicated immediately. On a bad RS, data was never
replicated (at least, on the order of 10's of minutes which we waited).
{quote}
did this use existing tooling? or something we could improve based off of e.g.
the canary tool? at a minimum sounds like a good docs addition for the
troubleshooting stuff in operator-tools or the ref guide
> Improve insight into replication WAL readers hung on checkQuota
> ---
>
> Key: HBASE-24779
> URL: https://issues.apache.org/jira/browse/HBASE-24779
> Project: HBase
> Issue Type: Task
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
>
> Helped a customer this past weekend who, with a large number of
> RegionServers, has some RegionServers which replicated data to a peer without
> issues while other RegionServers did not.
> The number of queue logs varied over the past 24hrs in the same manner. Some
> spikes in queued logs into 100's of logs, but other times, only 1's-10's of
> logs were queued.
> We were able to validate that there were "good" and "bad" RegionServers by
> creating a test table, assigning it to a regionserver, enabling replication
> on that table, and validating if the local puts were replicated to a peer. On
> a good RS, data was replicated immediately. On a bad RS, data was never
> replicated (at least, on the order of 10's of minutes which we waited).
> On the "bad RS", we were able to observe that the \{{wal-reader}} thread(s)
> on that RS were spending time in a Thread.sleep() in a different location
> than the other. Specifically it was sitting in the
> {{ReplicationSourceWALReader#checkQuota()}}'s sleep call, _not_ the
> {{handleEmptyWALBatch()}} method on the same class.
> My only assumption is that, somehow, these RegionServers got into a situation
> where they "allocated" memory from the quota but never freed it. Then,
> because the WAL reader thinks it has no free memory, it blocks indefinitely
> and there are no pending edits to ship and (thus) free that memory. A cursory
> glance at the code gives me a _lot_ of anxiety around places where we don't
> properly clean it up (e.g. batches that fail to ship, dropping a peer). As a
> first stab, let me add some more debugging so we can actually track this
> state properly for the operators and their sanity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24732) Send announce email
[ https://issues.apache.org/jira/browse/HBASE-24732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24732. -- Resolution: Fixed > Send announce email > --- > > Key: HBASE-24732 > URL: https://issues.apache.org/jira/browse/HBASE-24732 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > Use the template found on > https://hbase.apache.org/book.html#hbase.release.announcement to mail the > specified lists. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24777) InfoServer support ipv6 host and port
[ https://issues.apache.org/jira/browse/HBASE-24777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165860#comment-17165860 ] Viraj Jasani commented on HBASE-24777: -- Thanks [~chenyechao] for the contribution. > InfoServer support ipv6 host and port > - > > Key: HBASE-24777 > URL: https://issues.apache.org/jira/browse/HBASE-24777 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 2.3.0, 2.1.9, 2.2.5 >Reporter: Yechao Chen >Assignee: Yechao Chen >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7 > > > ipv6 host and port is > [ a:::b:c:d]:16010 > ipv4 host and port is > a.b.c.d:16010 > we can use HostAndPort to handle it -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24779) Improve insight into replication WAL readers hung on checkQuota
[
https://issues.apache.org/jira/browse/HBASE-24779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-24779:
Component/s: Replication
> Improve insight into replication WAL readers hung on checkQuota
> ---
>
> Key: HBASE-24779
> URL: https://issues.apache.org/jira/browse/HBASE-24779
> Project: HBase
> Issue Type: Task
> Components: Replication
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
>
> Helped a customer this past weekend who, with a large number of
> RegionServers, has some RegionServers which replicated data to a peer without
> issues while other RegionServers did not.
> The number of queue logs varied over the past 24hrs in the same manner. Some
> spikes in queued logs into 100's of logs, but other times, only 1's-10's of
> logs were queued.
> We were able to validate that there were "good" and "bad" RegionServers by
> creating a test table, assigning it to a regionserver, enabling replication
> on that table, and validating if the local puts were replicated to a peer. On
> a good RS, data was replicated immediately. On a bad RS, data was never
> replicated (at least, on the order of 10's of minutes which we waited).
> On the "bad RS", we were able to observe that the \{{wal-reader}} thread(s)
> on that RS were spending time in a Thread.sleep() in a different location
> than the other. Specifically it was sitting in the
> {{ReplicationSourceWALReader#checkQuota()}}'s sleep call, _not_ the
> {{handleEmptyWALBatch()}} method on the same class.
> My only assumption is that, somehow, these RegionServers got into a situation
> where they "allocated" memory from the quota but never freed it. Then,
> because the WAL reader thinks it has no free memory, it blocks indefinitely
> and there are no pending edits to ship and (thus) free that memory. A cursory
> glance at the code gives me a _lot_ of anxiety around places where we don't
> properly clean it up (e.g. batches that fail to ship, dropping a peer). As a
> first stab, let me add some more debugging so we can actually track this
> state properly for the operators and their sanity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24779) Improve insight into replication WAL readers hung on checkQuota
[
https://issues.apache.org/jira/browse/HBASE-24779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165885#comment-17165885
]
Sean Busbey commented on HBASE-24779:
-
{quote}Helped a customer this past weekend who, with a large number of
RegionServers, has some RegionServers which replicated data to a peer without
issues while other RegionServers did not.{quote}
hbase 1-ish replication, hbase 2-ish replication, or hbase master-branch-ish
replication?
> Improve insight into replication WAL readers hung on checkQuota
> ---
>
> Key: HBASE-24779
> URL: https://issues.apache.org/jira/browse/HBASE-24779
> Project: HBase
> Issue Type: Task
> Components: Replication
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
>
> Helped a customer this past weekend who, with a large number of
> RegionServers, has some RegionServers which replicated data to a peer without
> issues while other RegionServers did not.
> The number of queue logs varied over the past 24hrs in the same manner. Some
> spikes in queued logs into 100's of logs, but other times, only 1's-10's of
> logs were queued.
> We were able to validate that there were "good" and "bad" RegionServers by
> creating a test table, assigning it to a regionserver, enabling replication
> on that table, and validating if the local puts were replicated to a peer. On
> a good RS, data was replicated immediately. On a bad RS, data was never
> replicated (at least, on the order of 10's of minutes which we waited).
> On the "bad RS", we were able to observe that the \{{wal-reader}} thread(s)
> on that RS were spending time in a Thread.sleep() in a different location
> than the other. Specifically it was sitting in the
> {{ReplicationSourceWALReader#checkQuota()}}'s sleep call, _not_ the
> {{handleEmptyWALBatch()}} method on the same class.
> My only assumption is that, somehow, these RegionServers got into a situation
> where they "allocated" memory from the quota but never freed it. Then,
> because the WAL reader thinks it has no free memory, it blocks indefinitely
> and there are no pending edits to ship and (thus) free that memory. A cursory
> glance at the code gives me a _lot_ of anxiety around places where we don't
> properly clean it up (e.g. batches that fail to ship, dropping a peer). As a
> first stab, let me add some more debugging so we can actually track this
> state properly for the operators and their sanity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24757) ReplicationSink should limit the batch rowcount for batch mutations based on hbase.rpc.rows.warning.threshold
[ https://issues.apache.org/jira/browse/HBASE-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-24757: - Summary: ReplicationSink should limit the batch rowcount for batch mutations based on hbase.rpc.rows.warning.threshold (was: ReplicationSink should limit the batch size for batch mutations based on hbase.rpc.rows.warning.threshold) > ReplicationSink should limit the batch rowcount for batch mutations based on > hbase.rpc.rows.warning.threshold > - > > Key: HBASE-24757 > URL: https://issues.apache.org/jira/browse/HBASE-24757 > Project: HBase > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7 > > > At times there are quite a large no of WAL Edits to ship as part of > Replication and sometimes replication queues accumulate huge list of Edits to > process. ReplicationSink at the sink server usually goes through all Edits > and creates map of table -> list of rows grouped by clusterIds, and performs > batch mutation of all rows per table level. However, there is no limit to no > of Rows that are sent as part of batch mutate call. If no of rows > limit > threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn > "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is > turned on, RS will reject the whole batch without processing. > We should let Replication Sink honour this threshold value and accordingly > keep the size lower per batch mutation call. > Replication triggered batch mutations should always be consumed but keeping > limit of mutation low enough will let the system function at the same pace > and without triggering redundant warnings. This will also restrict > exploitation of cpu cycles at the destination server. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24779) Improve insight into replication WAL readers hung on checkQuota
[
https://issues.apache.org/jira/browse/HBASE-24779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165821#comment-17165821
]
Wellington Chevreuil commented on HBASE-24779:
--
Interesting. To confirm this suspicion, a restart of such stuck RSes had
allowed replication to run smooth again?
> Improve insight into replication WAL readers hung on checkQuota
> ---
>
> Key: HBASE-24779
> URL: https://issues.apache.org/jira/browse/HBASE-24779
> Project: HBase
> Issue Type: Task
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
>
> Helped a customer this past weekend who, with a large number of
> RegionServers, has some RegionServers which replicated data to a peer without
> issues while other RegionServers did not.
> The number of queue logs varied over the past 24hrs in the same manner. Some
> spikes in queued logs into 100's of logs, but other times, only 1's-10's of
> logs were queued.
> We were able to validate that there were "good" and "bad" RegionServers by
> creating a test table, assigning it to a regionserver, enabling replication
> on that table, and validating if the local puts were replicated to a peer. On
> a good RS, data was replicated immediately. On a bad RS, data was never
> replicated (at least, on the order of 10's of minutes which we waited).
> On the "bad RS", we were able to observe that the \{{wal-reader}} thread(s)
> on that RS were spending time in a Thread.sleep() in a different location
> than the other. Specifically it was sitting in the
> {{ReplicationSourceWALReader#checkQuota()}}'s sleep call, _not_ the
> {{handleEmptyWALBatch()}} method on the same class.
> My only assumption is that, somehow, these RegionServers got into a situation
> where they "allocated" memory from the quota but never freed it. Then,
> because the WAL reader thinks it has no free memory, it blocks indefinitely
> and there are no pending edits to ship and (thus) free that memory. A cursory
> glance at the code gives me a _lot_ of anxiety around places where we don't
> properly clean it up (e.g. batches that fail to ship, dropping a peer). As a
> first stab, let me add some more debugging so we can actually track this
> state properly for the operators and their sanity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24779) Improve insight into replication WAL readers hung on checkQuota
[
https://issues.apache.org/jira/browse/HBASE-24779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165919#comment-17165919
]
Josh Elser commented on HBASE-24779:
{quote}To confirm this suspicion, a restart of such stuck RSes had allowed
replication to run smooth again?
{quote}
Yep.
{quote}did this use existing tooling? or something we could improve based off
of e.g. the canary tool? at a minimum sounds like a good docs addition for the
troubleshooting stuff in operator-tools or the ref guide
{quote}
Manual creation of a table and `assign`'ing it to RegionServers. We were not
able to validate if "bad" / "good" regionservers actually matched the
{{checkQuota()}} stack trace as I would've hoped.
{quote}
hbase 1-ish replication, hbase 2-ish replication, or hbase master-branch-ish
replication?
{quote}
This user was actually on 2-ish, but going off of git-log, it seems like this
logic has been here for some time. Surprising because I never ran into it
before. Make me curious how the heck it even happened and suspect of my
analysis to date.
> Improve insight into replication WAL readers hung on checkQuota
> ---
>
> Key: HBASE-24779
> URL: https://issues.apache.org/jira/browse/HBASE-24779
> Project: HBase
> Issue Type: Task
> Components: Replication
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
>
> Helped a customer this past weekend who, with a large number of
> RegionServers, has some RegionServers which replicated data to a peer without
> issues while other RegionServers did not.
> The number of queue logs varied over the past 24hrs in the same manner. Some
> spikes in queued logs into 100's of logs, but other times, only 1's-10's of
> logs were queued.
> We were able to validate that there were "good" and "bad" RegionServers by
> creating a test table, assigning it to a regionserver, enabling replication
> on that table, and validating if the local puts were replicated to a peer. On
> a good RS, data was replicated immediately. On a bad RS, data was never
> replicated (at least, on the order of 10's of minutes which we waited).
> On the "bad RS", we were able to observe that the \{{wal-reader}} thread(s)
> on that RS were spending time in a Thread.sleep() in a different location
> than the other. Specifically it was sitting in the
> {{ReplicationSourceWALReader#checkQuota()}}'s sleep call, _not_ the
> {{handleEmptyWALBatch()}} method on the same class.
> My only assumption is that, somehow, these RegionServers got into a situation
> where they "allocated" memory from the quota but never freed it. Then,
> because the WAL reader thinks it has no free memory, it blocks indefinitely
> and there are no pending edits to ship and (thus) free that memory. A cursory
> glance at the code gives me a _lot_ of anxiety around places where we don't
> properly clean it up (e.g. batches that fail to ship, dropping a peer). As a
> first stab, let me add some more debugging so we can actually track this
> state properly for the operators and their sanity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24757) ReplicationSink should limit the batch size for batch mutations based on hbase.rpc.rows.warning.threshold
[ https://issues.apache.org/jira/browse/HBASE-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-24757. -- Hadoop Flags: Reviewed Resolution: Fixed > ReplicationSink should limit the batch size for batch mutations based on > hbase.rpc.rows.warning.threshold > - > > Key: HBASE-24757 > URL: https://issues.apache.org/jira/browse/HBASE-24757 > Project: HBase > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7 > > > At times there are quite a large no of WAL Edits to ship as part of > Replication and sometimes replication queues accumulate huge list of Edits to > process. ReplicationSink at the sink server usually goes through all Edits > and creates map of table -> list of rows grouped by clusterIds, and performs > batch mutation of all rows per table level. However, there is no limit to no > of Rows that are sent as part of batch mutate call. If no of rows > limit > threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn > "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is > turned on, RS will reject the whole batch without processing. > We should let Replication Sink honour this threshold value and accordingly > keep the size lower per batch mutation call. > Replication triggered batch mutations should always be consumed but keeping > limit of mutation low enough will let the system function at the same pace > and without triggering redundant warnings. This will also restrict > exploitation of cpu cycles at the destination server. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24757) ReplicationSink should limit the batch rowcount for batch mutations based on hbase.rpc.rows.warning.threshold
[
https://issues.apache.org/jira/browse/HBASE-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166015#comment-17166015
]
Hudson commented on HBASE-24757:
Results for branch branch-2.2
[build #919 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/919/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/919//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/919//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/919//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ReplicationSink should limit the batch rowcount for batch mutations based on
> hbase.rpc.rows.warning.threshold
> -
>
> Key: HBASE-24757
> URL: https://issues.apache.org/jira/browse/HBASE-24757
> Project: HBase
> Issue Type: Improvement
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7
>
>
> At times there are quite a large no of WAL Edits to ship as part of
> Replication and sometimes replication queues accumulate huge list of Edits to
> process. ReplicationSink at the sink server usually goes through all Edits
> and creates map of table -> list of rows grouped by clusterIds, and performs
> batch mutation of all rows per table level. However, there is no limit to no
> of Rows that are sent as part of batch mutate call. If no of rows > limit
> threshold defined by hbase.rpc.rows.warning.threshold, we usually get warn
> "Large batch operation detected". If hbase.rpc.rows.size.threshold.reject is
> turned on, RS will reject the whole batch without processing.
> We should let Replication Sink honour this threshold value and accordingly
> keep the size lower per batch mutation call.
> Replication triggered batch mutations should always be consumed but keeping
> limit of mutation low enough will let the system function at the same pace
> and without triggering redundant warnings. This will also restrict
> exploitation of cpu cycles at the destination server.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24775) [hbtop] StoreFile size should be rounded off
[ https://issues.apache.org/jira/browse/HBASE-24775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166026#comment-17166026 ] Toshihiro Suzuki commented on HBASE-24775: -- Pushed to master, branch-2, branch-2.3, branch-2.2 and branch-1. Thank you for reviewing this! [~zhangduo] [~vjasani] > [hbtop] StoreFile size should be rounded off > > > Key: HBASE-24775 > URL: https://issues.apache.org/jira/browse/HBASE-24775 > Project: HBase > Issue Type: Bug > Components: hbtop >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.2.7 > > Attachments: Screen Shot.png, Screen Shot2.png > > > Currently, hbtop doesn't show StoreFile size rounded off, so when it's > reached 1GB it will be very ugly as follows: > !Screen Shot.png! > We should round off StoreFile size. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24775) [hbtop] StoreFile size should be rounded off
[ https://issues.apache.org/jira/browse/HBASE-24775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166026#comment-17166026 ] Toshihiro Suzuki edited comment on HBASE-24775 at 7/27/20, 11:37 PM: - Pushed to master, branch-2, branch-2.3, branch-2.2 and branch-1. Thank you for reviewing this! [~zhangduo] [~vjasani] was (Author: brfrn169): Pushed to master, branch-2, branch-2.3, branch-2.2 and branch-1. Thank you for reviewing this! [~zhangduo] [~vjasani] > [hbtop] StoreFile size should be rounded off > > > Key: HBASE-24775 > URL: https://issues.apache.org/jira/browse/HBASE-24775 > Project: HBase > Issue Type: Bug > Components: hbtop >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.2.7 > > Attachments: Screen Shot.png, Screen Shot2.png > > > Currently, hbtop doesn't show StoreFile size rounded off, so when it's > reached 1GB it will be very ugly as follows: > !Screen Shot.png! > We should round off StoreFile size. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24775) [hbtop] StoreFile size should be rounded off
[ https://issues.apache.org/jira/browse/HBASE-24775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki resolved HBASE-24775. -- Hadoop Flags: Reviewed Resolution: Fixed > [hbtop] StoreFile size should be rounded off > > > Key: HBASE-24775 > URL: https://issues.apache.org/jira/browse/HBASE-24775 > Project: HBase > Issue Type: Bug > Components: hbtop >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.2.7 > > Attachments: Screen Shot.png, Screen Shot2.png > > > Currently, hbtop doesn't show StoreFile size rounded off, so when it's > reached 1GB it will be very ugly as follows: > !Screen Shot.png! > We should round off StoreFile size. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166030#comment-17166030
]
Bharath Vissapragada commented on HBASE-23887:
--
Hey Danil, sorry for the delay on this. Been busy with things at work and
haven't had a chance to focus on this lately. Will get back by end of this week
or if others are free meanwhile, feel free to take it up.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-11676) Scan FORMATTER is not applied for columns using non-printable name in shell
[ https://issues.apache.org/jira/browse/HBASE-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliot Miller reassigned HBASE-11676: - Assignee: Elliot Miller > Scan FORMATTER is not applied for columns using non-printable name in shell > > > Key: HBASE-11676 > URL: https://issues.apache.org/jira/browse/HBASE-11676 > Project: HBase > Issue Type: Bug > Components: shell > Environment: 0.98 + hadoop2.2.0 >Reporter: t samkawa >Assignee: Elliot Miller >Priority: Minor > > The "FORMATTER" does not work if the target cell`s column uses binary name. > hbase> create "test1", "f" > hbase> incr "test1", "row1" , "f:a", 1 > hbase> incr "test1", "row1" , "f:\x11", 1 > hbase> scan "test1", COLUMNS=>["f:\x11:toLong","f:a:toLong"] > ROW COLUMN+CELL > row1column=f:\x11, ..., value=\x00\x00\x00\x00\x00\x00\x00\x01 > row1column=f:a, ..., value=1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165945#comment-17165945
]
Michael Stack commented on HBASE-15519:
---
So, yeah, the per-user I think needs working on. For example, here is what I
see for connections... i.e. all from one 'user', the APP_USER, all lumped
together, though from different IPs:
{code:java}
2020-07-27 17:21:49,576 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.189.210:33190, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:49,615 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.184.233:37032, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:49,992 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.188.92:42026, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:50,136 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.178.154:40568, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:50,894 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.185.40:59874, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:51,082 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.188.86:46592, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:51,470 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.181.25:56220, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:51,478 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.184.211:44888, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:51,644 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.185.34:45848, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:51,954 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.174.87:46344, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:52,342 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.191.23:40736, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:52,461 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.181.150:39626, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:52,494 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.186.83:44636, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:52,663 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.181.211:40950, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:52,996 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.184.147:37758, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService {code}
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24758) Avoid flooding replication source RSes logs when no sinks are available
[
https://issues.apache.org/jira/browse/HBASE-24758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165968#comment-17165968
]
Hudson commented on HBASE-24758:
Results for branch branch-2.3
[build #191 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/191/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/191/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/191/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/191/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/191/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Avoid flooding replication source RSes logs when no sinks are available
>
>
> Key: HBASE-24758
> URL: https://issues.apache.org/jira/browse/HBASE-24758
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.5
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> On HBaseInterClusterReplicationEndpoint.replicate, if no sinks are returned
> by ReplicationSinkManager, say remote peer is not available, we log message
> below and return false to source shipper thread, which then keeps retrying,
> flooding source RS log with the below messages:
> {noformat}
> WARN
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint:
> No replication sinks found, returning without replicating. The source should
> retry with the same set of edits.
> {noformat}
> This condition could also cause ReplicationSinkManager.chooseSinks to blow an
> NPE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24777) InfoServer support ipv6 host and port
[
https://issues.apache.org/jira/browse/HBASE-24777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166016#comment-17166016
]
Hudson commented on HBASE-24777:
Results for branch branch-2.2
[build #919 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/919/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/919//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/919//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/919//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> InfoServer support ipv6 host and port
> -
>
> Key: HBASE-24777
> URL: https://issues.apache.org/jira/browse/HBASE-24777
> Project: HBase
> Issue Type: Improvement
> Components: UI
>Affects Versions: 2.3.0, 2.1.9, 2.2.5
>Reporter: Yechao Chen
>Assignee: Yechao Chen
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7
>
>
> ipv6 host and port is
> [ a:::b:c:d]:16010
> ipv4 host and port is
> a.b.c.d:16010
> we can use HostAndPort to handle it
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reopened HBASE-15519: - > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24775) [hbtop] StoreFile size should be rounded off
[ https://issues.apache.org/jira/browse/HBASE-24775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki updated HBASE-24775: - Fix Version/s: 2.2.7 1.7.0 2.3.1 3.0.0-alpha-1 > [hbtop] StoreFile size should be rounded off > > > Key: HBASE-24775 > URL: https://issues.apache.org/jira/browse/HBASE-24775 > Project: HBase > Issue Type: Bug > Components: hbtop >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.2.7 > > Attachments: Screen Shot.png, Screen Shot2.png > > > Currently, hbtop doesn't show StoreFile size rounded off, so when it's > reached 1GB it will be very ugly as follows: > !Screen Shot.png! > We should round off StoreFile size. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24753) HA masters based on raft
[ https://issues.apache.org/jira/browse/HBASE-24753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166058#comment-17166058 ] Duo Zhang commented on HBASE-24753: --- Update: I was trying to make use of the sofa-jraft, but it depends on protobuf 3.x, so I tried to set hadoop.version to 3.3.0, as hadoop 3.3.0 has shaded protobuf, so we can purge all the protobuf 2.5 dependencies. But then I noticed that there is a problem for support hadoop 3.3.0 because of the conflict on jetty version. So I'm currently working on shade jetty to solve the problem first. > HA masters based on raft > > > Key: HBASE-24753 > URL: https://issues.apache.org/jira/browse/HBASE-24753 > Project: HBase > Issue Type: New Feature > Components: master >Reporter: Duo Zhang >Priority: Major > > For better availability, for moving bootstrap information from zookeeper to > our own service so finally we could remove the dependency on zookeeper > completely. > This has been in my mind for a long time, and since the there is a dicussion > in HBASE-11288 about how to storing root table, and also in HBASE-24749, we > want to have better performance on a filesystem can not support list and > rename well, where requires a storage engine at the bottom to store the > storefiles information for meta table, I think it is the time to throw this > idea out. > The basic solution is to build a raft group to store the bootstrap > information, for now it is cluster id(it is on the file system already?) and > the root table. For region servers they will always go to the leader to ask > for the information so they can always see the newest data, and for client, > we enable 'follower read', to reduce the load of the leader(and there are > some solutions to even let 'follower read' to always get the newest data in > raft). > With this solution in place, as long as root table will not be in a format of > region(we could just use rocksdb to store it locally), the cyclic dependency > in HBASE-24749 has also been solved, as we do not need to find a place to > store the storefiles information for root table any more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20226) Performance Improvement Taking Large Snapshots In Remote Filesystems
[ https://issues.apache.org/jira/browse/HBASE-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166068#comment-17166068 ] Bharath Vissapragada commented on HBASE-20226: -- bq. This is related to the work going on in HBASE-21098. It's likely that this Jira is no longer needed after that work is concluded. [~zyork] Why is this no longer needed? I think the attached patch parallelizes deletes by adding them to a thread pool while the parent jira (HBASE-21098) doesn't do it right? We are running into bottlenecks on this sequential delete and I saw this jira. Did I miss something? > Performance Improvement Taking Large Snapshots In Remote Filesystems > > > Key: HBASE-20226 > URL: https://issues.apache.org/jira/browse/HBASE-20226 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 1.4.0 > Environment: HBase 1.4.0 running on an AWS EMR cluster with the > hbase.rootdir set to point to a folder in S3 >Reporter: Saad Mufti >Priority: Minor > Attachments: HBASE-20226..01.patch > > > When taking a snapshot of any table, one of the last steps is to delete the > region manifests, which have already been rolled up into a larger overall > manifest and thus have redundant information. > This proposal is to do the deletion in a thread pool bounded by > hbase.snapshot.thread.pool.max . For large tables with a lot of regions, the > current single threaded deletion is taking longer than all the rest of the > snapshot tasks when the Hbase data and the snapshot folder are both in a > remote filesystem like S3. > I have a patch for this proposal almost ready and will submit it tomorrow for > feedback, although I haven't had a chance to write any tests yet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24665) MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
[
https://issues.apache.org/jira/browse/HBASE-24665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166035#comment-17166035
]
Hudson commented on HBASE-24665:
Results for branch branch-2
[build #2759 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2759/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2759/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2756/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2756/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2756/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
> --
>
> Key: HBASE-24665
> URL: https://issues.apache.org/jira/browse/HBASE-24665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.3.0, master, 2.1.10, 1.4.14, 2.2.6
>Reporter: wenfeiyi666
>Assignee: wenfeiyi666
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7
>
>
> when use multiwal, any a wal request roll, all wal will be together roll.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24758) Avoid flooding replication source RSes logs when no sinks are available
[
https://issues.apache.org/jira/browse/HBASE-24758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166036#comment-17166036
]
Hudson commented on HBASE-24758:
Results for branch branch-2
[build #2759 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2759/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2759/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2756/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2756/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2756/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Avoid flooding replication source RSes logs when no sinks are available
>
>
> Key: HBASE-24758
> URL: https://issues.apache.org/jira/browse/HBASE-24758
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.5
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> On HBaseInterClusterReplicationEndpoint.replicate, if no sinks are returned
> by ReplicationSinkManager, say remote peer is not available, we log message
> below and return false to source shipper thread, which then keeps retrying,
> flooding source RS log with the below messages:
> {noformat}
> WARN
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint:
> No replication sinks found, returning without replicating. The source should
> retry with the same set of edits.
> {noformat}
> This condition could also cause ReplicationSinkManager.chooseSinks to blow an
> NPE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24780) Index rebuilds should record scan timestamp to job counter
Geoffrey Jacoby created HBASE-24780: --- Summary: Index rebuilds should record scan timestamp to job counter Key: HBASE-24780 URL: https://issues.apache.org/jira/browse/HBASE-24780 Project: HBase Issue Type: Improvement Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby The index tool output tables (PHOENIX_INDEX_TOOL and PHOENIX_INDEX_TOOL_RESULT) are both keyed on the effective scan time of the index rebuild. This makes the table quite difficult to query if you know the index you're interested in but not the exact millisecond it was last rebuilt. Short term: record this timestamp in the Phoenix index rebuild job counters (this JIRA) Longer term: we might want to consider reversing the PK to both cut down on hotspotting and make the table easier to query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24780) Index rebuilds should record scan timestamp to job counter
[ https://issues.apache.org/jira/browse/HBASE-24780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby resolved HBASE-24780. - Resolution: Invalid (sigh) Wrong project, apologies for the noise. > Index rebuilds should record scan timestamp to job counter > -- > > Key: HBASE-24780 > URL: https://issues.apache.org/jira/browse/HBASE-24780 > Project: HBase > Issue Type: Improvement >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > > The index tool output tables (PHOENIX_INDEX_TOOL and > PHOENIX_INDEX_TOOL_RESULT) are both keyed on the effective scan time of the > index rebuild. This makes the table quite difficult to query if you know the > index you're interested in but not the exact millisecond it was last rebuilt. > Short term: record this timestamp in the Phoenix index rebuild job counters > (this JIRA) > Longer term: we might want to consider reversing the PK to both cut down on > hotspotting and make the table easier to query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24770) Reimplement the Constraints API and revisit the IA annotations on related classes
[ https://issues.apache.org/jira/browse/HBASE-24770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-24770. --- Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) Resolution: Fixed Merged to master. Thanks [~stack] for reviewing. > Reimplement the Constraints API and revisit the IA annotations on related > classes > - > > Key: HBASE-24770 > URL: https://issues.apache.org/jira/browse/HBASE-24770 > Project: HBase > Issue Type: Task > Components: Client, Coprocessors >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20226) Performance Improvement Taking Large Snapshots In Remote Filesystems
[
https://issues.apache.org/jira/browse/HBASE-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166082#comment-17166082
]
Ted Yu commented on HBASE-20226:
{code}
+if (v1Regions.size() > 0 || v2Regions.size() > 0) {
{code}
It seems the thread pool is needed when v1Regions.size()+v2Regions.size() > 1.
There are also a few findbugs warnings to be addressed.
> Performance Improvement Taking Large Snapshots In Remote Filesystems
>
>
> Key: HBASE-20226
> URL: https://issues.apache.org/jira/browse/HBASE-20226
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 1.4.0
> Environment: HBase 1.4.0 running on an AWS EMR cluster with the
> hbase.rootdir set to point to a folder in S3
>Reporter: Saad Mufti
>Priority: Minor
> Attachments: HBASE-20226..01.patch
>
>
> When taking a snapshot of any table, one of the last steps is to delete the
> region manifests, which have already been rolled up into a larger overall
> manifest and thus have redundant information.
> This proposal is to do the deletion in a thread pool bounded by
> hbase.snapshot.thread.pool.max . For large tables with a lot of regions, the
> current single threaded deletion is taking longer than all the rest of the
> snapshot tasks when the Hbase data and the snapshot folder are both in a
> remote filesystem like S3.
> I have a patch for this proposal almost ready and will submit it tomorrow for
> feedback, although I haven't had a chance to write any tests yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22620) When a cluster open replication,regionserver will not clean up the walLog references on zk due to no wal entry need to be replicated
[
https://issues.apache.org/jira/browse/HBASE-22620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166100#comment-17166100
]
leizhang commented on HBASE-22620:
--
I think this problem still exist in Hbase2.x , when i use Hbase2.2.5 , I
encounter the same problem .
> When a cluster open replication,regionserver will not clean up the walLog
> references on zk due to no wal entry need to be replicated
>
>
> Key: HBASE-22620
> URL: https://issues.apache.org/jira/browse/HBASE-22620
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.1.0, 1.4.8, 1.4.9, 2.2.5
>Reporter: leizhang
>Assignee: yaojingyi
>Priority: Major
> Attachments: HBASE-22620.branch-1.4.001.patch
>
>
> When I open the replication feature on my hbase cluster (20 regionserver
> nodes) and added a peer cluster, for example, I create a table with 3 regions
> with REPLICATION_SCOPE set to 1, which opened on 3 regionservers of 20. Due
> to no data（entryBatch） to replicate ,the left 17 nodes accumulate lots of
> wal references on the zk node
> "/hbase/replication/rs/\{resionserver}/\{peerId}/" and will not be cleaned
> up, which cause lots of wal file on hdfs will not be cleaned up either. When
> I check my test cluster after about four months, it accumulates about 5w wal
> files in the oldWal directory on hdfs. The source code shows that only there
> are data to be replicated, and after some data is replicated in the source
> endpoint, then it will executed the useless wal file check, and clean their
> references on zk, and the hdfs useless wal files will be cleaned up normally.
> So I think do we need other method to trigger the useless wal cleaning job in
> a replication cluster? May be in the replication progress report schedule
> task (just like ReplicationStatisticsTask.class)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22620) When a cluster open replication,regionserver will not clean up the walLog references on zk due to no wal entry need to be replicated
[
https://issues.apache.org/jira/browse/HBASE-22620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
leizhang updated HBASE-22620:
-
Affects Version/s: (was: 1.2.4)
2.2.5
> When a cluster open replication,regionserver will not clean up the walLog
> references on zk due to no wal entry need to be replicated
>
>
> Key: HBASE-22620
> URL: https://issues.apache.org/jira/browse/HBASE-22620
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.1.0, 1.4.8, 1.4.9, 2.2.5
>Reporter: leizhang
>Assignee: yaojingyi
>Priority: Major
> Attachments: HBASE-22620.branch-1.4.001.patch
>
>
> When I open the replication feature on my hbase cluster (20 regionserver
> nodes) and added a peer cluster, for example, I create a table with 3 regions
> with REPLICATION_SCOPE set to 1, which opened on 3 regionservers of 20. Due
> to no data（entryBatch） to replicate ,the left 17 nodes accumulate lots of
> wal references on the zk node
> "/hbase/replication/rs/\{resionserver}/\{peerId}/" and will not be cleaned
> up, which cause lots of wal file on hdfs will not be cleaned up either. When
> I check my test cluster after about four months, it accumulates about 5w wal
> files in the oldWal directory on hdfs. The source code shows that only there
> are data to be replicated, and after some data is replicated in the source
> endpoint, then it will executed the useless wal file check, and clean their
> references on zk, and the hdfs useless wal files will be cleaned up normally.
> So I think do we need other method to trigger the useless wal cleaning job in
> a replication cluster? May be in the replication progress report schedule
> task (just like ReplicationStatisticsTask.class)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-11686) Shell code should create a binding / irb workspace instead of polluting the root namespace
[
https://issues.apache.org/jira/browse/HBASE-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-11686.
---
Fix Version/s: 3.0.0-alpha-1
Hadoop Flags: Reviewed
Resolution: Fixed
Resolving.
Mind adding a release note [~bitoffdev] ? Thanks.
> Shell code should create a binding / irb workspace instead of polluting the
> root namespace
> --
>
> Key: HBASE-11686
> URL: https://issues.apache.org/jira/browse/HBASE-11686
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Sean Busbey
>Assignee: Elliot Miller
>Priority: Minor
> Fix For: 3.0.0-alpha-1
>
>
> Right now, the shell builds a list of commands and then injects them into the
> root exectution's context
> bin/hirb.rb
> {code}
> # Add commands to this namespace
> @shell.export_commands(self)
> {code}
> hbase-shell/src/main/ruby/shell.rb
> {code}
> def export_commands(where)
> ::Shell.commands.keys.each do |cmd|
> # here where is the IRB namespace
> # this method just adds the call to the specified command
> # which just references back to 'this' shell object
> # a decently extensible way to add commands
> where.send :instance_eval, <<-EOF
> def #{cmd}(*args)
> ret = @shell.command('#{cmd}', *args)
> puts
> return ret
> end
> EOF
> end
> end
> {code}
> This is an unclean abstraction. For one, it requires that there be an
> instance variable in the main namespace called '@shell' without making that
> clear in the docs. Additionally, it complicates maintenance by breaking
> isolation.
> We should update things so that shell can provide a binding for eval or a
> workspace for IRB execution and then use it directly when we construct our
> IRB session.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24753) HA masters based on raft
[ https://issues.apache.org/jira/browse/HBASE-24753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166154#comment-17166154 ] Anoop Sam John commented on HBASE-24753: bq.With this solution in place, as long as root table will not be in a format of region(we could just use rocksdb to store it locally), There is one interesting usecase by Cloud to drop a cluster and recreate it later on existing data. This was/is possible because we never store any persisting data locally but always on FS. I would say lets not break that. I read in another jira also says that the Root table data can be stored locally (RAFT will be in place) not on FS. I would say lets not do that. Let us continue to have the storage isolation. > HA masters based on raft > > > Key: HBASE-24753 > URL: https://issues.apache.org/jira/browse/HBASE-24753 > Project: HBase > Issue Type: New Feature > Components: master >Reporter: Duo Zhang >Priority: Major > > For better availability, for moving bootstrap information from zookeeper to > our own service so finally we could remove the dependency on zookeeper > completely. > This has been in my mind for a long time, and since the there is a dicussion > in HBASE-11288 about how to storing root table, and also in HBASE-24749, we > want to have better performance on a filesystem can not support list and > rename well, where requires a storage engine at the bottom to store the > storefiles information for meta table, I think it is the time to throw this > idea out. > The basic solution is to build a raft group to store the bootstrap > information, for now it is cluster id(it is on the file system already?) and > the root table. For region servers they will always go to the leader to ask > for the information so they can always see the newest data, and for client, > we enable 'follower read', to reduce the load of the leader(and there are > some solutions to even let 'follower read' to always get the newest data in > raft). > With this solution in place, as long as root table will not be in a format of > region(we could just use rocksdb to store it locally), the cyclic dependency > in HBASE-24749 has also been solved, as we do not need to find a place to > store the storefiles information for root table any more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11686) Shell code should create a binding / irb workspace instead of polluting the root namespace
[
https://issues.apache.org/jira/browse/HBASE-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166133#comment-17166133
]
Michael Stack commented on HBASE-11686:
---
Merged on master. If you make a PR I'll merge it to branch-2 [~bitoffdev]
(Didn't go back easy). If backport, do it as sub-issue. I'll close out this
one. Thanks for the patch.
> Shell code should create a binding / irb workspace instead of polluting the
> root namespace
> --
>
> Key: HBASE-11686
> URL: https://issues.apache.org/jira/browse/HBASE-11686
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Sean Busbey
>Assignee: Elliot Miller
>Priority: Minor
>
> Right now, the shell builds a list of commands and then injects them into the
> root exectution's context
> bin/hirb.rb
> {code}
> # Add commands to this namespace
> @shell.export_commands(self)
> {code}
> hbase-shell/src/main/ruby/shell.rb
> {code}
> def export_commands(where)
> ::Shell.commands.keys.each do |cmd|
> # here where is the IRB namespace
> # this method just adds the call to the specified command
> # which just references back to 'this' shell object
> # a decently extensible way to add commands
> where.send :instance_eval, <<-EOF
> def #{cmd}(*args)
> ret = @shell.command('#{cmd}', *args)
> puts
> return ret
> end
> EOF
> end
> end
> {code}
> This is an unclean abstraction. For one, it requires that there be an
> instance variable in the main namespace called '@shell' without making that
> clear in the docs. Additionally, it complicates maintenance by breaking
> isolation.
> We should update things so that shell can provide a binding for eval or a
> workspace for IRB execution and then use it directly when we construct our
> IRB session.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
