[GitHub] [hbase] Apache-HBase commented on pull request #1987: HBASE-24646 Set the log level for ScheduledChore to INFO in HBTU
Apache-HBase commented on pull request #1987: URL: https://github.com/apache/hbase/pull/1987#issuecomment-650493467 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | checkstyle | 1m 14s | master passed | | +1 :green_heart: | spotbugs | 2m 7s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | checkstyle | 1m 15s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 8s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 34m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1987/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1987 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 0301b4b102fc 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / af49b87367 | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1987/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1774: HBASE-24389 Introduce new master rpc methods to locate meta region through root region
Apache-HBase commented on pull request #1774: URL: https://github.com/apache/hbase/pull/1774#issuecomment-650492503 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 2s | No case conflicting files found. | | +0 :ok: | prototool | 0m 2s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-11288.splittable-meta Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 21s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | checkstyle | 2m 16s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | spotbugs | 6m 31s | HBASE-11288.splittable-meta passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-protocol-shaded | | +1 :green_heart: | checkstyle | 0m 32s | hbase-client: The patch generated 0 new + 182 unchanged - 1 fixed = 182 total (was 183) | | +1 :green_heart: | checkstyle | 0m 13s | The patch passed checkstyle in hbase-zookeeper | | +1 :green_heart: | checkstyle | 1m 16s | hbase-server: The patch generated 0 new + 443 unchanged - 5 fixed = 443 total (was 448) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 2m 20s | the patch passed | | +1 :green_heart: | spotbugs | 7m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | | | | 48m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/20/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1774 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool | | uname | Linux 43c70f6b9c4c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-11288.splittable-meta / f3d0b535b8 | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/20/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on pull request #1883: HBASE-24530 Introduce a split policy similar with SteppingSplitPolicy…
bsglz commented on pull request #1883: URL: https://github.com/apache/hbase/pull/1883#issuecomment-650492489 @wchevreuil Thought it again, if this new splitPolicy is useable in branch-2.2 and branch-2.3, but disappear in branch-2.4 without deprecated cycle, is it break the compatibility guarantee? IMO, maybe we can do as below: 1: In all actived branchs, add this new splitPolicy. 2: In minor branch set it as default policy and mark SteppingSpliyPolicy and IncreasingToUpperBoundRegionSplitPolicy as deprecated, at the same time fix the ConstantSizeRegionSplitPolicy. What do you think? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 opened a new pull request #1987: HBASE-24646 Set the log level for ScheduledChore to INFO in HBTU
Apache9 opened a new pull request #1987: URL: https://github.com/apache/hbase/pull/1987 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24635) Split TestMetaWithReplicas
[ https://issues.apache.org/jira/browse/HBASE-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146755#comment-17146755 ] Duo Zhang commented on HBASE-24635: --- Need to wait for HBASE-24562 to see if we want to port to branch-2.2. > Split TestMetaWithReplicas > -- > > Key: HBASE-24635 > URL: https://issues.apache.org/jira/browse/HBASE-24635 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0 > > > It will stop and then start a mini cluster every time after each test method, > so let's just split them into individual test files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24635) Split TestMetaWithReplicas
[ https://issues.apache.org/jira/browse/HBASE-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-24635: -- Fix Version/s: 2.4.0 2.3.1 3.0.0-alpha-1 > Split TestMetaWithReplicas > -- > > Key: HBASE-24635 > URL: https://issues.apache.org/jira/browse/HBASE-24635 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0 > > > It will stop and then start a mini cluster every time after each test method, > so let's just split them into individual test files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] infraio commented on a change in pull request #1730: HBASE-24289 Heterogeneous Storage for Date Tiered Compaction
infraio commented on a change in pull request #1730:
URL: https://github.com/apache/hbase/pull/1730#discussion_r446475799
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDateTieredCompactionPolicyHeterogeneousStorage.java
##
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HConstants;
+import
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import
org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.testclassification.SmallTests;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({ RegionServerTests.class, SmallTests.class })
+public class TestDateTieredCompactionPolicyHeterogeneousStorage
+extends AbstractTestDateTieredCompactionPolicy {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+
HBaseClassTestRule.forClass(TestDateTieredCompactionPolicyHeterogeneousStorage.class);
+ public static final String HOT_WINDOW_SP = "ALL_SSD";
+ public static final String WARM_WINDOW_SP = "ONE_SSD";
+ public static final String COLD_WINDOW_SP = "HOT";
+
+ @Override
+ protected void config() {
+super.config();
+
+// Set up policy
+conf.set(StoreEngine.STORE_ENGINE_CLASS_KEY,
+ "org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine");
+conf.setLong(CompactionConfiguration.DATE_TIERED_MAX_AGE_MILLIS_KEY, 100);
+conf.setLong(CompactionConfiguration.DATE_TIERED_INCOMING_WINDOW_MIN_KEY,
3);
+conf.setLong(ExponentialCompactionWindowFactory.BASE_WINDOW_MILLIS_KEY, 6);
+conf.setInt(ExponentialCompactionWindowFactory.WINDOWS_PER_TIER_KEY, 4);
+
conf.setBoolean(CompactionConfiguration.DATE_TIERED_SINGLE_OUTPUT_FOR_MINOR_COMPACTION_KEY,
+ false);
+
+// Special settings for compaction policy per window
+this.conf.setInt(CompactionConfiguration.HBASE_HSTORE_COMPACTION_MIN_KEY,
2);
+this.conf.setInt(CompactionConfiguration.HBASE_HSTORE_COMPACTION_MAX_KEY,
12);
+
this.conf.setFloat(CompactionConfiguration.HBASE_HSTORE_COMPACTION_RATIO_KEY,
1.2F);
+
+conf.setInt(HStore.BLOCKING_STOREFILES_KEY, 20);
+conf.setLong(HConstants.MAJOR_COMPACTION_PERIOD, 5);
+
+// Set Storage Policy for different type window
+
conf.setBoolean(CompactionConfiguration.DATE_TIERED_STORAGE_POLICY_ENABLE_KEY,
true);
+
conf.setLong(CompactionConfiguration.DATE_TIERED_HOT_WINDOW_AGE_MILLIS_KEY, 6);
+
conf.set(CompactionConfiguration.DATE_TIERED_HOT_WINDOW_STORAGE_POLICY_KEY,
HOT_WINDOW_SP);
+
conf.setLong(CompactionConfiguration.DATE_TIERED_WARM_WINDOW_AGE_MILLIS_KEY,
12);
+
conf.set(CompactionConfiguration.DATE_TIERED_WARM_WINDOW_STORAGE_POLICY_KEY,
WARM_WINDOW_SP);
+
conf.set(CompactionConfiguration.DATE_TIERED_COLD_WINDOW_STORAGE_POLICY_KEY,
COLD_WINDOW_SP);
+ }
+
+ /**
+ * Test for incoming window and is HOT window
+ * window start >= now - hot age
+ * @throws IOException with error
+ */
+ @Test
+ public void incomingWindowHot() throws IOException {
+long[] minTimestamps = new long[] { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0 };
+long[] maxTimestamps = new long[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15 };
+long[] sizes = new long[] { 30, 31, 32, 33, 34, 20, 21, 22, 23, 24, 25,
10, 11, 12, 13 };
+Map expected = new HashMap<>();
+// boundaries = { Long.MIN_VALUE, 12 }
+expected.put(12L, HOT_WINDOW_SP);
Review comment:
Explain more about the unit test? What is the different between this and
other tests? And there are some duplicate code in these test methods?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries
[GitHub] [hbase] infraio commented on a change in pull request #1730: HBASE-24289 Heterogeneous Storage for Date Tiered Compaction
infraio commented on a change in pull request #1730:
URL: https://github.com/apache/hbase/pull/1730#discussion_r446475459
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -1564,6 +1568,18 @@ public void
deleteChangedReaderObserver(ChangedReadersObserver o) {
return sfs;
}
+ // Set correct storage policy from the file name of DTCP.
+ // Rename file will not change the storage policy.
+ private void setStoragePolicyFromFileName(List newFiles) throws
IOException {
+String prefix = HConstants.STORAGE_POLICY_PREFIX;
+for (Path newFile : newFiles) {
+ if (newFile.getParent().getName().startsWith(prefix)) {
+CommonFSUtils.setStoragePolicy(fs.getFileSystem(), newFile,
Review comment:
Set storage policy for a file? This is not work before hdfs 3.3.0?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #1730: HBASE-24289 Heterogeneous Storage for Date Tiered Compaction
infraio commented on a change in pull request #1730:
URL: https://github.com/apache/hbase/pull/1730#discussion_r446475247
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDateTieredCompactionPolicyHeterogeneousStorage.java
##
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HConstants;
+import
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import
org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.testclassification.SmallTests;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+@Category({ RegionServerTests.class, SmallTests.class })
+public class TestDateTieredCompactionPolicyHeterogeneousStorage
+extends AbstractTestDateTieredCompactionPolicy {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+
HBaseClassTestRule.forClass(TestDateTieredCompactionPolicyHeterogeneousStorage.class);
+ public static final String HOT_WINDOW_SP = "ALL_SSD";
+ public static final String WARM_WINDOW_SP = "ONE_SSD";
+ public static final String COLD_WINDOW_SP = "HOT";
+
+ @Override
+ protected void config() {
+super.config();
+
+// Set up policy
+conf.set(StoreEngine.STORE_ENGINE_CLASS_KEY,
+ "org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine");
+conf.setLong(CompactionConfiguration.DATE_TIERED_MAX_AGE_MILLIS_KEY, 100);
+conf.setLong(CompactionConfiguration.DATE_TIERED_INCOMING_WINDOW_MIN_KEY,
3);
+conf.setLong(ExponentialCompactionWindowFactory.BASE_WINDOW_MILLIS_KEY, 6);
+conf.setInt(ExponentialCompactionWindowFactory.WINDOWS_PER_TIER_KEY, 4);
+
conf.setBoolean(CompactionConfiguration.DATE_TIERED_SINGLE_OUTPUT_FOR_MINOR_COMPACTION_KEY,
+ false);
+
+// Special settings for compaction policy per window
+this.conf.setInt(CompactionConfiguration.HBASE_HSTORE_COMPACTION_MIN_KEY,
2);
+this.conf.setInt(CompactionConfiguration.HBASE_HSTORE_COMPACTION_MAX_KEY,
12);
+
this.conf.setFloat(CompactionConfiguration.HBASE_HSTORE_COMPACTION_RATIO_KEY,
1.2F);
+
+conf.setInt(HStore.BLOCKING_STOREFILES_KEY, 20);
+conf.setLong(HConstants.MAJOR_COMPACTION_PERIOD, 5);
+
+// Set Storage Policy for different type window
+
conf.setBoolean(CompactionConfiguration.DATE_TIERED_STORAGE_POLICY_ENABLE_KEY,
true);
+
conf.setLong(CompactionConfiguration.DATE_TIERED_HOT_WINDOW_AGE_MILLIS_KEY, 6);
+
conf.set(CompactionConfiguration.DATE_TIERED_HOT_WINDOW_STORAGE_POLICY_KEY,
HOT_WINDOW_SP);
+
conf.setLong(CompactionConfiguration.DATE_TIERED_WARM_WINDOW_AGE_MILLIS_KEY,
12);
+
conf.set(CompactionConfiguration.DATE_TIERED_WARM_WINDOW_STORAGE_POLICY_KEY,
WARM_WINDOW_SP);
+
conf.set(CompactionConfiguration.DATE_TIERED_COLD_WINDOW_STORAGE_POLICY_KEY,
COLD_WINDOW_SP);
+ }
+
+ /**
+ * Test for incoming window and is HOT window
+ * window start >= now - hot age
+ * @throws IOException with error
+ */
+ @Test
+ public void incomingWindowHot() throws IOException {
Review comment:
Method name start with "test"? incomingWindowHot => testIncomingWindowHot
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #1730: HBASE-24289 Heterogeneous Storage for Date Tiered Compaction
infraio commented on a change in pull request #1730:
URL: https://github.com/apache/hbase/pull/1730#discussion_r446474765
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DateTieredStoreEngine.java
##
@@ -94,6 +94,7 @@ public void forceSelect(CompactionRequestImpl request) {
throws IOException {
if (request instanceof DateTieredCompactionRequest) {
return compactor.compact(request, ((DateTieredCompactionRequest)
request).getBoundaries(),
+ ((DateTieredCompactionRequest) request).getBoundariesPolicies(),
Review comment:
Use a local variable and no need to cast twice...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #1980: HBASE-24635 Split TestMetaWithReplicas
Apache9 merged pull request #1980: URL: https://github.com/apache/hbase/pull/1980 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-24646) Set the log level for ScheduledChore to INFO in HBTU
[
https://issues.apache.org/jira/browse/HBASE-24646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang reassigned HBASE-24646:
-
Assignee: Duo Zhang
> Set the log level for ScheduledChore to INFO in HBTU
>
>
> Key: HBASE-24646
> URL: https://issues.apache.org/jira/browse/HBASE-24646
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Now we changed the thead waken interval to 100ms in tests, which causes the
> test log beng flooded with
> {noformat}
> 2020-06-27 10:28:00,680 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
> 2020-06-27 10:28:00,680 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
> 2020-06-27 10:28:00,684 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
> 2020-06-27 10:28:00,684 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
> 2020-06-27 10:28:00,696 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
> 2020-06-27 10:28:00,696 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
> 2020-06-27 10:28:00,782 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
> 2020-06-27 10:28:00,782 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
> 2020-06-27 10:28:00,786 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
> {noformat}
> We could set the log level to INFO when starting a mini cluster in HBTU.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-24646) Set the log level for ScheduledChore to INFO in HBTU
[
https://issues.apache.org/jira/browse/HBASE-24646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-24646 started by Duo Zhang.
-
> Set the log level for ScheduledChore to INFO in HBTU
>
>
> Key: HBASE-24646
> URL: https://issues.apache.org/jira/browse/HBASE-24646
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Now we changed the thead waken interval to 100ms in tests, which causes the
> test log beng flooded with
> {noformat}
> 2020-06-27 10:28:00,680 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
> 2020-06-27 10:28:00,680 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
> 2020-06-27 10:28:00,684 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
> 2020-06-27 10:28:00,684 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
> 2020-06-27 10:28:00,696 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
> 2020-06-27 10:28:00,696 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
> 2020-06-27 10:28:00,782 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
> 2020-06-27 10:28:00,782 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
> 2020-06-27 10:28:00,786 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
> hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
> {noformat}
> We could set the log level to INFO when starting a mini cluster in HBTU.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24646) Set the log level for ScheduledChore to INFO in HBTU
Duo Zhang created HBASE-24646:
-
Summary: Set the log level for ScheduledChore to INFO in HBTU
Key: HBASE-24646
URL: https://issues.apache.org/jira/browse/HBASE-24646
Project: HBase
Issue Type: Task
Components: test
Reporter: Duo Zhang
Now we changed the thead waken interval to 100ms in tests, which causes the
test log beng flooded with
{noformat}
2020-06-27 10:28:00,680 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
2020-06-27 10:28:00,680 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
2020-06-27 10:28:00,684 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
2020-06-27 10:28:00,684 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
2020-06-27 10:28:00,696 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
2020-06-27 10:28:00,696 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
2020-06-27 10:28:00,782 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
2020-06-27 10:28:00,782 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms.
2020-06-27 10:28:00,786 DEBUG [regionserver/zhangduo-ubuntu:0.Chore.1]
hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms.
{noformat}
We could set the log level to INFO when starting a mini cluster in HBTU.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24623) SIGSEGV v ~StubRoutines::jbyte_disjoint_arraycopy
[
https://issues.apache.org/jira/browse/HBASE-24623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146736#comment-17146736
]
Duo Zhang commented on HBASE-24623:
---
So a broken WAL file?
IIRC we've fixed a bug related to this area before, where we release the byte
buffer before actually writing it out, which causes WAL splitting to fail.
> SIGSEGV v ~StubRoutines::jbyte_disjoint_arraycopy
> --
>
> Key: HBASE-24623
> URL: https://issues.apache.org/jira/browse/HBASE-24623
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Priority: Major
>
> In testing, 1% of a decent cluster went down with this seg fault in the vm:
> {code}
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f6659052410, pid=37208, tid=0x7f3c89453700
> #
> # JRE version: OpenJDK Runtime Environment (8.0_232-b09) (build 1.8.0_232-b09)
> # Java VM: OpenJDK 64-Bit Server VM (25.232-b09 mixed mode linux-amd64 )
> # Problematic frame:
> # v ~StubRoutines::jbyte_disjoint_arraycopy
> {code}
> Looking in the hs_err log, the crash happens in the same area. Here are a few
> of the stack traces:
> {code}
> Stack: [0x7f3c89353000,0x7f3c89454000], sp=0x7f3c89452110, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jbyte_disjoint_arraycopy
> J 17674 C2
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray([BLjava/nio/ByteBuffer;III)V
> (69 bytes) @ 0x7f665af000d1 [0x7f665aefffe0+0xf1]
> J 17732 C1
> org.apache.hadoop.hbase.CellUtil.copyQualifierTo(Lorg/apache/hadoop/hbase/Cell;[BI)I
> (59 bytes) @ 0x7f665bc440dc [0x7f665bc43b80+0x55c]
> j
> org.apache.hadoop.hbase.CellUtil.cloneQualifier(Lorg/apache/hadoop/hbase/Cell;)[B+12
> J 22278 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getQualifierArray()[B
> (5 bytes) @ 0x7f6659bd4784 [0x7f6659bd4760+0x24]
> j
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;Ljava/util/function/Function;)Ljava/lang/String;+97
> j
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;)Ljava/lang/String;+6
> j
> org.apache.hadoop.hbase.CellUtil.toString(Lorg/apache/hadoop/hbase/Cell;Z)Ljava/lang/String;+16
> j org.apache.hadoop.hbase.ByteBufferKeyValue.toString()Ljava/lang/String;+2
> j
> org.apache.hadoop.hbase.client.Mutation.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Mutation;+28
> J 22605 C2
> org.apache.hadoop.hbase.client.Put.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Put;
> (8 bytes) @ 0x7f665a982a04 [0x7f665a9829e0+0x24]
> J 22112 C2
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$MutationProto;Lorg/apache/hadoop/hbase/CellScanner;)Lorg/apache/hadoop/hbase/client/Put;
> (910 bytes) @ 0x7f665c706700 [0x7f665c706000+0x700]
> J 24084 C2
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Ljava/util/List;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;Z)V
> (646 bytes) @ 0x7f665cc21100 [0x7f665cc20c80+0x480]
> J 14696 C2
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionAction;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Ljava/util/List;JLorg/apache/hadoop/hbase/regionserver/RSRpcServices$RegionScannersCloseCallBack;Lorg/apache/hadoop/hbase/ipc/RpcCallContext;Lorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;)Ljava/util/List;
> (901 bytes) @ 0x7f665b722148 [0x7f665b7218e0+0x868]
> {code}
> Here's another:
> {code}
> Stack: [0x7edd015e2000,0x7edd016e3000], sp=0x7edd016e11b0, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jbyte_disjoint_arraycopy
> J 18255 C2
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray([BLjava/nio/ByteBuffer;III)V
> (69 bytes) @ 0x7f06d2593551 [0x7f06d2593460+0xf1]
> j
> org.apache.hadoop.hbase.PrivateCellUtil.copyTagsTo(Lorg/apache/hadoop/hbase/Cell;[BI)I+31
> j
> org.apache.hadoop.hbase.CellUtil.cloneTags(Lorg/apache/hadoop/hbase/Cell;)[B+12
> j org.apache.hadoop.hbase.ByteBufferKeyValue.getTagsArray()[B+1
> j
[GitHub] [hbase] Apache9 commented on a change in pull request #1774: HBASE-24389 Introduce new master rpc methods to locate meta region through root region
Apache9 commented on a change in pull request #1774:
URL: https://github.com/apache/hbase/pull/1774#discussion_r446471594
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/CatalogFamilyFormat.java
##
@@ -363,4 +370,27 @@ public static Delete removeRegionReplica(byte[] metaRow,
int replicaIndexToDelet
}
return deleteReplicaLocations;
}
+
+ private static byte[] buildLocateRegionStartRow(TableName tableName, byte[]
row,
+RegionLocateType locateType) {
+if (locateType.equals(RegionLocateType.BEFORE)) {
Review comment:
You mean the 'RegionLocateType'? Or just change the method name?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1774: HBASE-24389 Introduce new master rpc methods to locate meta region through root region
Apache9 commented on a change in pull request #1774:
URL: https://github.com/apache/hbase/pull/1774#discussion_r446471642
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionLocator.java
##
@@ -85,26 +118,41 @@ private boolean isMeta(TableName tableName) {
return TableName.isMetaTableName(tableName);
}
+ private AbstractAsyncTableRegionLocator
getOrCreateTableRegionLocator(TableName tableName) {
Review comment:
This is internal so just use the implementation class directly. Can have
an interface later if we think it is useful.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1774: HBASE-24389 Introduce new master rpc methods to locate meta region through root region
Apache9 commented on a change in pull request #1774:
URL: https://github.com/apache/hbase/pull/1774#discussion_r446471518
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/ClientMetaTableAccessor.java
##
@@ -164,26 +164,27 @@ private ClientMetaTableAccessor() {
/**
* Used to get all region locations for the specific table.
- * @param metaTable
* @param tableName table we're looking for, can be null for getting all
regions
* @return the list of region locations. The return value will be wrapped by
a
* {@link CompletableFuture}.
*/
public static CompletableFuture>
getTableHRegionLocations(
-AsyncTable metaTable, TableName tableName) {
+AsyncTable metaTable, TableName tableName,
+boolean excludeOfflinedSplitParents) {
CompletableFuture> future = new
CompletableFuture<>();
-addListener(getTableRegionsAndLocations(metaTable, tableName, true),
(locations, err) -> {
- if (err != null) {
-future.completeExceptionally(err);
- } else if (locations == null || locations.isEmpty()) {
-future.complete(Collections.emptyList());
- } else {
-List regionLocations =
- locations.stream().map(loc -> new HRegionLocation(loc.getFirst(),
loc.getSecond()))
-.collect(Collectors.toList());
-future.complete(regionLocations);
- }
-});
+addListener(getTableRegionsAndLocations(metaTable, tableName,
excludeOfflinedSplitParents),
Review comment:
At least in the Snapshot related code, we have special support for meta
table. So at master side, we need to expose this parameter. On whether to
expose this parameter at RPC level, I'm not sure yet. In the current design,
the cache server will use the rpc method to fetch all the records in root
table, maybe it is good to set this parameter to true so they will get all the
records? Not sure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1774: HBASE-24389 Introduce new master rpc methods to locate meta region through root region
Apache9 commented on a change in pull request #1774:
URL: https://github.com/apache/hbase/pull/1774#discussion_r446471410
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractAsyncTableRegionLocator.java
##
@@ -0,0 +1,308 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.HConstants.EMPTY_END_ROW;
+import static org.apache.hadoop.hbase.client.AsyncRegionLocatorHelper.isGood;
+import static
org.apache.hadoop.hbase.client.ConnectionUtils.createClosestRowAfter;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.commons.lang3.ObjectUtils;
+import org.apache.hadoop.hbase.HBaseIOException;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.RegionLocations;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * The base class for locating region of a table.
+ */
+@InterfaceAudience.Private
+abstract class AbstractAsyncTableRegionLocator {
Review comment:
The AbstractAsyncTableRegionLocator has a cache in it, for caching the
location of regions of a table. That's why I added a 'Table' in the class name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24588) Normalizer plan execution is not consistent between plan types
[
https://issues.apache.org/jira/browse/HBASE-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146720#comment-17146720
]
Hudson commented on HBASE-24588:
Results for branch master
[build #1769 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1769/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1769/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1600//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1769/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1769/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Normalizer plan execution is not consistent between plan types
> --
>
> Key: HBASE-24588
> URL: https://issues.apache.org/jira/browse/HBASE-24588
> Project: HBase
> Issue Type: Bug
> Components: master, Normalizer
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> I left a comment on a merged
> [commit|https://github.com/apache/hbase/commit/5d0e0fc5fd09bddb2d766d1e24e28e472961f454#r39987289]
> where a little discussion has blossomed. Right now the normalizer produces
> two types of plans: "split" or "merge". The master receives the list of plans
> and executes them in a simple loop. The way it does the actual execution is
> delegated off to the plan implementation.
> The bug I noticed is that the two implementations are subtly different. Both
> use async APIs to submit procedures, but "split" blocks on completion while
> "merge" does not. Furthermore, because "split" blocks, it's able to capture
> any exception that's thrown, while "merge" cannot.
> These implementations should be made consistent. My thinking at the moment is
> this {{execute}} method should instead be named {{submit}}, creating and
> returning the {{Future}} that represents whatever work it submitted, and the
> calling context should handle the resolution of those futures in a single
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24638) Edit doc on (offheap) memory management
[
https://issues.apache.org/jira/browse/HBASE-24638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146719#comment-17146719
]
Hudson commented on HBASE-24638:
Results for branch master
[build #1769 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1769/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1769/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1600//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1769/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1769/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Edit doc on (offheap) memory management
> ---
>
> Key: HBASE-24638
> URL: https://issues.apache.org/jira/browse/HBASE-24638
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Gave it a read over to try and figure current state of memory management in
> hbase-2.3.0. Updated it to reflect more of what the current state is.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24221) Support bulkLoadHFile by family
[
https://issues.apache.org/jira/browse/HBASE-24221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146716#comment-17146716
]
Hudson commented on HBASE-24221:
Results for branch branch-2
[build #2721 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Support bulkLoadHFile by family
> ---
>
> Key: HBASE-24221
> URL: https://issues.apache.org/jira/browse/HBASE-24221
> Project: HBase
> Issue Type: Improvement
> Components: HFile
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.2.4
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> Support bulkLoadHFile by family to avoid long time waiting of bulkLoadHFile
> because of compacting at server side
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24562) Stabilize master startup with meta replicas enabled
[
https://issues.apache.org/jira/browse/HBASE-24562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146715#comment-17146715
]
Hudson commented on HBASE-24562:
Results for branch branch-2
[build #2721 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Stabilize master startup with meta replicas enabled
> ---
>
> Key: HBASE-24562
> URL: https://issues.apache.org/jira/browse/HBASE-24562
> Project: HBase
> Issue Type: Improvement
> Components: meta, read replicas
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.5
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> This is related to HBASE-21624 .
> I created a separate ticket because in the original one a "complete solution
> for meta replicas" was requested and this is not one. I'm just trying to make
> master startup more stable by making assigning meta replicas asynchronous and
> preventing a potential assignment failure from crashing master.
> The idea is that starting master with less or even no meta replicas assigned
> is preferable to not having a running master.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24552) Replica region needs to check if primary region directory exists at file system in TransitRegionStateProcedure
[
https://issues.apache.org/jira/browse/HBASE-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146717#comment-17146717
]
Hudson commented on HBASE-24552:
Results for branch branch-2
[build #2721 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2721/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Replica region needs to check if primary region directory exists at file
> system in TransitRegionStateProcedure
>
>
> Key: HBASE-24552
> URL: https://issues.apache.org/jira/browse/HBASE-24552
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 2.3.0
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Major
>
> In hbase-1, it always runs into the situation that primary region has been
> closed/removed and replica region still stays in master's in-memory db and
> open at one of the region servers. Balancer can move this replica region to a
> new region server. During the region open, replica region does not check if
> primary region has been removed and moves forward. During store open, it will
> recreates primary region directory at hdfs and caused inconsistency.
>
> In hbase-2, things get much better. To prevent the above inconsistency from
> happening, it adds more checks for a replica region, i.e, if primary regions'
> directory exists and there is a .regioninfo under.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24637) Filter SKIP hinting regression
[ https://issues.apache.org/jira/browse/HBASE-24637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24637: Attachment: parse_call_trace.pl > Filter SKIP hinting regression > -- > > Key: HBASE-24637 > URL: https://issues.apache.org/jira/browse/HBASE-24637 > Project: HBase > Issue Type: Bug > Components: Filters, Performance, Scanners >Affects Versions: 2.2.5 >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: W-7665966-FAST_DIFF-FILTER_ALL.pdf, > W-7665966-Instrument-low-level-scan-details-branch-1.patch, > W-7665966-Instrument-low-level-scan-details-branch-2.2.patch, > parse_call_trace.pl > > > I have been looking into reported performance regressions in HBase 2 relative > to HBase 1. Depending on the test scenario, HBase 2 can demonstrate > significantly better microbenchmarks in a number of cases, and usually shows > improvement in whole cluster benchmarks like YCSB. > To assist in debugging I added methods to RpcServer for updating per-call > metrics that leverage the fact it puts a reference to the current Call into a > thread local and that all activity for a given RPC is processed by a single > thread context. I then instrumented ScanQueryMatcher (in branch-1) and its > various friends (in branch-2.2), StoreScanner, HFileReaderV2 and > HFileReaderV3 (in branch-1) and HFileReaderImpl (in branch-2.2), HFileBlock, > and DefaultMemStore (branch-1) and SegmentScanner (branch-2.2). Test tables > with one family and 1, 5, 10, 20, 50, and 100 distinct column-qualifiers per > row were created, snapshot, dropped, and cloned from the snapshot. Both 1.6 > and 2.2 versions under test operated on identical data files in HDFS. For > tests with 1.6 and 2.2 on the server side the same 1.6 PE client was used, to > ensure only the server side differed. > The results for pe --filterAll were revealing. See attached. > It appears a refactor to ScanQueryMatcher and friends has disabled the > ability of filters to provide meaningful SKIP hints, which disables an > optimization that avoids reseeking, leading to a serious and proportional > regression in reseek activity and time spent in that code path. So for > queries that use filters, there can be a substantial regression. > Other test cases that did not use filters did not show this regression. If > filters are not used the behavior of ScanQueryMatcher between 1.6 and 2.2 was > almost identical, as measured by counts of the hint types returned, whether > or not column or version trackers are called, and counts of store seeks or > reseeks. Regarding micro-timings, there was a 10% variance in my testing and > results generally fell within this range, except for the filter all case of > course. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24637) Filter SKIP hinting regression
[ https://issues.apache.org/jira/browse/HBASE-24637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24637: Attachment: (was: parse_call_trace.pl) > Filter SKIP hinting regression > -- > > Key: HBASE-24637 > URL: https://issues.apache.org/jira/browse/HBASE-24637 > Project: HBase > Issue Type: Bug > Components: Filters, Performance, Scanners >Affects Versions: 2.2.5 >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: W-7665966-FAST_DIFF-FILTER_ALL.pdf, > W-7665966-Instrument-low-level-scan-details-branch-1.patch, > W-7665966-Instrument-low-level-scan-details-branch-2.2.patch, > parse_call_trace.pl > > > I have been looking into reported performance regressions in HBase 2 relative > to HBase 1. Depending on the test scenario, HBase 2 can demonstrate > significantly better microbenchmarks in a number of cases, and usually shows > improvement in whole cluster benchmarks like YCSB. > To assist in debugging I added methods to RpcServer for updating per-call > metrics that leverage the fact it puts a reference to the current Call into a > thread local and that all activity for a given RPC is processed by a single > thread context. I then instrumented ScanQueryMatcher (in branch-1) and its > various friends (in branch-2.2), StoreScanner, HFileReaderV2 and > HFileReaderV3 (in branch-1) and HFileReaderImpl (in branch-2.2), HFileBlock, > and DefaultMemStore (branch-1) and SegmentScanner (branch-2.2). Test tables > with one family and 1, 5, 10, 20, 50, and 100 distinct column-qualifiers per > row were created, snapshot, dropped, and cloned from the snapshot. Both 1.6 > and 2.2 versions under test operated on identical data files in HDFS. For > tests with 1.6 and 2.2 on the server side the same 1.6 PE client was used, to > ensure only the server side differed. > The results for pe --filterAll were revealing. See attached. > It appears a refactor to ScanQueryMatcher and friends has disabled the > ability of filters to provide meaningful SKIP hints, which disables an > optimization that avoids reseeking, leading to a serious and proportional > regression in reseek activity and time spent in that code path. So for > queries that use filters, there can be a substantial regression. > Other test cases that did not use filters did not show this regression. If > filters are not used the behavior of ScanQueryMatcher between 1.6 and 2.2 was > almost identical, as measured by counts of the hint types returned, whether > or not column or version trackers are called, and counts of store seeks or > reseeks. Regarding micro-timings, there was a 10% variance in my testing and > results generally fell within this range, except for the filter all case of > course. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24637) Filter SKIP hinting regression
[ https://issues.apache.org/jira/browse/HBASE-24637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146705#comment-17146705 ] Andrew Kyle Purtell commented on HBASE-24637: - I am considering adding the instrumentation to our internal version of HBase, so I spent some time cleaning them up. Refreshed the patches attached here. That's the only update for today. > Filter SKIP hinting regression > -- > > Key: HBASE-24637 > URL: https://issues.apache.org/jira/browse/HBASE-24637 > Project: HBase > Issue Type: Bug > Components: Filters, Performance, Scanners >Affects Versions: 2.2.5 >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: W-7665966-FAST_DIFF-FILTER_ALL.pdf, > W-7665966-Instrument-low-level-scan-details-branch-1.patch, > W-7665966-Instrument-low-level-scan-details-branch-2.2.patch, > parse_call_trace.pl > > > I have been looking into reported performance regressions in HBase 2 relative > to HBase 1. Depending on the test scenario, HBase 2 can demonstrate > significantly better microbenchmarks in a number of cases, and usually shows > improvement in whole cluster benchmarks like YCSB. > To assist in debugging I added methods to RpcServer for updating per-call > metrics that leverage the fact it puts a reference to the current Call into a > thread local and that all activity for a given RPC is processed by a single > thread context. I then instrumented ScanQueryMatcher (in branch-1) and its > various friends (in branch-2.2), StoreScanner, HFileReaderV2 and > HFileReaderV3 (in branch-1) and HFileReaderImpl (in branch-2.2), HFileBlock, > and DefaultMemStore (branch-1) and SegmentScanner (branch-2.2). Test tables > with one family and 1, 5, 10, 20, 50, and 100 distinct column-qualifiers per > row were created, snapshot, dropped, and cloned from the snapshot. Both 1.6 > and 2.2 versions under test operated on identical data files in HDFS. For > tests with 1.6 and 2.2 on the server side the same 1.6 PE client was used, to > ensure only the server side differed. > The results for pe --filterAll were revealing. See attached. > It appears a refactor to ScanQueryMatcher and friends has disabled the > ability of filters to provide meaningful SKIP hints, which disables an > optimization that avoids reseeking, leading to a serious and proportional > regression in reseek activity and time spent in that code path. So for > queries that use filters, there can be a substantial regression. > Other test cases that did not use filters did not show this regression. If > filters are not used the behavior of ScanQueryMatcher between 1.6 and 2.2 was > almost identical, as measured by counts of the hint types returned, whether > or not column or version trackers are called, and counts of store seeks or > reseeks. Regarding micro-timings, there was a 10% variance in my testing and > results generally fell within this range, except for the filter all case of > course. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24637) Filter SKIP hinting regression
[ https://issues.apache.org/jira/browse/HBASE-24637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24637: Attachment: (was: W-7665966-Instrument-low-level-scan-details-branch-1.patch) > Filter SKIP hinting regression > -- > > Key: HBASE-24637 > URL: https://issues.apache.org/jira/browse/HBASE-24637 > Project: HBase > Issue Type: Bug > Components: Filters, Performance, Scanners >Affects Versions: 2.2.5 >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: W-7665966-FAST_DIFF-FILTER_ALL.pdf, > W-7665966-Instrument-low-level-scan-details-branch-1.patch, > W-7665966-Instrument-low-level-scan-details-branch-2.2.patch, > parse_call_trace.pl > > > I have been looking into reported performance regressions in HBase 2 relative > to HBase 1. Depending on the test scenario, HBase 2 can demonstrate > significantly better microbenchmarks in a number of cases, and usually shows > improvement in whole cluster benchmarks like YCSB. > To assist in debugging I added methods to RpcServer for updating per-call > metrics that leverage the fact it puts a reference to the current Call into a > thread local and that all activity for a given RPC is processed by a single > thread context. I then instrumented ScanQueryMatcher (in branch-1) and its > various friends (in branch-2.2), StoreScanner, HFileReaderV2 and > HFileReaderV3 (in branch-1) and HFileReaderImpl (in branch-2.2), HFileBlock, > and DefaultMemStore (branch-1) and SegmentScanner (branch-2.2). Test tables > with one family and 1, 5, 10, 20, 50, and 100 distinct column-qualifiers per > row were created, snapshot, dropped, and cloned from the snapshot. Both 1.6 > and 2.2 versions under test operated on identical data files in HDFS. For > tests with 1.6 and 2.2 on the server side the same 1.6 PE client was used, to > ensure only the server side differed. > The results for pe --filterAll were revealing. See attached. > It appears a refactor to ScanQueryMatcher and friends has disabled the > ability of filters to provide meaningful SKIP hints, which disables an > optimization that avoids reseeking, leading to a serious and proportional > regression in reseek activity and time spent in that code path. So for > queries that use filters, there can be a substantial regression. > Other test cases that did not use filters did not show this regression. If > filters are not used the behavior of ScanQueryMatcher between 1.6 and 2.2 was > almost identical, as measured by counts of the hint types returned, whether > or not column or version trackers are called, and counts of store seeks or > reseeks. Regarding micro-timings, there was a 10% variance in my testing and > results generally fell within this range, except for the filter all case of > course. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24637) Filter SKIP hinting regression
[ https://issues.apache.org/jira/browse/HBASE-24637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24637: Attachment: (was: W-7665966-Instrument-low-level-scan-details-branch-2.2.patch) > Filter SKIP hinting regression > -- > > Key: HBASE-24637 > URL: https://issues.apache.org/jira/browse/HBASE-24637 > Project: HBase > Issue Type: Bug > Components: Filters, Performance, Scanners >Affects Versions: 2.2.5 >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: W-7665966-FAST_DIFF-FILTER_ALL.pdf, > W-7665966-Instrument-low-level-scan-details-branch-1.patch, > W-7665966-Instrument-low-level-scan-details-branch-2.2.patch, > parse_call_trace.pl > > > I have been looking into reported performance regressions in HBase 2 relative > to HBase 1. Depending on the test scenario, HBase 2 can demonstrate > significantly better microbenchmarks in a number of cases, and usually shows > improvement in whole cluster benchmarks like YCSB. > To assist in debugging I added methods to RpcServer for updating per-call > metrics that leverage the fact it puts a reference to the current Call into a > thread local and that all activity for a given RPC is processed by a single > thread context. I then instrumented ScanQueryMatcher (in branch-1) and its > various friends (in branch-2.2), StoreScanner, HFileReaderV2 and > HFileReaderV3 (in branch-1) and HFileReaderImpl (in branch-2.2), HFileBlock, > and DefaultMemStore (branch-1) and SegmentScanner (branch-2.2). Test tables > with one family and 1, 5, 10, 20, 50, and 100 distinct column-qualifiers per > row were created, snapshot, dropped, and cloned from the snapshot. Both 1.6 > and 2.2 versions under test operated on identical data files in HDFS. For > tests with 1.6 and 2.2 on the server side the same 1.6 PE client was used, to > ensure only the server side differed. > The results for pe --filterAll were revealing. See attached. > It appears a refactor to ScanQueryMatcher and friends has disabled the > ability of filters to provide meaningful SKIP hints, which disables an > optimization that avoids reseeking, leading to a serious and proportional > regression in reseek activity and time spent in that code path. So for > queries that use filters, there can be a substantial regression. > Other test cases that did not use filters did not show this regression. If > filters are not used the behavior of ScanQueryMatcher between 1.6 and 2.2 was > almost identical, as measured by counts of the hint types returned, whether > or not column or version trackers are called, and counts of store seeks or > reseeks. Regarding micro-timings, there was a 10% variance in my testing and > results generally fell within this range, except for the filter all case of > course. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24637) Filter SKIP hinting regression
[ https://issues.apache.org/jira/browse/HBASE-24637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24637: Attachment: W-7665966-Instrument-low-level-scan-details-branch-2.2.patch W-7665966-Instrument-low-level-scan-details-branch-1.patch > Filter SKIP hinting regression > -- > > Key: HBASE-24637 > URL: https://issues.apache.org/jira/browse/HBASE-24637 > Project: HBase > Issue Type: Bug > Components: Filters, Performance, Scanners >Affects Versions: 2.2.5 >Reporter: Andrew Kyle Purtell >Priority: Major > Attachments: W-7665966-FAST_DIFF-FILTER_ALL.pdf, > W-7665966-Instrument-low-level-scan-details-branch-1.patch, > W-7665966-Instrument-low-level-scan-details-branch-2.2.patch, > parse_call_trace.pl > > > I have been looking into reported performance regressions in HBase 2 relative > to HBase 1. Depending on the test scenario, HBase 2 can demonstrate > significantly better microbenchmarks in a number of cases, and usually shows > improvement in whole cluster benchmarks like YCSB. > To assist in debugging I added methods to RpcServer for updating per-call > metrics that leverage the fact it puts a reference to the current Call into a > thread local and that all activity for a given RPC is processed by a single > thread context. I then instrumented ScanQueryMatcher (in branch-1) and its > various friends (in branch-2.2), StoreScanner, HFileReaderV2 and > HFileReaderV3 (in branch-1) and HFileReaderImpl (in branch-2.2), HFileBlock, > and DefaultMemStore (branch-1) and SegmentScanner (branch-2.2). Test tables > with one family and 1, 5, 10, 20, 50, and 100 distinct column-qualifiers per > row were created, snapshot, dropped, and cloned from the snapshot. Both 1.6 > and 2.2 versions under test operated on identical data files in HDFS. For > tests with 1.6 and 2.2 on the server side the same 1.6 PE client was used, to > ensure only the server side differed. > The results for pe --filterAll were revealing. See attached. > It appears a refactor to ScanQueryMatcher and friends has disabled the > ability of filters to provide meaningful SKIP hints, which disables an > optimization that avoids reseeking, leading to a serious and proportional > regression in reseek activity and time spent in that code path. So for > queries that use filters, there can be a substantial regression. > Other test cases that did not use filters did not show this regression. If > filters are not used the behavior of ScanQueryMatcher between 1.6 and 2.2 was > almost identical, as measured by counts of the hint types returned, whether > or not column or version trackers are called, and counts of store seeks or > reseeks. Regarding micro-timings, there was a 10% variance in my testing and > results generally fell within this range, except for the filter all case of > course. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24602) Add Increment and Append support to CheckAndMutate
[ https://issues.apache.org/jira/browse/HBASE-24602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki updated HBASE-24602: - Description: Currently, CheckAndMutate supports only Put and Delete. Supporting Increment and Append would be helpful for some use cases. (was: Currently, CheckAndMutate supports only Put, Delete and RowMutations. Supporting Increment and Append would be helpful for some use cases.) > Add Increment and Append support to CheckAndMutate > -- > > Key: HBASE-24602 > URL: https://issues.apache.org/jira/browse/HBASE-24602 > Project: HBase > Issue Type: New Feature >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently, CheckAndMutate supports only Put and Delete. Supporting Increment > and Append would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24623) SIGSEGV v ~StubRoutines::jbyte_disjoint_arraycopy
[
https://issues.apache.org/jira/browse/HBASE-24623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146698#comment-17146698
]
Michael Stack commented on HBASE-24623:
---
bq. Coming to your case, so when you see the issue in write, this comes while
replication?
Yeah. Replication alone, all is fine. If same cluster gets really heavy reads,
then issue. If heavy-reads and no Replication, all is fine too.
We are running a RSRpcServices.multi call provoked by a Replication #replay
from somewhere else in the cluster (hard to confirm).
> SIGSEGV v ~StubRoutines::jbyte_disjoint_arraycopy
> --
>
> Key: HBASE-24623
> URL: https://issues.apache.org/jira/browse/HBASE-24623
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Priority: Major
>
> In testing, 1% of a decent cluster went down with this seg fault in the vm:
> {code}
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f6659052410, pid=37208, tid=0x7f3c89453700
> #
> # JRE version: OpenJDK Runtime Environment (8.0_232-b09) (build 1.8.0_232-b09)
> # Java VM: OpenJDK 64-Bit Server VM (25.232-b09 mixed mode linux-amd64 )
> # Problematic frame:
> # v ~StubRoutines::jbyte_disjoint_arraycopy
> {code}
> Looking in the hs_err log, the crash happens in the same area. Here are a few
> of the stack traces:
> {code}
> Stack: [0x7f3c89353000,0x7f3c89454000], sp=0x7f3c89452110, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jbyte_disjoint_arraycopy
> J 17674 C2
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray([BLjava/nio/ByteBuffer;III)V
> (69 bytes) @ 0x7f665af000d1 [0x7f665aefffe0+0xf1]
> J 17732 C1
> org.apache.hadoop.hbase.CellUtil.copyQualifierTo(Lorg/apache/hadoop/hbase/Cell;[BI)I
> (59 bytes) @ 0x7f665bc440dc [0x7f665bc43b80+0x55c]
> j
> org.apache.hadoop.hbase.CellUtil.cloneQualifier(Lorg/apache/hadoop/hbase/Cell;)[B+12
> J 22278 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getQualifierArray()[B
> (5 bytes) @ 0x7f6659bd4784 [0x7f6659bd4760+0x24]
> j
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;Ljava/util/function/Function;)Ljava/lang/String;+97
> j
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;)Ljava/lang/String;+6
> j
> org.apache.hadoop.hbase.CellUtil.toString(Lorg/apache/hadoop/hbase/Cell;Z)Ljava/lang/String;+16
> j org.apache.hadoop.hbase.ByteBufferKeyValue.toString()Ljava/lang/String;+2
> j
> org.apache.hadoop.hbase.client.Mutation.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Mutation;+28
> J 22605 C2
> org.apache.hadoop.hbase.client.Put.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Put;
> (8 bytes) @ 0x7f665a982a04 [0x7f665a9829e0+0x24]
> J 22112 C2
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$MutationProto;Lorg/apache/hadoop/hbase/CellScanner;)Lorg/apache/hadoop/hbase/client/Put;
> (910 bytes) @ 0x7f665c706700 [0x7f665c706000+0x700]
> J 24084 C2
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Ljava/util/List;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;Z)V
> (646 bytes) @ 0x7f665cc21100 [0x7f665cc20c80+0x480]
> J 14696 C2
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionAction;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Ljava/util/List;JLorg/apache/hadoop/hbase/regionserver/RSRpcServices$RegionScannersCloseCallBack;Lorg/apache/hadoop/hbase/ipc/RpcCallContext;Lorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;)Ljava/util/List;
> (901 bytes) @ 0x7f665b722148 [0x7f665b7218e0+0x868]
> {code}
> Here's another:
> {code}
> Stack: [0x7edd015e2000,0x7edd016e3000], sp=0x7edd016e11b0, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jbyte_disjoint_arraycopy
> J 18255 C2
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray([BLjava/nio/ByteBuffer;III)V
> (69 bytes) @ 0x7f06d2593551 [0x7f06d2593460+0xf1]
> j
>
[GitHub] [hbase] saintstack commented on a change in pull request #1774: HBASE-24389 Introduce new master rpc methods to locate meta region through root region
saintstack commented on a change in pull request #1774:
URL: https://github.com/apache/hbase/pull/1774#discussion_r446446620
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractAsyncTableRegionLocator.java
##
@@ -0,0 +1,308 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.HConstants.EMPTY_END_ROW;
+import static org.apache.hadoop.hbase.client.AsyncRegionLocatorHelper.isGood;
+import static
org.apache.hadoop.hbase.client.ConnectionUtils.createClosestRowAfter;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.commons.lang3.ObjectUtils;
+import org.apache.hadoop.hbase.HBaseIOException;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.RegionLocations;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * The base class for locating region of a table.
+ */
+@InterfaceAudience.Private
+abstract class AbstractAsyncTableRegionLocator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(AbstractAsyncTableRegionLocator.class);
+
+ protected final AsyncConnectionImpl conn;
+
+ protected final TableName tableName;
+
+ protected final int maxConcurrent;
+
+ protected final TableRegionLocationCache cache;
+
+ protected static final class LocateRequest {
Review comment:
Let me resolve. I see that this is just a move of code. My fault for not
knowing this code well enough and realizing it not new.
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/CatalogFamilyFormat.java
##
@@ -363,4 +370,27 @@ public static Delete removeRegionReplica(byte[] metaRow,
int replicaIndexToDelet
}
return deleteReplicaLocations;
}
+
+ private static byte[] buildLocateRegionStartRow(TableName tableName, byte[]
row,
+RegionLocateType locateType) {
+if (locateType.equals(RegionLocateType.BEFORE)) {
Review comment:
nit... should be LocateRegionType to match the method names in here.
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/ClientMetaTableAccessor.java
##
@@ -164,26 +164,27 @@ private ClientMetaTableAccessor() {
/**
* Used to get all region locations for the specific table.
- * @param metaTable
* @param tableName table we're looking for, can be null for getting all
regions
* @return the list of region locations. The return value will be wrapped by
a
* {@link CompletableFuture}.
*/
public static CompletableFuture>
getTableHRegionLocations(
-AsyncTable metaTable, TableName tableName) {
+AsyncTable metaTable, TableName tableName,
+boolean excludeOfflinedSplitParents) {
CompletableFuture> future = new
CompletableFuture<>();
-addListener(getTableRegionsAndLocations(metaTable, tableName, true),
(locations, err) -> {
- if (err != null) {
-future.completeExceptionally(err);
- } else if (locations == null || locations.isEmpty()) {
-future.complete(Collections.emptyList());
- } else {
-List regionLocations =
- locations.stream().map(loc -> new HRegionLocation(loc.getFirst(),
loc.getSecond()))
-.collect(Collectors.toList());
-future.complete(regionLocations);
- }
-});
+addListener(getTableRegionsAndLocations(metaTable, tableName,
excludeOfflinedSplitParents),
Review comment:
Trying to understand why we need to support this flag
excludeOfflinedSplitParents
Snapshots need it? Are we trying to support snapshotting hbase:meta table?
I was going to suggest we not do this but thinking on it, it is probably a
useful utility to
[jira] [Commented] (HBASE-24221) Support bulkLoadHFile by family
[
https://issues.apache.org/jira/browse/HBASE-24221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146690#comment-17146690
]
Hudson commented on HBASE-24221:
Results for branch branch-2.2
[build #903 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/903/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/903//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/903//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/903//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Support bulkLoadHFile by family
> ---
>
> Key: HBASE-24221
> URL: https://issues.apache.org/jira/browse/HBASE-24221
> Project: HBase
> Issue Type: Improvement
> Components: HFile
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.2.4
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.5
>
>
> Support bulkLoadHFile by family to avoid long time waiting of bulkLoadHFile
> because of compacting at server side
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack commented on pull request #1774: HBASE-24389 Introduce new master rpc methods to locate meta region through root region
saintstack commented on pull request #1774: URL: https://github.com/apache/hbase/pull/1774#issuecomment-650433945 bq. OK, got it. So you are talking about the whole design and Im expecting you are commenting on the patch. Yeah. This patch presumes a design that is only starting to come out in our design doc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-24459) Move the locateMeta logic from AsyncMetaRegionTableLocator to ConnectionRegistry
[ https://issues.apache.org/jira/browse/HBASE-24459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146657#comment-17146657 ] Michael Stack edited comment on HBASE-24459 at 6/26/20, 10:03 PM: -- Duo has talked of keeping the ConnectionRegistry Interface read-only and simple so Implemention could go against an existing caching tier. Could also keep the cache tier in-cluster with Master farming out location updates on (rare) change via remote procedure infrastructure so any RegionServer can participate? Or simpler, ROOT Table Region replication across all Masters; active and backups. For implementation later. was (Author: stack): Duo has talked of keeping the ConnectionRegistry Interface read-only and simple so Implemention could go against an existing caching tier. Could also keep the cache tier in-cluster with Master farming out location updates on (rare) change via remote procedure infrastructure so any RegionServer can participate? For implementation later. > Move the locateMeta logic from AsyncMetaRegionTableLocator to > ConnectionRegistry > > > Key: HBASE-24459 > URL: https://issues.apache.org/jira/browse/HBASE-24459 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > > Now the related code is only in AsyncMetaRegionTableLocator, we could make > the actually implementation pluggable, so we do not always need to go to the > active master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1983: HBASE-24588 : Submit task for NormalizationPlan (#1933)
Apache-HBase commented on pull request #1983: URL: https://github.com/apache/hbase/pull/1983#issuecomment-650417303 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 0s | branch-2 passed | | +1 :green_heart: | compile | 1m 48s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 21s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 49s | the patch passed | | +1 :green_heart: | compile | 1m 52s | the patch passed | | +1 :green_heart: | javac | 1m 52s | the patch passed | | +1 :green_heart: | shadedjars | 6m 46s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 43s | hbase-client in the patch passed. | | -1 :x: | unit | 217m 6s | hbase-server in the patch failed. | | | | 254m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1983 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 257743fd5231 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a499eae1aa | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/testReport/ | | Max. process+thread count | 2911 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24459) Move the locateMeta logic from AsyncMetaRegionTableLocator to ConnectionRegistry
[ https://issues.apache.org/jira/browse/HBASE-24459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146657#comment-17146657 ] Michael Stack commented on HBASE-24459: --- Duo has talked of keeping the ConnectionRegistry Interface read-only and simple so Implemention could go against an existing caching tier. Could also keep the cache tier in-cluster with Master farming out location updates on (rare) change via remote procedure infrastructure so any RegionServer can participate? For implementation later. > Move the locateMeta logic from AsyncMetaRegionTableLocator to > ConnectionRegistry > > > Key: HBASE-24459 > URL: https://issues.apache.org/jira/browse/HBASE-24459 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > > Now the related code is only in AsyncMetaRegionTableLocator, we could make > the actually implementation pluggable, so we do not always need to go to the > active master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1985: HBASE-24588 : Submit task for NormalizationPlan (#1933)
Apache-HBase commented on pull request #1985: URL: https://github.com/apache/hbase/pull/1985#issuecomment-650411532 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 57s | branch-2.3 passed | | +1 :green_heart: | compile | 1m 44s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 6m 29s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 31s | hbase-client in branch-2.3 failed. | | -0 :warning: | javadoc | 0m 44s | hbase-server in branch-2.3 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 25s | the patch passed | | +1 :green_heart: | compile | 1m 45s | the patch passed | | +1 :green_heart: | javac | 1m 45s | the patch passed | | +1 :green_heart: | shadedjars | 6m 25s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 28s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 45s | hbase-client in the patch passed. | | -1 :x: | unit | 195m 32s | hbase-server in the patch failed. | | | | 230m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1985 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d581f34f7366 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 5e0935977d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/testReport/ | | Max. process+thread count | 2461 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1922: HBASE-24583 Normalizer can't actually merge empty regions when neighbor is larger than average size
ndimiduk commented on a change in pull request #1922:
URL: https://github.com/apache/hbase/pull/1922#discussion_r446418314
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -369,7 +369,8 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
}
final long currentSizeMb = getRegionSizeMB(current);
final long nextSizeMb = getRegionSizeMB(next);
- if (currentSizeMb + nextSizeMb < avgRegionSizeMb) {
+ // always merge away empty regions when they present themselves.
+ if (currentSizeMb == 0 || nextSizeMb == 0 || currentSizeMb + nextSizeMb
< avgRegionSizeMb) {
Review comment:
Oh, and `hbase.normalizer.min.region.count` as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1922: HBASE-24583 Normalizer can't actually merge empty regions when neighbor is larger than average size
ndimiduk commented on a change in pull request #1922:
URL: https://github.com/apache/hbase/pull/1922#discussion_r446413487
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -369,7 +369,8 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
}
final long currentSizeMb = getRegionSizeMB(current);
final long nextSizeMb = getRegionSizeMB(next);
- if (currentSizeMb + nextSizeMb < avgRegionSizeMb) {
+ // always merge away empty regions when they present themselves.
+ if (currentSizeMb == 0 || nextSizeMb == 0 || currentSizeMb + nextSizeMb
< avgRegionSizeMb) {
Review comment:
I think what we need is to unset the default values of
`hbase.normalizer.merge.min_region_age.days` and
`hbase.normalizer.merge.min_region_size.mb`. If someone wants the features,
make them opt-in.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1985: HBASE-24588 : Submit task for NormalizationPlan (#1933)
Apache-HBase commented on pull request #1985: URL: https://github.com/apache/hbase/pull/1985#issuecomment-650390234 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 31s | branch-2.3 passed | | +1 :green_heart: | compile | 1m 20s | branch-2.3 passed | | +1 :green_heart: | shadedjars | 4m 56s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 10s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 4m 58s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 18s | hbase-client in the patch passed. | | -1 :x: | unit | 152m 19s | hbase-server in the patch failed. | | | | 179m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1985 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cd30327355b4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 5e0935977d | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/testReport/ | | Max. process+thread count | 4306 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1983: HBASE-24588 : Submit task for NormalizationPlan (#1933)
Apache-HBase commented on pull request #1983: URL: https://github.com/apache/hbase/pull/1983#issuecomment-650377859 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 8s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 10s | branch-2 passed | | +1 :green_heart: | compile | 1m 33s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 44s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 30s | hbase-client in branch-2 failed. | | -0 :warning: | javadoc | 0m 38s | hbase-server in branch-2 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 55s | the patch passed | | +1 :green_heart: | compile | 1m 35s | the patch passed | | +1 :green_heart: | javac | 1m 35s | the patch passed | | +1 :green_heart: | shadedjars | 5m 46s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 27s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 21s | hbase-client in the patch passed. | | -1 :x: | unit | 130m 20s | hbase-server in the patch failed. | | | | 161m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1983 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 297bf455715e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a499eae1aa | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/testReport/ | | Max. process+thread count | 3831 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1922: HBASE-24583 Normalizer can't actually merge empty regions when neighbor is larger than average size
ndimiduk commented on a change in pull request #1922:
URL: https://github.com/apache/hbase/pull/1922#discussion_r446387518
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -369,7 +369,8 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
}
final long currentSizeMb = getRegionSizeMB(current);
final long nextSizeMb = getRegionSizeMB(next);
- if (currentSizeMb + nextSizeMb < avgRegionSizeMb) {
+ // always merge away empty regions when they present themselves.
+ if (currentSizeMb == 0 || nextSizeMb == 0 || currentSizeMb + nextSizeMb
< avgRegionSizeMb) {
Review comment:
Argh. This idea is in direct contradiction to the feature implemented by
"The minimum size for a region to be considered for a merge, in whole MBs."
```
static final String MERGE_MIN_REGION_SIZE_MB_KEY =
"hbase.normalizer.merge.min_region_size.mb";
static final int DEFAULT_MERGE_MIN_REGION_SIZE_MB = 1;
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-20819) Use TableDescriptor to replace HTableDescriptor in hbase-shell module
[ https://issues.apache.org/jira/browse/HBASE-20819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliot Miller updated HBASE-20819: -- Release Note: Removes HBase::Admin.hcd and HBase::Admin.update_htd_from_arg from hbase-shell. Removes 8 constants from HBaseAdmin in the hbase-shell module: CACHE_DATA_IN_L1, COMPARATOR, COMPARATOR_IGNORE_REPLICATION, ENCODE_ON_DISK, IS_MOB_BYTES, LENGTH, MOB_COMPACT_PARTITION_POLICY_BYTES, MOB_THRESHOLD_BYTES. (was: Removes HBase::Admin.hcd and HBase::Admin.update_htd_from_arg from hbase-shell. Removes 11 constants from HBaseAdmin in the hbase-shell module: CACHE_DATA_IN_L1, COMPARATOR, COMPARATOR_IGNORE_REPLICATION, ENCODE_ON_DISK, FOREVER, IS_MOB_BYTES, IS_ROOT, LENGTH, MOB_COMPACT_PARTITION_POLICY_BYTES, MOB_THRESHOLD_BYTES, REPLICATION_SCOPE_BYTES.) > Use TableDescriptor to replace HTableDescriptor in hbase-shell module > - > > Key: HBASE-20819 > URL: https://issues.apache.org/jira/browse/HBASE-20819 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 2.0.0 >Reporter: Xiaolin Ha >Assignee: Elliot Miller >Priority: Minor > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-20819.branch-2.001.patch, > HBASE-20819.branch-2.002.patch, > HBaseConstants-b5563432922268c7a16deacbb51bfba89c0a2aba.txt, > HBaseConstants-cf2aa593e590133b0c76d3723b4074b28b55dcc9.txt, > HBaseConstants-diff.txt > > > HTableDescriptor is deprecated as of release 2.0.0, and will be removed in > 3.0.0. This patch replaces all usages of HTableDescriptor and > HColumnDescriptor in the hbase-shell module so that HTableDescriptor can be > removed. > There a few other consequences of this change: > * Ruby methods relating to HTableDescriptor and HColumnDescriptor have been > removed. This is noted in "Release Note" on this issue. > * We no longer import constants from HTableDescriptor and HColumnDescriptor > into the ruby HBaseConstants module. Instead, we import them from > ColumnFamilyDescriptorBuilder and TableDescriptorBuilder. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-20819) Use TableDescriptor to replace HTableDescriptor in hbase-shell module
[ https://issues.apache.org/jira/browse/HBASE-20819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-20819. --- Resolution: Fixed Merged. Thanks for nice contrib [~bitoffdev] (thanks for reviews [~zhangduo] and [~busbey]) > Use TableDescriptor to replace HTableDescriptor in hbase-shell module > - > > Key: HBASE-20819 > URL: https://issues.apache.org/jira/browse/HBASE-20819 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 2.0.0 >Reporter: Xiaolin Ha >Assignee: Elliot Miller >Priority: Minor > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-20819.branch-2.001.patch, > HBASE-20819.branch-2.002.patch, > HBaseConstants-b5563432922268c7a16deacbb51bfba89c0a2aba.txt, > HBaseConstants-cf2aa593e590133b0c76d3723b4074b28b55dcc9.txt, > HBaseConstants-diff.txt > > > HTableDescriptor is deprecated as of release 2.0.0, and will be removed in > 3.0.0. This patch replaces all usages of HTableDescriptor and > HColumnDescriptor in the hbase-shell module so that HTableDescriptor can be > removed. > There a few other consequences of this change: > * Ruby methods relating to HTableDescriptor and HColumnDescriptor have been > removed. This is noted in "Release Note" on this issue. > * We no longer import constants from HTableDescriptor and HColumnDescriptor > into the ruby HBaseConstants module. Instead, we import them from > ColumnFamilyDescriptorBuilder and TableDescriptorBuilder. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
saintstack merged pull request #1959: URL: https://github.com/apache/hbase/pull/1959 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1980: HBASE-24635 Split TestMetaWithReplicas
Apache-HBase commented on pull request #1980: URL: https://github.com/apache/hbase/pull/1980#issuecomment-650358418 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 0s | master passed | | +1 :green_heart: | compile | 1m 48s | master passed | | +1 :green_heart: | shadedjars | 6m 44s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 45s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 49s | the patch passed | | +1 :green_heart: | compile | 1m 46s | the patch passed | | +1 :green_heart: | javac | 1m 46s | the patch passed | | +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 44s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 196m 12s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 1m 4s | hbase-it in the patch passed. | | | | 230m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1980/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1980 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 76906c3c3d39 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1980/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1980/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1980/1/testReport/ | | Max. process+thread count | 3507 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-it U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1980/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24645) Include dump files in the jenkins archive
Nick Dimiduk created HBASE-24645:
Summary: Include dump files in the jenkins archive
Key: HBASE-24645
URL: https://issues.apache.org/jira/browse/HBASE-24645
Project: HBase
Issue Type: Task
Components: build
Reporter: Nick Dimiduk
Tracking down jenkins test failures in the face of
{{org.apache.maven.surefire.booter.SurefireBooterForkException:
ExecutionException The forked VM terminated without properly saying goodbye. VM
crash or System.exit called?}} is difficult right now. Let's include the "dump"
files in our archives so that there's something to look at, maybe find a usable
stack trace.
The failures I want to track look like
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test) on
project hbase-common: There are test failures.
[ERROR]
[ERROR] Please refer to
/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3@2/component/hbase-common/target/surefire-reports
for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump,
[date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd
/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3@2/component/hbase-common
&& /usr/lib/jvm/jdk8u232-b09/jre/bin/java -enableassertions
-Dhbase.build.id=2020-06-24T15:53:27Z -Xmx2200m
-Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true -Djdk.net.URLClassPath.disableClassPathURLCheck=true
-Dorg.apache.hbase.thirdparty.io.netty.leakDetection.level=advanced
-Dio.netty.eventLoopThreads=3 -jar
/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3@2/component/hbase-common/target/surefire/surefirebooter9093790277421313959.jar
/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2.3@2/component/hbase-common/target/surefire
2020-06-24T15-53-31_857-jvmRun1 surefire8040542579565444758tmp
surefire_783341445535876751188tmp
[ERROR] Process Exit Code: 0
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #6: HBASE-23105: Download lib double conversion, fizz, update folly
phrocker commented on a change in pull request #6:
URL: https://github.com/apache/hbase-native-client/pull/6#discussion_r446358645
##
File path: src/hbase/client/async-batch-rpc-retrying-caller.cc
##
@@ -70,7 +70,8 @@ AsyncBatchRpcRetryingCaller::~AsyncBatchRpcRetryingCaller() {}
template
Future>> AsyncBatchRpcRetryingCaller::Call() {
GroupAndSend(actions_, 1);
- return collectAll(action2futures_);
+ // use the executor to convert he SemiFuture to a Future.
+ return std::move(collectAll(action2futures_)).via(cpu_pool_.get());
Review comment:
yeah I'm surprised to see that there too. This is something I'm usually
good at picking up in reviews of others' code, so it's kind of embarrassing
that I didn't notice this myself, but I got pretty careless when I pulled this
commit out from the original PR. Removed!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #6: HBASE-23105: Download lib double conversion, fizz, update folly
phrocker commented on a change in pull request #6:
URL: https://github.com/apache/hbase-native-client/pull/6#discussion_r446354713
##
File path: CMakeLists.txt
##
@@ -154,6 +160,9 @@ endif (OPENSSL_FOUND)
if (DOWNLOAD_DEPENDENCIES)
+ download_doubleconversion(${CMAKE_CURRENT_SOURCE_DIR}
${CMAKE_CURRENT_BINARY_DIR})
+ download_fizz(${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR})
Review comment:
I pushed a commit but will ping you after I've made some additional
changes regarding this. I'm going to re-run through some tests. thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1986: HBASE-24581 Skip compaction request/check for replica regions at the …
Apache-HBase commented on pull request #1986: URL: https://github.com/apache/hbase/pull/1986#issuecomment-650340047 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun opened a new pull request #1986: HBASE-24581 Skip compaction request/check for replica regions at the …
huaxiangsun opened a new pull request #1986: URL: https://github.com/apache/hbase/pull/1986 …early stage. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #1922: HBASE-24583 Normalizer can't actually merge empty regions when neighbor is larger than average size
huaxiangsun commented on a change in pull request #1922:
URL: https://github.com/apache/hbase/pull/1922#discussion_r446344983
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -369,7 +369,8 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
}
final long currentSizeMb = getRegionSizeMB(current);
final long nextSizeMb = getRegionSizeMB(next);
- if (currentSizeMb + nextSizeMb < avgRegionSizeMb) {
+ // always merge away empty regions when they present themselves.
+ if (currentSizeMb == 0 || nextSizeMb == 0 || currentSizeMb + nextSizeMb
< avgRegionSizeMb) {
Review comment:
Yeah, this is a good idea!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1985: HBASE-24588 : Submit task for NormalizationPlan (#1933)
Apache-HBase commented on pull request #1985: URL: https://github.com/apache/hbase/pull/1985#issuecomment-650329335 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.3 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 49s | branch-2.3 passed | | +1 :green_heart: | checkstyle | 1m 48s | branch-2.3 passed | | +1 :green_heart: | spotbugs | 3m 9s | branch-2.3 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | -0 :warning: | checkstyle | 1m 14s | hbase-server: The patch generated 2 new + 103 unchanged - 2 fixed = 105 total (was 105) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 6s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 20s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 46m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1985 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux c3007ceb4f59 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.3 / 5e0935977d | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1985/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #1933: HBASE-24588 : Submit task for NormalizationPlan
virajjasani commented on pull request #1933: URL: https://github.com/apache/hbase/pull/1933#issuecomment-650329509 > Looks like there were consistent unit test failures in TestNormalizerOnCluster. let's keep an eye on it. Yeah, they pass locally all the time, on the build, it was [failed-to-read] xml error. So yes, will keep an eye. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun merged pull request #1971: Backport: HBASE-24552 Replica region needs to check if primary region directory…
huaxiangsun merged pull request #1971: URL: https://github.com/apache/hbase/pull/1971 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #1971: Backport: HBASE-24552 Replica region needs to check if primary region directory…
huaxiangsun commented on pull request #1971: URL: https://github.com/apache/hbase/pull/1971#issuecomment-650329376 This is the failure, jvm exits, it does not say which test case caused the exit. Do not think it is related with my patch. ``` [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test) on project hbase-server: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1971/yetus-jdk11-hadoop3-check/src/hbase-server/target/surefire-reports for the individual test results. [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1971/yetus-jdk11-hadoop3-check/src/hbase-server && /usr/lib/jvm/jdk-11.0.6+10/bin/java --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED -enableassertions -Dhbase.build.id=2020-06-25T17:24:59Z -Xmx2200m -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djdk.net.URLClassPath.disableClassPathURLCheck=true -Dorg.apache.hbase.thirdparty.io.netty.leakDetection.level=advanced -Dio.netty.eventLoopThreads=3 -jar /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1971/yetus-jdk11-hadoop3-check/src/hbase-server/target/surefire/surefirebooter6724336873935557668.jar /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1971/yetus-jdk11-hadoop3-check/src/hbase-server/target/surefire 2020-06-25T17-26-21_545-jvmRun3 surefire6655336917994257031tmp surefire_89311486818001122343460tmp [ERROR] Process Exit Code: 0 [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1971/yetus-jdk11-hadoop3-check/src/hbase-server && /usr/lib/jvm/jdk-11.0.6+10/bin/java --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED -enableassertions -Dhbase.build.id=2020-06-25T17:24:59Z -Xmx2200m -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djdk.net.URLClassPath.disableClassPathURLCheck=true -Dorg.apache.hbase.thirdparty.io.netty.leakDetection.level=advanced -Dio.netty.eventLoopThreads=3 -jar /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1971/yetus-jdk11-hadoop3-check/src/hbase-server/target/surefire/surefirebooter6724336873935557668.jar /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1971/yetus-jdk11-hadoop3-check/src/hbase-server/target/surefire 2020-06-25T17-26-21_545-jvmRun3 surefire6655336917994257031tmp surefire_89311486818001122343460tmp [ERROR] Process Exit Code: 0 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:513) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:460) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:301) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:249) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1217) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1063) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:889) [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117) [ERROR] at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:190) [ERROR] at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:186) [ERROR] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [ERROR] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [ERROR] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [ERROR] at
[GitHub] [hbase] Apache-HBase commented on pull request #1980: HBASE-24635 Split TestMetaWithReplicas
Apache-HBase commented on pull request #1980: URL: https://github.com/apache/hbase/pull/1980#issuecomment-650327992 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 40s | master passed | | +1 :green_heart: | compile | 1m 20s | master passed | | +1 :green_heart: | shadedjars | 5m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 23s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 5m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 136m 41s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 1m 10s | hbase-it in the patch passed. | | | | 163m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1980/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1980 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fb52b67d835f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1980/1/testReport/ | | Max. process+thread count | 4615 (vs. ulimit of 12500) | | modules | C: hbase-server hbase-it U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1980/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun merged pull request #1972: Backport: HBASE-24552 Replica region needs to check if primary region directory…
huaxiangsun merged pull request #1972: URL: https://github.com/apache/hbase/pull/1972 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
Apache-HBase commented on pull request #1959: URL: https://github.com/apache/hbase/pull/1959#issuecomment-650322428 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 21s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | checkstyle | 0m 42s | master passed | | +1 :green_heart: | spotbugs | 1m 17s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 30s | the patch passed | | -0 :warning: | checkstyle | 0m 31s | hbase-client: The patch generated 1 new + 27 unchanged - 0 fixed = 28 total (was 27) | | -0 :warning: | rubocop | 0m 22s | The patch generated 117 new + 478 unchanged - 95 fixed = 595 total (was 573) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 14m 5s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 1m 33s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. | | | | 38m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1959 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle rubocop | | uname | Linux 25e181d9d92e 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 71af97cd05 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | rubocop | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1983: HBASE-24588 : Submit task for NormalizationPlan (#1933)
Apache-HBase commented on pull request #1983: URL: https://github.com/apache/hbase/pull/1983#issuecomment-650321039 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 36s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 41s | branch-2 passed | | +1 :green_heart: | spotbugs | 3m 9s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 13s | the patch passed | | -0 :warning: | checkstyle | 1m 6s | hbase-server: The patch generated 2 new + 103 unchanged - 2 fixed = 105 total (was 105) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 39s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 38m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1983 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux e84d5745b889 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / a499eae1aa | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1983/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
Apache-HBase commented on pull request #1959: URL: https://github.com/apache/hbase/pull/1959#issuecomment-650320797 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 23s | master passed | | +1 :green_heart: | compile | 0m 54s | master passed | | +1 :green_heart: | shadedjars | 5m 51s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 29s | hbase-client in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 13s | the patch passed | | +1 :green_heart: | compile | 0m 51s | the patch passed | | +1 :green_heart: | javac | 0m 51s | the patch passed | | +1 :green_heart: | shadedjars | 6m 25s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 29s | hbase-client in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 15s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 7m 4s | hbase-shell in the patch passed. | | | | 35m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1959 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c8ad984b907c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 71af97cd05 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/testReport/ | | Max. process+thread count | 2263 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
Apache-HBase commented on pull request #1959: URL: https://github.com/apache/hbase/pull/1959#issuecomment-650319359 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 29s | master passed | | +1 :green_heart: | compile | 0m 45s | master passed | | +1 :green_heart: | shadedjars | 5m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 35s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 3s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 7m 11s | hbase-shell in the patch passed. | | | | 31m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1959 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dd6dee58bc37 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 71af97cd05 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/testReport/ | | Max. process+thread count | 2246 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1984: HBASE-24644 Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
Apache-HBase commented on pull request #1984: URL: https://github.com/apache/hbase/pull/1984#issuecomment-650318684 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 21s | master passed | | +0 :ok: | refguide | 5m 33s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 5m 51s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 22m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1984/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1984 | | Optional Tests | dupname asflicense refguide | | uname | Linux 02c0823a279e 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 51ad588565 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1984/1/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1984/1/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 66 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1984/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz edited a comment on pull request #1883: HBASE-24530 Introduce a split policy similar with SteppingSplitPolicy…
bsglz edited a comment on pull request #1883: URL: https://github.com/apache/hbase/pull/1883#issuecomment-650316101 > @bsglz , sorry for the delay. So majority of people on the DISCUSS thread opined towards #3. I think @busbey suggestion to fix IncreasingToUpperBoundRegionSplitPolicy for minor updates only is reasonable, so maybe we can merge this current one to branch-2.2 and branch-2.3, then for master and branch-2 we fix IncreasingToUpperBoundRegionSplitPolicy. Ok, will do it in a follow-on issue. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on pull request #1883: HBASE-24530 Introduce a split policy similar with SteppingSplitPolicy…
bsglz commented on pull request #1883: URL: https://github.com/apache/hbase/pull/1883#issuecomment-650316101 > @bsglz , sorry for the delay. So majority of people on the DISCUSS thread opined towards #3. I think @busbey suggestion to fix IncreasingToUpperBoundRegionSplitPolicy for minor updates only is reasonable, so maybe we can merge this current one to branch-2.2 and branch-2.3, then for master and branch-2 we fix IncreasingToUpperBoundRegionSplitPolicy. Ok, will do it in a follow-on issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1979: HBASE-24639 RequestId Tracing Feature for Hbase
Apache-HBase commented on pull request #1979: URL: https://github.com/apache/hbase/pull/1979#issuecomment-650311420 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 20s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1.4 Compile Tests _ | | +0 :ok: | mvndep | 2m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 7m 43s | branch-1.4 passed | | +1 :green_heart: | compile | 1m 28s | branch-1.4 passed with JDK v1.8.0_252 | | +1 :green_heart: | compile | 1m 39s | branch-1.4 passed with JDK v1.7.0_262 | | +1 :green_heart: | checkstyle | 5m 31s | branch-1.4 passed | | +1 :green_heart: | shadedjars | 3m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 7s | branch-1.4 passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 1m 21s | branch-1.4 passed with JDK v1.7.0_262 | | +0 :ok: | spotbugs | 2m 48s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 6m 44s | branch-1.4 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 4s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | cc | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | compile | 1m 39s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | cc | 1m 39s | the patch passed | | +1 :green_heart: | javac | 1m 39s | the patch passed | | -1 :x: | checkstyle | 0m 44s | hbase-client: The patch generated 26 new + 351 unchanged - 5 fixed = 377 total (was 356) | | -1 :x: | checkstyle | 1m 44s | hbase-server: The patch generated 20 new + 90 unchanged - 2 fixed = 110 total (was 92) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 3m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 2m 21s | Patch does not cause any errors with Hadoop 2.7.7. | | +1 :green_heart: | hbaseprotoc | 1m 54s | the patch passed | | +1 :green_heart: | javadoc | 1m 0s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 1m 18s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | findbugs | 7m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 28s | hbase-protocol in the patch passed. | | -1 :x: | unit | 2m 24s | hbase-client in the patch failed. | | -1 :x: | unit | 148m 11s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 1m 1s | The patch does not generate ASF License warnings. | | | | 216m 21s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestAsyncProcess | | | hadoop.hbase.regionserver.TestRegionServerMetrics | | | hadoop.hbase.client.TestReplicasClient | | | hadoop.hbase.client.TestRpcControllerFactory | | | hadoop.hbase.client.TestMalformedCellFromClient | | | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles | | | hadoop.hbase.client.TestSnapshotCloneIndependence | | | hadoop.hbase.coprocessor.TestHTableWrapper | | | hadoop.hbase.coprocessor.TestBatchCoprocessorEndpoint | | | hadoop.hbase.client.TestFromClientSide3 | | | hadoop.hbase.client.TestMultiParallel | | | hadoop.hbase.client.TestMultiRespectsLimits | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1979/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1979 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux 838b63b04572 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | |
[GitHub] [hbase] ndimiduk commented on a change in pull request #1922: HBASE-24583 Normalizer can't actually merge empty regions when neighbor is larger than average size
ndimiduk commented on a change in pull request #1922:
URL: https://github.com/apache/hbase/pull/1922#discussion_r446324754
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -369,7 +369,8 @@ private boolean skipForMerge(final RegionStates
regionStates, final RegionInfo r
}
final long currentSizeMb = getRegionSizeMB(current);
final long nextSizeMb = getRegionSizeMB(next);
- if (currentSizeMb + nextSizeMb < avgRegionSizeMb) {
+ // always merge away empty regions when they present themselves.
+ if (currentSizeMb == 0 || nextSizeMb == 0 || currentSizeMb + nextSizeMb
< avgRegionSizeMb) {
Review comment:
@huaxiangsun I've been thinking about this and haven't come to a formula
that I like. So let me propose this: rather than basing this decision on a
scaling factor, how about we decide on some arbitrary value to use as meaning
"effectively 0" in size. Say, any region <= 10mb will be considered as
"effectively 0-sized" and we'll merge it into it's neighbor when possible. I
supposed we can make this configurable as well...
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1922: HBASE-24583 Normalizer can't actually merge empty regions when neighbor is larger than average size
ndimiduk commented on a change in pull request #1922:
URL: https://github.com/apache/hbase/pull/1922#discussion_r446323076
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
##
@@ -78,4 +82,30 @@ public void execute(Admin admin) {
LOG.error("Error during region merge: ", ex);
}
}
+
+ @Override
+ public boolean equals(Object o) {
+if (this == o) {
+ return true;
+}
+
+if (o == null || getClass() != o.getClass()) {
+ return false;
+}
+
+MergeNormalizationPlan that = (MergeNormalizationPlan) o;
+
+return new EqualsBuilder()
+ .append(firstRegion, that.firstRegion)
+ .append(secondRegion, that.secondRegion)
+ .isEquals();
+ }
+
+ @Override
+ public int hashCode() {
+return new HashCodeBuilder(17, 37)
Review comment:
From
[javadoc](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#equals(java.lang.Object))
on `Object#equals(Object)`
> Note that it is generally necessary to override the `hashCode` method
whenever this method is overridden, so as to maintain the general contract for
the `hashCode` method, which states that equal objects must have equal hash
codes.
And besides since I want to treat these objects like POJOs, it is useful to
be able to use them in maps and sets.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1984: HBASE-24644 Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
Apache-HBase commented on pull request #1984: URL: https://github.com/apache/hbase/pull/1984#issuecomment-650309424 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1922: HBASE-24583 Normalizer can't actually merge empty regions when neighbor is larger than average size
ndimiduk commented on a change in pull request #1922:
URL: https://github.com/apache/hbase/pull/1922#discussion_r446319764
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
##
@@ -78,4 +82,30 @@ public void execute(Admin admin) {
LOG.error("Error during region merge: ", ex);
}
}
+
+ @Override
+ public boolean equals(Object o) {
+if (this == o) {
+ return true;
+}
+
+if (o == null || getClass() != o.getClass()) {
+ return false;
+}
+
+MergeNormalizationPlan that = (MergeNormalizationPlan) o;
+
+return new EqualsBuilder()
+ .append(firstRegion, that.firstRegion)
+ .append(secondRegion, that.secondRegion)
+ .isEquals();
+ }
+
+ @Override
+ public int hashCode() {
Review comment:
Actually, I think i'd like to keep them. I'm finding they're handy for
some of the experimental changes I've been trying. I'd prefer keep them for now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
Apache-HBase commented on pull request #1959: URL: https://github.com/apache/hbase/pull/1959#issuecomment-650304489 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 47s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 3s | master passed | | +1 :green_heart: | checkstyle | 0m 44s | master passed | | +1 :green_heart: | spotbugs | 1m 16s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 38s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-client: The patch generated 1 new + 27 unchanged - 0 fixed = 28 total (was 27) | | -0 :warning: | rubocop | 0m 23s | The patch generated 116 new + 478 unchanged - 95 fixed = 594 total (was 573) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 14m 44s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 1m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 41m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1959 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle rubocop | | uname | Linux 691d482fbab7 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 71af97cd05 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | rubocop | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work started] (HBASE-24644) Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
[ https://issues.apache.org/jira/browse/HBASE-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24644 started by Nick Dimiduk. > Add a clause to the book noting that sometimes we short-circuit the > deprecation cycle > - > > Key: HBASE-24644 > URL: https://issues.apache.org/jira/browse/HBASE-24644 > Project: HBase > Issue Type: Task > Components: community, documentation >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > Let's add a note to the book that describes the circumstances around > HBASE-21782 and how that can result in code not following our stated > deprecation cycle guidelines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21782) LoadIncrementalHFiles should not be IA.Public
[ https://issues.apache.org/jira/browse/HBASE-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146529#comment-17146529 ] Nick Dimiduk commented on HBASE-21782: -- And there's a release note as well. Yes, I think this issue is will documented. Filed HBASE-24644 and posted a patch. > LoadIncrementalHFiles should not be IA.Public > - > > Key: HBASE-21782 > URL: https://issues.apache.org/jira/browse/HBASE-21782 > Project: HBase > Issue Type: Task > Components: mapreduce >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: bulkload > Fix For: 3.0.0-alpha-1, 2.2.0 > > Attachments: HBASE-21782-v1.patch, HBASE-21782.patch > > > It is an implementation class, so some of the methods which are only supposed > to be used by replication sink are also public to users. And it exposes > methods which take Table and Connection as parameter and inside the > implementation we assume that they are HTable and ConnectionImplementation, > which will be a pain when we want to replace the sync client implementation > with async client. > Here I think we should make the implementation class as > IA.LimitPrivate(TOOL), and introduce an interface for bulking hfiles > programmatically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani opened a new pull request #1985: HBASE-24588 : Submit task for NormalizationPlan (#1933)
virajjasani opened a new pull request #1985: URL: https://github.com/apache/hbase/pull/1985 Signed-off-by: Nick Dimiduk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #1984: HBASE-24644 Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
ndimiduk opened a new pull request #1984: URL: https://github.com/apache/hbase/pull/1984 Here's an exception where we willingly don't follow the guidelines as stated. Let's make that clear. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24562) Stabilize master startup with meta replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-24562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-24562: - Affects Version/s: 2.4.0 2.3.0 2.2.5 > Stabilize master startup with meta replicas enabled > --- > > Key: HBASE-24562 > URL: https://issues.apache.org/jira/browse/HBASE-24562 > Project: HBase > Issue Type: Improvement > Components: meta, read replicas >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.5 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0 > > > This is related to HBASE-21624 . > I created a separate ticket because in the original one a "complete solution > for meta replicas" was requested and this is not one. I'm just trying to make > master startup more stable by making assigning meta replicas asynchronous and > preventing a potential assignment failure from crashing master. > The idea is that starting master with less or even no meta replicas assigned > is preferable to not having a running master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24562) Stabilize master startup with meta replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-24562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146526#comment-17146526 ] Wellington Chevreuil commented on HBASE-24562: -- Had fixed conflicts and pushed into branch-2 and branch-2.3. Tried on branch-2.2 as well, but got TestMetaWithReplicas.testFailedReplicaAssigment test failing. [~bszabolcs], could you make a branch-2.2 compatible PR? > Stabilize master startup with meta replicas enabled > --- > > Key: HBASE-24562 > URL: https://issues.apache.org/jira/browse/HBASE-24562 > Project: HBase > Issue Type: Improvement > Components: meta, read replicas >Affects Versions: 3.0.0-alpha-1 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0-alpha-1 > > > This is related to HBASE-21624 . > I created a separate ticket because in the original one a "complete solution > for meta replicas" was requested and this is not one. I'm just trying to make > master startup more stable by making assigning meta replicas asynchronous and > preventing a potential assignment failure from crashing master. > The idea is that starting master with less or even no meta replicas assigned > is preferable to not having a running master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24644) Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
[ https://issues.apache.org/jira/browse/HBASE-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reassigned HBASE-24644: Assignee: Nick Dimiduk > Add a clause to the book noting that sometimes we short-circuit the > deprecation cycle > - > > Key: HBASE-24644 > URL: https://issues.apache.org/jira/browse/HBASE-24644 > Project: HBase > Issue Type: Task > Components: community, documentation >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > > Let's add a note to the book that describes the circumstances around > HBASE-21782 and how that can result in code not following our stated > deprecation cycle guidelines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24562) Stabilize master startup with meta replicas enabled
[ https://issues.apache.org/jira/browse/HBASE-24562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-24562: - Fix Version/s: 2.4.0 2.3.1 > Stabilize master startup with meta replicas enabled > --- > > Key: HBASE-24562 > URL: https://issues.apache.org/jira/browse/HBASE-24562 > Project: HBase > Issue Type: Improvement > Components: meta, read replicas >Affects Versions: 3.0.0-alpha-1 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0 > > > This is related to HBASE-21624 . > I created a separate ticket because in the original one a "complete solution > for meta replicas" was requested and this is not one. I'm just trying to make > master startup more stable by making assigning meta replicas asynchronous and > preventing a potential assignment failure from crashing master. > The idea is that starting master with less or even no meta replicas assigned > is preferable to not having a running master. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24644) Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
Nick Dimiduk created HBASE-24644: Summary: Add a clause to the book noting that sometimes we short-circuit the deprecation cycle Key: HBASE-24644 URL: https://issues.apache.org/jira/browse/HBASE-24644 Project: HBase Issue Type: Task Components: community, documentation Reporter: Nick Dimiduk Let's add a note to the book that describes the circumstances around HBASE-21782 and how that can result in code not following our stated deprecation cycle guidelines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani opened a new pull request #1983: HBASE-24588 : Submit task for NormalizationPlan (#1933)
virajjasani opened a new pull request #1983: URL: https://github.com/apache/hbase/pull/1983 Signed-off-by: Nick Dimiduk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24644) Add a clause to the book noting that sometimes we short-circuit the deprecation cycle
[
https://issues.apache.org/jira/browse/HBASE-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146525#comment-17146525
]
Nick Dimiduk commented on HBASE-24644:
--
{quote}
Usually we should keep the deprecated classes for a whole major release, as
what we describe in the ref guide.
But here I define the IA.Public annotation on LoadIncrementalHFiles as a
mistake, especially that it exposes several IA.Private classes, so I used a
more aggressive policy to drop the class.
IIRC I've gotten consensus from the community. We should add a special note on
this in our ref guide too.
{quote}
> Add a clause to the book noting that sometimes we short-circuit the
> deprecation cycle
> -
>
> Key: HBASE-24644
> URL: https://issues.apache.org/jira/browse/HBASE-24644
> Project: HBase
> Issue Type: Task
> Components: community, documentation
>Reporter: Nick Dimiduk
>Priority: Minor
>
> Let's add a note to the book that describes the circumstances around
> HBASE-21782 and how that can result in code not following our stated
> deprecation cycle guidelines.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] wchevreuil commented on pull request #1883: HBASE-24530 Introduce a split policy similar with SteppingSplitPolicy…
wchevreuil commented on pull request #1883: URL: https://github.com/apache/hbase/pull/1883#issuecomment-650299753 @bsglz , sorry for the delay. So majority of people on the DISCUSS thread opined towards #3. I think @busbey suggestion to fix IncreasingToUpperBoundRegionSplitPolicy for minor updates only is reasonable, so maybe we can merge this current one to branch-2.2 and branch-2.3, then for master and branch-2 we fix IncreasingToUpperBoundRegionSplitPolicy. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24643) Replace Cluster#primariesOfRegionsPerServer from int array to treemap
Huaxiang Sun created HBASE-24643: Summary: Replace Cluster#primariesOfRegionsPerServer from int array to treemap Key: HBASE-24643 URL: https://issues.apache.org/jira/browse/HBASE-24643 Project: HBase Issue Type: Improvement Components: Balancer Affects Versions: 2.3.0 Reporter: Huaxiang Sun Assignee: Huaxiang Sun Currently, primariesOfRegionsPerServer is an int array, moveRegion does heavy work by searching the array (linearly) and insert/remove an element requires allocating/copying the whole array. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
Apache-HBase commented on pull request #1959: URL: https://github.com/apache/hbase/pull/1959#issuecomment-650297656 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 15s | master passed | | +1 :green_heart: | compile | 0m 50s | master passed | | +1 :green_heart: | shadedjars | 5m 45s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 28s | hbase-client in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 5m 46s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 26s | hbase-client in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 9s | hbase-client in the patch passed. | | -1 :x: | unit | 1m 41s | hbase-shell in the patch failed. | | | | 28m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1959 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 14d0db47cea6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 71af97cd05 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-shell.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/testReport/ | | Max. process+thread count | 2129 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
Apache-HBase commented on pull request #1959: URL: https://github.com/apache/hbase/pull/1959#issuecomment-650296731 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 26s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 5m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 5m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 2s | hbase-client in the patch passed. | | -1 :x: | unit | 2m 11s | hbase-shell in the patch failed. | | | | 26m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1959 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dc1f385806fd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 71af97cd05 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-shell.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/testReport/ | | Max. process+thread count | 2106 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1959/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24588) Normalizer plan execution is not consistent between plan types
[
https://issues.apache.org/jira/browse/HBASE-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146500#comment-17146500
]
Viraj Jasani commented on HBASE-24588:
--
True, let me create two backport PRs for branch-2 and branch-2.3. After QA
build results, we can merge them.
> Normalizer plan execution is not consistent between plan types
> --
>
> Key: HBASE-24588
> URL: https://issues.apache.org/jira/browse/HBASE-24588
> Project: HBase
> Issue Type: Bug
> Components: master, Normalizer
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> I left a comment on a merged
> [commit|https://github.com/apache/hbase/commit/5d0e0fc5fd09bddb2d766d1e24e28e472961f454#r39987289]
> where a little discussion has blossomed. Right now the normalizer produces
> two types of plans: "split" or "merge". The master receives the list of plans
> and executes them in a simple loop. The way it does the actual execution is
> delegated off to the plan implementation.
> The bug I noticed is that the two implementations are subtly different. Both
> use async APIs to submit procedures, but "split" blocks on completion while
> "merge" does not. Furthermore, because "split" blocks, it's able to capture
> any exception that's thrown, while "merge" cannot.
> These implementations should be made consistent. My thinking at the moment is
> this {{execute}} method should instead be named {{submit}}, creating and
> returning the {{Future}} that represents whatever work it submitted, and the
> calling context should handle the resolution of those futures in a single
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24588) Normalizer plan execution is not consistent between plan types
[
https://issues.apache.org/jira/browse/HBASE-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146494#comment-17146494
]
Nick Dimiduk commented on HBASE-24588:
--
This is a good fix as the existing behavior is broken. +1 for branch-2.3 and
2.3.0rc1.
> Normalizer plan execution is not consistent between plan types
> --
>
> Key: HBASE-24588
> URL: https://issues.apache.org/jira/browse/HBASE-24588
> Project: HBase
> Issue Type: Bug
> Components: master, Normalizer
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> I left a comment on a merged
> [commit|https://github.com/apache/hbase/commit/5d0e0fc5fd09bddb2d766d1e24e28e472961f454#r39987289]
> where a little discussion has blossomed. Right now the normalizer produces
> two types of plans: "split" or "merge". The master receives the list of plans
> and executes them in a simple loop. The way it does the actual execution is
> delegated off to the plan implementation.
> The bug I noticed is that the two implementations are subtly different. Both
> use async APIs to submit procedures, but "split" blocks on completion while
> "merge" does not. Furthermore, because "split" blocks, it's able to capture
> any exception that's thrown, while "merge" cannot.
> These implementations should be made consistent. My thinking at the moment is
> this {{execute}} method should instead be named {{submit}}, creating and
> returning the {{Future}} that represents whatever work it submitted, and the
> calling context should handle the resolution of those futures in a single
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on pull request #1933: HBASE-24588 : Submit task for NormalizationPlan
ndimiduk commented on pull request #1933: URL: https://github.com/apache/hbase/pull/1933#issuecomment-650292872 Looks like there were consistent unit test failures in TestNormalizerOnCluster. let's keep an eye on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-native-client] bharathv commented on a change in pull request #6: HBASE-23105: Download lib double conversion, fizz, update folly
bharathv commented on a change in pull request #6: URL: https://github.com/apache/hbase-native-client/pull/6#discussion_r446305722 ## File path: cmake/DownloadDoubleConversion.cmake ## @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +## Download facebook's folly library. +## SOURCE_DIR is typically the cmake source directory +function(download_doubleconversion SOURCE_DIR BUILD_DIR) Review comment: Here you go. https://issues.apache.org/jira/browse/HBASE-24641. I think this one makes everyone's life easier :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24641) Add linter checks for patches
[ https://issues.apache.org/jira/browse/HBASE-24641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146488#comment-17146488 ] Bharath Vissapragada commented on HBASE-24641: -- [~phrocker] FYI. Also, do you have a preference for a static analysis tool? > Add linter checks for patches > - > > Key: HBASE-24641 > URL: https://issues.apache.org/jira/browse/HBASE-24641 > Project: HBase > Issue Type: Sub-task > Components: Client, native-client >Reporter: Bharath Vissapragada >Priority: Major > > I've worked with clang-tidy before and I think it works well. There is also a > [helper > script|https://clang.llvm.org/extra/doxygen/clang-tidy-diff_8py_source.html] > that runs clang-tidy on a patch rather than the entire source tree. > - pull in clang as a dependency > - Define .clang-format > - Run clang-tidy-diff on each checkin > There are also other tools like cpplint from Google. I don't have any > preference, so either works for me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24642) Apache Yetus integration
Bharath Vissapragada created HBASE-24642: Summary: Apache Yetus integration Key: HBASE-24642 URL: https://issues.apache.org/jira/browse/HBASE-24642 Project: HBase Issue Type: Sub-task Reporter: Bharath Vissapragada Assignee: Bharath Vissapragada Now that we have a clean test run with all the tests passing on trunk (with HBase trunk) , lets get the Yetus integration with test-patch utility working. That makes it much easier to implement a precommit with github. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24641) Add linter checks for patches
Bharath Vissapragada created HBASE-24641: Summary: Add linter checks for patches Key: HBASE-24641 URL: https://issues.apache.org/jira/browse/HBASE-24641 Project: HBase Issue Type: Sub-task Components: Client, native-client Reporter: Bharath Vissapragada I've worked with clang-tidy before and I think it works well. There is also a [helper script|https://clang.llvm.org/extra/doxygen/clang-tidy-diff_8py_source.html] that runs clang-tidy on a patch rather than the entire source tree. - pull in clang as a dependency - Define .clang-format - Run clang-tidy-diff on each checkin There are also other tools like cpplint from Google. I don't have any preference, so either works for me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] busbey commented on a change in pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
busbey commented on a change in pull request #1959: URL: https://github.com/apache/hbase/pull/1959#discussion_r446299722 ## File path: hbase-shell/src/main/ruby/hbase_constants.rb ## @@ -109,8 +109,8 @@ def self.promote_constants(constants) end end - promote_constants(org.apache.hadoop.hbase.HColumnDescriptor.constants) - promote_constants(org.apache.hadoop.hbase.HTableDescriptor.constants) + promote_constants(org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder.constants) Review comment: That's great work, thanks! Analysis looks reasonable to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bitoffdev commented on a change in pull request #1959: HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module
bitoffdev commented on a change in pull request #1959: URL: https://github.com/apache/hbase/pull/1959#discussion_r446296466 ## File path: hbase-shell/src/main/ruby/hbase_constants.rb ## @@ -109,8 +109,8 @@ def self.promote_constants(constants) end end - promote_constants(org.apache.hadoop.hbase.HColumnDescriptor.constants) - promote_constants(org.apache.hadoop.hbase.HTableDescriptor.constants) + promote_constants(org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder.constants) Review comment: I just pushed a commit that adds back three constants, but leaves the rest out. I left a [comment on Jira](https://issues.apache.org/jira/browse/HBASE-20819?focusedCommentId=17146470#comment-17146470) with my rationale for each of the constants. **Let me know if you disagree with any of my conclusions, and I can easily add more constants back** . Specifically for the [MOB constants](https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ColumnFamilyDescriptorBuilder.java#L163), we added setters for all the MOB key-value pairs to our builder. I think including these _private_ constants will encourage users to go against our defined interface. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24588) Normalizer plan execution is not consistent between plan types
[
https://issues.apache.org/jira/browse/HBASE-24588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146475#comment-17146475
]
Viraj Jasani commented on HBASE-24588:
--
Thank you [~ndimiduk] for the review.
I believe we can have this in 2.3, but I am fine with 2.3.1 also, you have
already taken so many such last moment Jiras and I don't want to give more
headache :)
So let me keep this till branch-2 for now?
> Normalizer plan execution is not consistent between plan types
> --
>
> Key: HBASE-24588
> URL: https://issues.apache.org/jira/browse/HBASE-24588
> Project: HBase
> Issue Type: Bug
> Components: master, Normalizer
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> I left a comment on a merged
> [commit|https://github.com/apache/hbase/commit/5d0e0fc5fd09bddb2d766d1e24e28e472961f454#r39987289]
> where a little discussion has blossomed. Right now the normalizer produces
> two types of plans: "split" or "merge". The master receives the list of plans
> and executes them in a simple loop. The way it does the actual execution is
> delegated off to the plan implementation.
> The bug I noticed is that the two implementations are subtly different. Both
> use async APIs to submit procedures, but "split" blocks on completion while
> "merge" does not. Furthermore, because "split" blocks, it's able to capture
> any exception that's thrown, while "merge" cannot.
> These implementations should be made consistent. My thinking at the moment is
> this {{execute}} method should instead be named {{submit}}, creating and
> returning the {{Future}} that represents whatever work it submitted, and the
> calling context should handle the resolution of those futures in a single
> place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] virajjasani merged pull request #1933: HBASE-24588 : Submit task for NormalizationPlan
virajjasani merged pull request #1933: URL: https://github.com/apache/hbase/pull/1933 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-20819) Use TableDescriptor to replace HTableDescriptor in hbase-shell module
[
https://issues.apache.org/jira/browse/HBASE-20819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146470#comment-17146470
]
Elliot Miller commented on HBASE-20819:
---
h2. Which Constants Should We Keep?
Below are my opinions on which constants we should keep, based on my last
comment.
h3. L1 Cache Constants
* {color:#de350b}CACHE_DATA_IN_L1{color}
** Fact: Zero usages in HBase repo (via grep)
** Fact: This constant pertains to a feature that is marked for removal in
HBase 3.0.0
*** "From HBase 2.0.0 onwards, the notions of L1 and L2 have been deprecated."
[See HBase Book|http://hbase.apache.org/book.html#offheap.blockcache]
** Conclusion: Delete it!
h3. Comparator Constants
Both of these constants (while public) are marked with
{{@InterfaceAudience.Private}}, and seem to be for internal use only. I'm
inclined to leave them out.
* {color:#de350b}COMPARATOR{color}
* {color:#de350b}COMPARATOR_IGNORE_REPLICATION{color}
h3. MOB (Medium OBjects) Constants
All three of these constants were copied from HTableDescriptor to
ColumnFamilyDescriptorBuilder. In this process, they went from public to
private and three new setters were added to ColumnFamilyDescriptorBuilder:
setMobCompactPartitionPolicy, setMobEnabled, and setMobThreshold. *I worry that
if we include the constants in the shell, it will encourage users to circumvent
our new builder interface setters and directly mess with values.* For that
reason, I think we should not include these constants.
* {color:#ff}IS_MOB_BYTES{color}
* {color:#ff}MOB_COMPACT_PARTITION_POLICY_BYTES{color}
* {color:#ff}MOB_THRESHOLD_BYTES{color}
h3. Replication Scope Constants
* {color:#00875a}REPLICATION_SCOPE_BYTES{color}
** Copied from HColumnDescriptor to ColumnFamilyDescriptor and it's public! We
should *definitely keep this one*.
h3. MISC Constants
* {color:#de350b}ENCODE_ON_DISK{color}
** Zero usages in entire HBase repo
** No longer used... In fact, there was a [comment on the definition
line|https://github.com/apache/hbase/blob/fe5865d9e3ffc7b0c0fdef4980295f6ca07298a2/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java#L74]
added in 2013 stating that it was no longer used
* {color:#00875a}FOREVER{color}
** Moved to HConstants.FOREVER
* {color:#00875a}IS_ROOT{color}
** Used in TableSchemaModel... I'm guessing the constant will be moved to
HConstants in some follow-up work to remove HTableDescriptor from the entire
repo before the 3.0.0 release. For now, I'll keep it.
* {color:#de350b}LENGTH{color}
** As far as I can tell, it hasn't been in use since
[HBASE-1304|https://github.com/apache/hbase/commit/6af4292630daca370de7fce3a2b2b3401cd1abfa]
(2009)
> Use TableDescriptor to replace HTableDescriptor in hbase-shell module
> -
>
> Key: HBASE-20819
> URL: https://issues.apache.org/jira/browse/HBASE-20819
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Elliot Miller
>Priority: Minor
> Fix For: 3.0.0-alpha-1
>
> Attachments: HBASE-20819.branch-2.001.patch,
> HBASE-20819.branch-2.002.patch,
> HBaseConstants-b5563432922268c7a16deacbb51bfba89c0a2aba.txt,
> HBaseConstants-cf2aa593e590133b0c76d3723b4074b28b55dcc9.txt,
> HBaseConstants-diff.txt
>
>
> HTableDescriptor is deprecated as of release 2.0.0, and will be removed in
> 3.0.0. This patch replaces all usages of HTableDescriptor and
> HColumnDescriptor in the hbase-shell module so that HTableDescriptor can be
> removed.
> There a few other consequences of this change:
> * Ruby methods relating to HTableDescriptor and HColumnDescriptor have been
> removed. This is noted in "Release Note" on this issue.
> * We no longer import constants from HTableDescriptor and HColumnDescriptor
> into the ruby HBaseConstants module. Instead, we import them from
> ColumnFamilyDescriptorBuilder and TableDescriptorBuilder.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
