[GitHub] [hbase] Apache-HBase commented on pull request #1789: HBASE-22738 Fallback to default group to choose RS when there are no …
Apache-HBase commented on pull request #1789: URL: https://github.com/apache/hbase/pull/1789#issuecomment-635120785 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | shadedjars | 5m 47s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 5m 37s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 38s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 128m 12s | hbase-server in the patch failed. | | | | 153m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1789/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1789 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ee95cadd84ef 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1789/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1789/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1789/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1789/2/testReport/ | | Max. process+thread count | 3866 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1789/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24451) Remove the HasThread because the related bug had been fixed since jdk7
[ https://issues.apache.org/jira/browse/HBASE-24451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Wang updated HBASE-24451: --- Description: The HasThread was introduced as a wrapper of Thread to workaround a deadlock bug, and this bug had fixed since jdk7, so I think it is the time to remove it as we are using jdk8+. Related links: [https://issues.apache.org/jira/browse/HBASE-4367|https://slack-redir.net/link?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHBASE-4367] [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6915621|https://slack-redir.net/link?url=https%3A%2F%2Fbugs.java.com%2Fbugdatabase%2Fview_bug.do%3Fbug_id%3D6915621] was: The HasThread was introduced as a wrapper of Thread to workaround a deadlock bug, and this bug had fixed since jdk7, so I think it time to remove it as we are using jdk8+. Related links: [https://issues.apache.org/jira/browse/HBASE-4367|https://slack-redir.net/link?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHBASE-4367] [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6915621|https://slack-redir.net/link?url=https%3A%2F%2Fbugs.java.com%2Fbugdatabase%2Fview_bug.do%3Fbug_id%3D6915621] > Remove the HasThread because the related bug had been fixed since jdk7 > -- > > Key: HBASE-24451 > URL: https://issues.apache.org/jira/browse/HBASE-24451 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > The HasThread was introduced as a wrapper of Thread to workaround a deadlock > bug, and this bug had fixed since jdk7, so I think it is the time to remove > it as we are using jdk8+. > > Related links: > > [https://issues.apache.org/jira/browse/HBASE-4367|https://slack-redir.net/link?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHBASE-4367] > > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6915621|https://slack-redir.net/link?url=https%3A%2F%2Fbugs.java.com%2Fbugdatabase%2Fview_bug.do%3Fbug_id%3D6915621] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24451) Remove the HasThread because the related bug had been fixed since jdk7
Zheng Wang created HBASE-24451: -- Summary: Remove the HasThread because the related bug had been fixed since jdk7 Key: HBASE-24451 URL: https://issues.apache.org/jira/browse/HBASE-24451 Project: HBase Issue Type: Improvement Components: util Reporter: Zheng Wang Assignee: Zheng Wang The HasThread was introduced as a wrapper of Thread to workaround a deadlock bug, and this bug had fixed since jdk7, so I think it time to remove it as we are using jdk8+. Related links: [https://issues.apache.org/jira/browse/HBASE-4367|https://slack-redir.net/link?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHBASE-4367] [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6915621|https://slack-redir.net/link?url=https%3A%2F%2Fbugs.java.com%2Fbugdatabase%2Fview_bug.do%3Fbug_id%3D6915621] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1795: HBASE-24189 WALSplit recreates region dirs for deleted table with rec…
Apache-HBase commented on pull request #1795: URL: https://github.com/apache/hbase/pull/1795#issuecomment-635102127 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | master passed | | +1 :green_heart: | checkstyle | 1m 6s | master passed | | +1 :green_heart: | spotbugs | 1m 59s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 23s | the patch passed | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 3s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 32m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1795/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1795 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 70db1ca5d6fc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1795/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1795/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
huaxiangsun commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r431580567
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitPlanFirstComparator.java
##
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.normalizer;
+
+import java.util.Comparator;
+
+/**
+ * Comparator class that gives higher priority to {@link
SplitNormalizationPlan}.
+ */
+class SplitPlanFirstComparator implements Comparator {
Review comment:
The PlanComparator compares plans based on if it is a split or merge. It
puts all split plan at the head of the list and merge plans put to the end of
the list.
Based on the addToList order, split plans are already added first, followed
by merge plans, so my understanding is that this sort does nothing.
plans.sort(planComparator);
We can get rid of this comparator totally and simplify the code. Something
missing in my understanding?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #1791: HBASE-23202 ExportSnapshot (import) will fail if copying files to roo…
huaxiangsun commented on pull request #1791: URL: https://github.com/apache/hbase/pull/1791#issuecomment-635095629 Test failure is a flaky, HBASE-24429 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1792: HBASE-24447: Contribute a Test class that shows some examples for usi…
Apache-HBase commented on pull request #1792: URL: https://github.com/apache/hbase/pull/1792#issuecomment-635093780 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 50s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 32s | branch-2 passed | | +1 :green_heart: | compile | 1m 24s | branch-2 passed | | +1 :green_heart: | shadedjars | 7m 25s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 53s | hbase-server in branch-2 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 59s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed | | +1 :green_heart: | javac | 1m 11s | the patch passed | | +1 :green_heart: | shadedjars | 6m 27s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 229m 29s | hbase-server in the patch failed. | | | | 261m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1792 | | JIRA Issue | HBASE-24447 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dc015a82510e 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 60c125367b | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/testReport/ | | Max. process+thread count | 2794 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn opened a new pull request #1795: HBASE-24189 WALSplit recreates region dirs for deleted table with rec…
anoopsjohn opened a new pull request #1795: URL: https://github.com/apache/hbase/pull/1795 …overed edits data. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #1782: HBASE-24431 RSGroupInfo add configuration map to store something extra
infraio commented on a change in pull request #1782:
URL: https://github.com/apache/hbase/pull/1782#discussion_r431573270
##
File path:
hbase-protocol-shaded/src/main/protobuf/server/rsgroup/RSGroupAdmin.proto
##
@@ -183,4 +191,7 @@ service RSGroupAdminService {
rpc RenameRSGroup(RenameRSGroupRequest)
returns (RenameRSGroupResponse);
+
Review comment:
This has been added to master.proto. Why add this here too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-24189) WALSplit recreates region dirs for deleted table with recovered edits data
[ https://issues.apache.org/jira/browse/HBASE-24189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-24189: --- Summary: WALSplit recreates region dirs for deleted table with recovered edits data (was: Regionserver recreates region folders in HDFS after replaying WAL with removed table entries) > WALSplit recreates region dirs for deleted table with recovered edits data > -- > > Key: HBASE-24189 > URL: https://issues.apache.org/jira/browse/HBASE-24189 > Project: HBase > Issue Type: Bug > Components: regionserver, wal >Affects Versions: 2.2.4 > Environment: * HDFS 3.1.3 > * HBase 2.1.4 > * OpenJDK 8 >Reporter: Andrey Elenskiy >Assignee: Anoop Sam John >Priority: Major > > Under the following scenario region directories in HDFS can be recreated with > only recovered.edits in them: > # Create table "test" > # Put into "test" > # Delete table "test" > # Create table "test" again > # Crash the regionserver to which the put has went to force the WAL replay > # Region directory in old table is recreated in new table > # hbase hbck returns inconsistency > This appears to happen due to the fact that WALs are not cleaned up once a > table is deleted and they still contain the edits from old table. I've tried > wal_roll command on the regionserver before crashing it, but it doesn't seem > to help as under some circumstances there are still WAL files around. The > only solution that works consistently is to restart regionserver before > creating the table at step 4 because that triggers log cleanup on startup: > [https://github.com/apache/hbase/blob/f3ee9b8aa37dd30d34ff54cd39fb9b4b6d22e683/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L508|https://github.com/apache/hbase/blob/f3ee9b8aa37dd30d34ff54cd39fb9b4b6d22e683/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L508)] > > Truncating a table also would be a workaround by in our case it's a no-go as > we create and delete tables in our tests which run back to back (create table > in the beginning of the test and delete in the end of the test). > A nice option in our case would be to provide hbase shell utility to force > clean up of log files manually as I realize that it's not really viable to > clean all of those up every time some table is removed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] infraio commented on a change in pull request #1789: HBASE-22738 Fallback to default group to choose RS when there are no …
infraio commented on a change in pull request #1789: URL: https://github.com/apache/hbase/pull/1789#discussion_r431571750 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java ## @@ -77,6 +81,23 @@ private volatile RSGroupInfoManager rsGroupInfoManager; private LoadBalancer internalBalancer; + /** Define the config key of if fallback group is enabled */ + public static final String FALLBACK_GROUP_ENABLED_KEY = "hbase.rsgroup.fallback.groups.enable"; + public static final boolean FALLBACK_GROUP_ENABLED_DEFAULT = false; + + /** Define the config key of fallback groups */ + public static final String FALLBACK_GROUPS_KEY = "hbase.rsgroup.fallback.groups"; + public static final String FALLBACK_GROUPS_DEFAULT = RSGroupInfo.DEFAULT_GROUP; + + /** Define the config key of interval to correct fallback regions */ + static final String FALLBACK_CORRECT_INTERVAL_KEY = "hbase.rsgroup.fallback.correct.interval"; + static final int FALLBACK_CORRECT_INTERVAL_DEFAULT = 10 * 60 * 1000; // 10min + + private boolean fallbackEnabled; + private List fallbackGroups; + @VisibleForTesting + FallbackCorrector fallbackCorrector; Review comment: Do we need this chore? This should be work of RSGroupBalancer? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #1789: HBASE-22738 Fallback to default group to choose RS when there are no …
infraio commented on a change in pull request #1789: URL: https://github.com/apache/hbase/pull/1789#discussion_r431571480 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java ## @@ -77,6 +81,23 @@ private volatile RSGroupInfoManager rsGroupInfoManager; private LoadBalancer internalBalancer; + /** Define the config key of if fallback group is enabled */ + public static final String FALLBACK_GROUP_ENABLED_KEY = "hbase.rsgroup.fallback.groups.enable"; Review comment: This config doesn't need be introduced? Enable this when config FALLBACK_GROUPS_KEY. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #1789: HBASE-22738 Fallback to default group to choose RS when there are no …
infraio commented on a change in pull request #1789: URL: https://github.com/apache/hbase/pull/1789#discussion_r431571294 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java ## @@ -77,6 +81,23 @@ private volatile RSGroupInfoManager rsGroupInfoManager; private LoadBalancer internalBalancer; + /** Define the config key of if fallback group is enabled */ + public static final String FALLBACK_GROUP_ENABLED_KEY = "hbase.rsgroup.fallback.groups.enable"; + public static final boolean FALLBACK_GROUP_ENABLED_DEFAULT = false; + + /** Define the config key of fallback groups */ + public static final String FALLBACK_GROUPS_KEY = "hbase.rsgroup.fallback.groups"; Review comment: groups? Allow fallback to more than one group? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24450) There was a partial failure due to IO when attempting to load
[ https://issues.apache.org/jira/browse/HBASE-24450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118282#comment-17118282 ] niuyulin commented on HBASE-24450: -- client side will retry, we have config "hbase.bulkload.retries.number" > There was a partial failure due to IO when attempting to load > - > > Key: HBASE-24450 > URL: https://issues.apache.org/jira/browse/HBASE-24450 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.4 >Reporter: yukunpeng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1789: HBASE-22738 Fallback to default group to choose RS when there are no …
Apache-HBase commented on pull request #1789: URL: https://github.com/apache/hbase/pull/1789#issuecomment-635081799 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | master passed | | +1 :green_heart: | checkstyle | 1m 13s | master passed | | +1 :green_heart: | spotbugs | 2m 9s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 42s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 10s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 35m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1789/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1789 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux ac3fa8d1f2eb 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1789/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24428) Priority compaction for recently split daughter regions
[
https://issues.apache.org/jira/browse/HBASE-24428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118277#comment-17118277
]
Hudson commented on HBASE-24428:
Results for branch branch-2
[build #2680 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2680/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2680/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2680/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2680/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2680/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Priority compaction for recently split daughter regions
> ---
>
> Key: HBASE-24428
> URL: https://issues.apache.org/jira/browse/HBASE-24428
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.5
>
>
> We observe that under hotspotting conditions that splitting will proceed very
> slowly and the "_Cannot split region due to reference files being there_" log
> line will be logged excessively. (branch-1 based production.) This is because
> after a region is split it must be compacted before it can be split again.
> Reference files must be replaced by real HFiles, normal housekeeping
> performed during compaction. However if the regionserver is under excessive
> load, its compaction queues may become deep. The daughters of a recently
> split hotspotting region may themselves continue to hotspot and will rapidly
> need to split again. If the scheduled compaction work to remove/replace
> reference files is queued hundreds or thousands of compaction queue elements
> behind current, the recently split daughter regions will not be able to split
> again for a long time and may grow very large, producing additional
> complications (very large regions, very deep replication queues).
> To help avoid this condition we should prioritize the compaction of recently
> split daughter regions. Compaction requests include a {{priority}} field and
> CompactionRequest implements a comparator that sorts by this field. We
> already detect when a compaction request involves a region that has reference
> files, to ensure that it gets selected to be eligible for compaction, but we
> do not seem to prioritize the requests for post-split housekeeping. Split
> work should be placed at the top of the queue. Ensure that this is happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24434) fix building cpp-example DemoClient
[
https://issues.apache.org/jira/browse/HBASE-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118278#comment-17118278
]
Hudson commented on HBASE-24434:
Results for branch branch-2
[build #2680 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2680/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2680/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2680/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2680/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2680/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> fix building cpp-example DemoClient
> ---
>
> Key: HBASE-24434
> URL: https://issues.apache.org/jira/browse/HBASE-24434
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
>Reporter: Alexander Myasnikov
>Priority: Major
> Labels: thrift
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.4.0
>
> Attachments: HBASE-24434-branch-1.patch, HBASE-24434-branch-2.patch,
> HBASE-24434-master.patch
>
>
> Changed boost::shared_ptr to std::shared_ptr for compatibility with apache
> thrift v0.13.
> https://github.com/apache/hbase/pull/1775
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24450) There was a partial failure due to IO when attempting to load
[
https://issues.apache.org/jira/browse/HBASE-24450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118274#comment-17118274
]
yukunpeng edited comment on HBASE-24450 at 5/28/20, 3:51 AM:
-
When bulk load some hfiles, IO exception occurred in hadoop, how to retry
{code:java}
regionserver.SecureBulkLoadManager: Failed to complete bulk load
java.io.IOException: Failed rename of
hdfs://node1:9000/tmp/load/ontime/715505590901669888/f/8ccf9d873d114d248efd497c15a2007a
to
hdfs://node1:9000/Hydrogen/data/default/ontime/82c3619f9a6b2c3ddc73092deb239a85/f/4d6fa87ae5e24899b5a84dd944fa3c96_SeqId_84_
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:480)
at
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:891)
{code}
{code:java}
2020-05-28 11:48:47,301 ERROR
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
regionserver.SecureBulkLoadManager: Failed to complete bulk load
java.io.IOException: Failed rename of
hdfs://node1:9000/tmp/load/ontime/715505590901669888/f/b137252855764f51be61746b0bd8179a
to
hdfs://node1:9000/Hydrogen/data/default/ontime/82c3619f9a6b2c3ddc73092deb239a85/f/acf774e955064d319d7485168f6fcf49_SeqId_78_
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:480)
{code}
was (Author: yukunpeng):
When bulk load some hfiles, IO exception occurred in hadoop, how to retry
> There was a partial failure due to IO when attempting to load
> -
>
> Key: HBASE-24450
> URL: https://issues.apache.org/jira/browse/HBASE-24450
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.4
>Reporter: yukunpeng
>Priority: Minor
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24450) There was a partial failure due to IO when attempting to load
[ https://issues.apache.org/jira/browse/HBASE-24450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yukunpeng updated HBASE-24450: -- Attachment: (was: image-2020-05-28-11-44-14-142.png) > There was a partial failure due to IO when attempting to load > - > > Key: HBASE-24450 > URL: https://issues.apache.org/jira/browse/HBASE-24450 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.4 >Reporter: yukunpeng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24450) There was a partial failure due to IO when attempting to load
[ https://issues.apache.org/jira/browse/HBASE-24450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118274#comment-17118274 ] yukunpeng commented on HBASE-24450: --- When bulk load some hfiles, IO exception occurred in hadoop, how to retry > There was a partial failure due to IO when attempting to load > - > > Key: HBASE-24450 > URL: https://issues.apache.org/jira/browse/HBASE-24450 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.4 >Reporter: yukunpeng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HBASE-24450) There was a partial failure due to IO when attempting to load
[ https://issues.apache.org/jira/browse/HBASE-24450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yukunpeng updated HBASE-24450: -- Comment: was deleted (was: !image-2020-05-28-11-44-14-142.png!) > There was a partial failure due to IO when attempting to load > - > > Key: HBASE-24450 > URL: https://issues.apache.org/jira/browse/HBASE-24450 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.4 >Reporter: yukunpeng >Priority: Minor > Attachments: image-2020-05-28-11-44-14-142.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24289) Heterogeneous Storage for Date Tiered Compaction
[
https://issues.apache.org/jira/browse/HBASE-24289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118273#comment-17118273
]
Guanghao Zhang commented on HBASE-24289:
{quote}any other date-based compaction policies or other writes (wals and cf
maybe?)
{quote}
All wal is created for current write. It doesn't need date-based.
{quote}A writer factory that returns writers on different storage/folder based
on date ranges would be a more reusable component. We can call it date-based
HDFS storage assignment.
{quote}
I thought this shoule be a new issue. It is not this proposal's goal. And now
the 2.x hdfs didn't have api about create a file writer with storage policy.
The api only finish write first then set storage policy. Then move the data to
meet storage policy. So this is not friendly.
{quote}Another change I recommend is for the policy to take a list of date
boundaries for ranges to assign (potentially) different HDFS storages. It can
be empty so all are on the same storage, or multiple for different storages.
This makes it backward compatible and easy for future extension or adoption.
{quote}
Can you explain more about this? The hot, warm, cold classification is not
enough?
> Heterogeneous Storage for Date Tiered Compaction
>
>
> Key: HBASE-24289
> URL: https://issues.apache.org/jira/browse/HBASE-24289
> Project: HBase
> Issue Type: New Feature
> Components: Compaction
>Reporter: Mengqing Peng
>Assignee: Mengqing Peng
>Priority: Major
>
> Support DateTiredCompaction(HBASE-15181) for cold and hot data separation,
> support different storage policies for different time periods of data to get
> better performance, for example, we can configure the data of last 1 month in
> SSD, and 1 month ago data was in HDD.
> design doc:
> https://docs.google.com/document/d/1fk_EWLNnxniwt3gDjUS_apQ3cPzn90AmvDT1wkirvKE/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24450) There was a partial failure due to IO when attempting to load
yukunpeng created HBASE-24450: - Summary: There was a partial failure due to IO when attempting to load Key: HBASE-24450 URL: https://issues.apache.org/jira/browse/HBASE-24450 Project: HBase Issue Type: Bug Affects Versions: 2.2.4 Reporter: yukunpeng -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24450) There was a partial failure due to IO when attempting to load
[ https://issues.apache.org/jira/browse/HBASE-24450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118269#comment-17118269 ] yukunpeng commented on HBASE-24450: --- !image-2020-05-28-11-44-14-142.png! > There was a partial failure due to IO when attempting to load > - > > Key: HBASE-24450 > URL: https://issues.apache.org/jira/browse/HBASE-24450 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.4 >Reporter: yukunpeng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24450) There was a partial failure due to IO when attempting to load
[ https://issues.apache.org/jira/browse/HBASE-24450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yukunpeng updated HBASE-24450: -- Attachment: image-2020-05-28-11-44-14-142.png > There was a partial failure due to IO when attempting to load > - > > Key: HBASE-24450 > URL: https://issues.apache.org/jira/browse/HBASE-24450 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.4 >Reporter: yukunpeng >Priority: Minor > Attachments: image-2020-05-28-11-44-14-142.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1743: HBASE-24403 FsDelegationToken Should Cache Token After Acquired A New One
Apache-HBase commented on pull request #1743: URL: https://github.com/apache/hbase/pull/1743#issuecomment-635062821 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | master passed | | +1 :green_heart: | checkstyle | 1m 17s | master passed | | +1 :green_heart: | spotbugs | 2m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | -0 :warning: | checkstyle | 1m 16s | hbase-server: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 32s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 37m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1743 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 4ba8e689a5a6 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1792: HBASE-24447: Contribute a Test class that shows some examples for usi…
Apache-HBase commented on pull request #1792: URL: https://github.com/apache/hbase/pull/1792#issuecomment-635062664 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 45s | branch-2 passed | | +1 :green_heart: | compile | 0m 56s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 14s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 4m 57s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 132m 54s | hbase-server in the patch failed. | | | | 155m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1792 | | JIRA Issue | HBASE-24447 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 60a0a61174bc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 60c125367b | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/testReport/ | | Max. process+thread count | 4040 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1743: HBASE-24403 FsDelegationToken Should Cache Token After Acquired A New One
Apache-HBase commented on pull request #1743: URL: https://github.com/apache/hbase/pull/1743#issuecomment-635061935 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 6m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 54s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 9m 5s | hbase-server in the patch failed. | | | | 35m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1743 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 057040e2d721 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/testReport/ | | Max. process+thread count | 519 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1743: HBASE-24403 FsDelegationToken Should Cache Token After Acquired A New One
Apache-HBase commented on pull request #1743: URL: https://github.com/apache/hbase/pull/1743#issuecomment-635060887 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 5m 50s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 4s | the patch passed | | +1 :green_heart: | shadedjars | 5m 43s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 7m 3s | hbase-server in the patch failed. | | | | 31m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1743 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f9b483178674 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/testReport/ | | Max. process+thread count | 740 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1793: HBASE-23941 : FilterBy operator support in get_slowlog_responses API
Apache-HBase commented on pull request #1793: URL: https://github.com/apache/hbase/pull/1793#issuecomment-635056963 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 34s | master passed | | +1 :green_heart: | compile | 3m 3s | master passed | | +1 :green_heart: | shadedjars | 6m 36s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 44s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | compile | 2m 46s | the patch passed | | +1 :green_heart: | javac | 2m 46s | the patch passed | | +1 :green_heart: | shadedjars | 6m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 54s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 329m 1s | root in the patch failed. | | | | 368m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e37c3fc218b4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/testReport/ | | Max. process+thread count | 5337 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-15550) Backport HBASE-12220 hedgedRead metrics to 1.2+
[ https://issues.apache.org/jira/browse/HBASE-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-15550: -- Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > Backport HBASE-12220 hedgedRead metrics to 1.2+ > --- > > Key: HBASE-15550 > URL: https://issues.apache.org/jira/browse/HBASE-15550 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.3.0, 1.2.2 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 1.7.0 > > Attachments: 15550.branch-1.2.txt > > > hedgedRead metrics are in master branch only. They were put in branch-1 but > then reverted because old versions of hadoop didn't supported hedgedReads. > 1.2+ has minimal hadoop version. Retry backport. > FYI [~phobos182] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ddupg commented on pull request #1788: backport: HBASE-24416 RegionNormalizer spliting region should not be limited by…
ddupg commented on pull request #1788: URL: https://github.com/apache/hbase/pull/1788#issuecomment-635053759 > @ddupg multiple tests are failing. Can you please take a look once? How are they doing locally? Thanks @virajjasani for taking a look here. The following is unit log, it seems like that failed UTs are unrelated with this PR. > [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? Let me trigger UTs running again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-15550) Backport HBASE-12220 hedgedRead metrics to 1.2+
[ https://issues.apache.org/jira/browse/HBASE-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-15550: -- Component/s: metrics > Backport HBASE-12220 hedgedRead metrics to 1.2+ > --- > > Key: HBASE-15550 > URL: https://issues.apache.org/jira/browse/HBASE-15550 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.3.0, 1.2.2 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 1.7.0 > > Attachments: 15550.branch-1.2.txt > > > hedgedRead metrics are in master branch only. They were put in branch-1 but > then reverted because old versions of hadoop didn't supported hedgedReads. > 1.2+ has minimal hadoop version. Retry backport. > FYI [~phobos182] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-15550) Backport HBASE-12220 hedgedRead metrics to 1.2+
[ https://issues.apache.org/jira/browse/HBASE-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-15550: -- Fix Version/s: 1.7.0 > Backport HBASE-12220 hedgedRead metrics to 1.2+ > --- > > Key: HBASE-15550 > URL: https://issues.apache.org/jira/browse/HBASE-15550 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.0, 1.2.2 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 1.7.0 > > Attachments: 15550.branch-1.2.txt > > > hedgedRead metrics are in master branch only. They were put in branch-1 but > then reverted because old versions of hadoop didn't supported hedgedReads. > 1.2+ has minimal hadoop version. Retry backport. > FYI [~phobos182] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15550) Backport HBASE-12220 hedgedRead metrics to 1.2+
[ https://issues.apache.org/jira/browse/HBASE-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118234#comment-17118234 ] Reid Chan commented on HBASE-15550: --- It is back-ported to branch-1, fixed in 1.7 > Backport HBASE-12220 hedgedRead metrics to 1.2+ > --- > > Key: HBASE-15550 > URL: https://issues.apache.org/jira/browse/HBASE-15550 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.3.0, 1.2.2 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Attachments: 15550.branch-1.2.txt > > > hedgedRead metrics are in master branch only. They were put in branch-1 but > then reverted because old versions of hadoop didn't supported hedgedReads. > 1.2+ has minimal hadoop version. Retry backport. > FYI [~phobos182] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24435) Bring back hedged reads metrics to branch-1
[ https://issues.apache.org/jira/browse/HBASE-24435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118232#comment-17118232 ] Reid Chan commented on HBASE-24435: --- Take the credit and assign yourself after that is done. Thanks for the contribution. > Bring back hedged reads metrics to branch-1 > --- > > Key: HBASE-24435 > URL: https://issues.apache.org/jira/browse/HBASE-24435 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 1.4.13 >Reporter: Javier Akira Luca de Tena >Priority: Minor > Fix For: 1.7.0 > > > Seems at some point, the hedged metrics were added to > MetricsRegionServerWrapperImpl (they were added without mentioning Jira > ticket, but may be it was https://issues.apache.org/jira/browse/HBASE-12220 ?) > > However, because of compilation issue, it was reverted: > https://issues.apache.org/jira/browse/HBASE-12246 > > And never re-applied for branch-1: > https://issues.apache.org/jira/browse/HBASE-12246?focusedCommentId=16425102=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16425102 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24435) Bring back hedged reads metrics to branch-1
[ https://issues.apache.org/jira/browse/HBASE-24435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118230#comment-17118230 ] Reid Chan commented on HBASE-24435: --- [~akiraluca] Please send a request to dev mail for being added to the contributor list. I couldn't assign the ticket to you, since you're not in the list. > Bring back hedged reads metrics to branch-1 > --- > > Key: HBASE-24435 > URL: https://issues.apache.org/jira/browse/HBASE-24435 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 1.4.13 >Reporter: Javier Akira Luca de Tena >Priority: Minor > Fix For: 1.7.0 > > > Seems at some point, the hedged metrics were added to > MetricsRegionServerWrapperImpl (they were added without mentioning Jira > ticket, but may be it was https://issues.apache.org/jira/browse/HBASE-12220 ?) > > However, because of compilation issue, it was reverted: > https://issues.apache.org/jira/browse/HBASE-12246 > > And never re-applied for branch-1: > https://issues.apache.org/jira/browse/HBASE-12246?focusedCommentId=16425102=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16425102 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24435) Bring back hedged reads metrics to branch-1
[ https://issues.apache.org/jira/browse/HBASE-24435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan resolved HBASE-24435. --- Hadoop Flags: Reviewed Resolution: Fixed > Bring back hedged reads metrics to branch-1 > --- > > Key: HBASE-24435 > URL: https://issues.apache.org/jira/browse/HBASE-24435 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 1.4.13 >Reporter: Javier Akira Luca de Tena >Priority: Minor > Fix For: 1.7.0 > > > Seems at some point, the hedged metrics were added to > MetricsRegionServerWrapperImpl (they were added without mentioning Jira > ticket, but may be it was https://issues.apache.org/jira/browse/HBASE-12220 ?) > > However, because of compilation issue, it was reverted: > https://issues.apache.org/jira/browse/HBASE-12246 > > And never re-applied for branch-1: > https://issues.apache.org/jira/browse/HBASE-12246?focusedCommentId=16425102=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16425102 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24435) Bring back hedged reads metrics to branch-1
[ https://issues.apache.org/jira/browse/HBASE-24435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-24435: -- Component/s: metrics > Bring back hedged reads metrics to branch-1 > --- > > Key: HBASE-24435 > URL: https://issues.apache.org/jira/browse/HBASE-24435 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 1.4.13 >Reporter: Javier Akira Luca de Tena >Priority: Minor > Fix For: 1.7.0 > > > Seems at some point, the hedged metrics were added to > MetricsRegionServerWrapperImpl (they were added without mentioning Jira > ticket, but may be it was https://issues.apache.org/jira/browse/HBASE-12220 ?) > > However, because of compilation issue, it was reverted: > https://issues.apache.org/jira/browse/HBASE-12246 > > And never re-applied for branch-1: > https://issues.apache.org/jira/browse/HBASE-12246?focusedCommentId=16425102=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16425102 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24435) Bring back hedged reads metrics to branch-1
[ https://issues.apache.org/jira/browse/HBASE-24435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-24435: -- Issue Type: Improvement (was: Bug) > Bring back hedged reads metrics to branch-1 > --- > > Key: HBASE-24435 > URL: https://issues.apache.org/jira/browse/HBASE-24435 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 1.4.13 >Reporter: Javier Akira Luca de Tena >Priority: Minor > Fix For: 1.7.0 > > > Seems at some point, the hedged metrics were added to > MetricsRegionServerWrapperImpl (they were added without mentioning Jira > ticket, but may be it was https://issues.apache.org/jira/browse/HBASE-12220 ?) > > However, because of compilation issue, it was reverted: > https://issues.apache.org/jira/browse/HBASE-12246 > > And never re-applied for branch-1: > https://issues.apache.org/jira/browse/HBASE-12246?focusedCommentId=16425102=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16425102 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24435) Bring back hedged reads metrics to branch-1
[ https://issues.apache.org/jira/browse/HBASE-24435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-24435: -- Fix Version/s: 1.7.0 > Bring back hedged reads metrics to branch-1 > --- > > Key: HBASE-24435 > URL: https://issues.apache.org/jira/browse/HBASE-24435 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.13 >Reporter: Javier Akira Luca de Tena >Priority: Minor > Fix For: 1.7.0 > > > Seems at some point, the hedged metrics were added to > MetricsRegionServerWrapperImpl (they were added without mentioning Jira > ticket, but may be it was https://issues.apache.org/jira/browse/HBASE-12220 ?) > > However, because of compilation issue, it was reverted: > https://issues.apache.org/jira/browse/HBASE-12246 > > And never re-applied for branch-1: > https://issues.apache.org/jira/browse/HBASE-12246?focusedCommentId=16425102=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16425102 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Reidddddd merged pull request #1781: HBASE-24435 Add hedgedReads and hedgedReadWins count metrics
Reidd merged pull request #1781: URL: https://github.com/apache/hbase/pull/1781 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] VicoWu commented on pull request #1743: HBASE-24403 FsDelegationToken Should Cache Token After Acquired A New One
VicoWu commented on pull request #1743: URL: https://github.com/apache/hbase/pull/1743#issuecomment-635050379 > @jojochuang > This issue is found by our cluster incident. > But my current commit still has some problem which need to be resolved before finally adopted: > > In this PR, I cache the token between RS and NameNode instead of always applying for a new one every time processing a file, so the token should be confirmed that it is not expired every time we use it; > So I think I need to add a check logic to verify its expiration time , if it is already or going to be expired, I will have to apply for a new one and overwrite the cache; @jojochuang I have added the token expiration processing logic and help to review it; The logic has updated to the PR's description part; This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24448) Flaky tests for region server in branch-1
[ https://issues.apache.org/jira/browse/HBASE-24448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan resolved HBASE-24448. --- Resolution: Not A Bug > Flaky tests for region server in branch-1 > - > > Key: HBASE-24448 > URL: https://issues.apache.org/jira/browse/HBASE-24448 > Project: HBase > Issue Type: Bug > Components: regionserver, test >Affects Versions: 1.7.0 >Reporter: Clara Xiong >Priority: Minor > > [https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1781/4/testReport/] > Tests failed after unrelated code style changes for metrics for branch-1. > Rerun passed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118226#comment-17118226 ] Guanghao Zhang commented on HBASE-11288: BTW: For higher availability master and reduce zk dependency, can we consider quorum master solution? Elect leader and store all things inside. The follower master can become active master soonly. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC
[
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118219#comment-17118219
]
Guanghao Zhang commented on HBASE-23296:
{quote}bq.But if the index and bloom miss, we may need three seeks. So only
index/bloom go to L1 may be suitable. The current patch do things like this.
{quote}
So the key point is a bigger L1 cache to cache all index/bloom blocks and the
heap size is not enough?
> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
> Issue Type: New Feature
> Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118217#comment-17118217 ] Guanghao Zhang commented on HBASE-11288: I thought we are consistent on this point: need a splittable meta and need a storage to store the meta regions location (more than 1). Then we have three choices now: # zk node # root table # master local region For zk node, we want to reduce zk dependency, so this is not a good choice? For root table, I thought the previous problems about meta table will become the problems about root table. root table should be splitable? If root table cannot splitable, which regionserver to host root table's region, a isolated region or share with user regions? In our production use case (300+ RegionServer, 200K+ Regions, 1K+ tables), the meta region is isolated on one regionserver, not share with the user regions. This is a waste of machine, but for better stability. And we have many hack logic for meta regions, such as assign meta first, donot replicate meta wal, balancer give a bigger weight for meta region, the new root table need all these hack, too? You can found these by grep isMetaTable or isMetaRegion calls in our codebase... For master local region, the master was light load so we use small heap for master. And deploy many master on one share machine. There may be many locate request when many client bootstrap or failover? This is a challange for master. In our production use case , we meet one case is too many regionserver report slow down master... Another problem is the availability of master. In the past, master didn't response user access request so the lower availability is not a big deal. We can restart master at any time. In a big cluser, the master bootstrap may take almost 1 minute to load all things from zookeeper or meta table or receive the report from regionserver. But if master serve the root table, it may need higher availability. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24389) Introduce a new master rpc service to locate meta region through root region
[
https://issues.apache.org/jira/browse/HBASE-24389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118214#comment-17118214
]
Michael Stack commented on HBASE-24389:
---
Mind doing a bit of a write up on what this is doing. On quick review this adds
a new 'Service' that does the below API:
{code}
enum RegionLocateType {
REGION_LOCATE_TYPE_BEFORE =1;
REGION_LOCATE_TYPE_CURRENT = 2;
REGION_LOCATE_TYPE_AFTER = 3;
}
message LocateMetaRegionRequest {
required bytes row = 1;
required RegionLocateType locateType = 2;
}
message LocateMetaRegionResponse {
repeated RegionLocation meta_locations = 1;
}
message GetAllMetaRegionLocationsRequest {
required bool exclude_offlined_split_parents = 1;
}
message GetAllMetaRegionLocationsResponse {
repeated RegionLocation meta_locations = 1;
}
/**
* Implements the RPCs needed by clients to look up the location of meta region.
*/
service ClientLocateMetaRegionService {
rpc LocateMetaRegion(LocateMetaRegionRequest)
returns(LocateMetaRegionResponse);
rpc GetAllMetaRegionLocations(GetAllMetaRegionLocationsRequest)
returns(GetAllMetaRegionLocationsResponse);
}
{code}
Master is hard-coded to mediate the lookup? Doesn't seem to go via Registry?
(Didn't spend much time looking) Or a client config. would allow providing
different impls?
This is new API that is not how we usually do the Read of location. Do we want
to implement RegionLocation on this?
https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/RegionLocator.html
Would client impl do get all regions or would it use the one-at-a time? Does
this API imply anything about meta table size? Would we be able to have a
table of 10k Regions?
Thanks
> Introduce a new master rpc service to locate meta region through root region
>
>
> Key: HBASE-24389
> URL: https://issues.apache.org/jira/browse/HBASE-24389
> Project: HBase
> Issue Type: Sub-task
> Components: master, meta
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> And also reimlement the client logic to use the new api to locate meta
> region. In general, this will remove the assumption at client side that there
> is only one meta region.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] VicoWu commented on a change in pull request #1764: HBASE-24420 Avoid Meaningless Retry Attempts in Unrecoverable Failure
VicoWu commented on a change in pull request #1764:
URL: https://github.com/apache/hbase/pull/1764#discussion_r431533385
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java
##
@@ -382,22 +382,27 @@ public String prepareBulkLoad(final byte[] family, final
String srcPath, boolean
throw new IOException("Path does not reference a file: " + p);
}
- // Check to see if the source and target filesystems are the same
- if (!FSUtils.isSameHdfs(conf, srcFs, fs)) {
-LOG.debug("Bulk-load file " + srcPath + " is on different filesystem
than " +
+ try {
Review comment:
Overall, in this part, I just extend the exception catch domain from
previous just `fs.rename()` to the whole HDFS operation, they need obviously
the same process logic because they are all hdfs operations;
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] VicoWu commented on a change in pull request #1764: HBASE-24420 Avoid Meaningless Retry Attempts in Unrecoverable Failure
VicoWu commented on a change in pull request #1764:
URL: https://github.com/apache/hbase/pull/1764#discussion_r431532580
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java
##
@@ -382,22 +382,27 @@ public String prepareBulkLoad(final byte[] family, final
String srcPath, boolean
throw new IOException("Path does not reference a file: " + p);
}
- // Check to see if the source and target filesystems are the same
- if (!FSUtils.isSameHdfs(conf, srcFs, fs)) {
-LOG.debug("Bulk-load file " + srcPath + " is on different filesystem
than " +
+ try {
Review comment:
From the code , we could find that if exception is thrown correctly
instead of swallowed, the retry attempts number could be controlled by
`hbase.client.retries.number` and `hbase.bulkload.retries.retryOnIOException`
instead of falling into a retry disaster and user cannot do anything because
the retry attempts number are controlled by number of regions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] VicoWu commented on a change in pull request #1764: HBASE-24420 Avoid Meaningless Retry Attempts in Unrecoverable Failure
VicoWu commented on a change in pull request #1764:
URL: https://github.com/apache/hbase/pull/1764#discussion_r431532167
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java
##
@@ -382,22 +382,27 @@ public String prepareBulkLoad(final byte[] family, final
String srcPath, boolean
throw new IOException("Path does not reference a file: " + p);
}
- // Check to see if the source and target filesystems are the same
- if (!FSUtils.isSameHdfs(conf, srcFs, fs)) {
-LOG.debug("Bulk-load file " + srcPath + " is on different filesystem
than " +
+ try {
Review comment:
[
BulkLoadHFilesTool.java#396](https://github.com/apache/hbase/blob/a9205f8f4d98ee672c0c6aa9cafa5ef2afc6aab5/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L397)
This is the cliend-side retry logic if exception/error happened.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24371) Add more details when print CompactionConfiguration info
[
https://issues.apache.org/jira/browse/HBASE-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118206#comment-17118206
]
Hudson commented on HBASE-24371:
Results for branch branch-2.2
[build #878 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add more details when print CompactionConfiguration info
>
>
> Key: HBASE-24371
> URL: https://issues.apache.org/jira/browse/HBASE-24371
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.6
>
>
> Now the CompactionConfiguration info logs look like
> {code}
> 2020-05-14 15:32:45,468 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,468 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,469 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,469 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> {code}
> We don't have any details on which region and which column family the
> CompactionConfiguration for. It would be better to include the region name
> and column family detail also in this log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24379) CatalogJanitor misreports region holes when there are actually over laps.
[
https://issues.apache.org/jira/browse/HBASE-24379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118208#comment-17118208
]
Hudson commented on HBASE-24379:
Results for branch branch-2.2
[build #878 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> CatalogJanitor misreports region holes when there are actually over laps.
> -
>
> Key: HBASE-24379
> URL: https://issues.apache.org/jira/browse/HBASE-24379
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 2.3.0
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Major
>
> We found a case that there is actually an overlap, but a region hole is
> reported.
> r1: [aa, bb), r2: [cc, dd), r3: [a, cc)
>
> In this case, there are only overlaps from "a" to "d". However, hole (r1, r2)
> is reported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24428) Priority compaction for recently split daughter regions
[
https://issues.apache.org/jira/browse/HBASE-24428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118207#comment-17118207
]
Hudson commented on HBASE-24428:
Results for branch branch-2.2
[build #878 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/878//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Priority compaction for recently split daughter regions
> ---
>
> Key: HBASE-24428
> URL: https://issues.apache.org/jira/browse/HBASE-24428
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.5
>
>
> We observe that under hotspotting conditions that splitting will proceed very
> slowly and the "_Cannot split region due to reference files being there_" log
> line will be logged excessively. (branch-1 based production.) This is because
> after a region is split it must be compacted before it can be split again.
> Reference files must be replaced by real HFiles, normal housekeeping
> performed during compaction. However if the regionserver is under excessive
> load, its compaction queues may become deep. The daughters of a recently
> split hotspotting region may themselves continue to hotspot and will rapidly
> need to split again. If the scheduled compaction work to remove/replace
> reference files is queued hundreds or thousands of compaction queue elements
> behind current, the recently split daughter regions will not be able to split
> again for a long time and may grow very large, producing additional
> complications (very large regions, very deep replication queues).
> To help avoid this condition we should prioritize the compaction of recently
> split daughter regions. Compaction requests include a {{priority}} field and
> CompactionRequest implements a comparator that sorts by this field. We
> already detect when a compaction request involves a region that has reference
> files, to ensure that it gets selected to be eligible for compaction, but we
> do not seem to prioritize the requests for post-split housekeeping. Split
> work should be placed at the top of the queue. Ensure that this is happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] busbey commented on a change in pull request #1714: HBASE-24373 Implement JvmMetrics in HBase instead of using the one in hadoop
busbey commented on a change in pull request #1714: URL: https://github.com/apache/hbase/pull/1714#discussion_r431527682 ## File path: hbase-logging/src/main/java/org/apache/log4j/Logger.java ## @@ -0,0 +1,28 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.log4j; + +import org.apache.yetus.audience.InterfaceAudience; + +/** + * Just for making hadoop work without log4j Review comment: If these are for making Hadoop work, they should be in the hadoop-compat module? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] busbey commented on a change in pull request #1714: HBASE-24373 Implement JvmMetrics in HBase instead of using the one in hadoop
busbey commented on a change in pull request #1714:
URL: https://github.com/apache/hbase/pull/1714#discussion_r431527371
##
File path: hbase-logging/src/main/java/org/apache/log4j/Logger.java
##
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.log4j;
+
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Just for making hadoop work without log4j
+ */
+@InterfaceAudience.Private
+public class Logger {
Review comment:
This doesn't end up in our client facing artifacts right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1792: HBASE-24447: Contribute a Test class that shows some examples for usi…
Apache-HBase commented on pull request #1792: URL: https://github.com/apache/hbase/pull/1792#issuecomment-635023960 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 48s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 9s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 2s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 14s | the patch passed | | -0 :warning: | checkstyle | 1m 4s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 12m 42s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 36m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1792 | | JIRA Issue | HBASE-24447 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 56c3bd0c0748 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 60c125367b | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24438) Stale ServerCrashProcedure task in HBase Master UI
[
https://issues.apache.org/jira/browse/HBASE-24438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118186#comment-17118186
]
Michael Stack commented on HBASE-24438:
---
So, what is the issue? The staleness? Or difference between 'Procedures and
Locks' and Tasks list? Thanks [~timoha]
> Stale ServerCrashProcedure task in HBase Master UI
> --
>
> Key: HBASE-24438
> URL: https://issues.apache.org/jira/browse/HBASE-24438
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 2.2.4
> Environment: HBase 2.2.4
> HDFS 3.1.3 with erasure coding enabled
> Kubernetes
>Reporter: Andrey Elenskiy
>Priority: Major
>
> Tasks section (show non-RPC Tasks) in HBase Master UI has stale entries with
> ServerCrashProcedure after master failover. The procedures have finished with
> SUCCESS on a previously active HBase master and aren't showing in "Procedures
> & Locks".
> Based on the logs, both of those regionserver were carrying hbase:meta (logs
> are sorted newest first grepped for those specific servers that have stale
> ServerCrashProcedures):
> {noformat}
> 2020-05-21 19:04:09,176 INFO [KeepAlivePEWorker-28]
> procedure2.ProcedureExecutor: Finished pid=38, state=SUCCESS;
> ServerCrashProcedure
> server=regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347,
> splitWal=true, meta=true in 2.5290sec
> 2020-05-21 19:04:08,962 INFO [KeepAlivePEWorker-28]
> procedure.ServerCrashProcedure: removed crashed server
> regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347 after
> splitting done
> 2020-05-21 19:04:08,747 INFO [KeepAlivePEWorker-28] master.SplitLogManager:
> dead splitlog workers
> [regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347]
> 2020-05-21 19:04:08,746 INFO [KeepAlivePEWorker-28] master.MasterWalManager:
> Log dir for server
> regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347 does not
> exist
> 2020-05-21 19:04:08,636 INFO [KeepAlivePEWorker-28]
> procedure.ServerCrashProcedure:
> regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347 had 0 regions
> 2020-05-21 19:04:08,529 INFO [KeepAlivePEWorker-28]
> procedure.ServerCrashProcedure: pid=38,
> state=RUNNABLE:SERVER_CRASH_ASSIGN_META, locked=true; ServerCrashProcedure
> server=regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347,
> splitWal=true, meta=true found RIT pid=20, ppid=18,
> state=RUNNABLE:REGION_STATE_TRANSITION_OPEN, locked=true;
> TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN;
> rit=ABNORMALLY_CLOSED,
> location=regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347,
> table=hbase:meta, region=1588230740
> 2020-05-21 19:04:08,422 INFO [KeepAlivePEWorker-28] master.SplitLogManager:
> Finished splitting (more than or equal to) 0 (0 bytes) in 0 log files in
> [hdfs://aeris/hbase/WALs/regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347-splitting]
> in 0ms
> 2020-05-21 19:04:08,416 INFO [KeepAlivePEWorker-28] master.SplitLogManager:
> hdfs://aeris/hbase/WALs/regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347-splitting
> dir is empty, no logs to split.
> 2020-05-21 19:04:08,414 INFO [KeepAlivePEWorker-28] master.SplitLogManager:
> dead splitlog workers
> [regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347]
> 2020-05-21 19:04:08,300 INFO [KeepAlivePEWorker-28]
> procedure.ServerCrashProcedure: Start pid=38,
> state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure
> server=regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347,
> splitWal=true, meta=true
> 2020-05-21 19:04:06,544 INFO [RegionServerTracker-0]
> assignment.AssignmentManager: Scheduled SCP pid=38 for
> regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347
> (carryingMeta=true)
> regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@14e57294[Write
> locks = 1, Read locks = 0], oldState=ONLINE.
> 2020-05-21 19:04:06,434 INFO [RegionServerTracker-0] master.ServerManager:
> Processing expiration of
> regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347 on
> hbasemaster-0.hbase.hbase.svc.cluster.local,16000,1590087665366
> 2020-05-21 19:04:06,434 INFO [RegionServerTracker-0]
> master.RegionServerTracker: RegionServer ephemeral node deleted, processing
> expiration [regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347]
> ...
> 2020-05-21 19:04:04,711 INFO [KeepAlivePEWorker-27]
> procedure2.ProcedureExecutor: Finished pid=37, state=SUCCESS;
> ServerCrashProcedure
> server=regionserver-0.hbase.hbase.svc.cluster.local,16020,1590087787010,
> splitWal=true, meta=true in 997msec
> 2020-05-21 19:04:04,497 INFO
[GitHub] [hbase] Apache-HBase commented on pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache-HBase commented on pull request #1774: URL: https://github.com/apache/hbase/pull/1774#issuecomment-635019009 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 53s | master passed | | +1 :green_heart: | compile | 2m 37s | master passed | | +1 :green_heart: | shadedjars | 6m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 33s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 56s | the patch passed | | +1 :green_heart: | compile | 2m 29s | the patch passed | | +1 :green_heart: | javac | 2m 29s | the patch passed | | +1 :green_heart: | shadedjars | 6m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 46s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 59s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 28s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 51s | hbase-zookeeper in the patch passed. | | -1 :x: | unit | 225m 49s | hbase-server in the patch failed. | | | | 262m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1774 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 702a20b0a633 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/testReport/ | | Max. process+thread count | 2555 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1793: HBASE-23941 : FilterBy operator support in get_slowlog_responses API
Apache-HBase commented on pull request #1793: URL: https://github.com/apache/hbase/pull/1793#issuecomment-635017402 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 58s | master passed | | +1 :green_heart: | compile | 3m 16s | master passed | | +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 14s | root in master failed. | | -0 :warning: | javadoc | 0m 26s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 35s | the patch passed | | +1 :green_heart: | compile | 3m 6s | the patch passed | | +1 :green_heart: | javac | 3m 6s | the patch passed | | +1 :green_heart: | shadedjars | 6m 24s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 45s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 13s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 214m 11s | root in the patch failed. | | | | 250m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 84166ac79140 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/testReport/ | | Max. process+thread count | 2847 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1792: HBASE-24447: Contribute a Test class that shows some examples for usi…
Apache-HBase commented on pull request #1792: URL: https://github.com/apache/hbase/pull/1792#issuecomment-635010715 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 20s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 13s | branch-2 passed | | +1 :green_heart: | compile | 1m 13s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 47s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 44s | hbase-server in branch-2 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 42s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 6m 40s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 209m 20s | hbase-server in the patch failed. | | | | 239m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1792 | | JIRA Issue | HBASE-24447 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c32ec60edf8d 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 1e86ff09d7 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/1/testReport/ | | Max. process+thread count | 2581 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1794: HBASE-24449: Generate version.h include file during compilation.
Apache-HBase commented on pull request #1794: URL: https://github.com/apache/hbase/pull/1794#issuecomment-635009624 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | compile | 0m 51s | master passed | | +1 :green_heart: | shadedjars | 5m 42s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 17s | hbase-common in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 5m 38s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | hbase-common in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 22s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 0m 26s | hbase-assembly in the patch passed. | | | | 26m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1794/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1794 | | JIRA Issue | HBASE-24449 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0ae016c7f7de 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1794/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1794/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1794/1/testReport/ | | Max. process+thread count | 268 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-assembly U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1794/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1794: HBASE-24449: Generate version.h include file during compilation.
Apache-HBase commented on pull request #1794: URL: https://github.com/apache/hbase/pull/1794#issuecomment-635009277 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 40s | master passed | | +1 :green_heart: | compile | 0m 48s | master passed | | +1 :green_heart: | shadedjars | 5m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 0m 26s | hbase-assembly in the patch passed. | | | | 25m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1794/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1794 | | JIRA Issue | HBASE-24449 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cf1a99424f0a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1794/1/testReport/ | | Max. process+thread count | 339 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-assembly U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1794/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1794: HBASE-24449: Generate version.h include file during compilation.
Apache-HBase commented on pull request #1794: URL: https://github.com/apache/hbase/pull/1794#issuecomment-635007327 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 7s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 11m 31s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 24s | The patch does not generate ASF License warnings. | | | | 19m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1794/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1794 | | JIRA Issue | HBASE-24449 | | Optional Tests | dupname asflicense xml hadoopcheck shellcheck shelldocs | | uname | Linux b0bc251742c1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Max. process+thread count | 70 (vs. ulimit of 12500) | | modules | C: hbase-assembly hbase-common U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1794/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache-HBase commented on pull request #1774: URL: https://github.com/apache/hbase/pull/1774#issuecomment-635006849 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 17s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 6m 12s | master passed | | +1 :green_heart: | compile | 3m 7s | master passed | | +1 :green_heart: | shadedjars | 6m 22s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 15s | hbase-zookeeper in master failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 30s | the patch passed | | +1 :green_heart: | compile | 2m 59s | the patch passed | | +1 :green_heart: | javac | 2m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 20s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 15s | hbase-zookeeper in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 1s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 15s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 46s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 186m 53s | hbase-server in the patch passed. | | | | 226m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1774 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 73ed931e6550 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-zookeeper.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-zookeeper.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/testReport/ | | Max. process+thread count | 3217 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-native-client] bharathv commented on a change in pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
bharathv commented on a change in pull request #2:
URL: https://github.com/apache/hbase-native-client/pull/2#discussion_r431501510
##
File path: cmake/DownloadFolly.cmake
##
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+## Download facebook's folly library.
+## SOURCE_DIR is typically the cmake source directory
+## BINARY_DIR is the build directory, typically 'build'
+
+function(download_folly SOURCE_DIR BINARY_DIR)
+
+
+ ExternalProject_Add(
+ facebook-folly-proj
+ GIT_REPOSITORY "https://github.com/facebook/folly.git;
+ GIT_TAG "v2020.05.18.00"
+ SOURCE_DIR "${BINARY_DIR}/dependencies/facebook-folly-proj-src"
+ CMAKE_ARGS ${PASSTHROUGH_CMAKE_ARGS}
Review comment:
https://github.com/apache/hbase/pull/1794 is what I was talking about. I
couldn't add you to the PR for review but feel free to take a look. Hope it
helps you in this effort.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase-native-client] bharathv commented on a change in pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
bharathv commented on a change in pull request #2:
URL: https://github.com/apache/hbase-native-client/pull/2#discussion_r431501510
##
File path: cmake/DownloadFolly.cmake
##
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+## Download facebook's folly library.
+## SOURCE_DIR is typically the cmake source directory
+## BINARY_DIR is the build directory, typically 'build'
+
+function(download_folly SOURCE_DIR BINARY_DIR)
+
+
+ ExternalProject_Add(
+ facebook-folly-proj
+ GIT_REPOSITORY "https://github.com/facebook/folly.git;
+ GIT_TAG "v2020.05.18.00"
+ SOURCE_DIR "${BINARY_DIR}/dependencies/facebook-folly-proj-src"
+ CMAKE_ARGS ${PASSTHROUGH_CMAKE_ARGS}
Review comment:
https://github.com/apache/hbase/pull/1794 is what I was talking about. I
couldn't add you to the PR for review but feel free to talk a look. Hope it
helps you in this effort.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv opened a new pull request #1794: HBASE-24449: Generate version.h include file during compilation.
bharathv opened a new pull request #1794: URL: https://github.com/apache/hbase/pull/1794 This is needed for the hbase-native-client to compile. Also ships the includes in the assembly tar. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24449) Generate version.h include file during compilation
Bharath Vissapragada created HBASE-24449:
Summary: Generate version.h include file during compilation
Key: HBASE-24449
URL: https://issues.apache.org/jira/browse/HBASE-24449
Project: HBase
Issue Type: Sub-task
Components: Client, native-client
Affects Versions: 3.0.0-alpha-1
Reporter: Bharath Vissapragada
Assignee: Bharath Vissapragada
Curious case of {{version.h}}
hbase-native-client relies on this include file to compile. However that is not
generated by the HBase repo. I see no trace of it in the git log, wonder how it
even worked until now.
I think it will be useful to generate it during compilation and package it with
the release tar ball. This helps [~phrocker] in HBASE-24400.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22041) [k8s] The crashed node exists in onlineServer forever, and if it holds the meta data, master will start up hang.
[
https://issues.apache.org/jira/browse/HBASE-22041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118159#comment-17118159
]
Michael Stack commented on HBASE-22041:
---
The TRACE-level logs really help. Thanks [~timoha]
It looks like sun.net.inetaddr.ttl == networkaddress.cache.ttl.
If set to 10, that might be too long looking at the log.
I see this in the log around the time that regionserver-2 came back up:
{code}
2020-05-21 17:28:55,010 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=48,queue=3,port=16000]
master.ServerManager: STARTUP: Server
regionserver-2.hbase.hbase.svc.cluster.local,16020,1590082132059 came back up,
removed it from the dead servers list
{code}
What above says is that the old server instance,
regionserver-2.hbase.hbase.svc.cluster.local,16020,1590081637367, has been
replaced in the Master by the
regionserver-2.hbase.hbase.svc.cluster.local,16020,1590082132059 (see the
differing timestamps).
A little later in the log I see this:
{code}
2020-05-21 17:28:55,745 DEBUG [RSProcedureDispatcher-pool3-t31]
master.ServerManager: New admin connection to
regionserver-2.hbase.hbase.svc.cluster.local,16020,1590082132059
2020-05-21 17:28:55,745 TRACE [RSProcedureDispatcher-pool3-t31]
ipc.NettyRpcConnection: Connecting to
regionserver-2.hbase.hbase.svc.cluster.local/10.128.14.39:16020
...
{code}
i.e. the ServerManager is processing the 'new' server part of which is setting
up a new connection.
Logging is a little sparse here but code looks like it should do right
thing.. The ServerManager asks to get the admin instance which ends up in
here
{code}
@Override
public AdminProtos.AdminService.BlockingInterface getAdmin(ServerName
serverName)
throws IOException {
checkClosed();
if (isDeadServer(serverName)) {
throw new RegionServerStoppedException(serverName + " is dead.");
}
String key =
getStubKey(AdminProtos.AdminService.BlockingInterface.class.getName(),
serverName,
this.hostnamesCanChange);
return (AdminProtos.AdminService.BlockingInterface)
computeIfAbsentEx(stubs, key, () -> {
BlockingRpcChannel channel =
this.rpcClient.createBlockingRpcChannel(serverName, user, rpcTimeout);
return AdminProtos.AdminService.newBlockingStub(channel);
});
}
{code}
The call to getStubKey should be doing a new lookup (hostnamesCanChange
defaults true -- see HBASE-14544) but it is happening at~17:28:55,745.
The new server is registered at the Master at ~17:28:55,010.
When did K8S register in DNS the new pod?
What happens if you run w/ -Dsun.net.inetaddr.ttl=1 instead of 10?
> [k8s] The crashed node exists in onlineServer forever, and if it holds the
> meta data, master will start up hang.
>
>
> Key: HBASE-22041
> URL: https://issues.apache.org/jira/browse/HBASE-22041
> Project: HBase
> Issue Type: Bug
>Reporter: lujie
>Priority: Critical
> Attachments: bug.zip, hbasemaster.log, normal.zip
>
>
> while master fresh boot, we crash (kill- 9) the RS who hold meta. we find
> that the master startup fails and print thounds of logs like:
> {code:java}
> 2019-03-13 01:09:54,896 WARN [RSProcedureDispatcher-pool4-t1]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to java.net.ConnectException: Call to
> hadoop14/172.16.1.131:16020 failed on connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> syscall:getsockopt(..) failed: Connection refused:
> hadoop14/172.16.1.131:16020, try=0, retrying...
> 2019-03-13 01:09:55,004 WARN [RSProcedureDispatcher-pool4-t2]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=1, retrying...
> 2019-03-13 01:09:55,114 WARN [RSProcedureDispatcher-pool4-t3]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=2, retrying...
> 2019-03-13 01:09:55,219 WARN [RSProcedureDispatcher-pool4-t4]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
>
[GitHub] [hbase] ndimiduk commented on a change in pull request #1791: HBASE-23202 ExportSnapshot (import) will fail if copying files to roo…
ndimiduk commented on a change in pull request #1791:
URL: https://github.com/apache/hbase/pull/1791#discussion_r431486691
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
##
@@ -251,6 +260,25 @@ private void refreshCache() throws IOException {
this.snapshots.putAll(newSnapshots);
}
+ @VisibleForTesting
+ List getSnapshotsInProgress() throws IOException {
+List snapshotInProgress = Lists.newArrayList();
+// only add those files to the cache, but not to the known snapshots
+FileStatus[] snapshotsInProgress = CommonFSUtils.listStatus(fs,
+ new Path(snapshotDir, SnapshotDescriptionUtils.SNAPSHOT_TMP_DIR_NAME));
+
+if (!ArrayUtils.isEmpty(snapshotsInProgress)) {
+ for (FileStatus snapshot : snapshotsInProgress) {
+try {
+
snapshotInProgress.addAll(fileInspector.filesUnderSnapshot(snapshot.getPath()));
+} catch (CorruptedSnapshotException cse) {
+ LOG.debug("Corrupted in-progress snapshot file exception, ignored.",
cse);
Review comment:
`CorruptedSnapshotException` contains a copy of `SnapshotDescription`.
Where this is logged, instead of logging the exception, it can grab the snaps
tho info out of the description field to build a more meaningful log message.
I don't know about the log level. I guess it depends on if you think this is
a really severe problem or not. If not, `debug` is fine. If it's the only sign
an operator has of a bad problem, or it should be `info` or `warn`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (HBASE-24448) Flaky tests for region server in branch-1
Clara Xiong created HBASE-24448: --- Summary: Flaky tests for region server in branch-1 Key: HBASE-24448 URL: https://issues.apache.org/jira/browse/HBASE-24448 Project: HBase Issue Type: Bug Components: regionserver, test Affects Versions: 1.7.0 Reporter: Clara Xiong [https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1781/4/testReport/] Tests failed after unrelated code style changes for metrics for branch-1. Rerun passed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1792: HBASE-24447: Contribute a Test class that shows some examples for usi…
Apache-HBase commented on pull request #1792: URL: https://github.com/apache/hbase/pull/1792#issuecomment-634983048 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 39s | branch-2 passed | | +1 :green_heart: | compile | 0m 54s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 14s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 4m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 132m 52s | hbase-server in the patch failed. | | | | 155m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1792 | | JIRA Issue | HBASE-24447 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b531eb76facf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 1e86ff09d7 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/1/testReport/ | | Max. process+thread count | 3722 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1792/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1743: HBASE-24403 FsDelegationToken should cache token after acquired a new one
Apache-HBase commented on pull request #1743: URL: https://github.com/apache/hbase/pull/1743#issuecomment-634981783 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 27s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 6m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | the patch passed | | +1 :green_heart: | compile | 1m 8s | the patch passed | | +1 :green_heart: | javac | 1m 8s | the patch passed | | +1 :green_heart: | shadedjars | 6m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 198m 11s | hbase-server in the patch failed. | | | | 227m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1743 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1748c40183a9 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/3/testReport/ | | Max. process+thread count | 2876 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786: URL: https://github.com/apache/hbase/pull/1786#discussion_r431473726 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java ## @@ -18,126 +17,436 @@ */ package org.apache.hadoop.hbase.master.normalizer; +import java.io.IOException; +import java.time.Instant; +import java.time.Period; import java.util.ArrayList; import java.util.Collections; import java.util.Comparator; import java.util.List; - -import org.apache.hadoop.hbase.HBaseIOException; +import java.util.Objects; +import java.util.function.BooleanSupplier; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseInterfaceAudience; +import org.apache.hadoop.hbase.RegionMetrics; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.Size; import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.client.MasterSwitchType; import org.apache.hadoop.hbase.client.RegionInfo; +import org.apache.hadoop.hbase.client.TableDescriptor; +import org.apache.hadoop.hbase.master.MasterServices; +import org.apache.hadoop.hbase.master.RegionState; +import org.apache.hadoop.hbase.master.assignment.RegionStates; import org.apache.hadoop.hbase.master.normalizer.NormalizationPlan.PlanType; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; import org.apache.yetus.audience.InterfaceAudience; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils; /** * Simple implementation of region normalizer. Logic in use: * - * Get all regions of a given table - * Get avg size S of each region (by total size of store files reported in RegionMetrics) - * Seek every single region one by one. If a region R0 is bigger than S * 2, it is kindly - * requested to split. Thereon evaluate the next region R1 - * Otherwise, if R0 + R1 is smaller than S, R0 and R1 are kindly requested to merge. Thereon - * evaluate the next region R2 - * Otherwise, R1 is evaluated + * Get all regions of a given table + * Get avg size S of the regions in the table (by total size of store files reported in + * RegionMetrics) + * For each region R0, if R0 is bigger than S * 2, it is kindly requested to split. + * Otherwise, for the next region in the chain R1, if R0 + R1 is smaller then S, R0 and R1 + * are kindly requested to merge. + * + * + * The following parameters are configurable: Review comment: I've done some word-smithing here. However, the bulk of the documentation on this feature appears to have been copy-pasted from a post to Hortonworks forums. Rather than edit this work, I would prefer we start from scratch with our own documentation, so as to not mis-represent the work of that individual. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1791: HBASE-23202 ExportSnapshot (import) will fail if copying files to roo…
Apache-HBase commented on pull request #1791: URL: https://github.com/apache/hbase/pull/1791#issuecomment-634968890 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 47s | master passed | | +1 :green_heart: | compile | 1m 9s | master passed | | +1 :green_heart: | shadedjars | 6m 25s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 48s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 52s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 6m 26s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 236m 34s | hbase-server in the patch failed. | | | | 267m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1791/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1791 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6f84e372f262 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 48e9835adb | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1791/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1791/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1791/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1791/1/testReport/ | | Max. process+thread count | 3084 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1791/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #1791: HBASE-23202 ExportSnapshot (import) will fail if copying files to roo…
huaxiangsun commented on a change in pull request #1791:
URL: https://github.com/apache/hbase/pull/1791#discussion_r431469964
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
##
@@ -251,6 +260,25 @@ private void refreshCache() throws IOException {
this.snapshots.putAll(newSnapshots);
}
+ @VisibleForTesting
+ List getSnapshotsInProgress() throws IOException {
+List snapshotInProgress = Lists.newArrayList();
+// only add those files to the cache, but not to the known snapshots
+FileStatus[] snapshotsInProgress = CommonFSUtils.listStatus(fs,
+ new Path(snapshotDir, SnapshotDescriptionUtils.SNAPSHOT_TMP_DIR_NAME));
+
+if (!ArrayUtils.isEmpty(snapshotsInProgress)) {
+ for (FileStatus snapshot : snapshotsInProgress) {
+try {
+
snapshotInProgress.addAll(fileInspector.filesUnderSnapshot(snapshot.getPath()));
+} catch (CorruptedSnapshotException cse) {
+ LOG.debug("Corrupted in-progress snapshot file exception, ignored.",
cse);
Review comment:
It will be hard to put table name here. For an example, in the context
of ExportSnapshot job, at the destination cluster, the .snapshotinfo is written
to tmp directory first. There is no table existing yet, table may be cloned
from snapshot after ExportSnapshot job is done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #1786: HBASE-24418 Consolidate Normalizer implementations
saintstack commented on pull request #1786: URL: https://github.com/apache/hbase/pull/1786#issuecomment-634963002 Interesting. Wonder why the change was made. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #1791: HBASE-23202 ExportSnapshot (import) will fail if copying files to roo…
huaxiangsun commented on a change in pull request #1791:
URL: https://github.com/apache/hbase/pull/1791#discussion_r431464432
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
##
@@ -251,6 +260,25 @@ private void refreshCache() throws IOException {
this.snapshots.putAll(newSnapshots);
}
+ @VisibleForTesting
+ List getSnapshotsInProgress() throws IOException {
+List snapshotInProgress = Lists.newArrayList();
+// only add those files to the cache, but not to the known snapshots
+FileStatus[] snapshotsInProgress = CommonFSUtils.listStatus(fs,
+ new Path(snapshotDir, SnapshotDescriptionUtils.SNAPSHOT_TMP_DIR_NAME));
+
+if (!ArrayUtils.isEmpty(snapshotsInProgress)) {
+ for (FileStatus snapshot : snapshotsInProgress) {
+try {
+
snapshotInProgress.addAll(fileInspector.filesUnderSnapshot(snapshot.getPath()));
+} catch (CorruptedSnapshotException cse) {
+ LOG.debug("Corrupted in-progress snapshot file exception, ignored.",
cse);
Review comment:
It is hard to say. The CorruptedSnapshotException will only be thrown
out when it reads a corrupted .snapshotinfo file. There are two cases, one is
normal, which .snapshotinfo is being written and it reads incomplete one; the
other is that it is a real corrupted .snapshotinfo. The original code assumes
the first case (maybe in reality, there are more first cases than the second
one). Change it to info level? The exception has the following info
```
throw new CorruptedSnapshotException("Couldn't read snapshot info from:" +
snapshotInfo, e);
```
It is the path of .snapshotinfo, it has snapshot name embedded in the path
(does not have table name). Let me see if table name can be added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on pull request #1786: URL: https://github.com/apache/hbase/pull/1786#issuecomment-634960241 > Generally, I think the default is too aggressive. @saintstack Looks liked we originally ran the normalizer on a 30min period. This was reduced to 5 min for 2.0, according to https://hbase.apache.org/book.html#upgrade2.0.changed.defaults This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1793: HBASE-23941 : FilterBy operator support in get_slowlog_responses API
Apache-HBase commented on pull request #1793: URL: https://github.com/apache/hbase/pull/1793#issuecomment-634955841 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 36s | master passed | | +1 :green_heart: | checkstyle | 1m 55s | master passed | | +0 :ok: | refguide | 4m 57s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 14m 26s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | -0 :warning: | checkstyle | 2m 0s | root: The patch generated 2 new + 76 unchanged - 0 fixed = 78 total (was 76) | | -0 :warning: | rubocop | 0m 11s | The patch generated 7 new + 326 unchanged - 1 fixed = 333 total (was 327) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 4m 48s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 11m 8s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 6m 2s | the patch passed | | +1 :green_heart: | spotbugs | 15m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | | | | 77m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1793 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool rubocop refguide | | uname | Linux bee38ded3f0c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-general-check/output/branch-site/book.html | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | rubocop | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 137 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24434) fix building cpp-example DemoClient
[ https://issues.apache.org/jira/browse/HBASE-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-24434: - Component/s: (was: native-client) > fix building cpp-example DemoClient > --- > > Key: HBASE-24434 > URL: https://issues.apache.org/jira/browse/HBASE-24434 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0 >Reporter: Alexander Myasnikov >Priority: Major > Labels: thrift > Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.4.0 > > Attachments: HBASE-24434-branch-1.patch, HBASE-24434-branch-2.patch, > HBASE-24434-master.patch > > > Changed boost::shared_ptr to std::shared_ptr for compatibility with apache > thrift v0.13. > https://github.com/apache/hbase/pull/1775 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] huaxiangsun edited a comment on pull request #1786: HBASE-24418 Consolidate Normalizer implementations
huaxiangsun edited a comment on pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#issuecomment-634951336
```
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
index 6b75ed7881..eeb9bef0c8 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
@@ -396,13 +396,13 @@ public class TestSimpleRegionNormalizer {
final byte[][] keys = {
null,
Bytes.toBytes("aa"),
- Bytes.toBytes("aa1"),
Bytes.toBytes("aa1!"),
+ Bytes.toBytes("aa1"),
Bytes.toBytes("aa2"),
null,
};
final List regionInfos = createRegionInfos(tableName, keys);
-final Map regionSizes =
createRegionSizesMap(regionInfos, 3, 1, 3, 1, 5);
+final Map regionSizes =
createRegionSizesMap(regionInfos, 3, 1, 1, 3, 5);
setupMocksForNormalizer(regionSizes, regionInfos);
// Compute the plan, no merge plan returned as they are not adjacent.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun edited a comment on pull request #1786: HBASE-24418 Consolidate Normalizer implementations
huaxiangsun edited a comment on pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#issuecomment-634951336
`diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
index 6b75ed7881..eeb9bef0c8 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
@@ -396,13 +396,13 @@ public class TestSimpleRegionNormalizer {
final byte[][] keys = {
null,
Bytes.toBytes("aa"),
- Bytes.toBytes("aa1"),
Bytes.toBytes("aa1!"),
+ Bytes.toBytes("aa1"),
Bytes.toBytes("aa2"),
null,
};
final List regionInfos = createRegionInfos(tableName, keys);
-final Map regionSizes =
createRegionSizesMap(regionInfos, 3, 1, 3, 1, 5);
+final Map regionSizes =
createRegionSizesMap(regionInfos, 3, 1, 1, 3, 5);
setupMocksForNormalizer(regionSizes, regionInfos);
// Compute the plan, no merge plan returned as they are not adjacent.`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #1786: HBASE-24418 Consolidate Normalizer implementations
huaxiangsun commented on pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#issuecomment-634951336
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
index 6b75ed7881..eeb9bef0c8 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
@@ -396,13 +396,13 @@ public class TestSimpleRegionNormalizer {
final byte[][] keys = {
null,
Bytes.toBytes("aa"),
- Bytes.toBytes("aa1"),
Bytes.toBytes("aa1!"),
+ Bytes.toBytes("aa1"),
Bytes.toBytes("aa2"),
null,
};
final List regionInfos = createRegionInfos(tableName, keys);
-final Map regionSizes =
createRegionSizesMap(regionInfos, 3, 1, 3, 1, 5);
+final Map regionSizes =
createRegionSizesMap(regionInfos, 3, 1, 1, 3, 5);
setupMocksForNormalizer(regionSizes, regionInfos);
// Compute the plan, no merge plan returned as they are not adjacent.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
huaxiangsun commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r431452897
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
##
@@ -69,517 +78,347 @@
public static final HBaseClassTestRule CLASS_RULE =
HBaseClassTestRule.forClass(TestSimpleRegionNormalizer.class);
- private static final Logger LOG =
LoggerFactory.getLogger(TestSimpleRegionNormalizer.class);
-
- private RegionNormalizer normalizer;
+ private Configuration conf;
+ private SimpleRegionNormalizer normalizer;
private MasterServices masterServices;
@Rule
public TestName name = new TestName();
- @Test
- public void testPlanComparator() {
-Comparator comparator = new
SimpleRegionNormalizer.PlanComparator();
-NormalizationPlan splitPlan1 = new SplitNormalizationPlan(null, null);
-NormalizationPlan splitPlan2 = new SplitNormalizationPlan(null, null);
-NormalizationPlan mergePlan1 = new MergeNormalizationPlan(null, null);
-NormalizationPlan mergePlan2 = new MergeNormalizationPlan(null, null);
-
-assertEquals(0, comparator.compare(splitPlan1, splitPlan2));
-assertEquals(0, comparator.compare(splitPlan2, splitPlan1));
-assertEquals(0, comparator.compare(mergePlan1, mergePlan2));
-assertEquals(0, comparator.compare(mergePlan2, mergePlan1));
-assertTrue(comparator.compare(splitPlan1, mergePlan1) < 0);
-assertTrue(comparator.compare(mergePlan1, splitPlan1) > 0);
+ @Before
+ public void before() {
+conf = HBaseConfiguration.create();
}
@Test
- public void testNoNormalizationForMetaTable() throws HBaseIOException {
+ public void testNoNormalizationForMetaTable() {
TableName testTable = TableName.META_TABLE_NAME;
List RegionInfo = new ArrayList<>();
Map regionSizes = new HashMap<>();
setupMocksForNormalizer(regionSizes, RegionInfo);
-List plans = normalizer.computePlanForTable(testTable);
-assertNull(plans);
+List plans = normalizer.computePlansForTable(testTable);
+assertThat(plans, empty());
}
@Test
- public void testNoNormalizationIfTooFewRegions() throws HBaseIOException {
+ public void testNoNormalizationIfTooFewRegions() {
final TableName tableName = TableName.valueOf(name.getMethodName());
-List RegionInfo = new ArrayList<>();
-Map regionSizes = new HashMap<>();
-RegionInfo hri1 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("aaa"))
-.setEndKey(Bytes.toBytes("bbb"))
-.build();
-RegionInfo.add(hri1);
-regionSizes.put(hri1.getRegionName(), 10);
-
-RegionInfo hri2 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("bbb"))
-.setEndKey(Bytes.toBytes("ccc"))
-.build();
-RegionInfo.add(hri2);
-regionSizes.put(hri2.getRegionName(), 15);
+final List regionInfos = createRegionInfos(tableName, 2);
+final Map regionSizes = createRegionSizesMap(regionInfos,
10, 15);
+setupMocksForNormalizer(regionSizes, regionInfos);
-setupMocksForNormalizer(regionSizes, RegionInfo);
-List plans = normalizer.computePlanForTable(tableName);
-assertNull(plans);
+List plans = normalizer.computePlansForTable(tableName);
+assertThat(plans, empty());
}
@Test
- public void testNoNormalizationOnNormalizedCluster() throws HBaseIOException
{
+ public void testNoNormalizationOnNormalizedCluster() {
final TableName tableName = TableName.valueOf(name.getMethodName());
-List RegionInfo = new ArrayList<>();
-Map regionSizes = new HashMap<>();
-
-RegionInfo hri1 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("aaa"))
-.setEndKey(Bytes.toBytes("bbb"))
-.build();
-RegionInfo.add(hri1);
-regionSizes.put(hri1.getRegionName(), 10);
-
-RegionInfo hri2 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("bbb"))
-.setEndKey(Bytes.toBytes("ccc"))
-.build();
-RegionInfo.add(hri2);
-regionSizes.put(hri2.getRegionName(), 15);
-
-RegionInfo hri3 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("ccc"))
-.setEndKey(Bytes.toBytes("ddd"))
-.build();
-RegionInfo.add(hri3);
-regionSizes.put(hri3.getRegionName(), 8);
-
-RegionInfo hri4 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("ddd"))
-.setEndKey(Bytes.toBytes("eee"))
-.build();
-regionSizes.put(hri4.getRegionName(), 10);
+final List regionInfos = createRegionInfos(tableName, 4);
+final Map regionSizes = createRegionSizesMap(regionInfos,
10, 15, 8, 10);
+setupMocksForNormalizer(regionSizes, regionInfos);
-setupMocksForNormalizer(regionSizes, RegionInfo);
-List plans = normalizer.computePlanForTable(tableName);
-assertNull(plans);
+List plans =
[GitHub] [hbase] huaxiangsun commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
huaxiangsun commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r431452085
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
##
@@ -69,517 +78,347 @@
public static final HBaseClassTestRule CLASS_RULE =
HBaseClassTestRule.forClass(TestSimpleRegionNormalizer.class);
- private static final Logger LOG =
LoggerFactory.getLogger(TestSimpleRegionNormalizer.class);
-
- private RegionNormalizer normalizer;
+ private Configuration conf;
+ private SimpleRegionNormalizer normalizer;
private MasterServices masterServices;
@Rule
public TestName name = new TestName();
- @Test
- public void testPlanComparator() {
-Comparator comparator = new
SimpleRegionNormalizer.PlanComparator();
-NormalizationPlan splitPlan1 = new SplitNormalizationPlan(null, null);
-NormalizationPlan splitPlan2 = new SplitNormalizationPlan(null, null);
-NormalizationPlan mergePlan1 = new MergeNormalizationPlan(null, null);
-NormalizationPlan mergePlan2 = new MergeNormalizationPlan(null, null);
-
-assertEquals(0, comparator.compare(splitPlan1, splitPlan2));
-assertEquals(0, comparator.compare(splitPlan2, splitPlan1));
-assertEquals(0, comparator.compare(mergePlan1, mergePlan2));
-assertEquals(0, comparator.compare(mergePlan2, mergePlan1));
-assertTrue(comparator.compare(splitPlan1, mergePlan1) < 0);
-assertTrue(comparator.compare(mergePlan1, splitPlan1) > 0);
+ @Before
+ public void before() {
+conf = HBaseConfiguration.create();
}
@Test
- public void testNoNormalizationForMetaTable() throws HBaseIOException {
+ public void testNoNormalizationForMetaTable() {
TableName testTable = TableName.META_TABLE_NAME;
List RegionInfo = new ArrayList<>();
Map regionSizes = new HashMap<>();
setupMocksForNormalizer(regionSizes, RegionInfo);
-List plans = normalizer.computePlanForTable(testTable);
-assertNull(plans);
+List plans = normalizer.computePlansForTable(testTable);
+assertThat(plans, empty());
}
@Test
- public void testNoNormalizationIfTooFewRegions() throws HBaseIOException {
+ public void testNoNormalizationIfTooFewRegions() {
final TableName tableName = TableName.valueOf(name.getMethodName());
-List RegionInfo = new ArrayList<>();
-Map regionSizes = new HashMap<>();
-RegionInfo hri1 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("aaa"))
-.setEndKey(Bytes.toBytes("bbb"))
-.build();
-RegionInfo.add(hri1);
-regionSizes.put(hri1.getRegionName(), 10);
-
-RegionInfo hri2 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("bbb"))
-.setEndKey(Bytes.toBytes("ccc"))
-.build();
-RegionInfo.add(hri2);
-regionSizes.put(hri2.getRegionName(), 15);
+final List regionInfos = createRegionInfos(tableName, 2);
+final Map regionSizes = createRegionSizesMap(regionInfos,
10, 15);
+setupMocksForNormalizer(regionSizes, regionInfos);
-setupMocksForNormalizer(regionSizes, RegionInfo);
-List plans = normalizer.computePlanForTable(tableName);
-assertNull(plans);
+List plans = normalizer.computePlansForTable(tableName);
+assertThat(plans, empty());
}
@Test
- public void testNoNormalizationOnNormalizedCluster() throws HBaseIOException
{
+ public void testNoNormalizationOnNormalizedCluster() {
final TableName tableName = TableName.valueOf(name.getMethodName());
-List RegionInfo = new ArrayList<>();
-Map regionSizes = new HashMap<>();
-
-RegionInfo hri1 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("aaa"))
-.setEndKey(Bytes.toBytes("bbb"))
-.build();
-RegionInfo.add(hri1);
-regionSizes.put(hri1.getRegionName(), 10);
-
-RegionInfo hri2 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("bbb"))
-.setEndKey(Bytes.toBytes("ccc"))
-.build();
-RegionInfo.add(hri2);
-regionSizes.put(hri2.getRegionName(), 15);
-
-RegionInfo hri3 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("ccc"))
-.setEndKey(Bytes.toBytes("ddd"))
-.build();
-RegionInfo.add(hri3);
-regionSizes.put(hri3.getRegionName(), 8);
-
-RegionInfo hri4 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("ddd"))
-.setEndKey(Bytes.toBytes("eee"))
-.build();
-regionSizes.put(hri4.getRegionName(), 10);
+final List regionInfos = createRegionInfos(tableName, 4);
+final Map regionSizes = createRegionSizesMap(regionInfos,
10, 15, 8, 10);
+setupMocksForNormalizer(regionSizes, regionInfos);
-setupMocksForNormalizer(regionSizes, RegionInfo);
-List plans = normalizer.computePlanForTable(tableName);
-assertNull(plans);
+List plans =
[jira] [Commented] (HBASE-24434) fix building cpp-example DemoClient
[ https://issues.apache.org/jira/browse/HBASE-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118110#comment-17118110 ] Bharath Vissapragada commented on HBASE-24434: -- Committed to branch-1/2/2.3. Thanks for your contribution. Appreciate if you can use "git format-patch" next time you attach patches. That includes additional meta information unlike a plain diff. Also, can you please send an email to d...@hbase.apache.org so that you can assign this jira to yourself? Some jira admin on the list will add your id in the right places. I don't have access to that and I'm not able to assign this jira to you. > fix building cpp-example DemoClient > --- > > Key: HBASE-24434 > URL: https://issues.apache.org/jira/browse/HBASE-24434 > Project: HBase > Issue Type: Bug > Components: Client, native-client >Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0 >Reporter: Alexander Myasnikov >Priority: Major > Labels: thrift > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-24434-branch-1.patch, HBASE-24434-branch-2.patch, > HBASE-24434-master.patch > > > Changed boost::shared_ptr to std::shared_ptr for compatibility with apache > thrift v0.13. > https://github.com/apache/hbase/pull/1775 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24434) fix building cpp-example DemoClient
[ https://issues.apache.org/jira/browse/HBASE-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada resolved HBASE-24434. -- Fix Version/s: 2.4.0 1.7.0 2.3.0 Resolution: Fixed > fix building cpp-example DemoClient > --- > > Key: HBASE-24434 > URL: https://issues.apache.org/jira/browse/HBASE-24434 > Project: HBase > Issue Type: Bug > Components: Client, native-client >Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0 >Reporter: Alexander Myasnikov >Priority: Major > Labels: thrift > Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.4.0 > > Attachments: HBASE-24434-branch-1.patch, HBASE-24434-branch-2.patch, > HBASE-24434-master.patch > > > Changed boost::shared_ptr to std::shared_ptr for compatibility with apache > thrift v0.13. > https://github.com/apache/hbase/pull/1775 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1743: HBASE-24403 FsDelegationToken should cache token after acquired a new one
Apache-HBase commented on pull request #1743: URL: https://github.com/apache/hbase/pull/1743#issuecomment-634949224 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | compile | 1m 4s | master passed | | +1 :green_heart: | shadedjars | 5m 44s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 5m 40s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 124m 22s | hbase-server in the patch passed. | | | | 149m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1743 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux abad2d11cf31 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/3/testReport/ | | Max. process+thread count | 3980 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1791: HBASE-23202 ExportSnapshot (import) will fail if copying files to roo…
Apache-HBase commented on pull request #1791: URL: https://github.com/apache/hbase/pull/1791#issuecomment-634948652 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 0m 58s | master passed | | +1 :green_heart: | shadedjars | 6m 6s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 6m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 197m 11s | hbase-server in the patch passed. | | | | 223m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1791/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1791 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1ab2864134df 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 48e9835adb | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1791/1/testReport/ | | Max. process+thread count | 3593 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1791/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24371) Add more details when print CompactionConfiguration info
[
https://issues.apache.org/jira/browse/HBASE-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118104#comment-17118104
]
Hudson commented on HBASE-24371:
Results for branch branch-2
[build #2679 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2679/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2679/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2679/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2679/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2679/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add more details when print CompactionConfiguration info
>
>
> Key: HBASE-24371
> URL: https://issues.apache.org/jira/browse/HBASE-24371
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.6
>
>
> Now the CompactionConfiguration info logs look like
> {code}
> 2020-05-14 15:32:45,468 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,468 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,469 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,469 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> {code}
> We don't have any details on which region and which column family the
> CompactionConfiguration for. It would be better to include the region name
> and column family detail also in this log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118103#comment-17118103 ] Michael Stack commented on HBASE-11288: --- Updated Split Meta Design link so sharable. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24371) Add more details when print CompactionConfiguration info
[
https://issues.apache.org/jira/browse/HBASE-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118100#comment-17118100
]
Hudson commented on HBASE-24371:
Results for branch branch-2.3
[build #108 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/108/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/108/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/108/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/108/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/108/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add more details when print CompactionConfiguration info
>
>
> Key: HBASE-24371
> URL: https://issues.apache.org/jira/browse/HBASE-24371
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.4.0, 2.2.6
>
>
> Now the CompactionConfiguration info logs look like
> {code}
> 2020-05-14 15:32:45,468 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,468 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,469 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2020-05-14 15:32:45,469 INFO
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020]
> compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files
> [3, 10); ratio 1.20; off-peak ratio 5.00; throttle point 2684354560;
> major period 60480, major jitter 0.50, min locality to compact
> 0.00; tiered compaction: max_age 9223372036854775807, incoming window min
> 6, compaction policy for tiered window
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
> single output for minor true, compaction window factory
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> {code}
> We don't have any details on which region and which column family the
> CompactionConfiguration for. It would be better to include the region name
> and column family detail also in this log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache-HBase commented on pull request #1774: URL: https://github.com/apache/hbase/pull/1774#issuecomment-634940763 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 50s | master passed | | +1 :green_heart: | checkstyle | 2m 12s | master passed | | +1 :green_heart: | spotbugs | 7m 34s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | -0 :warning: | checkstyle | 0m 31s | hbase-client: The patch generated 1 new + 242 unchanged - 2 fixed = 243 total (was 244) | | -0 :warning: | checkstyle | 1m 20s | hbase-server: The patch generated 2 new + 455 unchanged - 5 fixed = 457 total (was 460) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 12m 21s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 2m 40s | the patch passed | | +1 :green_heart: | spotbugs | 10m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | | | | 56m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1774 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool | | uname | Linux fd8c5240dd11 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 79d56499ca | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/6/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r431442449
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
##
@@ -69,517 +78,347 @@
public static final HBaseClassTestRule CLASS_RULE =
HBaseClassTestRule.forClass(TestSimpleRegionNormalizer.class);
- private static final Logger LOG =
LoggerFactory.getLogger(TestSimpleRegionNormalizer.class);
-
- private RegionNormalizer normalizer;
+ private Configuration conf;
+ private SimpleRegionNormalizer normalizer;
private MasterServices masterServices;
@Rule
public TestName name = new TestName();
- @Test
- public void testPlanComparator() {
-Comparator comparator = new
SimpleRegionNormalizer.PlanComparator();
-NormalizationPlan splitPlan1 = new SplitNormalizationPlan(null, null);
-NormalizationPlan splitPlan2 = new SplitNormalizationPlan(null, null);
-NormalizationPlan mergePlan1 = new MergeNormalizationPlan(null, null);
-NormalizationPlan mergePlan2 = new MergeNormalizationPlan(null, null);
-
-assertEquals(0, comparator.compare(splitPlan1, splitPlan2));
-assertEquals(0, comparator.compare(splitPlan2, splitPlan1));
-assertEquals(0, comparator.compare(mergePlan1, mergePlan2));
-assertEquals(0, comparator.compare(mergePlan2, mergePlan1));
-assertTrue(comparator.compare(splitPlan1, mergePlan1) < 0);
-assertTrue(comparator.compare(mergePlan1, splitPlan1) > 0);
+ @Before
+ public void before() {
+conf = HBaseConfiguration.create();
}
@Test
- public void testNoNormalizationForMetaTable() throws HBaseIOException {
+ public void testNoNormalizationForMetaTable() {
TableName testTable = TableName.META_TABLE_NAME;
List RegionInfo = new ArrayList<>();
Map regionSizes = new HashMap<>();
setupMocksForNormalizer(regionSizes, RegionInfo);
-List plans = normalizer.computePlanForTable(testTable);
-assertNull(plans);
+List plans = normalizer.computePlansForTable(testTable);
+assertThat(plans, empty());
}
@Test
- public void testNoNormalizationIfTooFewRegions() throws HBaseIOException {
+ public void testNoNormalizationIfTooFewRegions() {
final TableName tableName = TableName.valueOf(name.getMethodName());
-List RegionInfo = new ArrayList<>();
-Map regionSizes = new HashMap<>();
-RegionInfo hri1 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("aaa"))
-.setEndKey(Bytes.toBytes("bbb"))
-.build();
-RegionInfo.add(hri1);
-regionSizes.put(hri1.getRegionName(), 10);
-
-RegionInfo hri2 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("bbb"))
-.setEndKey(Bytes.toBytes("ccc"))
-.build();
-RegionInfo.add(hri2);
-regionSizes.put(hri2.getRegionName(), 15);
+final List regionInfos = createRegionInfos(tableName, 2);
+final Map regionSizes = createRegionSizesMap(regionInfos,
10, 15);
+setupMocksForNormalizer(regionSizes, regionInfos);
-setupMocksForNormalizer(regionSizes, RegionInfo);
-List plans = normalizer.computePlanForTable(tableName);
-assertNull(plans);
+List plans = normalizer.computePlansForTable(tableName);
+assertThat(plans, empty());
}
@Test
- public void testNoNormalizationOnNormalizedCluster() throws HBaseIOException
{
+ public void testNoNormalizationOnNormalizedCluster() {
final TableName tableName = TableName.valueOf(name.getMethodName());
-List RegionInfo = new ArrayList<>();
-Map regionSizes = new HashMap<>();
-
-RegionInfo hri1 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("aaa"))
-.setEndKey(Bytes.toBytes("bbb"))
-.build();
-RegionInfo.add(hri1);
-regionSizes.put(hri1.getRegionName(), 10);
-
-RegionInfo hri2 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("bbb"))
-.setEndKey(Bytes.toBytes("ccc"))
-.build();
-RegionInfo.add(hri2);
-regionSizes.put(hri2.getRegionName(), 15);
-
-RegionInfo hri3 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("ccc"))
-.setEndKey(Bytes.toBytes("ddd"))
-.build();
-RegionInfo.add(hri3);
-regionSizes.put(hri3.getRegionName(), 8);
-
-RegionInfo hri4 = RegionInfoBuilder.newBuilder(tableName)
-.setStartKey(Bytes.toBytes("ddd"))
-.setEndKey(Bytes.toBytes("eee"))
-.build();
-regionSizes.put(hri4.getRegionName(), 10);
+final List regionInfos = createRegionInfos(tableName, 4);
+final Map regionSizes = createRegionSizesMap(regionInfos,
10, 15, 8, 10);
+setupMocksForNormalizer(regionSizes, regionInfos);
-setupMocksForNormalizer(regionSizes, RegionInfo);
-List plans = normalizer.computePlanForTable(tableName);
-assertNull(plans);
+List plans =
[GitHub] [hbase] huaxiangsun commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
huaxiangsun commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r431383688
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -400,6 +401,8 @@ public void run() {
private final LockManager lockManager = new LockManager(this);
private RSGroupBasedLoadBalancer balancer;
+ // a lock to prevent concurrent normalization actions.
+ private final ReentrantLock normalizationInProgressLock = new
ReentrantLock();
Review comment:
It can also run from cli.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -1911,43 +1912,51 @@ public boolean normalizeRegions() throws IOException {
return false;
}
-synchronized (this.normalizer) {
+if (!normalizationInProgressLock.tryLock()) {
// Don't run the normalizer concurrently
+ LOG.info("Normalization already in progress. Skipping request.");
+} else {
+ try {
+List allEnabledTables = new ArrayList<>(
+ tableStateManager.getTablesInStates(TableState.State.ENABLED));
+Collections.shuffle(allEnabledTables);
- List allEnabledTables = new ArrayList<>(
-this.tableStateManager.getTablesInStates(TableState.State.ENABLED));
-
- Collections.shuffle(allEnabledTables);
-
- for (TableName table : allEnabledTables) {
-TableDescriptor tblDesc = getTableDescriptors().get(table);
-if (table.isSystemTable() || (tblDesc != null &&
-!tblDesc.isNormalizationEnabled())) {
- LOG.trace("Skipping normalization for {}, as it's either system"
- + " table or doesn't have auto normalization turned on", table);
- continue;
-}
+for (TableName table : allEnabledTables) {
+ if (table.isSystemTable()) {
+continue;
+ }
+ final TableDescriptor tblDesc = getTableDescriptors().get(table);
+ if (tblDesc != null && !tblDesc.isNormalizationEnabled()) {
+LOG.debug(
+ "Skipping {} because normalization is disabled in its table
properties.", table);
+continue;
+ }
-// make one last check that the cluster isn't shutting down before
proceeding.
-if (skipRegionManagementAction("region normalizer")) {
- return false;
-}
+ // make one last check that the cluster isn't shutting down before
proceeding.
+ if (skipRegionManagementAction("region normalizer")) {
+return false;
+ }
-final List plans =
this.normalizer.computePlanForTable(table);
-if (CollectionUtils.isEmpty(plans)) {
- return true;
-}
+ final List plans =
normalizer.computePlansForTable(table);
+ if (CollectionUtils.isEmpty(plans)) {
+return true;
+ }
-try (final Admin admin =
asyncClusterConnection.toConnection().getAdmin()) {
- for (NormalizationPlan plan : plans) {
-plan.execute(admin);
-if (plan.getType() == PlanType.SPLIT) {
- splitPlanCount++;
-} else if (plan.getType() == PlanType.MERGE) {
- mergePlanCount++;
+ try (final Admin admin =
asyncClusterConnection.toConnection().getAdmin()) {
Review comment:
Sorry, this is an enhancement, can
final Admin admin = asyncClusterConnection.toConnection().getAdmin() be
moved to before
for (TableName table : allEnabledTables) {
so it wont create a admin instance for each table?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -18,126 +17,436 @@
*/
package org.apache.hadoop.hbase.master.normalizer;
+import java.io.IOException;
+import java.time.Instant;
+import java.time.Period;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
-
-import org.apache.hadoop.hbase.HBaseIOException;
+import java.util.Objects;
+import java.util.function.BooleanSupplier;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.hbase.RegionMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.Size;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.MasterSwitchType;
import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.assignment.RegionStates;
import org.apache.hadoop.hbase.master.normalizer.NormalizationPlan.PlanType;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
[GitHub] [hbase] brfrn169 commented on pull request #1648: HBASE-8458 Support for batch version of checkAndMutate()
brfrn169 commented on pull request #1648: URL: https://github.com/apache/hbase/pull/1648#issuecomment-634936795 Ping @joshelser This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r431437782
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -1911,43 +1912,51 @@ public boolean normalizeRegions() throws IOException {
return false;
}
-synchronized (this.normalizer) {
+if (!normalizationInProgressLock.tryLock()) {
Review comment:
It hangs out here on the master because, sadly, a bunch of logic is
implemented in `HMaster`. All the stuff about deciding which tables to
normalize and actual execution of the generated plans happens in
`Master#normalizeRegions` (under the same lock we were just discussing). I
don't love this design, but I think it does make sense that a normalizer
algorithm exist mostly outside of the context of the master context in which it
runs. If I were to make the master more of a composition, I would implement a
class called something like `NormalizationService`, which was responsible for
instantiating the instance of the `Normalizer` algorithm, registering the
`Chore` and being the delegation target for invocations coming from RPC. The
lock would be internal to this `NormalizationService`.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -400,6 +401,8 @@ public void run() {
private final LockManager lockManager = new LockManager(this);
private RSGroupBasedLoadBalancer balancer;
+ // a lock to prevent concurrent normalization actions.
+ private final ReentrantLock normalizationInProgressLock = new
ReentrantLock();
Review comment:
There is a `RegionNormalizerChore`, yes. It invokes
`master.normalizeRegions()`. However, the same method can also be called from
`MasterRpcServices`, so this lock (previously a `synchronized(normalizer)
block`) prevents the chore and an operator invocation from colliding.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -797,7 +800,6 @@ protected void initializeZKBasedSystemTrackers()
this.balancer.setConf(conf);
this.normalizer = RegionNormalizerFactory.getRegionNormalizer(conf);
this.normalizer.setMasterServices(this);
-this.normalizer.setMasterRpcServices((MasterRpcServices)rpcServices);
Review comment:
It was using `masterRpcServices` to reach out to check the split and
merge switches. I've replaced that with direct calls using `masterServices`.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -1911,43 +1912,51 @@ public boolean normalizeRegions() throws IOException {
return false;
}
-synchronized (this.normalizer) {
+if (!normalizationInProgressLock.tryLock()) {
// Don't run the normalizer concurrently
+ LOG.info("Normalization already in progress. Skipping request.");
+} else {
+ try {
+List allEnabledTables = new ArrayList<>(
+ tableStateManager.getTablesInStates(TableState.State.ENABLED));
+Collections.shuffle(allEnabledTables);
- List allEnabledTables = new ArrayList<>(
-this.tableStateManager.getTablesInStates(TableState.State.ENABLED));
-
- Collections.shuffle(allEnabledTables);
-
- for (TableName table : allEnabledTables) {
-TableDescriptor tblDesc = getTableDescriptors().get(table);
-if (table.isSystemTable() || (tblDesc != null &&
-!tblDesc.isNormalizationEnabled())) {
- LOG.trace("Skipping normalization for {}, as it's either system"
- + " table or doesn't have auto normalization turned on", table);
- continue;
-}
+for (TableName table : allEnabledTables) {
+ if (table.isSystemTable()) {
+continue;
+ }
+ final TableDescriptor tblDesc = getTableDescriptors().get(table);
+ if (tblDesc != null && !tblDesc.isNormalizationEnabled()) {
+LOG.debug(
+ "Skipping {} because normalization is disabled in its table
properties.", table);
+continue;
+ }
-// make one last check that the cluster isn't shutting down before
proceeding.
-if (skipRegionManagementAction("region normalizer")) {
- return false;
-}
+ // make one last check that the cluster isn't shutting down before
proceeding.
+ if (skipRegionManagementAction("region normalizer")) {
+return false;
+ }
-final List plans =
this.normalizer.computePlanForTable(table);
-if (CollectionUtils.isEmpty(plans)) {
- return true;
-}
+ final List plans =
normalizer.computePlansForTable(table);
+ if (CollectionUtils.isEmpty(plans)) {
+return true;
Review comment:
Yikes! good point.
##
File path:
[jira] [Commented] (HBASE-22285) A normalizer which merges very small size regions with adjacent regions.(MergeToNormalize)
[ https://issues.apache.org/jira/browse/HBASE-22285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118076#comment-17118076 ] Sean Busbey commented on HBASE-22285: - bq. Same comment re: this being a question for the release manager. This is a new feature, so I don't think it's appropriate for a patch release. to reinforce this, no new features in maintenance releases please. if we need to get new features out faster we should just release minor releases at a more regular cadence. > A normalizer which merges very small size regions with adjacent > regions.(MergeToNormalize) > -- > > Key: HBASE-22285 > URL: https://issues.apache.org/jira/browse/HBASE-22285 > Project: HBase > Issue Type: New Feature > Components: master >Reporter: Aman Poonia >Assignee: Aman Poonia >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.0 > > > There are scenarios where we have seen around 5% of total regions with a size > of 0 bytes and another 5-6 % regions with size in a few bytes. These kinds of > regions increase with time considering we have TTL over the rows. > After exploring the option of RegionNormalizer and doing some quick runs we > found that that is not suitable considering it also splits the regions and > merges to normalize. What we really want is to split as per Split policy and > merge very small regions with adjacent regions to make sure we reduce 0-byte > regions. > We can plugin this normalizer using the property > "hbase.master.normalizer.class" -- This message was sent by Atlassian Jira (v8.3.4#803005)
