[GitHub] [hbase] Apache-HBase commented on pull request #1746: HBASE-24388 Introduce a 'local root region' at master side to store t…
Apache-HBase commented on pull request #1746: URL: https://github.com/apache/hbase/pull/1746#issuecomment-632497694 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 33m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 22s | master passed | | +1 :green_heart: | checkstyle | 2m 13s | master passed | | +0 :ok: | refguide | 4m 45s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 4m 0s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 2m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 19s | the patch passed | | -0 :warning: | checkstyle | 1m 8s | hbase-server: The patch generated 1 new + 166 unchanged - 4 fixed = 167 total (was 170) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 4m 36s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 11m 4s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 45s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | | | | 85m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1746/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1746 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle refguide xml | | uname | Linux cd7fb531bd8f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 80b64ef4dc | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1746/2/artifact/yetus-general-check/output/branch-site/book.html | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1746/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1746/2/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-procedure hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1746/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun opened a new pull request #1759: HBASE-24376 MergeNormalizer is merging non-adjacent regions and causi…
huaxiangsun opened a new pull request #1759: URL: https://github.com/apache/hbase/pull/1759 …ng region overlaps/holes. (#1734) Signed-off-by: Viraj Jasani Signed-off-by: Jan Hentschel Signed-off-by: Nick Dimiduk Signed-off-by: stack This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun opened a new pull request #1758: Backport: HBASE-24376 MergeNormalizer is merging non-adjacent regions and causi…
huaxiangsun opened a new pull request #1758: URL: https://github.com/apache/hbase/pull/1758 …ng region overlaps/holes. (#1734) Signed-off-by: Viraj Jasani Signed-off-by: Jan Hentschel Signed-off-by: Nick Dimiduk Signed-off-by: stack This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio merged pull request #1757: HBASE-24410 Generate CHANGES.md and RELEASENOTES.md for 2.2.5 (addendum)
infraio merged pull request #1757: URL: https://github.com/apache/hbase/pull/1757 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ramkrish86 commented on a change in pull request #1747: HBASE-24405 : Document hbase:slowlog related operations
ramkrish86 commented on a change in pull request #1747: URL: https://github.com/apache/hbase/pull/1747#discussion_r429048682 ## File path: src/main/asciidoc/_chapters/hbase-default.adoc ## @@ -2246,6 +2246,23 @@ The percent of region server RPC threads failed to abort RS. `false` +[[hbase.regionserver.slowlog.systable.enabled]] +*`hbase.regionserver.slowlog.systable.enabled`*:: ++ +.Description + + Should be enabled only if hbase.regionserver.slowlog.buffer.enabled is enabled. + If enabled (true), all slow/large RPC logs would be persisted to system table + hbase:slowlog (in addition to in-memory ring buffer at each RegionServer). + The records are stored in increasing order of time. + Operators can scan the table with various combination of ColumnValueFilter. Review comment: Generally i feel that the querying will be like for given time range give me all the slow logs? Then they can drill down. So if they want to query with TS they can specify the Time range correct? Probably you can provide a sample input and output for that too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ramkrish86 commented on a change in pull request #1747: HBASE-24405 : Document hbase:slowlog related operations
ramkrish86 commented on a change in pull request #1747: URL: https://github.com/apache/hbase/pull/1747#discussion_r429048211 ## File path: src/main/asciidoc/_chapters/ops_mgt.adoc ## @@ -2079,6 +2079,98 @@ Examples: +[[slow_log_responses_from_systable]] + Get Slow/Large Response Logs from System table hbase:slowlog + +The above section provides details about Admin APIs: + +* get_slowlog_responses +* get_largelog_responses +* clear_slowlog_responses + +All of the above APIs access online in-memory ring buffers from +individual RegionServers and accumulate logs from ring buffers to display +to end user. However, they can only be used for an ongoing slow/large RPC calls +and user can retrieve ongoing issues with RPC logs. After RegionServer is +restarted, all the objects held in memory of that RegionServer will be cleaned up +and previous problematic logs are lost. What if we want to persist all these logs? +What if we want to store them in such a manner that operator can get all historical +records with some filters? e.g get me all large/slow RPC logs that are triggered by +user1 and are related to region: +cluster_test,,1589635796466.aa45e1571d533f5ed0bb31cdccaaf9cf. + +If we have a system table that stores such logs in increasing (not so strictly though) +order of time, it can definitely help operators debug some historical events +(scan, get, put, compaction, flush etc) with detailed inputs. + +Config which enabled system table to be created and store all log events is +`hbase.regionserver.slowlog.systable.enabled`. + +The default value for this config is `false`. If provided `true` +(Note: `hbase.regionserver.slowlog.buffer.enabled` should also be `true`), +HMaster will create system table `hbase:slowlog` and each RegionServer, upon +determining any slow/large RPC logs at RpcServer level, will put them in in-memory +ring buffer (since `hbase.regionserver.slowlog.buffer.enabled` is true) and Review comment: The one inside the () are redundant I think. You can remove them and still it makese sense. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24155) When running the tests, a tremendous number of connections are put into TIME_WAIT.
[ https://issues.apache.org/jira/browse/HBASE-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113749#comment-17113749 ] Mark Robert Miller commented on HBASE-24155: It took me a bit longer, but I ended up tracking this down a bit further. Raising the socket cache size and expiration for hdfs had helped a fair amount, but there still 50% the number of sockets getting made, a lot of it I tracked to *ReplicationSourceWALReader* and it's reset to look for additional data to read. > When running the tests, a tremendous number of connections are put into > TIME_WAIT. > -- > > Key: HBASE-24155 > URL: https://issues.apache.org/jira/browse/HBASE-24155 > Project: HBase > Issue Type: Test > Components: test >Reporter: Mark Robert Miller >Priority: Major > > When you run the test suite and monitor the number of connections in > TIME_WAIT, it appears that a very large number of connections do not end up > with a proper connection close lifecycle or perhaps proper reuse. > Given connections can stay in TIME_WAIT from 1-4 minutes depending on OS/Env, > running the tests faster or with more tests in parallel increases the > TIME_WAIT connection buildup. Some tests spin up a very, very large number of > connections and if the wrong ones run at the same time, this can also greatly > increase the number of connections put into TIME_WAIT. This can have a > dramatic affect on performance (as it can take longer to create a new > connection) or flat out fail or timeout. > In my experience, a much, much smaller number of connections in a test suite > would end up in TIME_WAIT when connection handling is all correct. > Notes to come in comments below. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #1753: HBAE-24408 Introduce a general 'local region' to store data on master
saintstack commented on a change in pull request #1753:
URL: https://github.com/apache/hbase/pull/1753#discussion_r429033369
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseTimeToLiveFileCleaner.java
##
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.cleaner;
+
+import java.time.Instant;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Base class for time to live file cleaner.
+ */
+@InterfaceAudience.Private
+public abstract class BaseTimeToLiveFileCleaner extends BaseLogCleanerDelegate
{
+
+ private static final Logger LOG =
+LoggerFactory.getLogger(BaseTimeToLiveFileCleaner.class.getName());
+
+ private static final DateTimeFormatter FORMATTER =
+DateTimeFormatter.ISO_DATE_TIME.withZone(ZoneOffset.systemDefault());
+
+ // Configured time a log can be kept after it was closed
+ private long ttlMs;
+
+ private boolean stopped = false;
+
+ @Override
+ public final void setConf(Configuration conf) {
+super.setConf(conf);
+this.ttlMs = getTtlMs(conf);
+ }
+
+ @Override
+ public boolean isFileDeletable(FileStatus status) {
+// Files are validated for the second time here,
+// if it causes a bottleneck this logic needs refactored
+if (!valiateFilename(status.getPath())) {
+ return true;
+}
+long currentTime = EnvironmentEdgeManager.currentTime();
+long time = status.getModificationTime();
+long life = currentTime - time;
+
+if (LOG.isTraceEnabled()) {
+ LOG.trace("File life:{}ms, ttl:{}ms, current:{}, from{}", life, ttlMs,
+FORMATTER.format(Instant.ofEpochMilli(currentTime)),
+FORMATTER.format(Instant.ofEpochMilli(time)));
+}
+if (life < 0) {
+ LOG.warn("Found a file ({}) newer than current time ({} < {}), probably
a clock skew",
+status.getPath(), FORMATTER.format(Instant.ofEpochMilli(currentTime)),
+FORMATTER.format(Instant.ofEpochMilli(time)));
+ return false;
+}
+return life > ttlMs;
+ }
+
+ @Override
+ public void stop(String why) {
+this.stopped = true;
Review comment:
Someone logs the 'why'?
##
File path: hbase-common/src/main/resources/hbase-default.xml
##
@@ -125,7 +125,7 @@ possible configurations would overwhelm and obscure the
important.
hbase.master.logcleaner.plugins
-
org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner
+
org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner,org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner
Review comment:
We want the procedureWALCleaner here still ? Different from the general
master local store cleaner?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
##
@@ -86,8 +87,9 @@ public LogCleaner(final int period, final Stoppable stopper,
Configuration conf,
@Override
protected boolean validate(Path file) {
-return AbstractFSWALProvider.validateWALFilename(file.getName())
-|| MasterProcedureUtil.validateProcedureWALFilename(file.getName());
+return AbstractFSWALProvider.validateWALFilename(file.getName()) ||
+ MasterProcedureUtil.validateProcedureWALFilename(file.getName()) ||
+ file.getName().endsWith(LocalStore.ARCHIVED_WAL_SUFFIX);
}
@Override
Review comment:
Man, we should have called this file the WALCleaner, not LogCleaner. Not
your fault.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseTimeToLiveFileCleaner.java
##
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license
[jira] [Created] (HBASE-24416) RegionNormalizer spliting region should not be limited by hbase.normalizer.min.region.count
Sun Xin created HBASE-24416:
---
Summary: RegionNormalizer spliting region should not be limited by
hbase.normalizer.min.region.count
Key: HBASE-24416
URL: https://issues.apache.org/jira/browse/HBASE-24416
Project: HBase
Issue Type: Improvement
Affects Versions: 3.0.0-alpha-1
Reporter: Sun Xin
Assignee: Sun Xin
Fix For: 3.0.0-alpha-1
In method computePlanForTable of SimpleRegionNormalizer:
we will skip spliting region if the number of regions in the table is less than
hbase.normalizer.min.region.count, even if there is a huge region in the table.
{code:java}
...
if (tableRegions == null || tableRegions.size() < minRegionCount) {
...
return null;
}
...
// get region split plan
if (splitEnabled) {
List splitPlans = getSplitNormalizationPlan(table);
if (splitPlans != null) {
plans.addAll(splitPlans);
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1753: HBAE-24408 Introduce a general 'local region' to store data on master
Apache-HBase commented on pull request #1753: URL: https://github.com/apache/hbase/pull/1753#issuecomment-632465912 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 15s | master passed | | +1 :green_heart: | compile | 1m 26s | master passed | | +1 :green_heart: | shadedjars | 6m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 2s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 58s | the patch passed | | +1 :green_heart: | compile | 1m 22s | the patch passed | | +1 :green_heart: | javac | 1m 22s | the patch passed | | +1 :green_heart: | shadedjars | 6m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 33s | hbase-common in the patch passed. | | -1 :x: | unit | 223m 13s | hbase-server in the patch failed. | | | | 254m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1753 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 32ca988acb72 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 80b64ef4dc | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/2/testReport/ | | Max. process+thread count | 2976 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1757: HBASE-24410 Generate CHANGES.md and RELEASENOTES.md for 2.2.5 (addendum)
Apache-HBase commented on pull request #1757: URL: https://github.com/apache/hbase/pull/1757#issuecomment-632454079 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.2 Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 20s | The patch does not generate ASF License warnings. | | | | 5m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1757/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1757 | | Optional Tests | dupname asflicense markdownlint | | uname | Linux 744c478443d7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1757/out/precommit/personality/provided.sh | | git revision | branch-2.2 / f76a601273 | | Max. process+thread count | 51 (vs. ulimit of 1) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1757/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (HBASE-24410) Generate CHANGES.md and RELEASENOTES.md for 2.2.5
[ https://issues.apache.org/jira/browse/HBASE-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reopened HBASE-24410: > Generate CHANGES.md and RELEASENOTES.md for 2.2.5 > - > > Key: HBASE-24410 > URL: https://issues.apache.org/jira/browse/HBASE-24410 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.2.5 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 2.2.5 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] infraio opened a new pull request #1757: HBASE-24410 Generate CHANGES.md and RELEASENOTES.md for 2.2.5 (addendum)
infraio opened a new pull request #1757: URL: https://github.com/apache/hbase/pull/1757 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24115) Relocate test-only REST "client" from src/ to test/ and mark Private
[ https://issues.apache.org/jira/browse/HBASE-24115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-24115. Resolution: Fixed > Relocate test-only REST "client" from src/ to test/ and mark Private > > > Key: HBASE-24115 > URL: https://issues.apache.org/jira/browse/HBASE-24115 > Project: HBase > Issue Type: Test > Components: REST, security >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 1.3.7, 1.7.0, 2.4.0, 2.1.10, > 1.4.14, 2.2.5 > > > Relocate test-only REST "client" from src/ to test/ and annotate as Private. > The classes o.a.h.h.rest.Remote* were developed to facilitate REST unit tests > and incorrectly committed to src/ . > Although this "breaks" compatibility by moving public classes to test jar and > marking them private, no attention has been paid to these classes with > respect to performance, convenience, or security. Consensus from various > discussions over the years is to move them to test/ as was intent of the > original committer, but misplaced by the same individual. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24115) Relocate test-only REST "client" from src/ to test/ and mark Private
[ https://issues.apache.org/jira/browse/HBASE-24115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-24115: --- Hadoop Flags: Incompatible change,Reviewed (was: Reviewed) Release Note: Relocate test-only REST RemoteHTable and RemoteAdmin from src/ to test/. And mark them as InterfaceAudience.Private. > Relocate test-only REST "client" from src/ to test/ and mark Private > > > Key: HBASE-24115 > URL: https://issues.apache.org/jira/browse/HBASE-24115 > Project: HBase > Issue Type: Test > Components: REST, security >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 1.3.7, 1.7.0, 2.4.0, 2.1.10, > 1.4.14, 2.2.5 > > > Relocate test-only REST "client" from src/ to test/ and annotate as Private. > The classes o.a.h.h.rest.Remote* were developed to facilitate REST unit tests > and incorrectly committed to src/ . > Although this "breaks" compatibility by moving public classes to test jar and > marking them private, no attention has been paid to these classes with > respect to performance, convenience, or security. Consensus from various > discussions over the years is to move them to test/ as was intent of the > original committer, but misplaced by the same individual. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-24115) Relocate test-only REST "client" from src/ to test/ and mark Private
[ https://issues.apache.org/jira/browse/HBASE-24115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reopened HBASE-24115: Reopen to add release note. > Relocate test-only REST "client" from src/ to test/ and mark Private > > > Key: HBASE-24115 > URL: https://issues.apache.org/jira/browse/HBASE-24115 > Project: HBase > Issue Type: Test > Components: REST, security >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 1.3.7, 1.7.0, 2.4.0, 2.1.10, > 1.4.14, 2.2.5 > > > Relocate test-only REST "client" from src/ to test/ and annotate as Private. > The classes o.a.h.h.rest.Remote* were developed to facilitate REST unit tests > and incorrectly committed to src/ . > Although this "breaks" compatibility by moving public classes to test jar and > marking them private, no attention has been paid to these classes with > respect to performance, convenience, or security. Consensus from various > discussions over the years is to move them to test/ as was intent of the > original committer, but misplaced by the same individual. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24402) Moving the meta region causes MetricsException when using above 2.6.0 hadoop version
[
https://issues.apache.org/jira/browse/HBASE-24402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113681#comment-17113681
]
Jeongdae Kim commented on HBASE-24402:
--
[~vincentpoon] [~enis] Can you take a look at this issue? I think other
versions also have this issue.
> Moving the meta region causes MetricsException when using above 2.6.0 hadoop
> version
>
>
> Key: HBASE-24402
> URL: https://issues.apache.org/jira/browse/HBASE-24402
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.13
>Reporter: Jeongdae Kim
>Assignee: Jeongdae Kim
>Priority: Minor
>
> When the meta region moved out from a region server then moved in to the one
> again, we got MetricsException below and the metric source for coprocessor
> couldn't be re-registered. it happened with hadoop 2.8.5
> {noformat}
> 2020-04-21 15:13:02,956 WARN [HBase-Metrics2-1] util.MBeans: Error creating
> MBean object name:
> Hadoop:service=HBase,name=RegionServer,sub=Coprocessor.Region.C
> P_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=HBase,name=RegionServer,sub=Coprocessor.Region.CP_org.
> apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:127)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:102)
> at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:121)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:64)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:222)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:100)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:239)
> at jdk.internal.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:320)
> at com.sun.proxy.$Proxy10.postStart(Unknown Source)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:193)
> at
> org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable.run(JmxCacheBuster.java:109)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] binlijin commented on a change in pull request #1592: HBASE-24244 Move Region Command fails to move region of a table to another RS which is a part of different RSGroup without giving
binlijin commented on a change in pull request #1592:
URL: https://github.com/apache/hbase/pull/1592#discussion_r429002260
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -2085,6 +2085,10 @@ public void move(final byte[] encodedRegionName, byte[]
destServerName) throws I
return;
}
}
+if (dest.equals(LoadBalancer.BOGUS_SERVER_NAME)) {
Review comment:
Since this is not merged, could you add more explanation about it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] VicoWu commented on pull request #1743: HBASE-24403 FsDelegationToken should cache token after acquired a new one
VicoWu commented on pull request #1743: URL: https://github.com/apache/hbase/pull/1743#issuecomment-632431531 > @VicoWu Do you have a Jira ticket related to this one? [HBASE-24403](https://issues.apache.org/jira/browse/HBASE-24403) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-12187) Review in source the paper "Simple Testing Can Prevent Most Critical Failures"
[ https://issues.apache.org/jira/browse/HBASE-12187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113667#comment-17113667 ] Matthew Foley commented on HBASE-12187: --- Not sure whether this ever got done or not, but seeing as the Jira is still open, I thought y'all might be interested to know that a couple days ago, [https://github.com/google/error-prone/pull/372] , consisting of these rules, was finally merged into [error-prone|https://github.com/google/error-prone]! No release yet, (last one was version 2.3.4 on 12/2/2019), but maybe someday :) > Review in source the paper "Simple Testing Can Prevent Most Critical Failures" > -- > > Key: HBASE-12187 > URL: https://issues.apache.org/jira/browse/HBASE-12187 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Priority: Critical > Attachments: HBASE-12187.patch, abortInOvercatch.warnings.txt, > emptyCatch.warnings.txt, todoInCatch.warnings.txt > > > Review the helpful paper > https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf > It describes 'catastrophic failures', especially issues where exceptions are > thrown but not properly handled. Their static analysis tool Aspirator turns > up a bunch of the obvious offenders (Lets add to test-patch.sh alongside > findbugs?). This issue is about going through code base making sub-issues to > root out these and others (Don't we have the test described in figure #6 > already? I thought we did? If we don't, need to add). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24115) Relocate test-only REST "client" from src/ to test/ and mark Private
[ https://issues.apache.org/jira/browse/HBASE-24115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113666#comment-17113666 ] Guanghao Zhang commented on HBASE-24115: Found this when release 2.2.5, [~apurtell] sir, we need a release note for this? > Relocate test-only REST "client" from src/ to test/ and mark Private > > > Key: HBASE-24115 > URL: https://issues.apache.org/jira/browse/HBASE-24115 > Project: HBase > Issue Type: Test > Components: REST, security >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 1.3.7, 1.7.0, 2.4.0, 2.1.10, > 1.4.14, 2.2.5 > > > Relocate test-only REST "client" from src/ to test/ and annotate as Private. > The classes o.a.h.h.rest.Remote* were developed to facilitate REST unit tests > and incorrectly committed to src/ . > Although this "breaks" compatibility by moving public classes to test jar and > marking them private, no attention has been paid to these classes with > respect to performance, convenience, or security. Consensus from various > discussions over the years is to move them to test/ as was intent of the > original committer, but misplaced by the same individual. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bsglz commented on a change in pull request #1737: HBASE-24382 Flush partial stores of region filtered by seqId when arc…
bsglz commented on a change in pull request #1737:
URL: https://github.com/apache/hbase/pull/1737#discussion_r428468856
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -2444,8 +2451,16 @@ public FlushResultImpl flushcache(boolean
forceFlushAllStores, boolean writeFlus
}
try {
-Collection specificStoresToFlush =
+Collection specificStoresToFlush = null;
+if (!forceFlushAllStores && families != null) {
+ specificStoresToFlush = stores.entrySet().stream()
+.filter(e -> families.contains(Bytes.toString(e.getKey(
Review comment:
Ok, will use byte[] instead of String.
In addition,since we do not need a sorted collection, i could iterate the
families to avoid use contains.
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase-native-client] phrocker commented on pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
phrocker commented on pull request #2: URL: https://github.com/apache/hbase-native-client/pull/2#issuecomment-632417275 @joshelser the patches are to ensure we have builds on the current compiler. Previously these succeeded because the standard accepted these as warnings. I moved the standard up a bit which caused those warnings to become errors. They are rectified in newer versions of zookeeper but I do not intend to update to 3.5 ( and likely can't? ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1753: HBAE-24408 Introduce a general 'local region' to store data on master
Apache-HBase commented on pull request #1753: URL: https://github.com/apache/hbase/pull/1753#issuecomment-632416699 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 18s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 0s | master passed | | +1 :green_heart: | checkstyle | 1m 37s | master passed | | +0 :ok: | refguide | 5m 34s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 2m 53s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | checkstyle | 0m 23s | The patch passed checkstyle in hbase-common | | +1 :green_heart: | checkstyle | 1m 12s | hbase-server: The patch generated 0 new + 113 unchanged - 4 fixed = 113 total (was 117) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 5m 19s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 12m 10s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 55m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1753 | | Optional Tests | dupname asflicense refguide xml spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux fd0f3d545e7c 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 80b64ef4dc | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/2/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/2/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
phrocker commented on a change in pull request #2: URL: https://github.com/apache/hbase-native-client/pull/2#discussion_r428980914 ## File path: NOTICE ## @@ -0,0 +1,15 @@ +Apache HBase +Copyright 2019 The Apache Software Foundation Review comment: ha! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113632#comment-17113632
]
Bharath Vissapragada commented on HBASE-23887:
--
Thanks for your detailed explanation, benchmarks and visualizations. I think I
see your intuition behind the idea of dynamically adapting the cache to control
thrashing at higher eviction rate. I have a few questions though, curious to
know your thoughts.
1. Your CM charts show that the block cache miss rate (first chart on the
bottom) increased almost 2x. While that is expected, since you are not
aggressively caching blocks, It is a bit concerning, I think. All the read
distributions show that the 95th and 99th %ile latencies are 3x bad generally
except for UNIFORM distribution. That is inline with your intuition too I guess
since your algorithm works well if the distribution of offsets is uniform. Even
with these drop in 95%ile and 99%ile latencies, average latency is still higher
without the patch (take LATEST for example), meaning there are a few outliers
with LRU and thats affecting the throughput and average latency? Did you get a
chance to dig into the results, how would you interpret them?
2. What would be pathological workload for your design? Do you foresee any
specific workload pattern that might be working very well with LRU but
regresses pretty bad without your patch?
3. Let's say you have mix of 90% scan and 10% non-scan workload. Assuming the
non-scan workload benefits from caching, wouldn't the performance for the
non-scan workload be non-deterministic depending on how the HFiles are laid out
(say before and after a compaction). If we are lucky and the block offsets for
the non-scan workload falls in the ideal range, we are good, otherwise they
aren't even considered for caching. Does your patch handle this some how or did
I miss it?
4. Can you please expand on how one chooses the values to configure for your
newly added params? Are there any rough guidelines?
5. In the following piece of code, When you consider 'bytesFreed', you don't
check whether the freed memory was from single access cache block or a multi
access cache block (both of them have equal weights). Should we consider
different weights for single and multi access cache blocks?
{noformat}
bytesFreed = cache.evict();
// If heavy cleaning BlockCache control.
// It helps avoid put too many blocks into BlockCache
// when evict() works very active.
if (bytesFreed > 0 && bytesFreed > cache.heavyEvictionBytesSizeLimit) {
cache.heavyEvictionCount++;
}
else {
cache.heavyEvictionCount = 0;
}
{noformat}
My point being, since this a segmented LRU with separate chunks of memory for
single and multi access blocks, constant eviction of single access cache blocks
probably signifies that there is scan based workload in progress (see the
following snippet of code). However constant eviction of multi blocks means
that there are cache hits and your optimization shouldn't kick in. Thoughts?
(Would this help get the cache miss count down?)
{noformat}
// this means no need to evict block in memory bucket,
// and we try best to make the ratio between single-bucket and
// multi-bucket is 1:2
long bytesRemain = s + m - bytesToFree;
if (3 * s <= bytesRemain) {
// single-bucket is small enough that no eviction happens for it
// hence all eviction goes from multi-bucket
bytesFreed = bucketMulti.free(bytesToFree);
} else if (3 * m <= 2 * bytesRemain) {
// multi-bucket is small enough that no eviction happens for it
// hence all eviction goes from single-bucket
bytesFreed = bucketSingle.free(bytesToFree);
} else {
// both buckets need to evict some blocks
bytesFreed = bucketSingle.free(s - bytesRemain / 3);
if (bytesFreed < bytesToFree) {
bytesFreed += bucketMulti.free(bytesToFree - bytesFreed);
}
}
{noformat}
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, cmp.png, evict_BC100_vs_BC23.png,
> read_requests_100pBC_vs_23pBC.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
[jira] [Commented] (HBASE-24347) Hadoop2 profiles are both active when pre-commit PR builds run
[ https://issues.apache.org/jira/browse/HBASE-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113627#comment-17113627 ] Josh Elser commented on HBASE-24347: bq. So what we should do here? Backport which issue firstly? I think we just need HBASE-24280 (patches attached here). I was trying to double check with Duo (as I missed this the first time, I wanted to make sure I got it right the second time). > Hadoop2 profiles are both active when pre-commit PR builds run > -- > > Key: HBASE-24347 > URL: https://issues.apache.org/jira/browse/HBASE-24347 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: Michael Stack >Assignee: Josh Elser >Priority: Major > Fix For: 2.3.0, 2.2.6 > > Attachments: HBASE-24280.001.branch-2.3.patch, > HBASE-24280.001.branch-2.patch > > > We need the magic done in the parent out in our precommit builds too. See how > https://github.com/apache/hbase/pull/1664 fails in hbase-rest w/ complaint > about jersey; this is a symptom of double hadoop2+hadoop3 profile activation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] joshelser commented on a change in pull request #1648: HBASE-8458 Support for batch version of checkAndMutate()
joshelser commented on a change in pull request #1648:
URL: https://github.com/apache/hbase/pull/1648#discussion_r428779997
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTable.java
##
@@ -344,6 +360,35 @@ default CheckAndMutateBuilder ifEquals(byte[] value) {
CompletableFuture thenMutate(RowMutations mutation);
}
+ /**
+ * checkAndMutate that atomically checks if a row matches the specified
condition. If it does,
+ * it performs the specified action.
+ *
+ * @param checkAndMutate The CheckAndMutate object.
+ * @return A {@link CompletableFuture}s that represent the result for the
CheckAndMutate.
+ */
+ CompletableFuture checkAndMutate(CheckAndMutate checkAndMutate);
+
+ /**
+ * Batch version of checkAndMutate.
Review comment:
Probably necessary to mention the consistency/expectations on this
method.
By that, I mean, each CheckAndMutate is still individually atomic. They are
batched only in the sense that they are sent to a RS in one RPC, but each
CheckAndMutate operation is still executed atomically (and thus, each may fail
independently of others). Is that correct?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
##
@@ -2793,18 +2793,93 @@ public MultiResponse multi(final RpcController rpcc,
final MultiRequest request)
long nonceGroup = request.hasNonceGroup() ? request.getNonceGroup() :
HConstants.NO_NONCE;
-// this will contain all the cells that we need to return. It's created
later, if needed.
-List cellsToReturn = null;
MultiResponse.Builder responseBuilder = MultiResponse.newBuilder();
RegionActionResult.Builder regionActionResultBuilder =
RegionActionResult.newBuilder();
-Boolean processed = null;
-RegionScannersCloseCallBack closeCallBack = null;
-RpcCallContext context = RpcServer.getCurrentCall().orElse(null);
this.rpcMultiRequestCount.increment();
this.requestCount.increment();
+ActivePolicyEnforcement spaceQuotaEnforcement =
getSpaceQuotaManager().getActiveEnforcements();
+
+// We no longer use MultiRequest#condition. Instead, we use
RegionAction#condition. The
+// following logic is for backward compatibility as old clients still use
+// MultiRequest#condition in case of checkAndMutate with RowMutations.
+if (request.hasCondition()) {
+ if (request.getRegionActionList().isEmpty()) {
+// If the region action list is empty, do nothing.
+responseBuilder.setProcessed(true);
+return responseBuilder.build();
+ }
+
+ RegionAction regionAction = request.getRegionAction(0);
+
+ // When request.hasCondition() is true, regionAction.getAtomic() should
be always true. So
+ // we can assume regionAction.getAtomic() is true here.
+ assert regionAction.getAtomic();
+
+ OperationQuota quota;
+ HRegion region;
+ RegionSpecifier regionSpecifier = regionAction.getRegion();
+
+ try {
+region = getRegion(regionSpecifier);
+quota = getRpcQuotaManager().checkQuota(region,
regionAction.getActionList());
+ } catch (IOException e) {
+failRegionAction(responseBuilder, regionActionResultBuilder,
regionAction, cellScanner, e);
+return responseBuilder.build();
+ }
+
+ boolean rejectIfFromClient = shouldRejectRequestsFromClient(region);
+ // We only allow replication in standby state and it will not set the
atomic flag.
+ if (rejectIfFromClient) {
+failRegionAction(responseBuilder, regionActionResultBuilder,
regionAction, cellScanner,
+ new DoNotRetryIOException(
+region.getRegionInfo().getRegionNameAsString() + " is in STANDBY
state"));
+quota.close();
+return responseBuilder.build();
+ }
+
+ try {
+Condition condition = request.getCondition();
+byte[] row = condition.getRow().toByteArray();
+byte[] family = condition.hasFamily() ?
condition.getFamily().toByteArray() : null;
+byte[] qualifier = condition.hasQualifier() ?
+ condition.getQualifier().toByteArray() : null;
+CompareOperator op = condition.hasCompareType() ?
+ CompareOperator.valueOf(condition.getCompareType().name()) : null;
+ByteArrayComparable comparator = condition.hasComparator() ?
+ ProtobufUtil.toComparator(condition.getComparator()) : null;
+Filter filter = condition.hasFilter() ?
+ ProtobufUtil.toFilter(condition.getFilter()) : null;
+TimeRange timeRange = condition.hasTimeRange() ?
+ ProtobufUtil.toTimeRange(condition.getTimeRange()) :
+ TimeRange.allTime();
+boolean processed =
+ checkAndRowMutate(region, regionAction.getActionList(), cellScanner,
row, family,
+qualifier, op, comparator, filter, timeRange,
regionActionResultBuilder,
+spaceQuotaEnforcement);
+
[GitHub] [hbase-native-client] joshelser commented on a change in pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
joshelser commented on a change in pull request #2:
URL: https://github.com/apache/hbase-native-client/pull/2#discussion_r428961966
##
File path: NOTICE
##
@@ -0,0 +1,15 @@
+Apache HBase
+Copyright 2019 The Apache Software Foundation
Review comment:
2020 now :)
##
File path: cmake/DownloadProject.CMakeLists.autogen.in
##
@@ -0,0 +1,19 @@
+# Distributed under the OSI-approved MIT License. See accompanying
+# file LICENSE or https://github.com/Crascit/DownloadProject for details.
Review comment:
Weird that Crascit doesn't have the normal MIT license header. I guess
we leave it rather than put the real MIT license text.
##
File path: src/hbase/test-util/mini-cluster.cc
##
@@ -34,7 +34,7 @@ JNIEnv *MiniCluster::CreateVM(JavaVM **jvm) {
char *classpath = getenv("CLASSPATH");
std::string clspath;
if (classpath == NULL || strstr(classpath, "-tests.jar") == NULL) {
-std::string
clsPathFilePath("../../../hbase-build-configuration/target/cached_classpath.txt");
+std::string clsPathFilePath("../target/cached_classpath.txt");
Review comment:
I reckon this doesn't work after removing the native client out of the
"main" HBase java repository.
##
File path: cmake/patches/zookeeper.3.4.14.buf
##
@@ -0,0 +1,3767 @@
+/**
Review comment:
What is this patching?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24411) Set version to 2.2.5 in branch-2.2 for first RC of 2.2.5
[
https://issues.apache.org/jira/browse/HBASE-24411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113598#comment-17113598
]
Hudson commented on HBASE-24411:
Results for branch branch-2.2
[build #873 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/873/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/873//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/873//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/873//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Set version to 2.2.5 in branch-2.2 for first RC of 2.2.5
>
>
> Key: HBASE-24411
> URL: https://issues.apache.org/jira/browse/HBASE-24411
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.2.5
>Reporter: Guanghao Zhang
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24410) Generate CHANGES.md and RELEASENOTES.md for 2.2.5
[
https://issues.apache.org/jira/browse/HBASE-24410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113597#comment-17113597
]
Hudson commented on HBASE-24410:
Results for branch branch-2.2
[build #873 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/873/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/873//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/873//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/873//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Generate CHANGES.md and RELEASENOTES.md for 2.2.5
> -
>
> Key: HBASE-24410
> URL: https://issues.apache.org/jira/browse/HBASE-24410
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.2.5
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.2.5
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] sguggilam commented on pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
sguggilam commented on pull request #1755: URL: https://github.com/apache/hbase/pull/1755#issuecomment-632369851 @apurtell for review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
Apache-HBase commented on pull request #1755: URL: https://github.com/apache/hbase/pull/1755#issuecomment-632363276 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 7m 2s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 8m 41s | branch-1 passed | | +1 :green_heart: | compile | 0m 39s | branch-1 passed with JDK v1.8.0_252 | | +1 :green_heart: | compile | 0m 45s | branch-1 passed with JDK v1.7.0_262 | | +1 :green_heart: | checkstyle | 1m 43s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | branch-1 passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 0m 39s | branch-1 passed with JDK v1.7.0_262 | | +0 :ok: | spotbugs | 2m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 51s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 55s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | javac | 0m 44s | the patch passed | | -1 :x: | checkstyle | 1m 39s | hbase-server: The patch generated 17 new + 279 unchanged - 2 fixed = 296 total (was 281) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 3m 32s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 6m 12s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 0m 53s | the patch passed with JDK v1.7.0_262 | | -1 :x: | findbugs | 4m 8s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 139m 18s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 190m 31s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | Sequence of calls to java.util.concurrent.ConcurrentHashMap may not be atomic in org.apache.hadoop.hbase.master.AssignmentManager.unassign(HRegionInfo, RegionState, int, ServerName, boolean, ServerName) At AssignmentManager.java:may not be atomic in org.apache.hadoop.hbase.master.AssignmentManager.unassign(HRegionInfo, RegionState, int, ServerName, boolean, ServerName) At AssignmentManager.java:[line 1982] | | Failed junit tests | hadoop.hbase.client.TestAdmin2 | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1755 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 854dba18eabe 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1755/out/precommit/personality/provided.sh | | git revision | branch-1 / 3235b56 | | Default Java | 1.7.0_262 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_252 /usr/lib/jvm/zulu-7-amd64:1.7.0_262 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/1/artifact/out/diff-checkstyle-hbase-server.txt | | findbugs | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/1/artifact/out/new-findbugs-hbase-server.html | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/1/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/1/testReport/ | | Max. process+thread count | 4654 (vs. ulimit of 1) | |
[GitHub] [hbase] Apache-HBase commented on pull request #1756: HBASE-24369 Provide more information about merged child regions in Hb…
Apache-HBase commented on pull request #1756: URL: https://github.com/apache/hbase/pull/1756#issuecomment-632356171 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 48s | master passed | | +1 :green_heart: | compile | 0m 56s | master passed | | +1 :green_heart: | shadedjars | 5m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | shadedjars | 5m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 139m 40s | hbase-server in the patch passed. | | | | 163m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1756/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1756 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1075a105a661 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 80b64ef4dc | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1756/1/testReport/ | | Max. process+thread count | 4366 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1756/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24244) Region move to different rsgroup server should fail fast.
[ https://issues.apache.org/jira/browse/HBASE-24244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Arshad updated HBASE-24244: Summary: Region move to different rsgroup server should fail fast. (was: Move Region Command fails to move region of a table to another RS which is a part of different RSGroup without giving error.) > Region move to different rsgroup server should fail fast. > - > > Key: HBASE-24244 > URL: https://issues.apache.org/jira/browse/HBASE-24244 > Project: HBase > Issue Type: Bug > Components: rsgroup >Reporter: Saurav Mehta >Assignee: Mohammad Arshad >Priority: Minor > > When I tried to move a region (which was the part of the Table T1 present in > RSG1 rsgroup) to RegionServer of another RSGroup RSG2, the move was not > successful and yet the output was "Took 3.4 seconds" instead of an error. > Also, during the move, the master logs did not show the destination address > as the RS address, it showed localhost which I think is misleading. > > STEPS TO RETRACE THE SCENARIO: > # Create a RSGroup 'RSG1' and add region servers into it. > # Create another RSGroup 'RSG2' and add region servers into it also. > # Create a table 't1' with multiple regions and move it to rsgroup 'RSG1'. > # Now try to move a region of table t1 from 'RSG1' to region server of > 'RSG2'. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1756: HBASE-24369 Provide more information about merged child regions in Hb…
Apache-HBase commented on pull request #1756: URL: https://github.com/apache/hbase/pull/1756#issuecomment-632295835 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 8s | master passed | | +1 :green_heart: | checkstyle | 1m 14s | master passed | | +1 :green_heart: | spotbugs | 2m 12s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 13s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 36m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1756/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1756 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 8cd13096079b 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 80b64ef4dc | | Max. process+thread count | 85 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1756/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-17163) Refactor ExportSnapshot, SnapshotInfo and remove FS references from it
[ https://issues.apache.org/jira/browse/HBASE-17163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-17163. --- Resolution: Later Resolving old issue w/ no work associated. > Refactor ExportSnapshot, SnapshotInfo and remove FS references from it > -- > > Key: HBASE-17163 > URL: https://issues.apache.org/jira/browse/HBASE-17163 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > > Refactor ExportSnapshot, SnapshotInfo and remove FS references from it. Also > fix a few lingering references to FS from snapshot code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] huaxiangsun commented on pull request #1756: HBASE-24369 Provide more information about merged child regions in Hb…
huaxiangsun commented on pull request #1756: URL: https://github.com/apache/hbase/pull/1756#issuecomment-632276264 A sample web page is at https://issues.apache.org/jira/browse/HBASE-24369?filter=-1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun opened a new pull request #1756: HBASE-24369 Provide more information about merged child regions in Hb…
huaxiangsun opened a new pull request #1756: URL: https://github.com/apache/hbase/pull/1756 …ck Overlaps section, which cannot be fixed immediately This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24369) Provide more information about merged child regions in Hbck Overlaps section, which cannot be fixed immediately
[ https://issues.apache.org/jira/browse/HBASE-24369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113454#comment-17113454 ] Huaxiang Sun commented on HBASE-24369: -- I have a patch and the sample screen is attached. !Screen Shot 2020-05-21 at 11.14.48 AM.png! > Provide more information about merged child regions in Hbck Overlaps > section, which cannot be fixed immediately > - > > Key: HBASE-24369 > URL: https://issues.apache.org/jira/browse/HBASE-24369 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 2.3.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Attachments: Screen Shot 2020-05-21 at 11.14.48 AM.png > > > Right now, in Master's hbck page, pairs of overlap regions are listed. > with Hbck2 -fixMeta, it will merge these overlap regions. However, if one of > the region is a newly merged child region and GC has not kicked in yet, they > are not fixable (merge will fail). > Users get confused as why after HBCK2 -fixMeta and still there are overlaps. > To avoid this confusion, propose a new subsection saying that master is doing > GC on these regions and these regions cannot be fixed/merged at this moment, > wait until these regions are ready (not showing up in this subsection) to > start the fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24369) Provide more information about merged child regions in Hbck Overlaps section, which cannot be fixed immediately
[ https://issues.apache.org/jira/browse/HBASE-24369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-24369: - Summary: Provide more information about merged child regions in Hbck Overlaps section, which cannot be fixed immediately (was: Add a new subsection under Hbck Overlaps section) > Provide more information about merged child regions in Hbck Overlaps > section, which cannot be fixed immediately > - > > Key: HBASE-24369 > URL: https://issues.apache.org/jira/browse/HBASE-24369 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 2.3.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Attachments: Screen Shot 2020-05-21 at 11.14.48 AM.png > > > Right now, in Master's hbck page, pairs of overlap regions are listed. > with Hbck2 -fixMeta, it will merge these overlap regions. However, if one of > the region is a newly merged child region and GC has not kicked in yet, they > are not fixable (merge will fail). > Users get confused as why after HBCK2 -fixMeta and still there are overlaps. > To avoid this confusion, propose a new subsection saying that master is doing > GC on these regions and these regions cannot be fixed/merged at this moment, > wait until these regions are ready (not showing up in this subsection) to > start the fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24369) Provide more information about merged child regions in Hbck Overlaps section, which cannot be fixed immediately
[ https://issues.apache.org/jira/browse/HBASE-24369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun updated HBASE-24369: - Attachment: Screen Shot 2020-05-21 at 11.14.48 AM.png > Provide more information about merged child regions in Hbck Overlaps > section, which cannot be fixed immediately > - > > Key: HBASE-24369 > URL: https://issues.apache.org/jira/browse/HBASE-24369 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 2.3.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Attachments: Screen Shot 2020-05-21 at 11.14.48 AM.png > > > Right now, in Master's hbck page, pairs of overlap regions are listed. > with Hbck2 -fixMeta, it will merge these overlap regions. However, if one of > the region is a newly merged child region and GC has not kicked in yet, they > are not fixable (merge will fail). > Users get confused as why after HBCK2 -fixMeta and still there are overlaps. > To avoid this confusion, propose a new subsection saying that master is doing > GC on these regions and these regions cannot be fixed/merged at this moment, > wait until these regions are ready (not showing up in this subsection) to > start the fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] sguggilam opened a new pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
sguggilam opened a new pull request #1755: URL: https://github.com/apache/hbase/pull/1755 …region close requests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-23279) Switch default block encoding to ROW_INDEX_V1
[ https://issues.apache.org/jira/browse/HBASE-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-23279: - Fix Version/s: (was: 2.4.0) (was: 3.0.0-alpha-1) > Switch default block encoding to ROW_INDEX_V1 > - > > Key: HBASE-23279 > URL: https://issues.apache.org/jira/browse/HBASE-23279 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Lars Hofhansl >Assignee: Viraj Jasani >Priority: Minor > Attachments: HBASE-23279.master.000.patch, > HBASE-23279.master.001.patch, HBASE-23279.master.002.patch, > HBASE-23279.master.003.patch, HBASE-23279.master.004.patch, > HBASE-23279.master.005.patch > > > Currently we set both block encoding and compression to NONE. > ROW_INDEX_V1 has many advantages and (almost) no disadvantages (the hfiles > are slightly larger about 3% or so). I think that would a better default than > NONE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[
https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113440#comment-17113440
]
Bharath Vissapragada commented on HBASE-18095:
--
> Maybe we could add an option called fallbackToZk?
Implementation wise, I think it's pretty simple to do that. We can have a
"compound" registry implementation that delegates to primary and secondary
registries but yes if all the masters are down and ZK servers are available,
the cluster is already in a bad shape. I think there is very little a client
could with the connection.
That said, after following the discussion here and in HBASE-11288, I think what
the current registry implementations lack (out of the box) is some sort of
dynamic resolver interface and re-configuration on the fly. Currently I think
everyone is exploiting their infrastructure specific solutions (DNS resolution,
lvs etc) to trick the rpcs to be load balanced. Was wondering if we can
abstract some of that out and make it pluggable in the client code.
For example, a simple interface like
{code:java}
interface RegistryEndPointResolver {
Set getRegistryEndPoints();
}
{code}
The default implementation would be based on the hbase-site.xml config and
polled every few seconds. But every one can plug their own implementation into
it based on say, DNS, existing service discovery solutions etc. Thoughts?
> Provide an option for clients to find the server hosting META that does not
> involve the ZooKeeper client
>
>
> Key: HBASE-18095
> URL: https://issues.apache.org/jira/browse/HBASE-18095
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Reporter: Andrew Kyle Purtell
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch
>
>
> Clients are required to connect to ZooKeeper to find the location of the
> regionserver hosting the meta table region. Site configuration provides the
> client a list of ZK quorum peers and the client uses an embedded ZK client to
> query meta location. Timeouts and retry behavior of this embedded ZK client
> are managed orthogonally to HBase layer settings and in some cases the ZK
> cannot manage what in theory the HBase client can, i.e. fail fast upon outage
> or network partition.
> We should consider new configuration settings that provide a list of
> well-known master and backup master locations, and with this information the
> client can contact any of the master processes directly. Any master in either
> active or passive state will track meta location and respond to requests for
> it with its cached last known location. If this location is stale, the client
> can ask again with a flag set that requests the master refresh its location
> cache and return the up-to-date location. Every client interaction with the
> cluster thus uses only HBase RPC as transport, with appropriate settings
> applied to the connection. The configuration toggle that enables this
> alternative meta location lookup should be false by default.
> This removes the requirement that HBase clients embed the ZK client and
> contact the ZK service directly at the beginning of the connection lifecycle.
> This has several benefits. ZK service need not be exposed to clients, and
> their potential abuse, yet no benefit ZK provides the HBase server cluster is
> compromised. Normalizing HBase client and ZK client timeout settings and
> retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer
> necessary.
> And, from [~ghelmling]: There is an additional complication here for
> token-based authentication. When a delegation token is used for SASL
> authentication, the client uses the cluster ID obtained from Zookeeper to
> select the token identifier to use. So there would also need to be some
> Zookeeper-less, unauthenticated way to obtain the cluster ID as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22041) [k8s] The crashed node exists in onlineServer forever, and if it holds the meta data, master will start up hang.
[
https://issues.apache.org/jira/browse/HBASE-22041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113423#comment-17113423
]
Andrey Elenskiy commented on HBASE-22041:
-
Attached entire hbasemaster log (hbasemaster.log) with TRACE enabled right
before trying to reproduce the issue.
The time I've triggered the issue was "Thu May 21 17:28:42 UTC 2020". And the
topology looked like so:
{noformat}
hbasemaster-0 10.128.25.30
hbasemaster-1 10.128.6.51
regionserver-0 10.128.53.53
regionserver-1 10.128.9.37
regionserver-2 10.128.14.39{noformat}
They way I trigger the issue is by picking a regionserver with 0 regions
(because it was restarted recently), triggering "balancer" and killing the
regionserver during the execution of balancer. In this case the regionserver I
killed was regionserver-2. Here's how topology looked like after regionserver 2
came back up:
{noformat}
hbasemaster-0 10.128.25.30
hbasemaster-1 10.128.6.51
regionserver-0 10.128.53.53
regionserver-1 10.128.9.37
regionserver-2 10.128.14.40{noformat}
You can see that regionserver-2 came back up with IP 10.128.14.40, but
hbasemaster still tries to contact 10.128.14.39
> [k8s] The crashed node exists in onlineServer forever, and if it holds the
> meta data, master will start up hang.
>
>
> Key: HBASE-22041
> URL: https://issues.apache.org/jira/browse/HBASE-22041
> Project: HBase
> Issue Type: Bug
>Reporter: lujie
>Priority: Critical
> Attachments: bug.zip, hbasemaster.log, normal.zip
>
>
> while master fresh boot, we crash (kill- 9) the RS who hold meta. we find
> that the master startup fails and print thounds of logs like:
> {code:java}
> 2019-03-13 01:09:54,896 WARN [RSProcedureDispatcher-pool4-t1]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to java.net.ConnectException: Call to
> hadoop14/172.16.1.131:16020 failed on connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> syscall:getsockopt(..) failed: Connection refused:
> hadoop14/172.16.1.131:16020, try=0, retrying...
> 2019-03-13 01:09:55,004 WARN [RSProcedureDispatcher-pool4-t2]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=1, retrying...
> 2019-03-13 01:09:55,114 WARN [RSProcedureDispatcher-pool4-t3]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=2, retrying...
> 2019-03-13 01:09:55,219 WARN [RSProcedureDispatcher-pool4-t4]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=3, retrying...
> 2019-03-13 01:09:55,324 WARN [RSProcedureDispatcher-pool4-t5]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=4, retrying...
> 2019-03-13 01:09:55,428 WARN [RSProcedureDispatcher-pool4-t6]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=5, retrying...
> 2019-03-13 01:09:55,533 WARN [RSProcedureDispatcher-pool4-t7]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=6, retrying...
> 2019-03-13 01:09:55,638 WARN
[jira] [Created] (HBASE-24415) Get all row ids in a table using Hbase Rest Server API
anusha created HBASE-24415: -- Summary: Get all row ids in a table using Hbase Rest Server API Key: HBASE-24415 URL: https://issues.apache.org/jira/browse/HBASE-24415 Project: HBase Issue Type: Wish Reporter: anusha How to get Row Ids in a table using Hbase Rest Server because I need to get all families with respective qualifiers using hbase rest server. By using row id we get families and qualifiers so i need how to get ROW IDS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22041) [k8s] The crashed node exists in onlineServer forever, and if it holds the meta data, master will start up hang.
[
https://issues.apache.org/jira/browse/HBASE-22041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Elenskiy updated HBASE-22041:
Attachment: hbasemaster.log
> [k8s] The crashed node exists in onlineServer forever, and if it holds the
> meta data, master will start up hang.
>
>
> Key: HBASE-22041
> URL: https://issues.apache.org/jira/browse/HBASE-22041
> Project: HBase
> Issue Type: Bug
>Reporter: lujie
>Priority: Critical
> Attachments: bug.zip, hbasemaster.log, normal.zip
>
>
> while master fresh boot, we crash (kill- 9) the RS who hold meta. we find
> that the master startup fails and print thounds of logs like:
> {code:java}
> 2019-03-13 01:09:54,896 WARN [RSProcedureDispatcher-pool4-t1]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to java.net.ConnectException: Call to
> hadoop14/172.16.1.131:16020 failed on connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> syscall:getsockopt(..) failed: Connection refused:
> hadoop14/172.16.1.131:16020, try=0, retrying...
> 2019-03-13 01:09:55,004 WARN [RSProcedureDispatcher-pool4-t2]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=1, retrying...
> 2019-03-13 01:09:55,114 WARN [RSProcedureDispatcher-pool4-t3]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=2, retrying...
> 2019-03-13 01:09:55,219 WARN [RSProcedureDispatcher-pool4-t4]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=3, retrying...
> 2019-03-13 01:09:55,324 WARN [RSProcedureDispatcher-pool4-t5]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=4, retrying...
> 2019-03-13 01:09:55,428 WARN [RSProcedureDispatcher-pool4-t6]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=5, retrying...
> 2019-03-13 01:09:55,533 WARN [RSProcedureDispatcher-pool4-t7]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=6, retrying...
> 2019-03-13 01:09:55,638 WARN [RSProcedureDispatcher-pool4-t8]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=7, retrying...
> 2019-03-13 01:09:55,755 WARN [RSProcedureDispatcher-pool4-t9]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=8, retrying...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] huaxiangsun commented on pull request #1734: HBASE-24376 MergeNormalizer is merging non-adjacent regions and causi…
huaxiangsun commented on pull request #1734: URL: https://github.com/apache/hbase/pull/1734#issuecomment-632229515 Thanks for the reviews! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun merged pull request #1734: HBASE-24376 MergeNormalizer is merging non-adjacent regions and causi…
huaxiangsun merged pull request #1734: URL: https://github.com/apache/hbase/pull/1734 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ramkrish86 commented on a change in pull request #1552: HBASE-24205 Create metric to know the number of reads that happens fr…
ramkrish86 commented on a change in pull request #1552:
URL: https://github.com/apache/hbase/pull/1552#discussion_r428789194
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -1002,6 +1012,14 @@ public Void call() throws IOException {
} finally {
this.lock.writeLock().unlock();
this.archiveLock.unlock();
+ // moving it after the unlocking so
+ // that metrics closure does not affect them
+ if (this.metricsStore != null) {
+metricsStore.close();
Review comment:
Yes. It will do the deregister.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ramkrish86 commented on a change in pull request #1552: HBASE-24205 Create metric to know the number of reads that happens fr…
ramkrish86 commented on a change in pull request #1552:
URL: https://github.com/apache/hbase/pull/1552#discussion_r428788807
##
File path:
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsStoreSourceImpl.java
##
@@ -0,0 +1,211 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Interns;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MetricsStoreSourceImpl implements MetricsStoreSource {
+
+ private MetricsStoreWrapper storeWrapper;
+ private MetricsStoreAggregateSourceImpl aggreagate;
+ private AtomicBoolean closed = new AtomicBoolean(false);
+
+ private String storeNamePrefix;
+ private final MetricRegistry registry;
+ private static final Logger LOG =
LoggerFactory.getLogger(MetricsStoreSourceImpl.class);
+ String storeReadsKey;
+
+ String memstoreReadsKey;
+ String fileReadsKey;
+ private final Counter storeReads;
+ private final Counter memstoreReads;
+ private final Counter fileReads;
+
+ public MetricsStoreSourceImpl(MetricsStoreWrapper storeWrapper,
+ MetricsStoreAggregateSourceImpl aggreagate) {
+this.storeWrapper = storeWrapper;
+this.aggreagate = aggreagate;
+aggreagate.register(this);
+
+LOG.debug("Creating new MetricsRegionSourceImpl for table " +
storeWrapper.getStoreName() + " "
++ storeWrapper.getRegionName());
+
+// we are using the hbase-metrics API
+registry = aggreagate.getMetricRegistry();
+
+storeNamePrefix = "Namespace_" + storeWrapper.getNamespace() + "_table_"
++ storeWrapper.getTableName() + "_region_" +
storeWrapper.getRegionName() + "_store_"
++ storeWrapper.getStoreName() + "_metric_";
+
+String suffix = "Count";
+
+storeReadsKey = storeNamePrefix + MetricsRegionServerSource.GET_KEY +
suffix;
+// all the counters are hbase-metrics API
+storeReads = registry.counter(storeReadsKey);
+
+memstoreReadsKey = storeNamePrefix +
MetricsRegionServerSource.MEMSTORE_GET_KEY + suffix;
+memstoreReads = registry.counter(memstoreReadsKey);
+
+fileReadsKey = storeNamePrefix + MetricsRegionServerSource.FILE_GET_KEY +
suffix;
+fileReads = registry.counter(fileReadsKey);
+
+ }
+
+ @Override
+ public void close() {
+boolean wasClosed = closed.getAndSet(true);
+
+// Has someone else already closed this for us?
+if (wasClosed) {
+ return;
+}
+
+// Before removing the metrics remove this region from the aggregate
region bean.
+// This should mean that it's unlikely that snapshot and close happen at
the same time.
+aggreagate.deregister(this);
+
+// While it's un-likely that snapshot and close happen at the same time
it's still possible.
+// So grab the lock to ensure that all calls to snapshot are done before
we remove the metrics
+synchronized (this) {
+ if (LOG.isTraceEnabled()) {
+LOG.trace("Removing store Metrics: " + storeWrapper.getStoreName());
+ }
+
+ registry.remove(storeReadsKey);
+ registry.remove(memstoreReadsKey);
+ registry.remove(fileReadsKey);
+
+ storeWrapper = null;
+}
+ }
+
+ @Override
+ public int compareTo(MetricsStoreSource source) {
+if (!(source instanceof MetricsStoreSourceImpl)) {
+ return -1;
+}
+
+MetricsStoreSourceImpl impl = (MetricsStoreSourceImpl) source;
+if (impl == null) {
+ return -1;
+}
+
+// TODO : make this better
+return Long.compare(this.storeWrapper.getStoreName().hashCode(),
+ impl.storeWrapper.getStoreName().hashCode());
+ }
+
+ @Override
+ public void updateGet() {
+storeReads.increment();
+ }
+
+ @Override
+ public void updateMemtoreGet() {
+memstoreReads.increment();
+ }
+
+ @Override
+ public void updateFileGet() {
+fileReads.increment();
+ }
+
+ @Override
+ public boolean
[GitHub] [hbase] ramkrish86 commented on a change in pull request #1552: HBASE-24205 Create metric to know the number of reads that happens fr…
ramkrish86 commented on a change in pull request #1552:
URL: https://github.com/apache/hbase/pull/1552#discussion_r428788326
##
File path:
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSourceImpl.java
##
@@ -302,6 +302,14 @@ void snapshot(MetricsRecordBuilder mrb, boolean ignored) {
regionNamePrefix + MetricsRegionSource.MAX_FLUSH_QUEUE_SIZE,
MetricsRegionSource.MAX_FLUSH_QUEUE_DESC),
this.regionWrapper.getMaxFlushQueueSize());
+ mrb.addCounter(
Review comment:
At the store level only it comes as per region per store. This is
something we already have. Jut adding those two new metric here. So should be
ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ramkrish86 commented on a change in pull request #1552: HBASE-24205 Create metric to know the number of reads that happens fr…
ramkrish86 commented on a change in pull request #1552: URL: https://github.com/apache/hbase/pull/1552#discussion_r428787492 ## File path: hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java ## @@ -402,6 +402,8 @@ String DELETE_BATCH_KEY = "deleteBatch"; String GET_SIZE_KEY = "getSize"; String GET_KEY = "get"; + String MEMSTORE_GET_KEY = "getsOnMemstore"; + String FILE_GET_KEY = "getsOnFile"; Review comment: Some of the gets that lands on the StoreScanners does not actually get accounted into actual get. Probably that row does not exist. So I thought it is better to track both. Also the overhead is very minimal. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24414) Extend HBASE-24205 to know the scan requests that comes to a memstore
ramkrishna.s.vasudevan created HBASE-24414: -- Summary: Extend HBASE-24205 to know the scan requests that comes to a memstore Key: HBASE-24414 URL: https://issues.apache.org/jira/browse/HBASE-24414 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 3.0.0-alpha-1 An extension to HBASE-24205 where we can try to get how many of the scan requests where trying to get the latest data. A heuristic should be good enough to know the pattern of the reads here too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1754: HBASE-24413 HBASE-22259 Removed deprecated getTimeStampOfLastShippedO…
Apache-HBase commented on pull request #1754: URL: https://github.com/apache/hbase/pull/1754#issuecomment-632221870 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 31s | master passed | | +1 :green_heart: | javadoc | 0m 16s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 14s | the patch passed | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 9s | hbase-shell in the patch passed. | | | | 18m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1754/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1754 | | Optional Tests | javac javadoc unit | | uname | Linux 182bf78873e1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | Default Java | 2020-01-14 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1754/1/testReport/ | | Max. process+thread count | 2271 (vs. ulimit of 12500) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1754/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1754: HBASE-24413 HBASE-22259 Removed deprecated getTimeStampOfLastShippedO…
Apache-HBase commented on pull request #1754: URL: https://github.com/apache/hbase/pull/1754#issuecomment-632220251 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | master passed | | +1 :green_heart: | javadoc | 0m 16s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | javadoc | 0m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 6m 58s | hbase-shell in the patch passed. | | | | 16m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1754/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1754 | | Optional Tests | javac javadoc unit | | uname | Linux 423e4fa41149 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1754/1/testReport/ | | Max. process+thread count | 2229 (vs. ulimit of 12500) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1754/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-21406) "status 'replication'" should not show SINK if the cluster does not act as sink
[
https://issues.apache.org/jira/browse/HBASE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113363#comment-17113363
]
Sean Busbey commented on HBASE-21406:
-
totally understandable. Add me as a reviewer on the PR once it's up so that
github will notify me?
> "status 'replication'" should not show SINK if the cluster does not act as
> sink
> ---
>
> Key: HBASE-21406
> URL: https://issues.apache.org/jira/browse/HBASE-21406
> Project: HBase
> Issue Type: Improvement
>Reporter: Daisuke Kobayashi
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HBASE-21406-branch-1.001.patch,
> HBASE-21406-master.001.patch, HBASE-21406-master.002.patch, Screen Shot
> 2018-10-31 at 18.12.54.png
>
>
> When replicating in 1 way, from source to target, {{status 'replication'}} on
> source always dumps SINK with meaningless metrics. It only makes sense when
> running the command on target cluster.
> {{status 'replication'}} on source, for example. {{AgeOfLastAppliedOp}} is
> always zero and {{TimeStampsOfLastAppliedOp}} does not get updated from the
> time the RS started since it's not acting as sink.
> {noformat}
> source-1.com
>SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=0,
> TimeStampsOfLastShippedOp=Mon Oct 29 23:44:14 PDT 2018, Replication Lag=0
>SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Oct 25
> 23:56:53 PDT 2018
> {noformat}
> {{status 'replication'}} on target works as expected. SOURCE is empty as it's
> not acting as source:
> {noformat}
> target-1.com
>SOURCE:
>SINK : AgeOfLastAppliedOp=70, TimeStampsOfLastAppliedOp=Mon Oct 29
> 23:44:08 PDT 2018
> {noformat}
> This is because {{getReplicationLoadSink}}, called in {{admin.rb}}, always
> returns a value (not null).
> 1.X
> https://github.com/apache/hbase/blob/rel/1.4.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L194-L204
> 2.X
> https://github.com/apache/hbase/blob/rel/2.0.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L392-L399
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
phrocker commented on a change in pull request #2: URL: https://github.com/apache/hbase-native-client/pull/2#discussion_r428779148 ## File path: cmake/patches/zookeeper.3.4.14.buf ## @@ -0,0 +1,3767 @@ +/** Review comment: This is a patch file. I should shorten this to use a diff'd patch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #1742: HBASE-24401 Cell size limit check on append should consider 0 or less…
wchevreuil commented on a change in pull request #1742:
URL: https://github.com/apache/hbase/pull/1742#discussion_r428778796
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -8265,14 +8265,17 @@ private WALEdit reckonDeltas(Operation op, Mutation
mutation, Durability effecti
break;
default: throw new UnsupportedOperationException(op.toString());
}
- int newCellSize = PrivateCellUtil.estimatedSerializedSizeOf(newCell);
- if (newCellSize > this.maxCellSize) {
-String msg = "Cell with size " + newCellSize + " exceeds limit of " +
this.maxCellSize
- + " bytes in region " + this;
-if (LOG.isDebugEnabled()) {
+ if (this.maxCellSize > 0) {
+int newCellSize = PrivateCellUtil.estimatedSerializedSizeOf(newCell);
+if (newCellSize > this.maxCellSize) {
+ String msg = "Cell with size " + newCellSize + " exceeds limit of "
+ this.maxCellSize
++ " bytes in region " + this;
LOG.debug(msg);
+ throw new DoNotRetryIOException(msg);
}
-throw new DoNotRetryIOException(msg);
+ } else {
+LOG.debug("Cell size check is disabled because of maxCellSize is set
to {}.",
Review comment:
> I removed this log, because there may be too many logs and not very
helpful.
Well, that's why I suggested it to be DEBUG. Logging it only once may not be
sufficient in cases where log rolls too quickly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
phrocker commented on a change in pull request #2:
URL: https://github.com/apache/hbase-native-client/pull/2#discussion_r428778571
##
File path: cmake/BuildTests.cmake
##
@@ -48,10 +48,13 @@ function(createTests testName)
target_include_directories(${testName} PRIVATE BEFORE
"${GTEST_INCLUDE_DIRS}")
target_include_directories(${testName} PRIVATE BEFORE
"${OPENSSL_INCLUDE_DIR}")
-target_link_libraries(hbaseclient-static ${PROTOBUF_LIBRARY})
-target_link_libraries(hbaseclient-static ${FOLLY_LIBRARIES})
+ #target_link_libraries(${testName} ${WANGLE_LIBRARIES})
Review comment:
These comments should be removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1754: HBASE-24413 HBASE-22259 Removed deprecated getTimeStampOfLastShippedO…
Apache-HBase commented on pull request #1754: URL: https://github.com/apache/hbase/pull/1754#issuecomment-632202400 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | rubocop | 0m 9s | There were no new rubocop issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 2m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1754/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1754 | | Optional Tests | dupname asflicense rubocop | | uname | Linux 55cf4a052e93 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | Max. process+thread count | 47 (vs. ulimit of 12500) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1754/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1753: HBAE-24408 Introduce a general 'local region' to store data on master
Apache-HBase commented on pull request #1753: URL: https://github.com/apache/hbase/pull/1753#issuecomment-632196225 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 41s | master passed | | +1 :green_heart: | checkstyle | 1m 36s | master passed | | +0 :ok: | refguide | 4m 43s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 2m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | -0 :warning: | checkstyle | 1m 5s | hbase-server: The patch generated 2 new + 113 unchanged - 4 fixed = 115 total (was 117) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 4m 47s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 12m 56s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 47m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1753 | | Optional Tests | dupname asflicense refguide xml spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux d55408bde2d9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-general-check/output/branch-site/book.html | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-general-check/output/whitespace-eol.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil opened a new pull request #1754: HBASE-24413 HBASE-22259 Removed deprecated getTimeStampOfLastShippedO…
wchevreuil opened a new pull request #1754: URL: https://github.com/apache/hbase/pull/1754 …p method from ReplicationLoadSource, but there were still references to this method on ruby admin.rb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1753: HBAE-24408 Introduce a general 'local region' to store data on master
Apache-HBase commented on pull request #1753: URL: https://github.com/apache/hbase/pull/1753#issuecomment-632189542 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 47s | master passed | | +1 :green_heart: | compile | 1m 37s | master passed | | +1 :green_heart: | shadedjars | 6m 42s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 21s | hbase-common in master failed. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 30s | the patch passed | | +1 :green_heart: | compile | 1m 30s | the patch passed | | +1 :green_heart: | javac | 1m 30s | the patch passed | | +1 :green_heart: | shadedjars | 5m 53s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 36s | hbase-common in the patch passed. | | -1 :x: | unit | 6m 59s | hbase-server in the patch failed. | | | | 38m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1753 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2208f381d759 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/testReport/ | | Max. process+thread count | 1172 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24413) HBASE-22259 Removed deprecated getTimeStampOfLastShippedOp method from ReplicationLoadSource, but there were still references to this method on ruby admin.rb
Wellington Chevreuil created HBASE-24413: Summary: HBASE-22259 Removed deprecated getTimeStampOfLastShippedOp method from ReplicationLoadSource, but there were still references to this method on ruby admin.rb Key: HBASE-24413 URL: https://issues.apache.org/jira/browse/HBASE-24413 Project: HBase Issue Type: Bug Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1753: HBAE-24408 Introduce a general 'local region' to store data on master
Apache-HBase commented on pull request #1753: URL: https://github.com/apache/hbase/pull/1753#issuecomment-632187035 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 45s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 5m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 1m 17s | the patch passed | | +1 :green_heart: | javac | 1m 17s | the patch passed | | +1 :green_heart: | shadedjars | 5m 28s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hbase-common in the patch passed. | | -1 :x: | unit | 6m 39s | hbase-server in the patch failed. | | | | 33m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1753 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 170b972deca9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/testReport/ | | Max. process+thread count | 776 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1753/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113328#comment-17113328 ] Andrew Kyle Purtell commented on HBASE-11288: - I think your point about a resolver is also interesting. We deliberately don’t (and can’t) control how either the ZK quorum string or list of master endpoints is constructed. We only require that at connection time, in configuration, it is available. I don’t think we are far apart here. Add the registry RPC service to regionservers too. Rename MasterRegistry with a role neutral naming. If you want to do more then the discussion becomes more nuanced but for example if the configuration string is a single host name that resolves to multiple address records then the registry client could be taught to handle this case and it would then just work along with the case where multiple hosts are listed in the string instead of discovered by DNS. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113312#comment-17113312 ] Andrew Kyle Purtell commented on HBASE-11288: - I’m sure an option for passing a static string of regionserver as a bootstrap set could be made to work, but per above answer a different master oriented model was preferred to minimize operational and configuration changes (especially if there is co-location of ZK and master service). Nothing precludes contribution of another type of registry that is backed by an HBase RPC endpoint hosted by the regionserver or another daemon role. If we had such a contribution I’d probably ask how difficult it would be to unify them so it becomes the operators choice what hosts from either role to include in the bootstrap string. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #1746: HBASE-24388 Introduce a 'local root region' at master side to store t…
saintstack commented on a change in pull request #1746:
URL: https://github.com/apache/hbase/pull/1746#discussion_r428743594
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
##
@@ -552,4 +553,6 @@ default SplitWALManager getSplitWALManager(){
* @return The state of the load balancer, or false if the load balancer
isn't defined.
*/
boolean isBalancerOn();
+
+ RootTable getRootTable();
Review comment:
If it the only consumer... yeah, maybe. No hurry.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1746: HBASE-24388 Introduce a 'local root region' at master side to store t…
saintstack commented on a change in pull request #1746:
URL: https://github.com/apache/hbase/pull/1746#discussion_r428743081
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/root/RootTable.java
##
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.root;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.Server;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.master.cleaner.DirScanPool;
+import org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil;
+import org.apache.hadoop.hbase.region.LocalRegion;
+import org.apache.hadoop.hbase.region.LocalRegionParams;
+import org.apache.hadoop.hbase.regionserver.BloomType;
+import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Used to store the location of meta region.
+ */
+@InterfaceAudience.Private
+public class RootTable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(RootTable.class);
+
+ static final String MAX_WALS_KEY = "hbase.root.table.region.maxwals";
+
+ private static final int DEFAULT_MAX_WALS = 10;
+
+ static final String USE_HSYNC_KEY = "hbase.root.table.region.wal.hsync";
+
+ static final String RING_BUFFER_SLOT_COUNT =
"hbase.root.table.region.maxwals";
+
+ private static final int DEFAULT_RING_BUFFER_SLOT_COUNT = 64;
+
+ static final String ROOT_TABLE_DIR = "RootTable";
+
+ static final String HFILECLEANER_PLUGINS =
"hbase.root.table.region.hfilecleaner.plugins";
+
+ static final String FLUSH_SIZE_KEY = "hbase.root.table.region.flush.size";
+
+ static final long DEFAULT_FLUSH_SIZE =
TableDescriptorBuilder.DEFAULT_MEMSTORE_FLUSH_SIZE;
+
+ static final String FLUSH_PER_CHANGES_KEY =
"hbase.root.table.region.flush.per.changes";
+
+ private static final long DEFAULT_FLUSH_PER_CHANGES = 1_000_000;
+
+ static final String FLUSH_INTERVAL_MS_KEY =
"hbase.root.table.region.flush.interval.ms";
+
+ // default to flush every 15 minutes, for safety
+ private static final long DEFAULT_FLUSH_INTERVAL_MS =
TimeUnit.MINUTES.toMillis(15);
+
+ static final String COMPACT_MIN_KEY = "hbase.root.table.region.compact.min";
+
+ private static final int DEFAULT_COMPACT_MIN = 4;
+
+ static final String ROLL_PERIOD_MS_KEY =
"hbase.root.table.region.walroll.period.ms";
+
+ private static final long DEFAULT_ROLL_PERIOD_MS =
TimeUnit.MINUTES.toMillis(15);
+
+ static final TableName TABLE_NAME = TableName.valueOf("master:root");
+
+ private static final TableDescriptor TABLE_DESC =
TableDescriptorBuilder.newBuilder(TABLE_NAME)
+
.setColumnFamily(ColumnFamilyDescriptorBuilder.newBuilder(HConstants.CATALOG_FAMILY)
+ .setMaxVersions(HConstants.DEFAULT_HBASE_META_VERSIONS).setInMemory(true)
+
.setBlocksize(HConstants.DEFAULT_HBASE_META_BLOCK_SIZE).setBloomFilterType(BloomType.ROWCOL)
+ .setDataBlockEncoding(DataBlockEncoding.ROW_INDEX_V1).build())
+.build();
+
+ private final Server server;
+
+ private final DirScanPool cleanerPool;
+
+ private LocalRegion region;
+
+ public RootTable(Server server, DirScanPool cleanerPool) {
+this.server = server;
+this.cleanerPool = cleanerPool;
+ }
+
+ public void initialize() throws IOException {
+LOG.info("Initializing root table...");
+LocalRegionParams params = new
LocalRegionParams().server(server).regionDirName(ROOT_TABLE_DIR)
Review comment:
Yeah, would be good if a general location for master local store.
This is an automated message
[GitHub] [hbase] saintstack commented on a change in pull request #1746: HBASE-24388 Introduce a 'local root region' at master side to store t…
saintstack commented on a change in pull request #1746: URL: https://github.com/apache/hbase/pull/1746#discussion_r428740739 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/region/LocalRegion.java ## @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.region; + +import static org.apache.hadoop.hbase.HConstants.HREGION_LOGDIR_NAME; + +import java.io.IOException; +import java.util.Collections; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.HBaseIOException; +import org.apache.hadoop.hbase.Server; +import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.RegionInfo; +import org.apache.hadoop.hbase.client.RegionInfoBuilder; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.TableDescriptor; +import org.apache.hadoop.hbase.master.HMaster; +import org.apache.hadoop.hbase.master.cleaner.HFileCleaner; +import org.apache.hadoop.hbase.regionserver.HRegion; +import org.apache.hadoop.hbase.regionserver.HRegion.FlushResult; +import org.apache.hadoop.hbase.regionserver.HRegionFileSystem; +import org.apache.hadoop.hbase.regionserver.RegionScanner; +import org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.CommonFSUtils; +import org.apache.hadoop.hbase.util.HFileArchiveUtil; +import org.apache.hadoop.hbase.util.RecoverLeaseFSUtils; +import org.apache.hadoop.hbase.wal.AbstractFSWALProvider; +import org.apache.hadoop.hbase.wal.WAL; +import org.apache.hadoop.hbase.wal.WALFactory; +import org.apache.yetus.audience.InterfaceAudience; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting; +import org.apache.hbase.thirdparty.com.google.common.math.IntMath; + +/** + * A region that stores data in a separated directory on WAL file system. Review comment: Pardon me. 'snowflakes' are purportedly unique; no two are alike. 'snowflaking' is making something 'special', a one-off. I was asking if we need to do the special trick where this local region runs on the WAL FS exclusively making it different to how other Regions do their storage spread across FS's... one for data and another for WAL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24408) Introduce a general 'local region' to store data on master
[ https://issues.apache.org/jira/browse/HBASE-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113288#comment-17113288 ] Michael Stack commented on HBASE-24408: --- I think this a good idea after reading HBASE-24388 PR. A general, persistent store could find other uses beyond procedure store and root table. Would the 'local' master region still host on the WAL FS? Or would it spread like other regions w/ data in data fs and WAL in WAL FS? > Introduce a general 'local region' to store data on master > -- > > Key: HBASE-24408 > URL: https://issues.apache.org/jira/browse/HBASE-24408 > Project: HBase > Issue Type: Task > Components: master >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.3.0 > > > We already have a local region to store the procedure data and when > implementing HBASE-11288, splittable meta, we are thinking of also storing > the data for root table in a local region. > Now in the patch for HBASE-24388, we introduced another local region to store > the data for root table, but maybe it is better to store the procedure data > and root table together in a single region(with different families). > And this should be done before 2.3.0, to prevent shipping the procedure store > region out in a release. Set it a blocker for 2.3.0. > Patch will be available soon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113290#comment-17113290 ] Andrew Kyle Purtell commented on HBASE-18095: - One objective of this work is to remove dependencies on ZK in the client for the normal client cases. That objective is defeated if there is a fallback. It would have to be optional. Now we have an optional fallback for an optional choice to remove dependencies on ZK that reintroduces a dependency on ZK? I don’t believe it provides much value anyway. It’s the same situation as if all your ZK quorum goes down. Someone or something (like an autoscaler or process supervisor) will have to start new instances in either case. > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Kyle Purtell >Assignee: Bharath Vissapragada >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch > > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #1753: HBAE-24408 Introduce a general 'local region' to store data on master
Apache9 commented on pull request #1753: URL: https://github.com/apache/hbase/pull/1753#issuecomment-632165806 Another benefit for this patch is that, now HFile will be stored on the root file system instead of wal filesystem, so we can reuse the HFileCleaner instead of creating a new one by our own. And it is also a good news if later we want to implement WAL on a system other than file system. Added a UT called TestLocalRegionOnTwoFileSystems to confirm it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #1753: HBAE-24408 Introduce a general 'local region' to store data on master
Apache9 commented on pull request #1753: URL: https://github.com/apache/hbase/pull/1753#issuecomment-632160822 The implementation is not perfect, as it initialize the store in RegionProcedureStore, and also the HFilePrinter and WALPrinter can not handle more than 1 family correctly. But anyway, it is enough for shipping with 2.3.0, later when we want to store more data in the region, we can refactor the code, maybe in HBASE-24388. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 opened a new pull request #1753: HBAE-24408 Introduce a general 'local region' to store data on master
Apache9 opened a new pull request #1753: URL: https://github.com/apache/hbase/pull/1753 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-native-client] phrocker opened a new pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
phrocker opened a new pull request #2: URL: https://github.com/apache/hbase-native-client/pull/2 Has test failures. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24400) Automatically download CMake Dependencies
[ https://issues.apache.org/jira/browse/HBASE-24400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marc Parisi updated HBASE-24400: Description: To improve the ability to build we should download and link a local version of dependencies ( in the build folder ) This will help with skew of versions and the ability to build the project. This will help the build process in docker and allow people to develop locally. this will also pave the way for future work to support was: To improve the ability to build we should download and link a local version of folly ( in the build folder ) This will help with skew of versions and the ability to build the project. > Automatically download CMake Dependencies > - > > Key: HBASE-24400 > URL: https://issues.apache.org/jira/browse/HBASE-24400 > Project: HBase > Issue Type: Sub-task >Reporter: Marc Parisi >Assignee: Marc Parisi >Priority: Major > > To improve the ability to build we should download and link a local version > of dependencies ( in the build folder ) > > This will help with skew of versions and the ability to build the project. > > This will help the build process in docker and allow people to develop > locally. this will also pave the way for future work to support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24400) Automatically download CMake Dependencies
[ https://issues.apache.org/jira/browse/HBASE-24400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marc Parisi updated HBASE-24400: Summary: Automatically download CMake Dependencies (was: Automatically download Folly in CMAKE if it does not exist. ) > Automatically download CMake Dependencies > - > > Key: HBASE-24400 > URL: https://issues.apache.org/jira/browse/HBASE-24400 > Project: HBase > Issue Type: Sub-task >Reporter: Marc Parisi >Assignee: Marc Parisi >Priority: Major > > To improve the ability to build we should download and link a local version > of folly ( in the build folder ) > > This will help with skew of versions and the ability to build the project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24386) TableSnapshotScanner support scan limit
[
https://issues.apache.org/jira/browse/HBASE-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113149#comment-17113149
]
Hudson commented on HBASE-24386:
Results for branch master
[build #1733 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1663//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/master/1733//artifact/output-integration/hadoop-3.log].
(note that this means we didn't check the Hadoop 3 shaded client)
> TableSnapshotScanner support scan limit
> ---
>
> Key: HBASE-24386
> URL: https://issues.apache.org/jira/browse/HBASE-24386
> Project: HBase
> Issue Type: Improvement
> Components: Scanners, snapshots
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.5
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24370) Avoid aggressive MergeRegion and GCMultipleMergedRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-24370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113152#comment-17113152
]
Hudson commented on HBASE-24370:
Results for branch master
[build #1733 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1663//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/master/1733//artifact/output-integration/hadoop-3.log].
(note that this means we didn't check the Hadoop 3 shaded client)
> Avoid aggressive MergeRegion and GCMultipleMergedRegionsProcedure
> --
>
> Key: HBASE-24370
> URL: https://issues.apache.org/jira/browse/HBASE-24370
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Major
>
> In
> [https://github.com/apache/hbase/blob/a40a0322a73add68d9cb0579abacdd6a2e41e8fb/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java#L478,
>
> |https://github.com/apache/hbase/blob/a40a0322a73add68d9cb0579abacdd6a2e41e8fb/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java#L478]
> prepareMergeRegion, it checks if one of merged parent regions is a merged
> child region and has not been GCed. If it is ready to GC, it will kick off a
> GCMultipleMergedRegionsProcedure and also start the MergeRegionProcedure.
> There is a race condition here. If MergeRegionProcedure finishes first, it
> will delete meta row for the merged child region. Then
> GCMultipleMergedRegionsProcedure runs, and because the newly added check, it
> thinks GC has been done and wont schedule GCRegionProcedure to clean up those
> merged parent regions. The end result is that these merged parent regions are
> left as orphans on Filesystem.
>
> [https://github.com/apache/hbase/blob/a40a0322a73add68d9cb0579abacdd6a2e41e8fb/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/GCMultipleMergedRegionsProcedure.java#L105]
>
> The proposed solution is to avoid being so aggressive, if it needs to kick
> off GCMultipleMergedRegionsProcedure, then abort MergeRegionProcedure and
> user can try MergeRegionProcedure later.
> [|https://github.com/apache/hbase/blob/a40a0322a73add68d9cb0579abacdd6a2e41e8fb/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java#L478]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24361) Make `RESTApiClusterManager` more resilient
[
https://issues.apache.org/jira/browse/HBASE-24361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113150#comment-17113150
]
Hudson commented on HBASE-24361:
Results for branch master
[build #1733 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1663//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/master/1733//artifact/output-integration/hadoop-3.log].
(note that this means we didn't check the Hadoop 3 shaded client)
> Make `RESTApiClusterManager` more resilient
> ---
>
> Key: HBASE-24361
> URL: https://issues.apache.org/jira/browse/HBASE-24361
> Project: HBase
> Issue Type: Test
> Components: integration tests
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> The Cloudera Manager API client in {{RESTApiClusterManager}} appears to
> assume that API calls sent to CM for process commands block on command
> completion. However, these commands are "asynchronous," queuing work in the
> background for execution. Update the client to track command submission and
> block on completion of that commandId. This allows this {{ClusterManager}} to
> conform to the expectations of the {{Actions}} that invoke it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23938) Replicate slow/large RPC calls to HDFS
[
https://issues.apache.org/jira/browse/HBASE-23938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113151#comment-17113151
]
Hudson commented on HBASE-23938:
Results for branch master
[build #1733 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1663//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1733/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/master/1733//artifact/output-integration/hadoop-3.log].
(note that this means we didn't check the Hadoop 3 shaded client)
> Replicate slow/large RPC calls to HDFS
> --
>
> Key: HBASE-23938
> URL: https://issues.apache.org/jira/browse/HBASE-23938
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: Screen Shot 2020-05-20 at 4.23.06 PM.png
>
>
> We should provide capability to replicate complete slow and large RPC logs to
> HDFS or create new system table in addition to Ring Buffer. This way we don't
> lose any of slow logs and operator can retrieve all the slow/large logs.
> Replicating logs to HDFS / creating new system table should be configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24412) Canary support check only one column family per region
niuyulin created HBASE-24412: Summary: Canary support check only one column family per region Key: HBASE-24412 URL: https://issues.apache.org/jira/browse/HBASE-24412 Project: HBase Issue Type: Improvement Components: canary Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1742: HBASE-24401 Cell size limit check on append should consider 0 or less…
Apache-HBase commented on pull request #1742: URL: https://github.com/apache/hbase/pull/1742#issuecomment-632058985 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 0s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 7m 7s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 38s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed | | +1 :green_heart: | javac | 1m 9s | the patch passed | | +1 :green_heart: | shadedjars | 7m 14s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 143m 32s | hbase-server in the patch passed. | | | | 174m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1742/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1742 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cf59d3d4e04d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1742/5/testReport/ | | Max. process+thread count | 3694 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1742/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #1552: HBASE-24205 Create metric to know the number of reads that happens fr…
anoopsjohn commented on a change in pull request #1552:
URL: https://github.com/apache/hbase/pull/1552#discussion_r428596325
##
File path:
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsStoreSourceImpl.java
##
@@ -0,0 +1,211 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Interns;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MetricsStoreSourceImpl implements MetricsStoreSource {
+
+ private MetricsStoreWrapper storeWrapper;
+ private MetricsStoreAggregateSourceImpl aggreagate;
+ private AtomicBoolean closed = new AtomicBoolean(false);
+
+ private String storeNamePrefix;
+ private final MetricRegistry registry;
+ private static final Logger LOG =
LoggerFactory.getLogger(MetricsStoreSourceImpl.class);
+ String storeReadsKey;
+
+ String memstoreReadsKey;
+ String fileReadsKey;
+ private final Counter storeReads;
+ private final Counter memstoreReads;
+ private final Counter fileReads;
+
+ public MetricsStoreSourceImpl(MetricsStoreWrapper storeWrapper,
+ MetricsStoreAggregateSourceImpl aggreagate) {
+this.storeWrapper = storeWrapper;
+this.aggreagate = aggreagate;
+aggreagate.register(this);
+
+LOG.debug("Creating new MetricsRegionSourceImpl for table " +
storeWrapper.getStoreName() + " "
Review comment:
Pls correct log
for store storeWrapper.getRegionName() + " : " + storeWrapper.getStoreName()
##
File path:
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsStoreSourceImpl.java
##
@@ -0,0 +1,211 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.hadoop.hbase.metrics.Counter;
+import org.apache.hadoop.hbase.metrics.Interns;
+import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class MetricsStoreSourceImpl implements MetricsStoreSource {
+
+ private MetricsStoreWrapper storeWrapper;
+ private MetricsStoreAggregateSourceImpl aggreagate;
+ private AtomicBoolean closed = new AtomicBoolean(false);
+
+ private String storeNamePrefix;
+ private final MetricRegistry registry;
+ private static final Logger LOG =
LoggerFactory.getLogger(MetricsStoreSourceImpl.class);
+ String storeReadsKey;
+
+ String memstoreReadsKey;
+ String fileReadsKey;
+ private final Counter storeReads;
+ private final Counter memstoreReads;
+ private final Counter fileReads;
Review comment:
Yes here.. No need to fileReads.
##
File path:
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
##
@@ -402,6 +402,8 @@
String DELETE_BATCH_KEY = "deleteBatch";
String GET_SIZE_KEY = "getSize";
String GET_KEY = "get";
+ String MEMSTORE_GET_KEY =
[GitHub] [hbase] Apache-HBase commented on pull request #1751: HBASE-24409: Exposing a function in HBase WALKey to set the table name
Apache-HBase commented on pull request #1751: URL: https://github.com/apache/hbase/pull/1751#issuecomment-632048140 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 10s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 48s | master passed | | +1 :green_heart: | compile | 1m 10s | master passed | | +1 :green_heart: | shadedjars | 6m 23s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 35s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 6m 23s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 186m 58s | hbase-server in the patch passed. | | | | 215m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1751/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1751 | | JIRA Issue | HBASE-24409 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2162f16b9c92 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1751/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1751/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1751/1/testReport/ | | Max. process+thread count | 2957 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1751/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1743: HBASE-24403 FsDelegationToken should cache token after acquired a new one
Apache-HBase commented on pull request #1743: URL: https://github.com/apache/hbase/pull/1743#issuecomment-632040961 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 0m 55s | master passed | | +1 :green_heart: | shadedjars | 5m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 143m 39s | hbase-server in the patch failed. | | | | 167m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1743 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6292c20d21b4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/2/testReport/ | | Max. process+thread count | 3910 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1743/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113080#comment-17113080
]
Francis Christopher Liu edited comment on HBASE-11288 at 5/21/20, 11:23 AM:
[~apurtell] thanks for the explanation, I'just have a few follow-up questions
if thats ok.
{quote}
In contrast, what if we start exporting a subset of regionservers. How does
that work? It's guaranteed to be different from how zk quorum strings worked.
{quote}
I pretty sure it's well thought out just understanding the thought process in
case we need to consider creating a regionserver registry so we understand the
tradeoff. My thinking was providing a static subset of regionservers the same
way that's being done with providing a static list of masters. Are we not able
to provide a static subset of masters in the current implementation?
{quote}
There are a lot more of them, they are randomly chosen , the connection info
has to dynamic instead of static (how is that collected? published to
clients?).
{quote}
I think there are a few ways to picking the static subset. Let me know if these
make sense. One way would be to pick regionservers that are part of the
regionserver group hosting the meta. Another way is to provide a seed list and
an api to allow the client to discover more if they need to someone mentioned
that his is how cassandra does it a while back when we were takling about a
regionserver registry. Another way is to provide just a single domain name and
that domain name contains a static subset list of regionserver ips which the
operator can choose to updated dynamically in case one is decomissioned or a
new on is added (this should work for the master approach too tho you would
need a resolver) this way the client config does not need to change.
was (Author: toffer):
[~apurtell] thanks for the explanation, I'just have a few follow-up questions
if thats ok.
{quote}
In contrast, what if we start exporting a subset of regionservers. How does
that work? It's guaranteed to be different from how zk quorum strings worked.
{quote}
I pretty sure it's well thought out just understanding the thought process in
case we need to consider creating a regionserver registry so we understand the
tradeoff. My thinking was providing a static subset of regionservers the same
way that's being done with providing a static list of masters. Are we not able
to provide a static subset of masters in the current implementation?
{quote}
There are a lot more of them, they are randomly chosen , the connection info
has to dynamic instead of static (how is that collected? published to
clients?).
{quote}
I think there are many ways to picking the static subset. One way would be to
pick regionservers that are part of the regionserver group hosting the meta.
Another way is to provide a seed list and an api to allow the client to
discover more if they need to someone mentioned that his is how cassandra does
it a while back when we were takling about a regionserver registry. Another way
is to provide just a single domain name and that domain name contains a static
subset list of regionserver ips which the operator can choose to updated
dynamically in case one is decomissioned or a new on is added (this should work
for the master approach too tho you would need a resolver) this way the client
config does not need to change.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] VicoWu commented on pull request #1743: HBASE-24403 FsDelegationToken should cache token after acquired a new one
VicoWu commented on pull request #1743: URL: https://github.com/apache/hbase/pull/1743#issuecomment-632031498 @jojochuang This issue is found by our cluster incident. But my current commit still has some problem which need to be resolved before finally adopted: In this PR, I cache the token between RS and NameNode instead of always applying for a new one every time processing a file, so the token should be confirmed that it is not expired every time we use it; So I think I need to add a check logic to verify its expiration time , if it is already or going to be expired, I will have to apply for a new one and overwrite the cache; This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1751: HBASE-24409: Exposing a function in HBase WALKey to set the table name
Apache-HBase commented on pull request #1751: URL: https://github.com/apache/hbase/pull/1751#issuecomment-632030193 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | master passed | | +1 :green_heart: | compile | 0m 58s | master passed | | +1 :green_heart: | shadedjars | 5m 39s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 5m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 147m 28s | hbase-server in the patch failed. | | | | 171m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1751/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1751 | | JIRA Issue | HBASE-24409 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f68ddb491b63 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1751/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1751/1/testReport/ | | Max. process+thread count | 4892 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1751/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1737: HBASE-24382 Flush partial stores of region filtered by seqId when arc…
Apache-HBase commented on pull request #1737: URL: https://github.com/apache/hbase/pull/1737#issuecomment-632028705 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 13s | master passed | | +1 :green_heart: | compile | 1m 0s | master passed | | +1 :green_heart: | shadedjars | 6m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 3s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 35s | hbase-server generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 192m 23s | hbase-server in the patch passed. | | | | 218m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1737/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1737 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bbe257225f7b 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / a8724e8120 | | Default Java | 1.8.0_232 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1737/5/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1737/5/testReport/ | | Max. process+thread count | 3230 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1737/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113085#comment-17113085
]
Francis Christopher Liu edited comment on HBASE-11288 at 5/21/20, 11:11 AM:
{quote}
At least we have to provide an API to fecth all the content in root table, as
the cache service needs it. So for HBCK it could also uses this API. But this
API will not be part of the client facing API, for client they just need a
simple locateMeta method.
{quote}
I see. You would need mutation apis as well to correct the errors.
was (Author: toffer):
{quote}
At least we have to provide an API to fecth all the content in root table, as
the cache service needs it. So for HBCK it could also uses this API. But this
API will not be part of the client facing API, for client they just need a
simple locateMeta method.
{quote}
I see. You would need mutation apis as well.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113085#comment-17113085
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
At least we have to provide an API to fecth all the content in root table, as
the cache service needs it. So for HBCK it could also uses this API. But this
API will not be part of the client facing API, for client they just need a
simple locateMeta method.
{quote}
I see. You would need mutation apis as well.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113080#comment-17113080
]
Francis Christopher Liu commented on HBASE-11288:
-
[~apurtell] thanks for the explanation, I'just have a few follow-up questions
if thats ok.
{quote}
In contrast, what if we start exporting a subset of regionservers. How does
that work? It's guaranteed to be different from how zk quorum strings worked.
{quote}
I pretty sure it's well thought out just understanding the thought process in
case we need to consider creating a regionserver registry so we understand the
tradeoff. My thinking was providing a static subset of regionservers the same
way that's being done with providing a static list of masters. Are we not able
to provide a static subset of masters in the current implementation?
{quote}
There are a lot more of them, they are randomly chosen , the connection info
has to dynamic instead of static (how is that collected? published to
clients?).
{quote}
I think there are many ways to picking the static subset. One way would be to
pick regionservers that are part of the regionserver group hosting the meta.
Another way is to provide a seed list and an api to allow the client to
discover more if they need to someone mentioned that his is how cassandra does
it a while back when we were takling about a regionserver registry. Another way
is to provide just a single domain name and that domain name contains a static
subset list of regionserver ips which the operator can choose to updated
dynamically in case one is decomissioned or a new on is added (this should work
for the master approach too tho you would need a resolver) this way the client
config does not need to change.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-21406) "status 'replication'" should not show SINK if the cluster does not act as sink
[
https://issues.apache.org/jira/browse/HBASE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113077#comment-17113077
]
Wellington Chevreuil commented on HBASE-21406:
--
Thanks [~busbey], working on the PR. Since it's been 1.5 year since the last
patch was proposed, there are lots of changes in master now that I need to sort
those out.
> "status 'replication'" should not show SINK if the cluster does not act as
> sink
> ---
>
> Key: HBASE-21406
> URL: https://issues.apache.org/jira/browse/HBASE-21406
> Project: HBase
> Issue Type: Improvement
>Reporter: Daisuke Kobayashi
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HBASE-21406-branch-1.001.patch,
> HBASE-21406-master.001.patch, HBASE-21406-master.002.patch, Screen Shot
> 2018-10-31 at 18.12.54.png
>
>
> When replicating in 1 way, from source to target, {{status 'replication'}} on
> source always dumps SINK with meaningless metrics. It only makes sense when
> running the command on target cluster.
> {{status 'replication'}} on source, for example. {{AgeOfLastAppliedOp}} is
> always zero and {{TimeStampsOfLastAppliedOp}} does not get updated from the
> time the RS started since it's not acting as sink.
> {noformat}
> source-1.com
>SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=0,
> TimeStampsOfLastShippedOp=Mon Oct 29 23:44:14 PDT 2018, Replication Lag=0
>SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Oct 25
> 23:56:53 PDT 2018
> {noformat}
> {{status 'replication'}} on target works as expected. SOURCE is empty as it's
> not acting as source:
> {noformat}
> target-1.com
>SOURCE:
>SINK : AgeOfLastAppliedOp=70, TimeStampsOfLastAppliedOp=Mon Oct 29
> 23:44:08 PDT 2018
> {noformat}
> This is because {{getReplicationLoadSink}}, called in {{admin.rb}}, always
> returns a value (not null).
> 1.X
> https://github.com/apache/hbase/blob/rel/1.4.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L194-L204
> 2.X
> https://github.com/apache/hbase/blob/rel/2.0.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L392-L399
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113072#comment-17113072 ] Duo Zhang commented on HBASE-18095: --- IIRC, one of the problem for MasterRegistry is that, if all the configured masters are gone, then we need to change the configuration to let client know the new masters. Maybe we could add an option called fallbackToZk? If enabled, when there are no masters available, we fallback to use the zookeeper to fetch all the active and backup masters and then update our configuration. Thoughts? Thanks. > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Kyle Purtell >Assignee: Bharath Vissapragada >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch > > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113067#comment-17113067
]
Duo Zhang commented on HBASE-11288:
---
{quote}
If that is something you need we could still have root table but not expose the
region api (possibly like we already do for mutations) to them just the api you
described? Also I'm wondering if we don't expose the scan apis at all then do
we plan to have hbck for root where it is located (eg in the master)?
{quote}
At least we have to provide an API to fecth all the content in root table, as
the cache service needs it. So for HBCK it could also uses this API. But this
API will not be part of the client facing API, for client they just need a
simple locateMeta method.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113058#comment-17113058
]
Francis Christopher Liu commented on HBASE-11288:
-
{quote}
For example, I'm not impressed by HBASE-18095 while lots of other committers
really love it. As in our deployments, we can also control how we deploy
zookeeper, not only HBase. Zookeeper supports a feature called observer node,
which can be considered as a cache of the zk cluster.
{quote}
I see, I agree with your sentiment and we do that too where appropriate. In
fact we were about to go down a similar route as you described with zookeeper
except for some limitations in early zookeeper versions prior to 3.5(?) which
did not have the "read only"(?) mode. The problem for us was that without that
mode there still had to be a quorum agreement for connections to the observer
even though the clients are doing only reads. This was unacceptable for us as
that was one of the things we observed was hammering the ensemble so we had to
go with a different approach.
More recenlty we also have implemented a Rest Registry because of security
requirements along the lines of what [~apurtell] described. Hence I too find
that it makes sense to abstract out zookeeper from the user so we have better
control of the attack surface. I suspect this requirement is going to become
more common.
{quote}
Then for me, it is also very easy to just implement a simple cache server, to
pull all the content in root from the active master and cache it in memory,
then to serve clients as the 'master' for locating meta. And I could also make
use of lvs, to spread load across multiple cache servers. And for users who can
not control things other than HBase, we could implement something like
HBASE-18095, to let backup masters to serve the locating meta requests.
{quote}
Yeah I think we would need an implemented solution for users as well. Or it
would seem a bit lopsided expecting users that worry about root traffic to
implement their own caching service.
{quote}
But if you use a hbase:root table, then you will expose the whole region API to
users. The API is powerful, but hard to simulate, so it will be really hard to
spread the load across multiple servers.
{quote}
If that is something you need we could still have root table but not expose the
region api (possibly like we already do for mutations) to them just the api you
described? Also I'm wondering if we don't expose the scan apis at all then do
we plan to have hbck for root where it is located (eg in the master)?
{quote}
Enable read replicas on root table? Seems a bit overkill and read replicas
itself is still not stable enough I suppose...
{quote}
It would seem overkill if we only did it for root. But we do it for meta today?
Or are people no longer using it for meta? There's no developer support in
getting it stable?
Please let me know what you think?
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
