[GitHub] [hbase] Apache-HBase commented on pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
Apache-HBase commented on pull request #2652: URL: https://github.com/apache/hbase/pull/2652#issuecomment-726581278 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 36s | master passed | | +1 :green_heart: | checkstyle | 1m 30s | master passed | | +1 :green_heart: | spotbugs | 2m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | checkstyle | 1m 30s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 14s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 42m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2652 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux ccd2d3b7ad9b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f89faf3ac8 | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
Apache9 commented on a change in pull request #2652:
URL: https://github.com/apache/hbase/pull/2652#discussion_r522733824
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestZKConnectionRegistry.java
##
@@ -125,4 +127,20 @@ public void testNoMetaAvailable() throws
InterruptedException {
}
}
}
+
+ /**
+ * Pass discontinuous list of znodes to registry getMetaRegionLocation.
Should work fine.
+ * It used to throw ArrayOutOfBoundsException. See HBASE-25280.
+ */
+ @Test
+ public void testDiscontinuousLocations() throws ExecutionException,
InterruptedException {
+Configuration conf = new Configuration(TEST_UTIL.getConfiguration());
+try (ZKConnectionRegistry registry = new ZKConnectionRegistry(conf)) {
+ CompletableFuture cf = new CompletableFuture<>();
+ List znodes = Arrays.asList(
+new String[] { ZNodePaths.META_ZNODE_PREFIX,
ZNodePaths.META_ZNODE_PREFIX + "_003" });
+ registry.getMetaRegionLocation(cf, znodes);
Review comment:
Better to not expose this method here but write fake znode on zk to
simulate discontinuous meta replicas? I do not think it is very difficult to
do...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[
https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231220#comment-17231220
]
Viraj Jasani commented on HBASE-25279:
--
HMaster creates two ZKWatchers:
1. Inherited from HRegionServer for tracking master, primary watcher: which is
closed properly once HRegionServer gets out of it's main loop i.e while
(!isStopped() && isHealthy())
2. For client to locate meta in ZK: which doesn't seem to be closed
{code:java}
if (clientQuorumServers != null && !clientZkObserverMode) {
// we need to take care of the ZK information synchronization
// if given client ZK are not observer nodes
ZKWatcher clientZkWatcher = new ZKWatcher(conf,
getProcessName() + ":" + rpcServices.getSocketAddress().getPort() +
"-clientZK", this,
false, true);
this.metaLocationSyncer = new MetaLocationSyncer(zooKeeper, clientZkWatcher,
this);
this.metaLocationSyncer.start();
this.masterAddressSyncer = new MasterAddressSyncer(zooKeeper,
clientZkWatcher, this);
this.masterAddressSyncer.start();
// set cluster id is a one-go effort
ZKClusterId.setClusterId(clientZkWatcher, fileSystemManager.getClusterId());
}
{code}
We can try defining clientZkWatcher as final instance variable and close with
shutdown() call soon after we delete clusterStateZNode?
{quote}In this case, these threads are tied to underlying ZK connections which
must be cleaned up. So, I don't think thats the right solution.
{quote}
On one hand, making this thread as daemon doesn't seem harmful at all, but on
the other hand, I agree that proper closures of ZKWatcher and thereby
underlying RecoverableZookeeper seem more important.
[~elserj] If you are observing an improper cleanup of this ExecutorService
(maybe as part of some unit tests), maybe we can take this opportunity to close
all ZKWatchers in master and RS (if required) as part of this Jira to make sure
we observe proper cleanup? WDYT?
> Non-daemon thread in ZKWatcher
> --
>
> Key: HBASE-25279
> URL: https://issues.apache.org/jira/browse/HBASE-25279
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
>
> ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons
> which will prevent clean shut downs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25281) Bulkload split hfile too many times due to unreasonable split point
[ https://issues.apache.org/jira/browse/HBASE-25281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-25281: - Component/s: tooling > Bulkload split hfile too many times due to unreasonable split point > --- > > Key: HBASE-25281 > URL: https://issues.apache.org/jira/browse/HBASE-25281 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > Attachments: bulkload.png > > > [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688] > the split point is not in middle, If the hfile span N regions, we need split > N-1 times now -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HBASE-12961) Negative values in read and write region server metrics
[ https://issues.apache.org/jira/browse/HBASE-12961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Thomas updated HBASE-12961: Comment: was deleted (was: The lists all Apache HBase Jiras included in CDH 5 that are not included in the Apache HBase base version 0.95.2. The hbase-0.95.2-cdh5.0.0-beta-1.CHANGES.txt file lists all changes included in CDH 5. The patch for each change can be found in the cloudera/patches directory in the release tarball. Best regards, Doris Hall from https://domywriting.com/blog/teaching-in-coronavirus-outbreak/) > Negative values in read and write region server metrics > > > Key: HBASE-12961 > URL: https://issues.apache.org/jira/browse/HBASE-12961 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Victoria >Assignee: Victoria >Priority: Minor > Fix For: 1.1.0, 0.98.11, 2.0.0 > > Attachments: HBASE-12961-2.0.0-v1.patch, HBASE-12961-v1.patch > > > HMaster web page ui, shows the read/write request per region server. They are > currently displayed by using 32 bit integers. Hence, if the servers are up > for a long time the values can be shown as negative. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25255) Master fails to initialize when creating rs group table
[
https://issues.apache.org/jira/browse/HBASE-25255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231213#comment-17231213
]
Duo Zhang commented on HBASE-25255:
---
Oh, the waitInitialized of MasterProedureEnv will wait until master is
initialized, which is not suitable for system table I suppose. Let me push an
addendum for wait for meta loaded for all the system tables.
> Master fails to initialize when creating rs group table
> ---
>
> Key: HBASE-25255
> URL: https://issues.apache.org/jira/browse/HBASE-25255
> Project: HBase
> Issue Type: Bug
> Components: master, rsgroup
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.4
>
> Attachments:
> TEST-org.apache.hadoop.hbase.rsgroup.TestRSGroupsKillRS.xml
>
>
> Saw this when setup TestRSGroupsKillRS
> {noformat}
> 2020-11-07 16:29:54,565 ERROR [master/e476f4f509a7:0:becomeActiveMaster]
> helpers.MarkerIgnoringBase(159): Failed to become active master
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.master.assignment.RegionStateNode.setProcedure(RegionStateNode.java:198)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createAssignProcedure(AssignmentManager.java:647)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.lambda$null$6(AssignmentManager.java:878)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
> at
> java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1699)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
> at
> java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at
> java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createAssignProcedures(AssignmentManager.java:879)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:759)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:775)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processOfflineRegions(AssignmentManager.java:1513)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1012)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2116)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:515)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25281) Bulkload split hfile too many times due to unreasonable split point
[ https://issues.apache.org/jira/browse/HBASE-25281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-25281: - Attachment: bulkload.png > Bulkload split hfile too many times due to unreasonable split point > --- > > Key: HBASE-25281 > URL: https://issues.apache.org/jira/browse/HBASE-25281 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > Attachments: bulkload.png > > > [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688] > the split point is not in middle, If the hfile span N regions, we need split > N-1 times now -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25281) Bulkload split hfile too many times due to unreasonable split point
niuyulin created HBASE-25281: Summary: Bulkload split hfile too many times due to unreasonable split point Key: HBASE-25281 URL: https://issues.apache.org/jira/browse/HBASE-25281 Project: HBase Issue Type: Improvement Reporter: niuyulin Assignee: niuyulin [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688] the split point is not in middle, If the hfile span N regions, we need split N-1 times now -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
saintstack commented on pull request #2652: URL: https://github.com/apache/hbase/pull/2652#issuecomment-726567532 Use a list as suggested above. Cleaner. New test fails before this fix. Failed test is related. WIll look later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2638: HBASE-25255 Master fails to initialize when creating rs group table
Apache9 commented on a change in pull request #2638:
URL: https://github.com/apache/hbase/pull/2638#discussion_r522685347
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
##
@@ -236,8 +236,8 @@ protected void
deserializeStateData(ProcedureStateSerializer serializer)
@Override
protected boolean waitInitialized(MasterProcedureEnv env) {
-if (getTableName().isSystemTable()) {
- // Creating system table is part of the initialization, so do not wait
here.
+if (getTableName().equals(TableName.NAMESPACE_TABLE_NAME)) {
Review comment:
OK, checked the code, on branch-2 we still do not need to create
namespace before loading meta. The creation of namespace is in
initClusterSchemaService, which is far after we finish loading meta...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-25255) Master fails to initialize when creating rs group table
[
https://issues.apache.org/jira/browse/HBASE-25255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-25255:
--
Fix Version/s: 2.3.4
2.2.7
2.4.0
3.0.0-alpha-1
> Master fails to initialize when creating rs group table
> ---
>
> Key: HBASE-25255
> URL: https://issues.apache.org/jira/browse/HBASE-25255
> Project: HBase
> Issue Type: Bug
> Components: master, rsgroup
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.4
>
> Attachments:
> TEST-org.apache.hadoop.hbase.rsgroup.TestRSGroupsKillRS.xml
>
>
> Saw this when setup TestRSGroupsKillRS
> {noformat}
> 2020-11-07 16:29:54,565 ERROR [master/e476f4f509a7:0:becomeActiveMaster]
> helpers.MarkerIgnoringBase(159): Failed to become active master
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.master.assignment.RegionStateNode.setProcedure(RegionStateNode.java:198)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createAssignProcedure(AssignmentManager.java:647)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.lambda$null$6(AssignmentManager.java:878)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
> at
> java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1699)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
> at
> java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at
> java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createAssignProcedures(AssignmentManager.java:879)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:759)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:775)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processOfflineRegions(AssignmentManager.java:1513)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1012)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2116)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:515)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #2638: HBASE-25255 Master fails to initialize when creating rs group table
Apache9 merged pull request #2638: URL: https://github.com/apache/hbase/pull/2638 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2638: HBASE-25255 Master fails to initialize when creating rs group table
Apache-HBase commented on pull request #2638: URL: https://github.com/apache/hbase/pull/2638#issuecomment-726549698 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 20s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 40s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 7m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 30s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 7m 16s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 196m 43s | hbase-server in the patch passed. | | | | 227m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2638/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2638 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2b99edac76f5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2638/4/testReport/ | | Max. process+thread count | 3373 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2638/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #1994: HBASE-24654 Allow unset table's rsgroup
infraio commented on a change in pull request #1994:
URL: https://github.com/apache/hbase/pull/1994#discussion_r522675608
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
##
@@ -541,7 +541,11 @@ public TableDescriptorBuilder setReplicationScope(int
scope) {
}
public TableDescriptorBuilder setRegionServerGroup(String group) {
-desc.setValue(RSGROUP_KEY, new Bytes(Bytes.toBytes(group)));
+if (group == null) {
Review comment:
How to unset RSGroup by shell? Add a unit test for shell too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
saintstack commented on a change in pull request #2652:
URL: https://github.com/apache/hbase/pull/2652#discussion_r522670863
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKConnectionRegistry.java
##
@@ -133,13 +133,17 @@ private static void tryComplete(MutableInt remaining,
HRegionLocation[] locs,
ServerName.valueOf(snProto.getHostName(), snProto.getPort(),
snProto.getStartCode()));
}
- private void getMetaRegionLocation(CompletableFuture future,
+ @VisibleForTesting
+ void getMetaRegionLocation(CompletableFuture future,
List metaReplicaZNodes) {
if (metaReplicaZNodes.isEmpty()) {
future.completeExceptionally(new IOException("No meta znode available"));
}
HRegionLocation[] locs = new HRegionLocation[metaReplicaZNodes.size()];
MutableInt remaining = new MutableInt(locs.length);
+// Do NOT use replicaid as index into locations array. The location set
may not be complete
Review comment:
Let me try and do what you suggest later @apache9 with a list and
pointers back to RegionLocations. Making an array that is maxReplicaId+1 is
going to have the tryComplete do more work than was asked for (see 'remaining'
param). Thanks for the help.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
Apache9 commented on a change in pull request #2652:
URL: https://github.com/apache/hbase/pull/2652#discussion_r522665883
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKConnectionRegistry.java
##
@@ -133,13 +133,17 @@ private static void tryComplete(MutableInt remaining,
HRegionLocation[] locs,
ServerName.valueOf(snProto.getHostName(), snProto.getPort(),
snProto.getStartCode()));
}
- private void getMetaRegionLocation(CompletableFuture future,
+ @VisibleForTesting
+ void getMetaRegionLocation(CompletableFuture future,
List metaReplicaZNodes) {
if (metaReplicaZNodes.isEmpty()) {
future.completeExceptionally(new IOException("No meta znode available"));
}
HRegionLocation[] locs = new HRegionLocation[metaReplicaZNodes.size()];
MutableInt remaining = new MutableInt(locs.length);
+// Do NOT use replicaid as index into locations array. The location set
may not be complete
Review comment:
The problem is for RegionLocations. As Huaxiang pointed out that in
RegionLocations we will handle the out of order replicas, I think it is fine to
do something like this here. Maybe we could use a List instead of an array? And
add a comment here to say that "we do not care about the order of the replicas
or if there are holes, the constructor of RegionLocations will handle this".
And we could add a Constructor for RegionLoations to accept a List. And for the
old constructor which accepts an array, just do this(Arrays.asList(locations));
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
Apache-HBase commented on pull request #2652: URL: https://github.com/apache/hbase/pull/2652#issuecomment-726524220 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 1m 24s | master passed | | +1 :green_heart: | shadedjars | 7m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 1m 23s | the patch passed | | +1 :green_heart: | javac | 1m 23s | the patch passed | | +1 :green_heart: | shadedjars | 7m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 7s | hbase-client in the patch passed. | | -1 :x: | unit | 202m 18s | hbase-server in the patch failed. | | | | 233m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2652 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c30d737a0d65 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/testReport/ | | Max. process+thread count | 3644 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2638: HBASE-25255 Master fails to initialize when creating rs group table
Apache-HBase commented on pull request #2638: URL: https://github.com/apache/hbase/pull/2638#issuecomment-726522310 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | master passed | | +1 :green_heart: | compile | 0m 56s | master passed | | +1 :green_heart: | shadedjars | 6m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 6m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 139m 36s | hbase-server in the patch passed. | | | | 165m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2638/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2638 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 35cac7372caf 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2638/4/testReport/ | | Max. process+thread count | 3795 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2638/4/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
saintstack commented on a change in pull request #2652:
URL: https://github.com/apache/hbase/pull/2652#discussion_r522642007
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKConnectionRegistry.java
##
@@ -133,13 +133,17 @@ private static void tryComplete(MutableInt remaining,
HRegionLocation[] locs,
ServerName.valueOf(snProto.getHostName(), snProto.getPort(),
snProto.getStartCode()));
}
- private void getMetaRegionLocation(CompletableFuture future,
+ @VisibleForTesting
+ void getMetaRegionLocation(CompletableFuture future,
List metaReplicaZNodes) {
if (metaReplicaZNodes.isEmpty()) {
future.completeExceptionally(new IOException("No meta znode available"));
}
HRegionLocation[] locs = new HRegionLocation[metaReplicaZNodes.size()];
MutableInt remaining = new MutableInt(locs.length);
+// Do NOT use replicaid as index into locations array. The location set
may not be complete
Review comment:
Thanks @huaxiangsun . That helps (Its also used here
https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKConnectionRegistry.java#L187
but if error, we don't get to this point because of ArrayOutOfBoundsException).
Let me just do what @apache9 suggests. Looks like there is precedent around
the handling of this situation. Let me follow the model (even if it means
double work as you point out).
Will be back in a bit w/ new patch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
huaxiangsun commented on a change in pull request #2652:
URL: https://github.com/apache/hbase/pull/2652#discussion_r522623604
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKConnectionRegistry.java
##
@@ -133,13 +133,17 @@ private static void tryComplete(MutableInt remaining,
HRegionLocation[] locs,
ServerName.valueOf(snProto.getHostName(), snProto.getPort(),
snProto.getStartCode()));
}
- private void getMetaRegionLocation(CompletableFuture future,
+ @VisibleForTesting
+ void getMetaRegionLocation(CompletableFuture future,
List metaReplicaZNodes) {
if (metaReplicaZNodes.isEmpty()) {
future.completeExceptionally(new IOException("No meta znode available"));
}
HRegionLocation[] locs = new HRegionLocation[metaReplicaZNodes.size()];
MutableInt remaining = new MutableInt(locs.length);
+// Do NOT use replicaid as index into locations array. The location set
may not be complete
Review comment:
Based on Duo and Stack's comments, I spend quite some time trying to
understand how `locs` is being used.
It is being used in
https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKConnectionRegistry.java#L162
It is used for initialize RegionLocations() object.
Within RegionLocations(), all logic is there trying to allocate a location
array with replica id as index.
https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/RegionLocations.java#L74
So it seems that at least in this place, it is fine to do in this way.
Otherwise, the logic is duplicated twice?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
Apache-HBase commented on pull request #2652: URL: https://github.com/apache/hbase/pull/2652#issuecomment-726496766 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 1m 35s | master passed | | +1 :green_heart: | shadedjars | 6m 36s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | compile | 1m 34s | the patch passed | | +1 :green_heart: | javac | 1m 34s | the patch passed | | +1 :green_heart: | shadedjars | 6m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 8s | hbase-client in the patch passed. | | -1 :x: | unit | 131m 19s | hbase-server in the patch failed. | | | | 163m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2652 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 18e49b508b83 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Default Java | AdoptOpenJDK-11.0.6+10 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/testReport/ | | Max. process+thread count | 4782 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
saintstack commented on a change in pull request #2652:
URL: https://github.com/apache/hbase/pull/2652#discussion_r522598513
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKConnectionRegistry.java
##
@@ -133,13 +133,17 @@ private static void tryComplete(MutableInt remaining,
HRegionLocation[] locs,
ServerName.valueOf(snProto.getHostName(), snProto.getPort(),
snProto.getStartCode()));
}
- private void getMetaRegionLocation(CompletableFuture future,
+ @VisibleForTesting
+ void getMetaRegionLocation(CompletableFuture future,
List metaReplicaZNodes) {
if (metaReplicaZNodes.isEmpty()) {
future.completeExceptionally(new IOException("No meta znode available"));
}
HRegionLocation[] locs = new HRegionLocation[metaReplicaZNodes.size()];
MutableInt remaining = new MutableInt(locs.length);
+// Do NOT use replicaid as index into locations array. The location set
may not be complete
Review comment:
Thanks for the review @Apache9
I need help understanding here why returning an array with empty slots is
the way to go instead? What you thinking? Is it that you are afraid that
elsewhere we are using replicaid as index?
When you say find the max replicaid, are you suggesting the max replicaid
present in the list metaReplicaZNodes (doable)? Or are you thinking the array
should be same as the configured replica count (hard -- hard to pass an active
table schema reference down to ConnectionRegistry)?
Thanks for the fast review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache9 merged pull request #42: HBASE-25270 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.m…
Apache9 merged pull request #42: URL: https://github.com/apache/hbase-thirdparty/pull/42 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25270) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25270. --- Hadoop Flags: Reviewed Resolution: Fixed Merged to master. Thanks [~zghao] for reviewing. > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1 > > > Key: HBASE-25270 > URL: https://issues.apache.org/jira/browse/HBASE-25270 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] huaxiangsun edited a comment on pull request #2640: Backport: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun edited a comment on pull request #2640: URL: https://github.com/apache/hbase/pull/2640#issuecomment-726485274 @Apache9 @saintstack I have rebased my branch to include HBASE-25272, please review. It shows lots of commits, but please look at the last two commits. Once Stack rebase HBASE-18070.branch-2, these previous commits will go away. `Huaxiang Sun HBASE-25126 Add load balance logic in hbase-client to distribute read… … 501ed34 Huaxiang Sun Remove ConnectionImplementation#setUseMetaReplicas() 7540dec` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #2640: Backport: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on pull request #2640: URL: https://github.com/apache/hbase/pull/2640#issuecomment-726485274 @Apache9 @saintstack I have rebased my branch to include HBASE-25272, please review. It shows lots of commits, but please look at the last two commits. Once Stack rebase HBASE-18070.branch-2, these previous commits will go away. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2638: HBASE-25255 Master fails to initialize when creating rs group table
Apache-HBase commented on pull request #2638: URL: https://github.com/apache/hbase/pull/2638#issuecomment-726483083 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | checkstyle | 1m 12s | master passed | | +1 :green_heart: | spotbugs | 2m 9s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | checkstyle | 1m 12s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 58s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 42m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2638/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2638 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 637ad54f1caf 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2638/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache-HBase commented on pull request #42: HBASE-25270 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.m…
Apache-HBase commented on pull request #42: URL: https://github.com/apache/hbase-thirdparty/pull/42#issuecomment-726480053 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 57s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 1m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-42/1/artifact/yetus-precommit-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase-thirdparty/pull/42 | | Optional Tests | dupname asflicense markdownlint | | uname | Linux d1e8208745f2 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux | | Build tool | maven | | git revision | master / 5f37d8d | | Max. process+thread count | 28 (vs. ulimit of 1000) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-Thirdparty-PreCommit/job/PR-42/1/console | | versions | git=2.20.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-thirdparty] Apache9 opened a new pull request #42: HBASE-25270 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.m…
Apache9 opened a new pull request #42: URL: https://github.com/apache/hbase-thirdparty/pull/42 …d for 3.4.1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (HBASE-25270) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-25270: --- > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1 > > > Key: HBASE-25270 > URL: https://issues.apache.org/jira/browse/HBASE-25270 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25270) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25270. --- Resolution: Fixed For generating changes and releasenotes > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1 > > > Key: HBASE-25270 > URL: https://issues.apache.org/jira/browse/HBASE-25270 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25270) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-25270: - Assignee: Duo Zhang > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1 > > > Key: HBASE-25270 > URL: https://issues.apache.org/jira/browse/HBASE-25270 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25270) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-25270: -- Fix Version/s: thirdparty-3.4.1 > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.1 > > > Key: HBASE-25270 > URL: https://issues.apache.org/jira/browse/HBASE-25270 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
Apache-HBase commented on pull request #2652: URL: https://github.com/apache/hbase/pull/2652#issuecomment-726463840 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | checkstyle | 1m 31s | master passed | | +1 :green_heart: | spotbugs | 3m 1s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 36s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 32s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 44m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2652 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 6efa3c2a1a40 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2652/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25255) Master fails to initialize when creating rs group table
[
https://issues.apache.org/jira/browse/HBASE-25255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231100#comment-17231100
]
Duo Zhang commented on HBASE-25255:
---
{quote}
Other system tables have same problem?
{quote}
It think so. In the PR here I moved the meta loaded event trigger after
processOfflineRegions, and modified CreateTableProcedure to always wait for
meta loaded so we will not create other system tables before
processOfflineRegions are done.
On branch-2 there is another problem that namespace table must be created
before meta table, so CreateTableProcedure need to check whether it is creating
namespace table, if so it should not wait for meta loaded event.
> Master fails to initialize when creating rs group table
> ---
>
> Key: HBASE-25255
> URL: https://issues.apache.org/jira/browse/HBASE-25255
> Project: HBase
> Issue Type: Bug
> Components: master, rsgroup
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Attachments:
> TEST-org.apache.hadoop.hbase.rsgroup.TestRSGroupsKillRS.xml
>
>
> Saw this when setup TestRSGroupsKillRS
> {noformat}
> 2020-11-07 16:29:54,565 ERROR [master/e476f4f509a7:0:becomeActiveMaster]
> helpers.MarkerIgnoringBase(159): Failed to become active master
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.master.assignment.RegionStateNode.setProcedure(RegionStateNode.java:198)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createAssignProcedure(AssignmentManager.java:647)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.lambda$null$6(AssignmentManager.java:878)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
> at
> java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1699)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
> at
> java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at
> java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createAssignProcedures(AssignmentManager.java:879)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:759)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:775)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processOfflineRegions(AssignmentManager.java:1513)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1012)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2116)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:515)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #2638: HBASE-25255 Master fails to initialize when creating rs group table
Apache9 commented on a change in pull request #2638:
URL: https://github.com/apache/hbase/pull/2638#discussion_r522563709
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
##
@@ -236,8 +236,8 @@ protected void
deserializeStateData(ProcedureStateSerializer serializer)
@Override
protected boolean waitInitialized(MasterProcedureEnv env) {
-if (getTableName().isSystemTable()) {
- // Creating system table is part of the initialization, so do not wait
here.
+if (getTableName().equals(TableName.NAMESPACE_TABLE_NAME)) {
Review comment:
For keeping the code same for both master and branch-2 for now, so it
will be easy to cherry-pick to branch-2. Could open a follow on issue to remove
it on master? Or I could open another PR for branch-2 after the master one is
merged and added this check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
Apache9 commented on a change in pull request #2652:
URL: https://github.com/apache/hbase/pull/2652#discussion_r522562320
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKConnectionRegistry.java
##
@@ -133,13 +133,17 @@ private static void tryComplete(MutableInt remaining,
HRegionLocation[] locs,
ServerName.valueOf(snProto.getHostName(), snProto.getPort(),
snProto.getStartCode()));
}
- private void getMetaRegionLocation(CompletableFuture future,
+ @VisibleForTesting
+ void getMetaRegionLocation(CompletableFuture future,
List metaReplicaZNodes) {
if (metaReplicaZNodes.isEmpty()) {
future.completeExceptionally(new IOException("No meta znode available"));
}
HRegionLocation[] locs = new HRegionLocation[metaReplicaZNodes.size()];
MutableInt remaining = new MutableInt(locs.length);
+// Do NOT use replicaid as index into locations array. The location set
may not be complete
Review comment:
I do not think use index instead of replicaId is the correct way to fix
the problem here.
You should use replicaId, and the problem here is the size of the array is
incorrect. You should find out the maximum replicaId and create the locs array
with maxReplicaId + 1 as its size.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack opened a new pull request #2652: HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConne…
saintstack opened a new pull request #2652: URL: https://github.com/apache/hbase/pull/2652 …ctionRegistry Use loop index rather than replicaid offsetting into locations array. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231063#comment-17231063 ] Bharath Vissapragada commented on HBASE-25279: -- > I'm not sure if we get into any troubles just marking this thread as daemon. > I would think not, it's just a smell to not be proactively closing stuff down > that we spawn I think daemon threads are useful if they are okay to be terminated abruptly and if there is not much state associated with them to be cleaned up. In this case, these threads are tied to underlying ZK connections which must be cleaned up. So, I don't think thats the right solution. Are you able to repro this at will? If so I can take a look. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25255) Master fails to initialize when creating rs group table
[
https://issues.apache.org/jira/browse/HBASE-25255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231059#comment-17231059
]
Guanghao Zhang commented on HBASE-25255:
{quote} it is possible that before meta loaded is finished, we could add the
region state node for rsgroup table to AssignmentManager with OFFLINE state,
and then we will call processOfflineRegions to assign the offline regions, but
at the same time, the CreateTableProcedure will assign it too, thus we get this
assertion error.
{quote}
Other system tables have same problem?
> Master fails to initialize when creating rs group table
> ---
>
> Key: HBASE-25255
> URL: https://issues.apache.org/jira/browse/HBASE-25255
> Project: HBase
> Issue Type: Bug
> Components: master, rsgroup
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Attachments:
> TEST-org.apache.hadoop.hbase.rsgroup.TestRSGroupsKillRS.xml
>
>
> Saw this when setup TestRSGroupsKillRS
> {noformat}
> 2020-11-07 16:29:54,565 ERROR [master/e476f4f509a7:0:becomeActiveMaster]
> helpers.MarkerIgnoringBase(159): Failed to become active master
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.master.assignment.RegionStateNode.setProcedure(RegionStateNode.java:198)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createAssignProcedure(AssignmentManager.java:647)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.lambda$null$6(AssignmentManager.java:878)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
> at
> java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1699)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
> at
> java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at
> java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createAssignProcedures(AssignmentManager.java:879)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:759)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:775)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processOfflineRegions(AssignmentManager.java:1513)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1012)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2116)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:515)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] infraio commented on a change in pull request #2638: HBASE-25255 Master fails to initialize when creating rs group table
infraio commented on a change in pull request #2638:
URL: https://github.com/apache/hbase/pull/2638#discussion_r522530830
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
##
@@ -236,8 +236,8 @@ protected void
deserializeStateData(ProcedureStateSerializer serializer)
@Override
protected boolean waitInitialized(MasterProcedureEnv env) {
-if (getTableName().isSystemTable()) {
- // Creating system table is part of the initialization, so do not wait
here.
+if (getTableName().equals(TableName.NAMESPACE_TABLE_NAME)) {
Review comment:
There are no NAMESPACE_TABLE now?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] z-york commented on a change in pull request #2650: HBASE-25278 Add CACHE_BLOCKS option to count shell command
z-york commented on a change in pull request #2650:
URL: https://github.com/apache/hbase/pull/2650#discussion_r522528641
##
File path: hbase-shell/src/main/ruby/shell/commands/count.rb
##
@@ -60,17 +71,27 @@ def count(table, params = {})
# If the second parameter is an integer, then it is the old command
syntax
params = { 'INTERVAL' => params } if params.is_a?(Integer)
+# Try to be nice and convert a string to a bool
+if params.include?('CACHE_BLOCKS') and
params['CACHE_BLOCKS'].is_a?(String)
+ if params['CACHE_BLOCKS'].downcase == 'true'
+params['CACHE_BLOCKS'] = true
+ elsif params['CACHE_BLOCKS'].downcase == 'false'
+params['CACHE_BLOCKS'] = false
+ end
Review comment:
Maybe worth throwing an error back if it isn't true or false?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25278) Add option to toggle CACHE_BLOCKS in count.rb
[ https://issues.apache.org/jira/browse/HBASE-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231057#comment-17231057 ] Zach York commented on HBASE-25278: --- Looks like an easy way to enable this use case. +1 Longer term, I wonder if it makes sense to add a warmup/cache blocks top level utility in hbase admin/shell since a user really shouldn't need to have to actually return any data if all they care about is warming the cache. This might give a better user experience for people who don't want to pay the region open penalty for prefetch on open, but still want to avoid a cold cache. > Add option to toggle CACHE_BLOCKS in count.rb > - > > Key: HBASE-25278 > URL: https://issues.apache.org/jira/browse/HBASE-25278 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.4.0 > > > A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is > to edit table.rb so that the `count` shell command will not instruct > RegionServers to not cache any data blocks. This is a quick+dirty way to > force a table to be loaded into block cache (i.e. for performance testing). > We can easily add another option to avoid having to edit the ruby files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231052#comment-17231052 ] Josh Elser commented on HBASE-25279: Yeah, I think you're right. I didn't go looking to see if it should've been closed down. In general, having any non-daemon threads in HBase processes is a smell (which is what caught my attention). Looks like HMaster creates an ZKWatcher but doesn't close it. Sounds like something else to follow-up on. I'm not sure if we get into any troubles just marking this thread as daemon. I would think not, it's just a smell to not be proactively closing stuff down that we spawn :) > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-726388823 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 48s | master passed | | +1 :green_heart: | compile | 1m 15s | master passed | | +1 :green_heart: | shadedjars | 7m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 7m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 36s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 205m 3s | hbase-server in the patch passed. | | | | 235m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 20a4a6f007c9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/testReport/ | | Max. process+thread count | 3522 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-726385370 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 33s | master passed | | +1 :green_heart: | compile | 1m 32s | master passed | | +1 :green_heart: | shadedjars | 7m 29s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 58s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 27s | the patch passed | | +1 :green_heart: | compile | 1m 30s | the patch passed | | +1 :green_heart: | javac | 1m 30s | the patch passed | | +1 :green_heart: | shadedjars | 7m 23s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 39s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 193m 23s | hbase-server in the patch passed. | | | | 226m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8d0d3efc3b20 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/testReport/ | | Max. process+thread count | 3292 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2651: HBASE-25279 Make ZKWatcher ExecutorService launch daemon threads
Apache-HBase commented on pull request #2651: URL: https://github.com/apache/hbase/pull/2651#issuecomment-726380757 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | checkstyle | 0m 15s | master passed | | +1 :green_heart: | spotbugs | 0m 35s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | +1 :green_heart: | checkstyle | 0m 14s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 9s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 34m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2651/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2651 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 00e0de20c518 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-zookeeper U: hbase-zookeeper | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2651/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2651: HBASE-25279 Make ZKWatcher ExecutorService launch daemon threads
Apache-HBase commented on pull request #2651: URL: https://github.com/apache/hbase/pull/2651#issuecomment-726377112 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 18s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 46s | master passed | | +1 :green_heart: | compile | 0m 20s | master passed | | +1 :green_heart: | shadedjars | 7m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 18s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 49s | the patch passed | | +1 :green_heart: | compile | 0m 20s | the patch passed | | +1 :green_heart: | javac | 0m 20s | the patch passed | | +1 :green_heart: | shadedjars | 7m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 47s | hbase-zookeeper in the patch passed. | | | | 30m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2651/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2651 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3e211fecc0df 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2651/1/testReport/ | | Max. process+thread count | 607 (vs. ulimit of 3) | | modules | C: hbase-zookeeper U: hbase-zookeeper | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2651/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2651: HBASE-25279 Make ZKWatcher ExecutorService launch daemon threads
Apache-HBase commented on pull request #2651: URL: https://github.com/apache/hbase/pull/2651#issuecomment-726375337 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 5s | master passed | | +1 :green_heart: | compile | 0m 22s | master passed | | +1 :green_heart: | shadedjars | 7m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 8s | the patch passed | | +1 :green_heart: | compile | 0m 20s | the patch passed | | +1 :green_heart: | javac | 0m 20s | the patch passed | | +1 :green_heart: | shadedjars | 6m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 45s | hbase-zookeeper in the patch passed. | | | | 27m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2651/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2651 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4a4a727bf169 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2651/1/testReport/ | | Max. process+thread count | 458 (vs. ulimit of 3) | | modules | C: hbase-zookeeper U: hbase-zookeeper | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2651/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231007#comment-17231007 ] Bharath Vissapragada commented on HBASE-25279: -- [~elserj] Not sure I follow. Doesn't that mean someone hasn't closed the watcher? If you shutdown without closing the watcher (and its underlying connection), hows that clean? > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] sguggilam commented on a change in pull request #2627: HBASE-25251 Enable configuration based enable/disable of Unsafe packa…
sguggilam commented on a change in pull request #2627: URL: https://github.com/apache/hbase/pull/2627#discussion_r522457859 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/UnsafeAvailChecker.java ## @@ -33,12 +35,16 @@ private static final Logger LOG = LoggerFactory.getLogger(UnsafeAvailChecker.class); private static boolean avail = false; private static boolean unaligned = false; + // Flag that can be set by clients to enable/disable Unsafe package usage + private static boolean useUnsafe; Review comment: @Apache9 I have made the PR simple and removed the code for replacing static with non-static references in Bytes class. Also I have removed the useUnsafe variable and just utilizing the avail variable based on the set config. Please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25267) Add SSL keystore type and truststore related configs for HBase RESTServer
[
https://issues.apache.org/jira/browse/HBASE-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230990#comment-17230990
]
Hudson commented on HBASE-25267:
Results for branch branch-2.2
[build #116 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add SSL keystore type and truststore related configs for HBase RESTServer
> -
>
> Key: HBASE-25267
> URL: https://issues.apache.org/jira/browse/HBASE-25267
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.4
>
>
> The RESTServer currently relies on the following parameters to configure SSL
> on the REST API:
> * {{hbase.rest.ssl.enabled}}
> * {{hbase.rest.ssl.keystore.store}}
> * {{hbase.rest.ssl.keystore.password}}
> * {{hbase.rest.ssl.keystore.keypassword}}
> * {{hbase.rest.ssl.exclude.cipher.suites}}
> * {{hbase.rest.ssl.include.cipher.suites}}
> * {{hbase.rest.ssl.exclude.protocols}}
> * {{hbase.rest.ssl.include.protocols}}
> In this patch I want to introduce the following new parameters:
> * {{hbase.rest.ssl.keystore.type}}
> * {{hbase.rest.ssl.truststore.store}}
> * {{hbase.rest.ssl.truststore.password}}
> * {{hbase.rest.ssl.truststore.type}}
> If any of the new the parameter is not provided, then we should fall-back to
> the current behaviour (e.g. assuming JKS keystore/truststore types, or no
> passwords, or no custom trust store file).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-25279: --- Status: Patch Available (was: Open) > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25276) Need to throw the original exception in HRegion#openHRegion
[
https://issues.apache.org/jira/browse/HBASE-25276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230986#comment-17230986
]
Hudson commented on HBASE-25276:
Results for branch branch-2.2
[build #116 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Need to throw the original exception in HRegion#openHRegion
> ---
>
> Key: HBASE-25276
> URL: https://issues.apache.org/jira/browse/HBASE-25276
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.4
>
>
> {code:java}
> protected HRegion openHRegion(final CancelableProgressable reporter)
> throws IOException {
> try {
> ..
> } catch(Throwable t) {
> // By coprocessor path wrong region will open failed,
> // MetricsRegionWrapperImpl is already init and not close,
> // add region close when open failed
> this.close(); // This may throw IOException too.
> throw t;
> }
> return this;
> }
> {code}
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L8144
> this.close() may throw IOException too. The original exception will not be
> recorded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24350) HBase table level replication metrics for shippedBytes are always 0
[
https://issues.apache.org/jira/browse/HBASE-24350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230988#comment-17230988
]
Hudson commented on HBASE-24350:
Results for branch branch-2.2
[build #116 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HBase table level replication metrics for shippedBytes are always 0
> ---
>
> Key: HBASE-24350
> URL: https://issues.apache.org/jira/browse/HBASE-24350
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.0
>Reporter: Sandeep Pal
>Assignee: Sandeep Pal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
>
> It was observed during some investigations that table level metrics for
> shippedBytes are always 0 consistently even though data is getting shipped.
> There are two problems with table-level metrics:
> # There are no table-level metrics for shipped bytes.
> # Another problem is that it's using `MetricsReplicationSourceSourceImpl`
> which is creating all source-level metrics at table level as well but updated
> only ageOfLastShippedOp. This reports lot of false/incorrect replication
> metrics at table level.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24779) Improve insight into replication WAL readers hung on checkQuota
[
https://issues.apache.org/jira/browse/HBASE-24779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230989#comment-17230989
]
Hudson commented on HBASE-24779:
Results for branch branch-2.2
[build #116 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve insight into replication WAL readers hung on checkQuota
> ---
>
> Key: HBASE-24779
> URL: https://issues.apache.org/jira/browse/HBASE-24779
> Project: HBase
> Issue Type: Task
> Components: Replication
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> Helped a customer this past weekend who, with a large number of
> RegionServers, has some RegionServers which replicated data to a peer without
> issues while other RegionServers did not.
> The number of queue logs varied over the past 24hrs in the same manner. Some
> spikes in queued logs into 100's of logs, but other times, only 1's-10's of
> logs were queued.
> We were able to validate that there were "good" and "bad" RegionServers by
> creating a test table, assigning it to a regionserver, enabling replication
> on that table, and validating if the local puts were replicated to a peer. On
> a good RS, data was replicated immediately. On a bad RS, data was never
> replicated (at least, on the order of 10's of minutes which we waited).
> On the "bad RS", we were able to observe that the \{{wal-reader}} thread(s)
> on that RS were spending time in a Thread.sleep() in a different location
> than the other. Specifically it was sitting in the
> {{ReplicationSourceWALReader#checkQuota()}}'s sleep call, _not_ the
> {{handleEmptyWALBatch()}} method on the same class.
> My only assumption is that, somehow, these RegionServers got into a situation
> where they "allocated" memory from the quota but never freed it. Then,
> because the WAL reader thinks it has no free memory, it blocks indefinitely
> and there are no pending edits to ship and (thus) free that memory. A cursory
> glance at the code gives me a _lot_ of anxiety around places where we don't
> properly clean it up (e.g. batches that fail to ship, dropping a peer). As a
> first stab, let me add some more debugging so we can actually track this
> state properly for the operators and their sanity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25280) [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry
Michael Stack created HBASE-25280:
-
Summary: [meta replicas] ArrayIndexOutOfBoundsException in
ZKConnectionRegistry
Key: HBASE-25280
URL: https://issues.apache.org/jira/browse/HBASE-25280
Project: HBase
Issue Type: Bug
Components: meta replicas
Affects Versions: HBASE-18070
Reporter: Michael Stack
Assignee: Michael Stack
Fix For: 3.0.0-alpha-1, 2.4.0
ITBLL Testing HBASE-18070 feature, [~huaxiangsun] found this:
{code:java}
2020-11-12 19:48:12,358 ERROR org.apache.hadoop.hbase.util.FutureUtils:
Unexpected error caught when processing
CompletableFuturejava.lang.ArrayIndexOutOfBoundsException: Index 3 out of
bounds for length 3 at
org.apache.hadoop.hbase.client.ZKConnectionRegistry.lambda$getMetaRegionLocation$2(ZKConnectionRegistry.java:180)
at
org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at
org.apache.hadoop.hbase.client.ZKConnectionRegistry.lambda$getAndConvert$0(ZKConnectionRegistry.java:78)
at
org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at
org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174)
at
org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342)
at java.base/java.lang.Thread.run(Thread.java:834) {code}
The code has been this way a long time but his running with four replicas seems
to have revealed a race exposed by using replicaid as index.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25003) Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3
[
https://issues.apache.org/jira/browse/HBASE-25003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230987#comment-17230987
]
Hudson commented on HBASE-25003:
Results for branch branch-2.2
[build #116 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/116//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3
> ---
>
> Key: HBASE-25003
> URL: https://issues.apache.org/jira/browse/HBASE-25003
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Affects Versions: 2.3.1, 2.2.5
>Reporter: Tamas Adami
>Assignee: Tamas Adami
>Priority: Major
> Fix For: 2.2.7, 2.3.4
>
>
> Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3.
> HBASE-24779 was written on top of HBASE-24350. Whilst it is possible to
> manually resolve the conflict to exclude the piece of code introduced by
> HBASE-24350, ideally we should also include HBASE-24350 in this package of
> fixes, as it deals with some inconsistencies about table level metrics for
> replication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] joshelser opened a new pull request #2651: HBASE-25279 Make ZKWatcher ExecutorService launch daemon threads
joshelser opened a new pull request #2651: URL: https://github.com/apache/hbase/pull/2651 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25279) Non-daemon thread in ZKWatcher
[ https://issues.apache.org/jira/browse/HBASE-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230984#comment-17230984 ] Josh Elser commented on HBASE-25279: [~bharathv], [~vjasani], be careful here please, gents. Marking the threads as daemon is super important. > Non-daemon thread in ZKWatcher > -- > > Key: HBASE-25279 > URL: https://issues.apache.org/jira/browse/HBASE-25279 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons > which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25279) Non-daemon thread in ZKWatcher
Josh Elser created HBASE-25279: -- Summary: Non-daemon thread in ZKWatcher Key: HBASE-25279 URL: https://issues.apache.org/jira/browse/HBASE-25279 Project: HBase Issue Type: Bug Reporter: Josh Elser Assignee: Josh Elser Fix For: 3.0.0-alpha-1 ZKWatcher spawns an ExecutorService which doesn't mark its threads as daemons which will prevent clean shut downs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] joshelser commented on pull request #2650: HBASE-25278 Add CACHE_BLOCKS option to count shell command
joshelser commented on pull request #2650: URL: https://github.com/apache/hbase/pull/2650#issuecomment-726357462 Validated this with a little local install. Wrote a bunch of data with `hbase pe`. Ran `count 'TestTable'` and did not see block cache usage grow. Ran `count 'TestTable', CACHE_BLOCKS=>true` and BC jumps up in use. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2650: HBASE-25278 Add CACHE_BLOCKS option to count shell command
Apache-HBase commented on pull request #2650: URL: https://github.com/apache/hbase/pull/2650#issuecomment-726352267 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | master passed | | +1 :green_heart: | javadoc | 0m 19s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 33s | the patch passed | | +1 :green_heart: | javadoc | 0m 15s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 7m 16s | hbase-shell in the patch passed. | | | | 18m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2650/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2650 | | Optional Tests | javac javadoc unit | | uname | Linux 18fdb70be54f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2650/1/testReport/ | | Max. process+thread count | 2363 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2650/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2650: HBASE-25278 Add CACHE_BLOCKS option to count shell command
Apache-HBase commented on pull request #2650: URL: https://github.com/apache/hbase/pull/2650#issuecomment-726352537 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 45s | master passed | | +1 :green_heart: | javadoc | 0m 15s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 39s | the patch passed | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 6m 55s | hbase-shell in the patch passed. | | | | 19m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2650/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2650 | | Optional Tests | javac javadoc unit | | uname | Linux 68814102f26b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2650/1/testReport/ | | Max. process+thread count | 1560 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2650/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2650: HBASE-25278 Add CACHE_BLOCKS option to count shell command
Apache-HBase commented on pull request #2650: URL: https://github.com/apache/hbase/pull/2650#issuecomment-726344805 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | -0 :warning: | rubocop | 0m 9s | The patch generated 8 new + 130 unchanged - 0 fixed = 138 total (was 130) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 2m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2650/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2650 | | Optional Tests | dupname asflicense rubocop | | uname | Linux acfb7dadb296 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | rubocop | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2650/1/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 51 (vs. ulimit of 3) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2650/1/console | | versions | git=2.17.1 maven=3.6.3 rubocop=0.80.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25278) Add option to toggle CACHE_BLOCKS in count.rb
[ https://issues.apache.org/jira/browse/HBASE-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-25278: --- Status: Patch Available (was: Open) > Add option to toggle CACHE_BLOCKS in count.rb > - > > Key: HBASE-25278 > URL: https://issues.apache.org/jira/browse/HBASE-25278 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.4.0 > > > A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is > to edit table.rb so that the `count` shell command will not instruct > RegionServers to not cache any data blocks. This is a quick+dirty way to > force a table to be loaded into block cache (i.e. for performance testing). > We can easily add another option to avoid having to edit the ruby files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] joshelser opened a new pull request #2650: HBASE-25278 Add CACHE_BLOCKS option to count shell command
joshelser opened a new pull request #2650: URL: https://github.com/apache/hbase/pull/2650 Expose an argument on the `count` command which is passed to the `setCacheBlocks` method on the Scan which the count command uses. This is a quick and dirty approach to read all of the blocks for a table into the block cache. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25278) Add option to toggle CACHE_BLOCKS in count.rb
Josh Elser created HBASE-25278: -- Summary: Add option to toggle CACHE_BLOCKS in count.rb Key: HBASE-25278 URL: https://issues.apache.org/jira/browse/HBASE-25278 Project: HBase Issue Type: New Feature Components: shell Reporter: Josh Elser Assignee: Josh Elser Fix For: 3.0.0-alpha-1, 2.4.0 A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is to edit table.rb so that the `count` shell command will not instruct RegionServers to not cache any data blocks. This is a quick+dirty way to force a table to be loaded into block cache (i.e. for performance testing). We can easily add another option to avoid having to edit the ruby files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
Apache-HBase commented on pull request #2483: URL: https://github.com/apache/hbase/pull/2483#issuecomment-726297524 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 33s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 51s | master passed | | +1 :green_heart: | checkstyle | 1m 28s | master passed | | +1 :green_heart: | spotbugs | 2m 37s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | -0 :warning: | checkstyle | 0m 12s | hbase-hadoop-compat: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | -0 :warning: | checkstyle | 1m 13s | hbase-server: The patch generated 5 new + 281 unchanged - 0 fixed = 286 total (was 281) | | -0 :warning: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 18m 44s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 45m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2483 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 6f9713994403 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 035c192eb6 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/artifact/yetus-general-check/output/diff-checkstyle-hbase-hadoop-compat.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | whitespace | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2483/22/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
Pankaj Kumar created HBASE-25277: Summary: postScannerFilterRow impacts Scan performance a lot in HBase 2.x Key: HBASE-25277 URL: https://issues.apache.org/jira/browse/HBASE-25277 Project: HBase Issue Type: Bug Components: Coprocessors, scan Reporter: Pankaj Kumar Assignee: Pankaj Kumar In our test we observed Scan performance is degraded by more than 60% in HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% of the time in postScannerFilterRow, however the coprocessors (AccessController, VisibilityController & ConstraintProcessor) does nothing in that hook. HBASE-14489 added the logic to avoid the call to postScannerFilterRow when not needed which is not working as expected in HBase 2.x. AccessController, VisibilityController & ConstraintProcessor override the postScannerFilterRow with dummy (same as RegionObserver) implementation, so RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the hook for all configured CPs while processing each row . Suppose we have configured 5 region CPs and there are 1 M rows in table, then there will be 1 * 5 M dummy call to postScannerFilterRow during whole table scan. We need to remove postScannerFilterRow hook from these CPs as these are not doing anything. Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow init logic, currently it is always TRUE even though we remove postScannerFilterRow hook from AccessController, VisibilityController & ConstraintProcessor, because we are finding postScannerFilterRow until it is found (we look in configured CP's super class also) or clazz is NULL. https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 Supper class of Object (super class of AccessController) will be NULL, so RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] gkanade commented on a change in pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
gkanade commented on a change in pull request #2483:
URL: https://github.com/apache/hbase/pull/2483#discussion_r522354131
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
##
@@ -3481,6 +3512,9 @@ public ScanResponse scan(final RpcController controller,
final ScanRequest reque
}
throw new ServiceException(e);
}
+if(rsh.isFullRegionScan) {
+ rpcFullScanRequestCount.increment();
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] gkanade commented on a change in pull request #2483: HBASE-25026 - Create a metric to track scans that have no start row and/or stop row
gkanade commented on a change in pull request #2483:
URL: https://github.com/apache/hbase/pull/2483#discussion_r522353826
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
##
@@ -3164,7 +3187,14 @@ private RegionScannerHolder newRegionScanner(ScanRequest
request, ScanResponse.B
builder.setMvccReadPoint(scanner.getMvccReadPoint());
builder.setTtl(scannerLeaseTimeoutPeriod);
String scannerName = String.valueOf(scannerId);
-return addScanner(scannerName, scanner, shipper, region,
scan.isNeedCursorResult());
+
+boolean isFullRegionScan = false;
+if(!region.getRegionInfo().getTable().isSystemTable()) {
Review comment:
done, didnt get the last line of your comment
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
##
@@ -1336,14 +1342,31 @@ private RegionScannerHolder addScanner(String
scannerName, RegionScanner s, Ship
} else {
closeCallback = new RegionScannerCloseCallBack(s);
}
+
+
+
RegionScannerHolder rsh =
-new RegionScannerHolder(scannerName, s, r, closeCallback,
shippedCallback, needCursor);
+new RegionScannerHolder(scannerName, s, r, closeCallback,
shippedCallback, needCursor, isFullRegionScan);
Review comment:
fixed
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
##
@@ -1336,14 +1342,31 @@ private RegionScannerHolder addScanner(String
scannerName, RegionScanner s, Ship
} else {
closeCallback = new RegionScannerCloseCallBack(s);
}
+
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #2640: Backport: HBASE-25126 Add load balance logic in hbase-client to distribute read…
huaxiangsun commented on pull request #2640: URL: https://github.com/apache/hbase/pull/2640#issuecomment-726239551 HBASE-18070.branch-2 needs to be rebased to include Duo's HBASE-27252. I rebased my branch and now the review messes up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25266) [hbase-operator-tools] Add a repair tool for moving stale regions dir not present in meta away from table dir
[ https://issues.apache.org/jira/browse/HBASE-25266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230821#comment-17230821 ] Wellington Chevreuil commented on HBASE-25266: -- Ah yeah, could add an additional wal player step. On the case we were working here, the "orphaned" regions were mainly results of merges that got interrupted via "hbck2 bypass", so no recovered.edits were present on the regions dirs. I can modify current PR to include that as well. Also, I had originally thought this to be performing too much work than what hbck2 commands normally do (uses existing hbck2 commands, plus bulkload, plus move dirs around, plus wal player now), so put it as a standalone one under hbase-tools module. My view is that hbck2 commands should be more granular, but [~busbey] also had suggested this to be under hbck2, so if you guys still think this fits better as another hbck2 command, I'm fine moving it there. > [hbase-operator-tools] Add a repair tool for moving stale regions dir not > present in meta away from table dir > - > > Key: HBASE-25266 > URL: https://issues.apache.org/jira/browse/HBASE-25266 > Project: HBase > Issue Type: New Feature >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > > This adds a new tool under *hbase-tools* module, that allows for moving aways > regions dirs existing under table's hdfs dir, but not in meta. This is useful > in cases where the region is not present in meta, but still has data on hdfs, > yet no holes in the table region chain is detected. > On such cases, the existing *HBCK2 addFsRegionsMissingInMeta* command isn't > ideal, as it would bring the region back in meta and cause overlaps. > This tool performs the following actions: > 1) Identifies regions in hdfs but not in meta using > *HBCK2.reportTablesWithMissingRegionsInMeta*; > 2) For each of these regions, sidelines the related dir to a temp folder; > 3) Bulkload hfiles from each sidelined region to the related table; > Sidelined regions are never removed from temp folder. Operators should remove > those manually, after they certified on data integrity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25253) Deprecated master carrys regions related methods and configs
[ https://issues.apache.org/jira/browse/HBASE-25253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenfeiyi666 updated HBASE-25253: Release Note: Since 2.4.0, deprecated all master carrys regions related methods(LoadBalancer,BaseLoadBalancer,ZNodeClearer) and configs(hbase.balancer.tablesOnMaster, hbase.balancer.tablesOnMaster.systemTablesOnly), they will be removed in 3.0.0. > Deprecated master carrys regions related methods and configs > > > Key: HBASE-25253 > URL: https://issues.apache.org/jira/browse/HBASE-25253 > Project: HBase > Issue Type: Sub-task > Components: Balancer, master >Reporter: Duo Zhang >Assignee: wenfeiyi666 >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > And also mention that this feature will be removed in 3.0.0. > This is the first step to undo master carrys regions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2649: HBASE-25110 Add heartbeat for ReplicationServer and dispatch replicat…
Apache-HBase commented on pull request #2649: URL: https://github.com/apache/hbase/pull/2649#issuecomment-726165480 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 43s | HBASE-24666 passed | | +1 :green_heart: | compile | 2m 31s | HBASE-24666 passed | | +1 :green_heart: | shadedjars | 7m 20s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 10s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 32s | the patch passed | | +1 :green_heart: | compile | 2m 35s | the patch passed | | +1 :green_heart: | javac | 2m 35s | the patch passed | | +1 :green_heart: | shadedjars | 7m 20s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 9s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 3s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 0m 40s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 208m 12s | hbase-server in the patch passed. | | | | 245m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2649 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux af2c75c8bdec 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / 34e49bc8f6 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/3/testReport/ | | Max. process+thread count | 3292 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25275) Upgrade asciidoctor
[ https://issues.apache.org/jira/browse/HBASE-25275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230703#comment-17230703 ] Peter Somogyi commented on HBASE-25275: --- Website build succeeded, pushed to branch-2. > Upgrade asciidoctor > --- > > Key: HBASE-25275 > URL: https://issues.apache.org/jira/browse/HBASE-25275 > Project: HBase > Issue Type: Bug > Components: website >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Blocker > > JRuby upgrade broke the site build as asciidoc depends on JRuby. > https://ci-hadoop.apache.org/job/HBase/job/hbase_generate_website/89/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25275) Upgrade asciidoctor
[ https://issues.apache.org/jira/browse/HBASE-25275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-25275: -- Fix Version/s: 2.4.0 3.0.0-alpha-1 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master and branch-2. Thanks for the reviews [~zhangduo] and [~vjasani]. > Upgrade asciidoctor > --- > > Key: HBASE-25275 > URL: https://issues.apache.org/jira/browse/HBASE-25275 > Project: HBase > Issue Type: Bug > Components: website >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.4.0 > > > JRuby upgrade broke the site build as asciidoc depends on JRuby. > https://ci-hadoop.apache.org/job/HBase/job/hbase_generate_website/89/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-12961) Negative values in read and write region server metrics
[ https://issues.apache.org/jira/browse/HBASE-12961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230678#comment-17230678 ] Doris Hall commented on HBASE-12961: The lists all Apache HBase Jiras included in CDH 5 that are not included in the Apache HBase base version 0.95.2. The hbase-0.95.2-cdh5.0.0-beta-1.CHANGES.txt file lists all changes included in CDH 5. The patch for each change can be found in the cloudera/patches directory in the release tarball. Best regards, Doris Hall from https://domywriting.com/blog/teaching-in-coronavirus-outbreak/ > Negative values in read and write region server metrics > > > Key: HBASE-12961 > URL: https://issues.apache.org/jira/browse/HBASE-12961 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Victoria >Assignee: Victoria >Priority: Minor > Fix For: 1.1.0, 0.98.11, 2.0.0 > > Attachments: HBASE-12961-2.0.0-v1.patch, HBASE-12961-v1.patch > > > HMaster web page ui, shows the read/write request per region server. They are > currently displayed by using 32 bit integers. Hence, if the servers are up > for a long time the values can be shown as negative. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2649: HBASE-25110 Add heartbeat for ReplicationServer and dispatch replicat…
Apache-HBase commented on pull request #2649: URL: https://github.com/apache/hbase/pull/2649#issuecomment-726120384 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 24s | HBASE-24666 passed | | +1 :green_heart: | compile | 1m 59s | HBASE-24666 passed | | +1 :green_heart: | shadedjars | 6m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 5s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | compile | 2m 2s | the patch passed | | +1 :green_heart: | javac | 2m 2s | the patch passed | | +1 :green_heart: | shadedjars | 6m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 47s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 0m 25s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 142m 53s | hbase-server in the patch passed. | | | | 173m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2649 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux eeaf8ff54eaf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / 34e49bc8f6 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/3/testReport/ | | Max. process+thread count | 4690 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25263) Change encryption key generation algorithm used in the HBase shell
[
https://issues.apache.org/jira/browse/HBASE-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230669#comment-17230669
]
Mate Szalay-Beko commented on HBASE-25263:
--
These are very good points, thanks!
{quote}... each HFile has all of the necessary metadata for supporting its
decryption stored within it ...
{quote}
Yes, I didn't think it through deep enough before asking the question. You are
right and this also means that we can change the derivation function without
causing problems with backward compatibility. We can even bulkload HFiles /
move snapshots to other cluster and all the "old" hfiles will be readable by
the "new" cluster (assuming the master encryption key is the same / still
known).
{quote}If our older key derivation function has been deemed insufficient then
all of the derived DEKs should probably also be treated as insufficient. The
guidance we give there is compaction will rewrite the files with newly derived
DEKs. Major compaction will rewrite most files, but not all, so there is scope
for further improvement here... We may want an unconditional major compaction
to ensure all DEKs are rotated.
{quote}
This is a valid point. Also an unconditional major compaction could be useful
for other things too (like changing HFile compression maybe?). I can check the
code and create a follow-up ticket later.
BTW in the documentation
([http://hbase.apache.org/book.html#hbase.encryption.server)] we only recommend
major compaction, we don't mention that this won't necessary rewrite all the
HFiles (I might need to update this):
{quote}Rotate the Data Key
To rotate the data key, first change the ColumnFamily key in the column
descriptor, then trigger a major compaction. When compaction is complete, all
HFiles will be re-encrypted using the new data key. Until the compaction
completes, the old HFiles will still be readable using the old key.
{quote}
Also a note for me: I should also update the documentation mentioning the key
derivation function we use in the HBase shell (and also mention that the users
will have a better control over the encryption key if they provide it from Java
code and not from the shell)
> Change encryption key generation algorithm used in the HBase shell
> --
>
> Key: HBASE-25263
> URL: https://issues.apache.org/jira/browse/HBASE-25263
> Project: HBase
> Issue Type: Improvement
> Components: encryption, shell
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
>
> This ticket is a follow-up of HBASE-25181, where several issues were
> discussed on the PR:
> 1. Currently we use {{PBKDF2WithHmacSHA1}} key generation algorithm to
> generate a secret key for HFile / WalFile encryption, when the user is
> defining a string encryption key in the hbase shell. This algorithm is not
> secure enough and not allowed in certain environments (e.g. on FIPS compliant
> clusters). Our plan is to change it to e.g. {{PBKDF2WithHmacSHA384}}. If
> this would break backward compatibility, then we should make this algorithm
> configurable.
> Smaller issues:
> 2. In {{EncryptionUtil.createEncryptionContext}} the various encryption
> config checks should throw IllegalStateExceptions instead of
> RuntimeExceptions.
> 3. Test cases in {{TestEncryptionTest.java}} should be broken down into
> smaller tests.
> 4. {{TestEncryptionDisabled.java}} should use {{ExpectedException}} JUnit
> rule to validate exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25275) Upgrade asciidoctor
[ https://issues.apache.org/jira/browse/HBASE-25275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230668#comment-17230668 ] Peter Somogyi commented on HBASE-25275: --- Merged PR to master. Once the website build job succeeds I'll push to branch-2 also. > Upgrade asciidoctor > --- > > Key: HBASE-25275 > URL: https://issues.apache.org/jira/browse/HBASE-25275 > Project: HBase > Issue Type: Bug > Components: website >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Blocker > > JRuby upgrade broke the site build as asciidoc depends on JRuby. > https://ci-hadoop.apache.org/job/HBase/job/hbase_generate_website/89/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] petersomogyi merged pull request #2647: HBASE-25275 Upgrade asciidoctor
petersomogyi merged pull request #2647: URL: https://github.com/apache/hbase/pull/2647 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25253) Deprecated master carrys regions related methods and configs
[ https://issues.apache.org/jira/browse/HBASE-25253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25253. --- Hadoop Flags: Reviewed Resolution: Fixed Pushed to master and branch-2. Thanks [~wenfeiyi666] for contributing. Please fill the release note about which methods and configs are deprecated. > Deprecated master carrys regions related methods and configs > > > Key: HBASE-25253 > URL: https://issues.apache.org/jira/browse/HBASE-25253 > Project: HBase > Issue Type: Sub-task > Components: Balancer, master >Reporter: Duo Zhang >Assignee: wenfeiyi666 >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > And also mention that this feature will be removed in 3.0.0. > This is the first step to undo master carrys regions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #2635: HBASE-25253 Deprecated master carrys regions related methods and configs
Apache9 merged pull request #2635: URL: https://github.com/apache/hbase/pull/2635 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2644: HBASE-25127 Enhance PerformanceEvaluation to profile meta replica performance.
Apache9 commented on a change in pull request #2644:
URL: https://github.com/apache/hbase/pull/2644#discussion_r52219
##
File path:
hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
##
@@ -1998,6 +2006,41 @@ protected void testTakedown() throws IOException {
super.testTakedown();
}
}
+ static class MetaRandomReadTest extends TableTest {
+private Random rd = new Random();
+private RegionLocator regionLocator;
+
+MetaRandomReadTest(Connection con, TestOptions options, Status status) {
+ super(con, options, status);
+ LOG.info("call getRegionLocation");
+}
+
+@Override
+void onStartup() throws IOException {
+ this.table = connection.getTable(TableName.valueOf("hbase:meta"));
+ this.regionLocator =
connection.getRegionLocator(TableName.valueOf("hbase:meta"));
+}
+
+@Override
+boolean testRow(final int i, final long startTime) throws IOException,
InterruptedException {
+ if (opts.randomSleep > 0) {
+Thread.sleep(rd.nextInt(opts.randomSleep));
+ }
+ HRegionLocation hRegionLocation =
regionLocator.getRegionLocation(Bytes.toBytes(Integer.toString(rd.nextInt(100)
+ 1)), true);
Review comment:
> Right. I was thinking to add trace in the case of troubleshooting but
not sure.
Then lt's remove it? Otherwise there will be a checkstyle warning.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-25272) Support scan on a specific replica
[ https://issues.apache.org/jira/browse/HBASE-25272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25272. --- Hadoop Flags: Reviewed Resolution: Fixed Merged to branch-2. Thanks all for reviewing. > Support scan on a specific replica > -- > > Key: HBASE-25272 > URL: https://issues.apache.org/jira/browse/HBASE-25272 > Project: HBase > Issue Type: Improvement > Components: Client, scan >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.4.0 > > > This is a missing part of the client library for sync client on branch-2, and > it is necessary when implementing meta replicas read. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #2627: HBASE-25251 Enable configuration based enable/disable of Unsafe packa…
Apache9 commented on a change in pull request #2627: URL: https://github.com/apache/hbase/pull/2627#discussion_r522106617 ## File path: hbase-common/src/main/java/org/apache/hadoop/hbase/util/UnsafeAvailChecker.java ## @@ -33,12 +35,16 @@ private static final Logger LOG = LoggerFactory.getLogger(UnsafeAvailChecker.class); private static boolean avail = false; private static boolean unaligned = false; + // Flag that can be set by clients to enable/disable Unsafe package usage + private static boolean useUnsafe; Review comment: I do not think adding configs to hbase-site.xml or hbase-default.xml can help here, as if we want to make it final then we need to initialize these flags in the static initialization section, where we do not pass in a Configuration. So I suggest we use a environment variable to control the behavior, i.e, add `-Dorg.apache.hbase.unsafe=false` in the start command and we will test this env variable in the static initialization section to determine whether to disable unsafe. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #2645: HBASE-25272 Support scan on a specific replica
Apache9 merged pull request #2645: URL: https://github.com/apache/hbase/pull/2645 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] catalin-luca commented on a change in pull request #2002: HBASE-24541 Add support to run LoadIncrementalHFiles in a distributed manner
catalin-luca commented on a change in pull request #2002:
URL: https://github.com/apache/hbase/pull/2002#discussion_r522100337
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFilesJob.java
##
@@ -0,0 +1,348 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mapreduce;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HColumnDescriptor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionLocator;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.io.HFileLink;
+import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapreduce.*;
+import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.*;
+
+public class LoadIncrementalHFilesJob extends Configured implements Tool {
+
+ private static final Log LOG =
LogFactory.getLog(LoadIncrementalHFilesJob.class);
+
+ public static final String ROOT_DIR = "loadincrementalhfilesjob.root.dir";
+ public static final String TABLE_NAME =
"loadincrementalhfilesjob.table.name";
+ public static final String MAX_MAP_TASK =
"loadincrementalhfilesjob.max.map.tasks";
+ public static final String DEPTH = "loadincrementalhfilesjob.depth";
+
+ @Override public int run(String[] args) throws Exception {
+Configuration configuration = HBaseConfiguration.create(getConf());
+
+return run(args[0], args[1], configuration);
+ }
+
+ @VisibleForTesting public int run(String rootPath, String table,
Configuration configuration)
+ throws Exception {
+Path basePath = new Path(rootPath);
+configuration.set(TABLE_NAME, table);
+configuration.set(ROOT_DIR, basePath.toString());
+
+if (!bulkload(basePath, table, configuration)) {
+ return 1;
+}
+
+return 0;
+ }
+
+ private boolean bulkload(Path rootDir, String table, Configuration
configuration)
+ throws Exception {
+Configuration jobConfiguration = new Configuration(configuration);
+
+applyNonOverridableOptions(jobConfiguration);
+
+Job job = Job.getInstance(jobConfiguration, "Bulkload-" + table + "-" +
rootDir.toString());
+
+TableMapReduceUtil.addDependencyJars(job);
+
+job.setJarByClass(LoadIncrementalHFilesJob.class);
+
+job.setInputFormatClass(BulkLoadInputFormat.class);
+job.setOutputFormatClass(NullOutputFormat.class);
+
+job.setMapperClass(BulkoadMapper.class);
+job.setMapOutputKeyClass(NullWritable.class);
+job.setMapOutputValueClass(NullWritable.class);
+
+job.setNumReduceTasks(0);
+
+return job.waitForCompletion(true);
+ }
+
+ private static void applyNonOverridableOptions(Configuration configuration) {
Review comment:
> IIRC, it would move to a temp dir first, if the load fails, file will
be left there in the pending dir, no? In that case, operators could react upon
what is left there in the temp dir.
Afaik, the temp directory is only used for the secure bulkload feature.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about
[GitHub] [hbase] catalin-luca commented on pull request #2002: HBASE-24541 Add support to run LoadIncrementalHFiles in a distributed manner
catalin-luca commented on pull request #2002: URL: https://github.com/apache/hbase/pull/2002#issuecomment-726070015 > > > > After inspecting thread dumps, I saw lots of time spent in re-creating HTTP connections. It seemed that the connections were not being re-used because the `HFile.Reader` was not reading all the bytes after seeking to read the trailer. In turn this causes the connection to get aborted and it can't be pooled. > > > > > > > > > So the issue is bound to JVM instance, since the better performance with multiple tasks only, but not with multiple threads on same JVM instance? > > > > > > Yup. > > Hum, It seems there's room for improvement in the reader code. Also, have you tried to increase fs.s3a.connection.maximum and fs.s3a.threads.max properties in hbase config? We are actually running over EMRFS in amazon. Increasing the equivalent max connections property did not yield any improvements. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25187) Improve SizeCachedKV variants initialization
[
https://issues.apache.org/jira/browse/HBASE-25187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230601#comment-17230601
]
Hudson commented on HBASE-25187:
Results for branch master
[build #123 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/123/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/123/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/123/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/123/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Improve SizeCachedKV variants initialization
>
>
> Key: HBASE-25187
> URL: https://issues.apache.org/jira/browse/HBASE-25187
> Project: HBase
> Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.4
>
>
> Currently in SizeCachedKV we get the rowlength and Key length from the
> buffers. This can be optimized because we can pass the keylen and row len
> while actually creating the cell while reading the cell from the block. Some
> times we see that the SizeCachedKV takes the max width in a flame graph -
> considering the fact we also do a sanity check on the created KV.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2649: HBASE-25110 Add heartbeat for ReplicationServer and dispatch replicat…
Apache-HBase commented on pull request #2649: URL: https://github.com/apache/hbase/pull/2649#issuecomment-726055966 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 49s | HBASE-24666 passed | | +1 :green_heart: | checkstyle | 1m 32s | HBASE-24666 passed | | +1 :green_heart: | spotbugs | 6m 31s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | -0 :warning: | checkstyle | 1m 11s | hbase-server: The patch generated 1 new + 23 unchanged - 0 fixed = 24 total (was 23) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 37s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 2m 3s | the patch passed | | +1 :green_heart: | spotbugs | 7m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 56m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2649 | | Optional Tests | dupname asflicense cc hbaseprotoc prototool spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux eed4134252c5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / 34e49bc8f6 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2002: HBASE-24541 Add support to run LoadIncrementalHFiles in a distributed manner
Apache-HBase commented on pull request #2002: URL: https://github.com/apache/hbase/pull/2002#issuecomment-726053711 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 10m 45s | Docker failed to build yetus/hbase:7ff737df0f. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/2002 | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2002/5/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25274) Regionserver crash by YouAreDeadException
[
https://issues.apache.org/jira/browse/HBASE-25274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hong updated HBASE-25274:
-
Priority: Blocker (was: Major)
> Regionserver crash by YouAreDeadException
> -
>
> Key: HBASE-25274
> URL: https://issues.apache.org/jira/browse/HBASE-25274
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 2.1.0
>Reporter: hong
>Priority: Blocker
>
> Hbase cluster is enabled kerberos;
> This problem occurs during cluster running, not during startup. So the
> Kerberos configuration should be fine
> hmaster log:
> {code:java}
> 2020-10-31 08:27:02,594 INFO
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch
> pid=32630, ppid=32629, state=RUNNABLE:REGION_TRANSITION_DISPATCH,
> locked=true; UnassignProcedure table=dm_sn_sa:t_agent_relation_info,
> region=66e7cf0d6ad81f90b5aeca2c74f26af6,
> server=cc1,16020,15988329396742020-10-31 08:27:02,594 INFO
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch
> pid=32630, ppid=32629, state=RUNNABLE:REGION_TRANSITION_DISPATCH,
> locked=true; UnassignProcedure table=dm_sn_sa:t_agent_relation_info,
> region=66e7cf0d6ad81f90b5aeca2c74f26af6,
> server=cc1,16020,15988329396742020-10-31 08:27:02,712 WARN
> org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher: Failed
> dispatch to server=cc1,16020,1598832939674
> try=0javax.security.sasl.SaslException: Call to cc1/192.168.68.94:16020
> failed on local exception: javax.security.sasl.SaslException: GSS initiate
> failed [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)] [Caused by
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]] at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:221) at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at
> org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at
> org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:326)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:312)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:304)
> at
> org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:326)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:312)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:304)
> at
> org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:366)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:326)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:312)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:304)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1426)
> at
>
[GitHub] [hbase] Apache-HBase commented on pull request #2645: HBASE-25272 Support scan on a specific replica
Apache-HBase commented on pull request #2645: URL: https://github.com/apache/hbase/pull/2645#issuecomment-726042334 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 15s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 7s | branch-2 passed | | +1 :green_heart: | compile | 1m 24s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 1m 24s | the patch passed | | +1 :green_heart: | javac | 1m 24s | the patch passed | | +1 :green_heart: | shadedjars | 6m 28s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 23s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 207m 37s | hbase-server in the patch passed. | | | | 241m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2645/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2645 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 677bf49a8734 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ec3f843b23 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2645/3/testReport/ | | Max. process+thread count | 2550 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2645/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25267) Add SSL keystore type and truststore related configs for HBase RESTServer
[
https://issues.apache.org/jira/browse/HBASE-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balazs Meszaros resolved HBASE-25267.
-
Fix Version/s: 2.3.4
2.2.7
2.4.0
3.0.0-alpha-1
Resolution: Fixed
> Add SSL keystore type and truststore related configs for HBase RESTServer
> -
>
> Key: HBASE-25267
> URL: https://issues.apache.org/jira/browse/HBASE-25267
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Mate Szalay-Beko
>Assignee: Mate Szalay-Beko
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.4
>
>
> The RESTServer currently relies on the following parameters to configure SSL
> on the REST API:
> * {{hbase.rest.ssl.enabled}}
> * {{hbase.rest.ssl.keystore.store}}
> * {{hbase.rest.ssl.keystore.password}}
> * {{hbase.rest.ssl.keystore.keypassword}}
> * {{hbase.rest.ssl.exclude.cipher.suites}}
> * {{hbase.rest.ssl.include.cipher.suites}}
> * {{hbase.rest.ssl.exclude.protocols}}
> * {{hbase.rest.ssl.include.protocols}}
> In this patch I want to introduce the following new parameters:
> * {{hbase.rest.ssl.keystore.type}}
> * {{hbase.rest.ssl.truststore.store}}
> * {{hbase.rest.ssl.truststore.password}}
> * {{hbase.rest.ssl.truststore.type}}
> If any of the new the parameter is not provided, then we should fall-back to
> the current behaviour (e.g. assuming JKS keystore/truststore types, or no
> passwords, or no custom trust store file).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2649: HBASE-25110 Add heartbeat for ReplicationServer and dispatch replicat…
Apache-HBase commented on pull request #2649: URL: https://github.com/apache/hbase/pull/2649#issuecomment-726028766 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 25s | HBASE-24666 passed | | +1 :green_heart: | compile | 1m 59s | HBASE-24666 passed | | +1 :green_heart: | shadedjars | 6m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 2m 2s | the patch passed | | +1 :green_heart: | javac | 2m 2s | the patch passed | | +1 :green_heart: | shadedjars | 6m 35s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 45s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 0m 26s | hbase-replication in the patch passed. | | -1 :x: | unit | 142m 25s | hbase-server in the patch failed. | | | | 173m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2649 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux fa47b35236f1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / 34e49bc8f6 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/2/testReport/ | | Max. process+thread count | 4376 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2649: HBASE-25110 Add heartbeat for ReplicationServer and dispatch replicat…
Apache-HBase commented on pull request #2649: URL: https://github.com/apache/hbase/pull/2649#issuecomment-726026216 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24666 Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 2s | HBASE-24666 passed | | +1 :green_heart: | compile | 2m 25s | HBASE-24666 passed | | +1 :green_heart: | shadedjars | 6m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 12s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 2m 23s | the patch passed | | +1 :green_heart: | javac | 2m 23s | the patch passed | | +1 :green_heart: | shadedjars | 6m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 59s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 0m 27s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 134m 15s | hbase-server in the patch passed. | | | | 168m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2649 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux 4d22540b9711 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / 34e49bc8f6 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/2/testReport/ | | Max. process+thread count | 4045 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-replication hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2649/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] meszibalu merged pull request #2642: HBASE-25267 Add SSL keystore type and truststore related configs for HBase RESTServer
meszibalu merged pull request #2642: URL: https://github.com/apache/hbase/pull/2642 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
