[jira] [Commented] (HBASE-25891) Remove dependence storing WAL filenames for backup
[
https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382994#comment-17382994
]
Mallikarjun commented on HBASE-25891:
-
This will be a much needed enhancement in terms of being able to extend this
functionality to rsgroups and things to come for hbase backup restore. Esp
before hbase 3.0 goes live, as this changes information stored in meta. Would
like someone to spend time in reviewing this. [~anoop.hbase] [~zhangduo]
> Remove dependence storing WAL filenames for backup
> --
>
> Key: HBASE-25891
> URL: https://issues.apache.org/jira/browse/HBASE-25891
> Project: HBase
> Issue Type: Improvement
> Components: backuprestore
>Affects Versions: 3.0.0-alpha-1
>Reporter: Mallikarjun
>Assignee: Mallikarjun
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> Context:
> Currently WAL logs are stored in `backup:system` meta table
> {code:java}
> // code placeholder
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> {code}
> Also, Every backup (Incremental and Full) performs a log roll just before
> taking backup and stores what was the timestamp at which log roll was
> performed per regionserver per backup using following format.
>
> {code:java}
> // code placeholder
> rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-2:16020
> column=meta:rs-log-ts, timestamp=1622887363301,value=\x00\x00\x01y\xDB\x81ar
> rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-3:16020
> column=meta:rs-log-ts, timestamp=1622887363294, value=\x00\x00\x01y\xDB\x81aP
> rslogts:hdfs://xx.xx.xx.xx:8020/tmp/backup_yaktest\x00preprod-dn-1:16020
> column=meta:rs-log-ts, timestamp=1622887363275,
> value=\x00\x00\x01y\xDB\x81\x85
> {code}
>
>
> There are 2 cases for which WAL log refrences stored in `backup:system` and
> are being used.
> 1. To cleanup WAL's for which backup is already taken using
> `BackupLogCleaner`
> Since log roll timestamp is stored as part of backup per regionserver. We can
> check all previous successfull backup's and then identify which logs are to
> be retained and which ones are to be cleaned up as follows
> * Identify which are the latest successful backups performed per table.
> * Per backup identified above, identify what is the oldest log rolled
> timestamp perfomed per regionserver per table.
> * All those WAL's which are older than oldest log rolled timestamp perfomed
> for any table backed can be removed by `BackupLogCleaner`
>
> 2. During incremental backup, to check system table if there are any
> duplicate WAL's for which backup is taken again.
> * Incremental backup already identifies which all WAL's to be backed up
> using `rslogts:` mentioned above.
> * Additionally it checks `wals:` to ensure no logs are backuped for second
> time. And this is redundant and not seen any extra benefit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] rda3mon commented on pull request #3359: HBASE-25891 remove dependence storing wal filenames for backup
rda3mon commented on pull request #3359: URL: https://github.com/apache/hbase/pull/3359#issuecomment-882229011 @Apache9 This is a much needed enhancement before going with 3.0 release. Can you kindly spare some time for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean edited a comment on pull request #3498: HBASE-26094 L2 BC should not be the victimhandler of L1 BC when using combined BC
YutSean edited a comment on pull request #3498: URL: https://github.com/apache/hbase/pull/3498#issuecomment-882217750 The change passed UTs locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on pull request #3498: HBASE-26094 L2 BC should not be the victimhandler of L1 BC when using combined BC
YutSean commented on pull request #3498: URL: https://github.com/apache/hbase/pull/3498#issuecomment-882217750 The change passed TUs locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26096) Cleanup the deprecated methods in HBTU related classes
[ https://issues.apache.org/jira/browse/HBASE-26096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26096: -- Component/s: test > Cleanup the deprecated methods in HBTU related classes > -- > > Key: HBASE-26096 > URL: https://issues.apache.org/jira/browse/HBASE-26096 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Priority: Major > > As it is now IA.LimitedPrivate, we should clean it up before the final 3.0.0 > release, to make the interface cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] joshelser commented on a change in pull request #3389: HBASE-25392 Direct insert compacted HFiles into data directory.
joshelser commented on a change in pull request #3389:
URL: https://github.com/apache/hbase/pull/3389#discussion_r671950730
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DirectStoreCompactor.java
##
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.regionserver.compactions;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.regionserver.HStore;
+import org.apache.hadoop.hbase.regionserver.HStoreFile;
+import org.apache.hadoop.hbase.regionserver.StoreFileWriter;
+import org.apache.yetus.audience.InterfaceAudience;
+
+/**
+ * Alternative Compactor implementation, this class extends
DefaultCompactor class,
Review comment:
nit: First sentence should tell us what this class does "Places the
compacted file directly into the store without the need for a rename" (and then
tell us implementation details).
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestDirectStoreCompactor.java
##
@@ -0,0 +1,135 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver.compactions;
+
+import static junit.framework.TestCase.assertEquals;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyBoolean;
+import static org.mockito.ArgumentMatchers.isNull;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.io.ByteBuffAllocator;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.hadoop.hbase.io.crypto.Encryption;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.regionserver.BloomType;
+import org.apache.hadoop.hbase.regionserver.HRegion;
+import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
+import org.apache.hadoop.hbase.regionserver.HStore;
+import org.apache.hadoop.hbase.regionserver.HStoreFile;
+import org.apache.hadoop.hbase.regionserver.StoreContext;
+import org.apache.hadoop.hbase.regionserver.StoreFileWriter;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.ChecksumType;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+import org.mockito.ArgumentCaptor;
+
+/**
+ * Test class for DirectStoreCompactor.
+ */
+@Category({
[jira] [Updated] (HBASE-26094) In branch-1 L2 BC should not be the victimhandler of L1 BC when using combined BC
[
https://issues.apache.org/jira/browse/HBASE-26094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yutong Xiao updated HBASE-26094:
Description:
Currently in branch-1, the block cache initialisation is as this code:
{code:java}
LruBlockCache l1 = getL1(conf);
// blockCacheDisabled is set as a side-effect of getL1Internal(), so check
it again after the call.
if (blockCacheDisabled) return null;
BlockCache l2 = getL2(conf);
if (l2 == null) {
GLOBAL_BLOCK_CACHE_INSTANCE = l1;
} else {
boolean useExternal = conf.getBoolean(EXTERNAL_BLOCKCACHE_KEY,
EXTERNAL_BLOCKCACHE_DEFAULT);
boolean combinedWithLru = conf.getBoolean(BUCKET_CACHE_COMBINED_KEY,
DEFAULT_BUCKET_CACHE_COMBINED);
if (useExternal) {
GLOBAL_BLOCK_CACHE_INSTANCE = new InclusiveCombinedBlockCache(l1, l2);
} else {
if (combinedWithLru) {
GLOBAL_BLOCK_CACHE_INSTANCE = new CombinedBlockCache(l1, l2);
} else {
// L1 and L2 are not 'combined'. They are connected via the
LruBlockCache victimhandler
// mechanism. It is a little ugly but works according to the
following: when the
// background eviction thread runs, blocks evicted from L1 will go to
L2 AND when we get
// a block from the L1 cache, if not in L1, we will search L2.
GLOBAL_BLOCK_CACHE_INSTANCE = l1;
}
}
l1.setVictimCache(l2);
}
{code}
As the code above, L2 will always be the victimhandler of L1, no matter if we
use combined blockcache or not. But as logic (in master & branch-2) L2 should
not be the victimhandler of L1 when using combined BC. We should set the
victimhandler only when we use InclusiveConbinedBC or we do not use CombinedBC.
was:As the logic, the L2 should not be the victimhandler of L1 cache. When
using InclusiveCombinedBlockCache, we should set L2 as the victimhandler of L1.
> In branch-1 L2 BC should not be the victimhandler of L1 BC when using
> combined BC
> -
>
> Key: HBASE-26094
> URL: https://issues.apache.org/jira/browse/HBASE-26094
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
>Affects Versions: 1.7.0
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Major
>
> Currently in branch-1, the block cache initialisation is as this code:
> {code:java}
> LruBlockCache l1 = getL1(conf);
> // blockCacheDisabled is set as a side-effect of getL1Internal(), so
> check it again after the call.
> if (blockCacheDisabled) return null;
> BlockCache l2 = getL2(conf);
> if (l2 == null) {
> GLOBAL_BLOCK_CACHE_INSTANCE = l1;
> } else {
> boolean useExternal = conf.getBoolean(EXTERNAL_BLOCKCACHE_KEY,
> EXTERNAL_BLOCKCACHE_DEFAULT);
> boolean combinedWithLru = conf.getBoolean(BUCKET_CACHE_COMBINED_KEY,
> DEFAULT_BUCKET_CACHE_COMBINED);
> if (useExternal) {
> GLOBAL_BLOCK_CACHE_INSTANCE = new InclusiveCombinedBlockCache(l1, l2);
> } else {
> if (combinedWithLru) {
> GLOBAL_BLOCK_CACHE_INSTANCE = new CombinedBlockCache(l1, l2);
> } else {
> // L1 and L2 are not 'combined'. They are connected via the
> LruBlockCache victimhandler
> // mechanism. It is a little ugly but works according to the
> following: when the
> // background eviction thread runs, blocks evicted from L1 will go
> to L2 AND when we get
> // a block from the L1 cache, if not in L1, we will search L2.
> GLOBAL_BLOCK_CACHE_INSTANCE = l1;
> }
> }
> l1.setVictimCache(l2);
> }
> {code}
> As the code above, L2 will always be the victimhandler of L1, no matter if we
> use combined blockcache or not. But as logic (in master & branch-2) L2 should
> not be the victimhandler of L1 when using combined BC. We should set the
> victimhandler only when we use InclusiveConbinedBC or we do not use
> CombinedBC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-26094) In branch-1 L2 BC should not be the victimhandler of L1 BC when using combined BC
[
https://issues.apache.org/jira/browse/HBASE-26094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yutong Xiao updated HBASE-26094:
Description:
Currently in branch-1, the block cache initialisation is:
{code:java}
LruBlockCache l1 = getL1(conf);
// blockCacheDisabled is set as a side-effect of getL1Internal(), so check
it again after the call.
if (blockCacheDisabled) return null;
BlockCache l2 = getL2(conf);
if (l2 == null) {
GLOBAL_BLOCK_CACHE_INSTANCE = l1;
} else {
boolean useExternal = conf.getBoolean(EXTERNAL_BLOCKCACHE_KEY,
EXTERNAL_BLOCKCACHE_DEFAULT);
boolean combinedWithLru = conf.getBoolean(BUCKET_CACHE_COMBINED_KEY,
DEFAULT_BUCKET_CACHE_COMBINED);
if (useExternal) {
GLOBAL_BLOCK_CACHE_INSTANCE = new InclusiveCombinedBlockCache(l1, l2);
} else {
if (combinedWithLru) {
GLOBAL_BLOCK_CACHE_INSTANCE = new CombinedBlockCache(l1, l2);
} else {
// L1 and L2 are not 'combined'. They are connected via the
LruBlockCache victimhandler
// mechanism. It is a little ugly but works according to the
following: when the
// background eviction thread runs, blocks evicted from L1 will go to
L2 AND when we get
// a block from the L1 cache, if not in L1, we will search L2.
GLOBAL_BLOCK_CACHE_INSTANCE = l1;
}
}
l1.setVictimCache(l2);
}
{code}
As the code above, L2 will always be the victimhandler of L1, no matter if we
use combined blockcache or not. But as logic (in master & branch-2) L2 should
not be the victimhandler of L1 when using combined BC. We should set the
victimhandler only when we use InclusiveConbinedBC or we do not use CombinedBC.
was:
Currently in branch-1, the block cache initialisation is as this code:
{code:java}
LruBlockCache l1 = getL1(conf);
// blockCacheDisabled is set as a side-effect of getL1Internal(), so check
it again after the call.
if (blockCacheDisabled) return null;
BlockCache l2 = getL2(conf);
if (l2 == null) {
GLOBAL_BLOCK_CACHE_INSTANCE = l1;
} else {
boolean useExternal = conf.getBoolean(EXTERNAL_BLOCKCACHE_KEY,
EXTERNAL_BLOCKCACHE_DEFAULT);
boolean combinedWithLru = conf.getBoolean(BUCKET_CACHE_COMBINED_KEY,
DEFAULT_BUCKET_CACHE_COMBINED);
if (useExternal) {
GLOBAL_BLOCK_CACHE_INSTANCE = new InclusiveCombinedBlockCache(l1, l2);
} else {
if (combinedWithLru) {
GLOBAL_BLOCK_CACHE_INSTANCE = new CombinedBlockCache(l1, l2);
} else {
// L1 and L2 are not 'combined'. They are connected via the
LruBlockCache victimhandler
// mechanism. It is a little ugly but works according to the
following: when the
// background eviction thread runs, blocks evicted from L1 will go to
L2 AND when we get
// a block from the L1 cache, if not in L1, we will search L2.
GLOBAL_BLOCK_CACHE_INSTANCE = l1;
}
}
l1.setVictimCache(l2);
}
{code}
As the code above, L2 will always be the victimhandler of L1, no matter if we
use combined blockcache or not. But as logic (in master & branch-2) L2 should
not be the victimhandler of L1 when using combined BC. We should set the
victimhandler only when we use InclusiveConbinedBC or we do not use CombinedBC.
> In branch-1 L2 BC should not be the victimhandler of L1 BC when using
> combined BC
> -
>
> Key: HBASE-26094
> URL: https://issues.apache.org/jira/browse/HBASE-26094
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
>Affects Versions: 1.7.0
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Major
>
> Currently in branch-1, the block cache initialisation is:
> {code:java}
> LruBlockCache l1 = getL1(conf);
> // blockCacheDisabled is set as a side-effect of getL1Internal(), so
> check it again after the call.
> if (blockCacheDisabled) return null;
> BlockCache l2 = getL2(conf);
> if (l2 == null) {
> GLOBAL_BLOCK_CACHE_INSTANCE = l1;
> } else {
> boolean useExternal = conf.getBoolean(EXTERNAL_BLOCKCACHE_KEY,
> EXTERNAL_BLOCKCACHE_DEFAULT);
> boolean combinedWithLru = conf.getBoolean(BUCKET_CACHE_COMBINED_KEY,
> DEFAULT_BUCKET_CACHE_COMBINED);
> if (useExternal) {
> GLOBAL_BLOCK_CACHE_INSTANCE = new InclusiveCombinedBlockCache(l1, l2);
> } else {
> if (combinedWithLru) {
> GLOBAL_BLOCK_CACHE_INSTANCE = new CombinedBlockCache(l1, l2);
> } else {
> // L1 and L2 are not 'combined'. They are connected via the
> LruBlockCache victimhandler
> // mechanism. It is a little ugly but works according to the
> following: when the
> //
[GitHub] [hbase] ZhaoBQ closed pull request #3451: HBASE-26056 Cell TTL set to -1 should mean never expire, the same as CF TTL
ZhaoBQ closed pull request #3451: URL: https://github.com/apache/hbase/pull/3451 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3437: HBASE-26037 Implement namespace and table level access control for thrift & thrift2
Apache9 commented on pull request #3437: URL: https://github.com/apache/hbase/pull/3437#issuecomment-882174874 Thank you for your patience! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on pull request #3437: HBASE-26037 Implement namespace and table level access control for thrift & thrift2
YutSean commented on pull request #3437: URL: https://github.com/apache/hbase/pull/3437#issuecomment-882174095 OK. I will reconstruct the code then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3437: HBASE-26037 Implement namespace and table level access control for thrift & thrift2
Apache9 commented on pull request #3437: URL: https://github.com/apache/hbase/pull/3437#issuecomment-882169159 I think we'd better align with the java client? It we think one method is better than two methods, then we should also change the java client. Otherwise I prefer we still keep the two separated methods. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on pull request #3437: HBASE-26037 Implement namespace and table level access control for thrift & thrift2
YutSean commented on pull request #3437: URL: https://github.com/apache/hbase/pull/3437#issuecomment-882168032 What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26096) Cleanup the deprecated methods in HBTU related classes
Duo Zhang created HBASE-26096: - Summary: Cleanup the deprecated methods in HBTU related classes Key: HBASE-26096 URL: https://issues.apache.org/jira/browse/HBASE-26096 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang As it is now IA.LimitedPrivate, we should clean it up before the final 3.0.0 release, to make the interface cleaner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26095) Modify our ref guide to mention the deprecated of HBTU and also how to make use of the new TestingHBaseCluster
Duo Zhang created HBASE-26095: - Summary: Modify our ref guide to mention the deprecated of HBTU and also how to make use of the new TestingHBaseCluster Key: HBASE-26095 URL: https://issues.apache.org/jira/browse/HBASE-26095 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26081) Copy HBTU to hbase-testing-util, rename the HBTU related classes in hbase-server and mark them as IA.LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-26081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26081. --- Resolution: Fixed Merged to master. Thanks [~stack] for reviewing. > Copy HBTU to hbase-testing-util, rename the HBTU related classes in > hbase-server and mark them as IA.LimitedPrivate > --- > > Key: HBASE-26081 > URL: https://issues.apache.org/jira/browse/HBASE-26081 > Project: HBase > Issue Type: Sub-task > Components: API, test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-2 > > > See the discussions in the parent issue on why we want to do this. > This is the final decision: > https://issues.apache.org/jira/browse/HBASE-13126?focusedCommentId=17358108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17358108 > Maybe a possible way is that, we mark HBTU as deprecated and introduce > another class for end users to use. And on the next major release, we just > introduce a another class in hbase-server for our internal usage, and copy > the whole HBTU related classes to hbase-testing-util module, and keep them > there for a whole major release, and then we remove them when the next major > release is out. So end users will have plenty of time to change their code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #3478: HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related …
Apache9 merged pull request #3478: URL: https://github.com/apache/hbase/pull/3478 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25973) Balancer should explain progress in a better way in log
[
https://issues.apache.org/jira/browse/HBASE-25973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382949#comment-17382949
]
Michael Stack commented on HBASE-25973:
---
[~apurt...@yahoo.com] merged the branch-2 PR.
> Balancer should explain progress in a better way in log
> ---
>
> Key: HBASE-25973
> URL: https://issues.apache.org/jira/browse/HBASE-25973
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 3.0.0-alpha-1
>Reporter: Clara Xiong
>Assignee: Clara Xiong
>Priority: Major
> Fix For: 2.5.0, 2.3.6, 3.0.0-alpha-2, 2.4.5
>
>
> In the log, balancer logs at info level at the beginning of run:
> {code}
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=277.3479243125063, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.3749771215224234); ServerLocalityCostFunction : (25.0,
> 0.5807483226644186); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (1000.0, 0.0019704142954972883);
> StoreFileCostFunction : (200.0, 0.3668512059459341); computedMaxSteps:
> 42270438200
> {code}
> the cost is reported without context, it is hard for operator to understand
> how unbalanced the cluster is for balancer and how much progress we are
> making.
> For a large cluster, the calculation can take a long time, we also need to
> let operator understand that it will take up to the max time to complete the
> calculation.
> At the end of computation:
> {code}
> balancer.StochasticLoadBalancer: Finished computing new load balance plan.
> Computation took PT40M0.006S to try 1036409 different iterations. Found a
> solution that moves 161926 regions; Going from a computed cost of
> 118.75715593924485 to a new cost of 1.5509126920967042
> {code}
> The time to compute the plan is also printed in a format that is not human
> readable. we also need to let operator understand that balancer is just
> submitting the plan and it be up to execution to complete the move.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26088) conn.getBufferedMutator(tableName) leaks thread executors and other problems
[
https://issues.apache.org/jira/browse/HBASE-26088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382946#comment-17382946
]
Rushabh Shah commented on HBASE-26088:
--
[~apurtell] If [~whitney13] is unable to provide a patch by Monday, I will put
up a PR. Thank you !
> conn.getBufferedMutator(tableName) leaks thread executors and other problems
>
>
> Key: HBASE-26088
> URL: https://issues.apache.org/jira/browse/HBASE-26088
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.4.13, 2.4.4
>Reporter: Whitney Jackson
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.5
>
>
> TL;DR: {{conn.getBufferedMutator(tableName)}} is dangerous in hbase client
> 2.4.4 and doesn't match documented behavior in 1.4.13.
> To work around the problems until fixed do this:
> {code:java}
> var mySingletonPool = HTable.getDefaultExecutor(hbaseConf);
> var params = new BufferedMutatorParams(tableName);
> params.pool(mySingletonPool);
> var myMutator = conn.getBufferedMutator(params);
> {code}
> And avoid code like this:
> {code:java}
> var myMutator = conn.getBufferedMutator(tableName);
> {code}
> The full story:
> My application started leaking threads after upgrading from hbase client
> 1.4.13 to 2.4.4. So much so that after less than a minute of runtime more
> that 30k threads are leaked and all available virtual memory on the box (> 50
> GB) is consumed. Other processes on the box start crashing with memory
> allocation errors. Even running {{ls}} at the shell fails with OS resource
> allocation failures.
> A thread dump after just a few seconds of runtime shows thousands of threads
> like this:
> {code:java}
> "htable-pool-0" #8841 prio=5 os_prio=0 cpu=0.15ms elapsed=7.49s
> tid=0x7efb6d2a1000 nid=0x57d2 waiting on condition [0x7ef8a6c38000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11.0.6/Native Method)
> - parking to wait for <0x0007e7cd6188> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.6/LockSupport.java:234)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@11.0.6/SynchronousQueue.java:462)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@11.0.6/SynchronousQueue.java:361)
> at
> java.util.concurrent.SynchronousQueue.poll(java.base@11.0.6/SynchronousQueue.java:937)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.6/ThreadPoolExecutor.java:1053)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.6/ThreadPoolExecutor.java:1114)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.6/ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(java.base@11.0.6/Thread.java:834)
> {code}
>
> Note: All the threads are labeled {{htable-pool-0}}. That suggests we're
> leaking thread executors not just threads. The {{htable-pool}} part indicates
> the problem is to do with {{HTable.getDefaultExecutor(conf)}} and the only
> part of my code that interacts with that is a call to
> {{conn.getBufferedMutator(tableName)}}.
>
> Looking at the hbase client code shows a few problems:
> 1) Neither 1.4.13 nor 2.4.4's behavior matches the documentation for
> {{conn.getBufferedMutator(tableName)}} which says:
> {quote}This BufferedMutator will use the Connection's ExecutorService.
> {quote}
> That suggests some singleton thread executor is being used which is not the
> case.
>
> 2) Under 1.4.13 you get a new {{ThreadPoolExecutor}} for every
> {{BufferedMutator}}. That's probably not what you want but you likely won't
> notice. I didn't. It's a code path I hadn't profiled much.
>
> 3) Under 2.4.4 you get a new {{ThreadPoolExecutor}} for every
> {{BufferedMutator}} *and* that {{ThreadPoolExecutor}} *is not* cleaned up
> after the {{Mutator}} is closed. Each completed {{ThreadPoolExecutor}}
> carries with it one thread which hangs around until a timeout value which
> defaults to 60 seconds.
> My application creates one {{BufferedMutator}} for every incoming stream and
> there are lots of streams, some of them are short lived so my code leaks
> threads fast under 2.4.4.
> Here's the part where a new executor is created for every {{BufferedMutator}}
> (it's similar for 1.4.13):
> [https://github.com/apache/hbase/blob/branch-2.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L420]
>
> The reason for the leak in 2.4.4 is the should-we/shouldn't-we cleanup logic
> added here:
> [https://github.com/apache/hbase/blob/branch-2.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L104]
> That
[GitHub] [hbase] Apache-HBase commented on pull request #3478: HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related …
Apache-HBase commented on pull request #3478: URL: https://github.com/apache/hbase/pull/3478#issuecomment-882116168 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 22s | Docker mode activated. | | -0 :warning: | yetus | 0m 11s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 8m 6s | master passed | | +1 :green_heart: | shadedjars | 9m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 8m 13s | the patch passed | | +1 :green_heart: | javac | 8m 13s | the patch passed | | +1 :green_heart: | shadedjars | 9m 1s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 6m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 50s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 26s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 46s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 0m 36s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 12m 5s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 0m 49s | hbase-http in the patch passed. | | +1 :green_heart: | unit | 1m 40s | hbase-asyncfs in the patch passed. | | +1 :green_heart: | unit | 2m 9s | hbase-procedure in the patch passed. | | +1 :green_heart: | unit | 212m 1s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 14m 11s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 1m 39s | hbase-testing-util in the patch passed. | | +1 :green_heart: | unit | 9m 11s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 5s | hbase-shell in the patch passed. | | +1 :green_heart: | unit | 3m 19s | hbase-endpoint in the patch passed. | | +1 :green_heart: | unit | 12m 56s | hbase-backup in the patch passed. | | +1 :green_heart: | unit | 1m 8s | hbase-it in the patch passed. | | +1 :green_heart: | unit | 5m 1s | hbase-rest in the patch passed. | | +1 :green_heart: | unit | 2m 15s | hbase-examples in the patch passed. | | +1 :green_heart: | unit | 1m 28s | hbase-shaded-testing-util-tester in the patch passed. | | +1 :green_heart: | unit | 1m 10s | hbase-client-project in the patch passed. | | +1 :green_heart: | unit | 1m 7s | hbase-shaded-client-project in the patch passed. | | | | 357m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3478 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 62ebe8952bb5 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 83d1bf1667 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/9/testReport/ | | Max. process+thread count | 3458 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded/hbase-shaded-testing-util-tester hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/9/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please
[jira] [Updated] (HBASE-25343) Avoid the failed meta replica region temporarily in Load Balance mode
[ https://issues.apache.org/jira/browse/HBASE-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25343: Fix Version/s: (was: 2.4.5) 2.4.6 > Avoid the failed meta replica region temporarily in Load Balance mode > - > > Key: HBASE-25343 > URL: https://issues.apache.org/jira/browse/HBASE-25343 > Project: HBase > Issue Type: Sub-task > Components: meta replicas >Affects Versions: 2.4.0 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6 > > > This is a follow-up enhancement with Stack, Duo. With the newly introduced > meta replica LoadBalance mode, if there is something wrong with one of meta > replica regions, the current logic is that it keeps trying until the meta > replica region is onlined again or it reports error, i.e, there is no HA at > LoadBalance mode. HA can be implemented if it reports timeout with one meta > replica region and tries another meta replica region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25433) There is no limit on the table name length when creating a table
[ https://issues.apache.org/jira/browse/HBASE-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25433: Fix Version/s: (was: 2.4.5) 2.4.6 > There is no limit on the table name length when creating a table > > > Key: HBASE-25433 > URL: https://issues.apache.org/jira/browse/HBASE-25433 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.2.3 >Reporter: ZouWeiMing >Priority: Major > Fix For: 2.5.0, 2.3.6, 3.0.0-alpha-2, 2.4.6 > > Attachments: attachment_tableNameLengthLimit.html > > > It occurs > Exeception(ERROR：java.util.concurrent.ExecutionExceptionjava.io.IOExpcetionException > in createDir) because of 8000 bytes length limits on HDFS directory name if > the length of HBase table name misses validation when creating. > So,we decide to verify the length of table name when creating a > table.IllegalArgumentExcetion will be thrown when length is larger than 8000 > bytes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25613) [Branch-2 and Master]Handle the NoNode exception in remove log replication in a better way then just log WARN
[ https://issues.apache.org/jira/browse/HBASE-25613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25613: Fix Version/s: (was: 2.4.5) 2.4.6 > [Branch-2 and Master]Handle the NoNode exception in remove log replication in > a better way then just log WARN > - > > Key: HBASE-25613 > URL: https://issues.apache.org/jira/browse/HBASE-25613 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Pal >Assignee: Sandeep Pal >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6 > > > Currently, when we see the NoNodeException in the replication source while > removing a log from ZK, we swallow that exception and log WARN. > > In certain cases, we might have the peer removed and corresponding logs > removing as well but the replication source continuous to run because of an > RPC failure or anything. > In stead of just log WARN we should check if the peer is removed, if it is > the case, we should terminate the source or try to execute the removePeer > workflow again. > > This would prevent the orphaned source execution infinitely. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25588) Excessive logging of "hbase.zookeeper.useMulti is deprecated. Default to true always."
[
https://issues.apache.org/jira/browse/HBASE-25588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25588:
Fix Version/s: (was: 2.4.5)
2.4.6
> Excessive logging of "hbase.zookeeper.useMulti is deprecated. Default to true
> always."
> --
>
> Key: HBASE-25588
> URL: https://issues.apache.org/jira/browse/HBASE-25588
> Project: HBase
> Issue Type: Bug
> Components: logging, Operability, Replication
>Affects Versions: 2.4.1
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6
>
>
> Log line printed frequently, can be as often as once per second:
> {noformat}
> 21:56:46.381 WARN
> [RS_REFRESH_PEER-regionserver/ip-172-31-51-4:8120-0.replicationSource,
> 1.replicationSource.shipperip-172-31-51-4.us-west-2.compute.internal%2C8120%2C1613596855628.ip-172-31-51-4.us-west-2.compute.internal%2C8120%2C1613596855628.default,1]
>org.apache.hadoop.hbase.zookeeper.ZKUtil - hbase.zookeeper.useMulti is
> deprecated. Default to true always.
> {noformat}
> The old configuration in place had hbase.zookeeper.useMulti set to true.
> This is the default and while I understand this setting is deprecated,
> constantly warning about it, especially when it conforms to the default, is
> log spam.
> Warn about this once over the lifetime of the replication source.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-25588) Excessive logging of "hbase.zookeeper.useMulti is deprecated. Default to true always."
[
https://issues.apache.org/jira/browse/HBASE-25588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell reassigned HBASE-25588:
---
Assignee: (was: Lokesh Khurana)
> Excessive logging of "hbase.zookeeper.useMulti is deprecated. Default to true
> always."
> --
>
> Key: HBASE-25588
> URL: https://issues.apache.org/jira/browse/HBASE-25588
> Project: HBase
> Issue Type: Bug
> Components: logging, Operability, Replication
>Affects Versions: 2.4.1
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.5
>
>
> Log line printed frequently, can be as often as once per second:
> {noformat}
> 21:56:46.381 WARN
> [RS_REFRESH_PEER-regionserver/ip-172-31-51-4:8120-0.replicationSource,
> 1.replicationSource.shipperip-172-31-51-4.us-west-2.compute.internal%2C8120%2C1613596855628.ip-172-31-51-4.us-west-2.compute.internal%2C8120%2C1613596855628.default,1]
>org.apache.hadoop.hbase.zookeeper.ZKUtil - hbase.zookeeper.useMulti is
> deprecated. Default to true always.
> {noformat}
> The old configuration in place had hbase.zookeeper.useMulti set to true.
> This is the default and while I understand this setting is deprecated,
> constantly warning about it, especially when it conforms to the default, is
> log spam.
> Warn about this once over the lifetime of the replication source.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25694) Improve the shell's 'status replication' command output
[ https://issues.apache.org/jira/browse/HBASE-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25694: Fix Version/s: (was: 2.4.5) 2.4.6 > Improve the shell's 'status replication' command output > --- > > Key: HBASE-25694 > URL: https://issues.apache.org/jira/browse/HBASE-25694 > Project: HBase > Issue Type: Task > Components: Operability, Replication, shell >Affects Versions: 2.4.2 >Reporter: Andrew Kyle Purtell >Assignee: Divyesh Chandra >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6 > > > The output of the shell's 'status "replication"' command could use some > cleaning up. > The basic format is: > version VERSION > N live servers > SERVER_1: >SOURCE: PeerID=PEERID, AgeOfLastShippedOp=160347, SizeOfLogQueue=49, > TimeStampsOfLastShippedOp=Thu Mar 25 01:55:21 UTC 2021, Replication Lag=161013 >SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Mar 25 > 01:49:53 UTC 2021 > ... > Issues: > - Per line format is KEY=VALUE, ... . Most KEYs are CamelCase but some have > spaces. Should all be camel case for consistency. > - Age is printed as long and timestamp is string (local TZ) format. I think > the string formatting of the timestamp makes it difficult to read. Lets just > print timestamps as longs. Or provide a non-default option for string > formatted timestamps. > - There is no sense of effective rate or bandwidth. Each source and sink line > emitted should provide some sense of current workload: rpcs/sec, bytes/sec, > cells/sec... something. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25624) Bound LoadBalancer's RegionLocationFinder cache
[ https://issues.apache.org/jira/browse/HBASE-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25624: Fix Version/s: (was: 2.4.5) 2.4.6 > Bound LoadBalancer's RegionLocationFinder cache > --- > > Key: HBASE-25624 > URL: https://issues.apache.org/jira/browse/HBASE-25624 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Operability >Affects Versions: 1.6.0, 2.4.1 >Reporter: Andrew Kyle Purtell >Assignee: Prathyusha >Priority: Major > Fix For: 2.5.0, 1.8.0, 3.0.0-alpha-2, 2.4.6 > > > We have a large table in production that causes the balancer's > RegionLocationFinder cache to consume 4 GB of heap, which, among other > factors, triggered OOMEs, and made us aware of this problem. > RegionLocationFinder embeds a cache backed by Guava's CacheLoader. The > RegionLocationFinder cache comes to consume heap for RegionInfos for all > table regions and all HDFS block locations of all store files for all regions > of all tables. > The only limit we pass to the CacheBuilder is an expiration time of 1440 > milliseconds for individual cache entries. That's 4 hours. That's much too > long; however, the cache also periodically refreshes itself, where the need > for a refresh is checked whenever BaseLoadBalancer calls > RegionLocationFinder's setClusterMetrics() method, which defeats the > expiration based limit anyway. > We should be bounding this cache with effective resource controls. Time based > expiry is fine but the periodic refresh logic must be removed to make it > effective. Implement size based limits too. CacheBuilder#maximumSize will > limit by number cache entries. This might be fine but > CacheBuilder#maximumWeight would be better, where weight is something > determined by the API user. In this case it can be an estimate of the heap > size of the hash map entries kept in the cache. > Default should remain unbounded. Specific bounds should be supported by new > site configuration options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25642) Fix or stop warning about already cached block
[
https://issues.apache.org/jira/browse/HBASE-25642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25642:
Fix Version/s: (was: 2.4.5)
2.4.6
> Fix or stop warning about already cached block
> --
>
> Key: HBASE-25642
> URL: https://issues.apache.org/jira/browse/HBASE-25642
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Operability, regionserver
>Affects Versions: 2.3.0
>Reporter: Nick Dimiduk
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6
>
>
> Our logs have as a fairly common occurrence:
> {noformat}
> 2021-03-05 22:24:31,034 WARN [StoreFileOpener-foo-1] hfile.BlockCacheUtil:
> Caching an already cached block: blah.bub. This is harmless and can happen in
> rare cases (see HBASE-8547)
> {noformat}
> If this is really harmless, why do we log? If it's not actually harmless,
> let's take another pass at fixing this situation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25823) TestSlowLogAccessor.testHigherSlowLogs repeatable failure
[
https://issues.apache.org/jira/browse/HBASE-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25823:
Fix Version/s: (was: 2.4.5)
2.4.6
> TestSlowLogAccessor.testHigherSlowLogs repeatable failure
> -
>
> Key: HBASE-25823
> URL: https://issues.apache.org/jira/browse/HBASE-25823
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Major
> Fix For: 2.4.6
>
>
> {noformat}
> [ERROR] TestSlowLogAccessor.testHigherSlowLogs:211 Waiting timed out after
> [7,000] msec{noformat}
> Repeatable failure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25915) ‘mobdir’ overlaps in HFileLink.mobPath
[ https://issues.apache.org/jira/browse/HBASE-25915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25915: Fix Version/s: (was: 2.4.5) (was: 1.4.14) 2.4.6 2.5.0 > ‘mobdir’ overlaps in HFileLink.mobPath > -- > > Key: HBASE-25915 > URL: https://issues.apache.org/jira/browse/HBASE-25915 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 3.0.0-alpha-1, 1.4.13, 2.4.3 >Reporter: wenfeiyi666 >Assignee: wenfeiyi666 >Priority: Minor > Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6 > > > ‘mobdir’ overlaps in HFileLink.mobPath, like > '/root/mobdir/mobdir/data/table/...' -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26088) conn.getBufferedMutator(tableName) leaks thread executors and other problems
[
https://issues.apache.org/jira/browse/HBASE-26088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26088:
Fix Version/s: 2.5.0
> conn.getBufferedMutator(tableName) leaks thread executors and other problems
>
>
> Key: HBASE-26088
> URL: https://issues.apache.org/jira/browse/HBASE-26088
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.4.13, 2.4.4
>Reporter: Whitney Jackson
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.5
>
>
> TL;DR: {{conn.getBufferedMutator(tableName)}} is dangerous in hbase client
> 2.4.4 and doesn't match documented behavior in 1.4.13.
> To work around the problems until fixed do this:
> {code:java}
> var mySingletonPool = HTable.getDefaultExecutor(hbaseConf);
> var params = new BufferedMutatorParams(tableName);
> params.pool(mySingletonPool);
> var myMutator = conn.getBufferedMutator(params);
> {code}
> And avoid code like this:
> {code:java}
> var myMutator = conn.getBufferedMutator(tableName);
> {code}
> The full story:
> My application started leaking threads after upgrading from hbase client
> 1.4.13 to 2.4.4. So much so that after less than a minute of runtime more
> that 30k threads are leaked and all available virtual memory on the box (> 50
> GB) is consumed. Other processes on the box start crashing with memory
> allocation errors. Even running {{ls}} at the shell fails with OS resource
> allocation failures.
> A thread dump after just a few seconds of runtime shows thousands of threads
> like this:
> {code:java}
> "htable-pool-0" #8841 prio=5 os_prio=0 cpu=0.15ms elapsed=7.49s
> tid=0x7efb6d2a1000 nid=0x57d2 waiting on condition [0x7ef8a6c38000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11.0.6/Native Method)
> - parking to wait for <0x0007e7cd6188> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.6/LockSupport.java:234)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@11.0.6/SynchronousQueue.java:462)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@11.0.6/SynchronousQueue.java:361)
> at
> java.util.concurrent.SynchronousQueue.poll(java.base@11.0.6/SynchronousQueue.java:937)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.6/ThreadPoolExecutor.java:1053)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.6/ThreadPoolExecutor.java:1114)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.6/ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(java.base@11.0.6/Thread.java:834)
> {code}
>
> Note: All the threads are labeled {{htable-pool-0}}. That suggests we're
> leaking thread executors not just threads. The {{htable-pool}} part indicates
> the problem is to do with {{HTable.getDefaultExecutor(conf)}} and the only
> part of my code that interacts with that is a call to
> {{conn.getBufferedMutator(tableName)}}.
>
> Looking at the hbase client code shows a few problems:
> 1) Neither 1.4.13 nor 2.4.4's behavior matches the documentation for
> {{conn.getBufferedMutator(tableName)}} which says:
> {quote}This BufferedMutator will use the Connection's ExecutorService.
> {quote}
> That suggests some singleton thread executor is being used which is not the
> case.
>
> 2) Under 1.4.13 you get a new {{ThreadPoolExecutor}} for every
> {{BufferedMutator}}. That's probably not what you want but you likely won't
> notice. I didn't. It's a code path I hadn't profiled much.
>
> 3) Under 2.4.4 you get a new {{ThreadPoolExecutor}} for every
> {{BufferedMutator}} *and* that {{ThreadPoolExecutor}} *is not* cleaned up
> after the {{Mutator}} is closed. Each completed {{ThreadPoolExecutor}}
> carries with it one thread which hangs around until a timeout value which
> defaults to 60 seconds.
> My application creates one {{BufferedMutator}} for every incoming stream and
> there are lots of streams, some of them are short lived so my code leaks
> threads fast under 2.4.4.
> Here's the part where a new executor is created for every {{BufferedMutator}}
> (it's similar for 1.4.13):
> [https://github.com/apache/hbase/blob/branch-2.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L420]
>
> The reason for the leak in 2.4.4 is the should-we/shouldn't-we cleanup logic
> added here:
> [https://github.com/apache/hbase/blob/branch-2.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L104]
> That might be ok if {{pool}} was being initialized there but in the
> {{conn.getBufferedMutator(tableName)}}
[jira] [Updated] (HBASE-26093) Replication is stuck due to zero length wal file in oldWALs directory [master/branch-2]
[ https://issues.apache.org/jira/browse/HBASE-26093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26093: Fix Version/s: 2.5.0 > Replication is stuck due to zero length wal file in oldWALs directory > [master/branch-2] > --- > > Key: HBASE-26093 > URL: https://issues.apache.org/jira/browse/HBASE-26093 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-2, 2.4.5 >Reporter: Bharath Vissapragada >Assignee: Rushabh Shah >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.5 > > > Clone of the parent issue for branch-2/master. Resolving the parent as the > issue is fixed in 1.7.1 and the jira needs to be in a resolved state for > release notes curation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25828) CompactionProgress WARNS: "totalCompactingKVs=N less than currentCompactedKVs=M"
[
https://issues.apache.org/jira/browse/HBASE-25828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25828:
Fix Version/s: (was: 2.4.5)
2.4.6
> CompactionProgress WARNS: "totalCompactingKVs=N less than
> currentCompactedKVs=M"
>
>
> Key: HBASE-25828
> URL: https://issues.apache.org/jira/browse/HBASE-25828
> Project: HBase
> Issue Type: Bug
> Components: Compaction, Operability, regionserver
>Affects Versions: 2.4.3
>Reporter: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6
>
>
> Similar to HBASE-25642, but compaction progress warnings.
> Lots of warnings like:
> {noformat}
> 2021-04-30 00:55:15,436 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=15589 less than
> currentCompactedKVs=21411
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,437 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=21916 less than
> currentCompactedKVs=33328
> {noformat}
> {noformat}
> 2021-04-30 00:55:15,438 WARN [regionserver/ip-172-31-63-65:8120]
> compactions.CompactionProgress: totalCompactingKVs=89731 less than
> currentCompactedKVs=92808
> {noformat}
> A couple of issues:
> - Is there a way to make CompactionProgress more reliable? I seem to recall
> this is the second or third time around the block on this.
> - This is annoying because compaction progress isn't always accurate, but
> this information is not used to determine anything significant, so WARN level
> is a bit much. (Or, if this is a real correctness problem, see point above.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25973) Balancer should explain progress in a better way in log
[
https://issues.apache.org/jira/browse/HBASE-25973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25973:
Fix Version/s: 2.5.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Balancer should explain progress in a better way in log
> ---
>
> Key: HBASE-25973
> URL: https://issues.apache.org/jira/browse/HBASE-25973
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 3.0.0-alpha-1
>Reporter: Clara Xiong
>Assignee: Clara Xiong
>Priority: Major
> Fix For: 2.5.0, 2.3.6, 3.0.0-alpha-2, 2.4.5
>
>
> In the log, balancer logs at info level at the beginning of run:
> {code}
> balancer.StochasticLoadBalancer: start StochasticLoadBalancer.balancer,
> initCost=277.3479243125063, functionCost=RegionCountSkewCostFunction :
> (500.0, 0.3749771215224234); ServerLocalityCostFunction : (25.0,
> 0.5807483226644186); RackLocalityCostFunction : (15.0, 0.0);
> TableSkewCostFunction : (1000.0, 0.0019704142954972883);
> StoreFileCostFunction : (200.0, 0.3668512059459341); computedMaxSteps:
> 42270438200
> {code}
> the cost is reported without context, it is hard for operator to understand
> how unbalanced the cluster is for balancer and how much progress we are
> making.
> For a large cluster, the calculation can take a long time, we also need to
> let operator understand that it will take up to the max time to complete the
> calculation.
> At the end of computation:
> {code}
> balancer.StochasticLoadBalancer: Finished computing new load balance plan.
> Computation took PT40M0.006S to try 1036409 different iterations. Found a
> solution that moves 161926 regions; Going from a computed cost of
> 118.75715593924485 to a new cost of 1.5509126920967042
> {code}
> The time to compute the plan is also printed in a format that is not human
> readable. we also need to let operator understand that balancer is just
> submitting the plan and it be up to execution to complete the move.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] apurtell merged pull request #3483: HBASE-25973 Balancer should explain progress in a better way in log -…
apurtell merged pull request #3483: URL: https://github.com/apache/hbase/pull/3483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-26006) Update ref guide about the 2.4.x release line
[ https://issues.apache.org/jira/browse/HBASE-26006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26006: Fix Version/s: (was: 2.4.5) 2.4.6 > Update ref guide about the 2.4.x release line > - > > Key: HBASE-26006 > URL: https://issues.apache.org/jira/browse/HBASE-26006 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Duo Zhang >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 2.4.6 > > > we do not have 2.4 in the compatibility matrix and release manager section. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26093) Replication is stuck due to zero length wal file in oldWALs directory [master/branch-2]
[ https://issues.apache.org/jira/browse/HBASE-26093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382895#comment-17382895 ] Andrew Kyle Purtell commented on HBASE-26093: - [~shahrs87] Are you planning to post a PR of this in the next day or two? Thinking about getting this in to 2.4.5. Please advise. > Replication is stuck due to zero length wal file in oldWALs directory > [master/branch-2] > --- > > Key: HBASE-26093 > URL: https://issues.apache.org/jira/browse/HBASE-26093 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-2, 2.4.5 >Reporter: Bharath Vissapragada >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-2, 2.4.5 > > > Clone of the parent issue for branch-2/master. Resolving the parent as the > issue is fixed in 1.7.1 and the jira needs to be in a resolved state for > release notes curation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26093) Replication is stuck due to zero length wal file in oldWALs directory [master/branch-2]
[ https://issues.apache.org/jira/browse/HBASE-26093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-26093: Fix Version/s: 2.4.5 3.0.0-alpha-2 > Replication is stuck due to zero length wal file in oldWALs directory > [master/branch-2] > --- > > Key: HBASE-26093 > URL: https://issues.apache.org/jira/browse/HBASE-26093 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-2, 2.4.5 >Reporter: Bharath Vissapragada >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-2, 2.4.5 > > > Clone of the parent issue for branch-2/master. Resolving the parent as the > issue is fixed in 1.7.1 and the jira needs to be in a resolved state for > release notes curation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26088) conn.getBufferedMutator(tableName) leaks thread executors and other problems
[
https://issues.apache.org/jira/browse/HBASE-26088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17382894#comment-17382894
]
Andrew Kyle Purtell commented on HBASE-26088:
-
Is someone planning a PR for the two line fix mentioned above? Otherwise I'll
take it for 2.4.5.
> conn.getBufferedMutator(tableName) leaks thread executors and other problems
>
>
> Key: HBASE-26088
> URL: https://issues.apache.org/jira/browse/HBASE-26088
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.4.13, 2.4.4
>Reporter: Whitney Jackson
>Priority: Critical
> Fix For: 3.0.0-alpha-2, 2.4.5
>
>
> TL;DR: {{conn.getBufferedMutator(tableName)}} is dangerous in hbase client
> 2.4.4 and doesn't match documented behavior in 1.4.13.
> To work around the problems until fixed do this:
> {code:java}
> var mySingletonPool = HTable.getDefaultExecutor(hbaseConf);
> var params = new BufferedMutatorParams(tableName);
> params.pool(mySingletonPool);
> var myMutator = conn.getBufferedMutator(params);
> {code}
> And avoid code like this:
> {code:java}
> var myMutator = conn.getBufferedMutator(tableName);
> {code}
> The full story:
> My application started leaking threads after upgrading from hbase client
> 1.4.13 to 2.4.4. So much so that after less than a minute of runtime more
> that 30k threads are leaked and all available virtual memory on the box (> 50
> GB) is consumed. Other processes on the box start crashing with memory
> allocation errors. Even running {{ls}} at the shell fails with OS resource
> allocation failures.
> A thread dump after just a few seconds of runtime shows thousands of threads
> like this:
> {code:java}
> "htable-pool-0" #8841 prio=5 os_prio=0 cpu=0.15ms elapsed=7.49s
> tid=0x7efb6d2a1000 nid=0x57d2 waiting on condition [0x7ef8a6c38000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11.0.6/Native Method)
> - parking to wait for <0x0007e7cd6188> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.6/LockSupport.java:234)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@11.0.6/SynchronousQueue.java:462)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@11.0.6/SynchronousQueue.java:361)
> at
> java.util.concurrent.SynchronousQueue.poll(java.base@11.0.6/SynchronousQueue.java:937)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.6/ThreadPoolExecutor.java:1053)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.6/ThreadPoolExecutor.java:1114)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.6/ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(java.base@11.0.6/Thread.java:834)
> {code}
>
> Note: All the threads are labeled {{htable-pool-0}}. That suggests we're
> leaking thread executors not just threads. The {{htable-pool}} part indicates
> the problem is to do with {{HTable.getDefaultExecutor(conf)}} and the only
> part of my code that interacts with that is a call to
> {{conn.getBufferedMutator(tableName)}}.
>
> Looking at the hbase client code shows a few problems:
> 1) Neither 1.4.13 nor 2.4.4's behavior matches the documentation for
> {{conn.getBufferedMutator(tableName)}} which says:
> {quote}This BufferedMutator will use the Connection's ExecutorService.
> {quote}
> That suggests some singleton thread executor is being used which is not the
> case.
>
> 2) Under 1.4.13 you get a new {{ThreadPoolExecutor}} for every
> {{BufferedMutator}}. That's probably not what you want but you likely won't
> notice. I didn't. It's a code path I hadn't profiled much.
>
> 3) Under 2.4.4 you get a new {{ThreadPoolExecutor}} for every
> {{BufferedMutator}} *and* that {{ThreadPoolExecutor}} *is not* cleaned up
> after the {{Mutator}} is closed. Each completed {{ThreadPoolExecutor}}
> carries with it one thread which hangs around until a timeout value which
> defaults to 60 seconds.
> My application creates one {{BufferedMutator}} for every incoming stream and
> there are lots of streams, some of them are short lived so my code leaks
> threads fast under 2.4.4.
> Here's the part where a new executor is created for every {{BufferedMutator}}
> (it's similar for 1.4.13):
> [https://github.com/apache/hbase/blob/branch-2.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L420]
>
> The reason for the leak in 2.4.4 is the should-we/shouldn't-we cleanup logic
> added here:
> [https://github.com/apache/hbase/blob/branch-2.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L104]
> That
[jira] [Updated] (HBASE-26088) conn.getBufferedMutator(tableName) leaks thread executors and other problems
[
https://issues.apache.org/jira/browse/HBASE-26088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-26088:
Fix Version/s: 2.4.5
3.0.0-alpha-2
> conn.getBufferedMutator(tableName) leaks thread executors and other problems
>
>
> Key: HBASE-26088
> URL: https://issues.apache.org/jira/browse/HBASE-26088
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.4.13, 2.4.4
>Reporter: Whitney Jackson
>Priority: Critical
> Fix For: 3.0.0-alpha-2, 2.4.5
>
>
> TL;DR: {{conn.getBufferedMutator(tableName)}} is dangerous in hbase client
> 2.4.4 and doesn't match documented behavior in 1.4.13.
> To work around the problems until fixed do this:
> {code:java}
> var mySingletonPool = HTable.getDefaultExecutor(hbaseConf);
> var params = new BufferedMutatorParams(tableName);
> params.pool(mySingletonPool);
> var myMutator = conn.getBufferedMutator(params);
> {code}
> And avoid code like this:
> {code:java}
> var myMutator = conn.getBufferedMutator(tableName);
> {code}
> The full story:
> My application started leaking threads after upgrading from hbase client
> 1.4.13 to 2.4.4. So much so that after less than a minute of runtime more
> that 30k threads are leaked and all available virtual memory on the box (> 50
> GB) is consumed. Other processes on the box start crashing with memory
> allocation errors. Even running {{ls}} at the shell fails with OS resource
> allocation failures.
> A thread dump after just a few seconds of runtime shows thousands of threads
> like this:
> {code:java}
> "htable-pool-0" #8841 prio=5 os_prio=0 cpu=0.15ms elapsed=7.49s
> tid=0x7efb6d2a1000 nid=0x57d2 waiting on condition [0x7ef8a6c38000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11.0.6/Native Method)
> - parking to wait for <0x0007e7cd6188> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.6/LockSupport.java:234)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@11.0.6/SynchronousQueue.java:462)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@11.0.6/SynchronousQueue.java:361)
> at
> java.util.concurrent.SynchronousQueue.poll(java.base@11.0.6/SynchronousQueue.java:937)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.6/ThreadPoolExecutor.java:1053)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.6/ThreadPoolExecutor.java:1114)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.6/ThreadPoolExecutor.java:628)
> at java.lang.Thread.run(java.base@11.0.6/Thread.java:834)
> {code}
>
> Note: All the threads are labeled {{htable-pool-0}}. That suggests we're
> leaking thread executors not just threads. The {{htable-pool}} part indicates
> the problem is to do with {{HTable.getDefaultExecutor(conf)}} and the only
> part of my code that interacts with that is a call to
> {{conn.getBufferedMutator(tableName)}}.
>
> Looking at the hbase client code shows a few problems:
> 1) Neither 1.4.13 nor 2.4.4's behavior matches the documentation for
> {{conn.getBufferedMutator(tableName)}} which says:
> {quote}This BufferedMutator will use the Connection's ExecutorService.
> {quote}
> That suggests some singleton thread executor is being used which is not the
> case.
>
> 2) Under 1.4.13 you get a new {{ThreadPoolExecutor}} for every
> {{BufferedMutator}}. That's probably not what you want but you likely won't
> notice. I didn't. It's a code path I hadn't profiled much.
>
> 3) Under 2.4.4 you get a new {{ThreadPoolExecutor}} for every
> {{BufferedMutator}} *and* that {{ThreadPoolExecutor}} *is not* cleaned up
> after the {{Mutator}} is closed. Each completed {{ThreadPoolExecutor}}
> carries with it one thread which hangs around until a timeout value which
> defaults to 60 seconds.
> My application creates one {{BufferedMutator}} for every incoming stream and
> there are lots of streams, some of them are short lived so my code leaks
> threads fast under 2.4.4.
> Here's the part where a new executor is created for every {{BufferedMutator}}
> (it's similar for 1.4.13):
> [https://github.com/apache/hbase/blob/branch-2.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L420]
>
> The reason for the leak in 2.4.4 is the should-we/shouldn't-we cleanup logic
> added here:
> [https://github.com/apache/hbase/blob/branch-2.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L104]
> That might be ok if {{pool}} was being initialized there but in the
>
[jira] [Updated] (HBASE-25923) Region state stuck in PENDING_OPEN
[
https://issues.apache.org/jira/browse/HBASE-25923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25923:
Status: Patch Available (was: Open)
> Region state stuck in PENDING_OPEN
> --
>
> Key: HBASE-25923
> URL: https://issues.apache.org/jira/browse/HBASE-25923
> Project: HBase
> Issue Type: Improvement
> Components: master, Region Assignment
>Affects Versions: 1.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 1.7.2
>
>
> Region will not be reassigned if encounters ConnectionClosingException, and
> then it will stuck in PENDING_OPEN state. Error logs are as follows,
> {code:java}
> INFO
> [jd-data-hbase02.gh.sankuai.com,16000,1621944138744-GeneralBulkAssigner-12]
> master.AssignmentManager: Unable to communicate with
> jd-data-hbase15.gh.sankuai.com,16020,1622026221268 in order to assign
> regions,
> org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
> jd-data-hbase15.gh.sankuai.com/10.78.96.166:16020 failed on local exception:
> org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to
> jd-data-hbase15.gh.sankuai.com/10.78.96.166:16020 is closing. Call id=19239,
> waitTime=1
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:289)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:25890)
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:798)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1744)
> at
> org.apache.hadoop.hbase.master.GeneralBulkAssigner$SingleServerBulkAssigner.run(GeneralBulkAssigner.java:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException:
> Connection to jd-data-hbase15.gh.sankuai.com/10.78.96.166:16020 is closing.
> Call id=19239, waitTime=1
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1083)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:863)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:580)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25923) Region state stuck in PENDING_OPEN
[
https://issues.apache.org/jira/browse/HBASE-25923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25923:
Fix Version/s: 1.7.2
> Region state stuck in PENDING_OPEN
> --
>
> Key: HBASE-25923
> URL: https://issues.apache.org/jira/browse/HBASE-25923
> Project: HBase
> Issue Type: Improvement
> Components: master, Region Assignment
>Affects Versions: 1.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 1.7.2
>
>
> Region will not be reassigned if encounters ConnectionClosingException, and
> then it will stuck in PENDING_OPEN state. Error logs are as follows,
> {code:java}
> INFO
> [jd-data-hbase02.gh.sankuai.com,16000,1621944138744-GeneralBulkAssigner-12]
> master.AssignmentManager: Unable to communicate with
> jd-data-hbase15.gh.sankuai.com,16020,1622026221268 in order to assign
> regions,
> org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
> jd-data-hbase15.gh.sankuai.com/10.78.96.166:16020 failed on local exception:
> org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to
> jd-data-hbase15.gh.sankuai.com/10.78.96.166:16020 is closing. Call id=19239,
> waitTime=1
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:289)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:25890)
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:798)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1744)
> at
> org.apache.hadoop.hbase.master.GeneralBulkAssigner$SingleServerBulkAssigner.run(GeneralBulkAssigner.java:203)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException:
> Connection to jd-data-hbase15.gh.sankuai.com/10.78.96.166:16020 is closing.
> Call id=19239, waitTime=1
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1083)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:863)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:580)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3478: HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related …
Apache-HBase commented on pull request #3478: URL: https://github.com/apache/hbase/pull/3478#issuecomment-882106824 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 12s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 10s | master passed | | +1 :green_heart: | compile | 9m 15s | master passed | | +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 7m 24s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 13s | the patch passed | | +1 :green_heart: | compile | 9m 16s | the patch passed | | +1 :green_heart: | javac | 9m 16s | the patch passed | | +1 :green_heart: | shadedjars | 8m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 7m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 50s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 23s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 46s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 0m 28s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 7m 42s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 0m 45s | hbase-http in the patch passed. | | +1 :green_heart: | unit | 1m 27s | hbase-asyncfs in the patch passed. | | +1 :green_heart: | unit | 1m 45s | hbase-procedure in the patch passed. | | +1 :green_heart: | unit | 137m 57s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 10m 16s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 1m 34s | hbase-testing-util in the patch passed. | | +1 :green_heart: | unit | 6m 25s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 7s | hbase-shell in the patch passed. | | +1 :green_heart: | unit | 2m 56s | hbase-endpoint in the patch passed. | | +1 :green_heart: | unit | 9m 5s | hbase-backup in the patch passed. | | +1 :green_heart: | unit | 1m 12s | hbase-it in the patch passed. | | +1 :green_heart: | unit | 3m 47s | hbase-rest in the patch passed. | | +1 :green_heart: | unit | 1m 48s | hbase-examples in the patch passed. | | +1 :green_heart: | unit | 1m 19s | hbase-shaded-testing-util-tester in the patch passed. | | +1 :green_heart: | unit | 1m 11s | hbase-client-project in the patch passed. | | +1 :green_heart: | unit | 1m 11s | hbase-shaded-client-project in the patch passed. | | | | 270m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3478 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ed24b47c312e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 83d1bf1667 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/9/testReport/ | | Max. process+thread count | 5054 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded/hbase-shaded-testing-util-tester hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/9/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log
[GitHub] [hbase] Apache-HBase commented on pull request #3478: HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related …
Apache-HBase commented on pull request #3478: URL: https://github.com/apache/hbase/pull/3478#issuecomment-882088682 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 10s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 59s | master passed | | +1 :green_heart: | compile | 14m 18s | master passed | | +1 :green_heart: | checkstyle | 7m 30s | master passed | | +1 :green_heart: | spotbugs | 13m 38s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 14m 23s | the patch passed | | +1 :green_heart: | javac | 0m 46s | hbase-common generated 0 new + 155 unchanged - 4 fixed = 155 total (was 159) | | +1 :green_heart: | javac | 0m 56s | hbase-client in the patch passed. | | -0 :warning: | javac | 0m 27s | hbase-zookeeper generated 1 new + 96 unchanged - 1 fixed = 97 total (was 97) | | +1 :green_heart: | javac | 0m 23s | hbase-replication in the patch passed. | | +1 :green_heart: | javac | 0m 30s | hbase-balancer in the patch passed. | | +1 :green_heart: | javac | 0m 28s | hbase-http in the patch passed. | | +1 :green_heart: | javac | 0m 26s | hbase-asyncfs in the patch passed. | | +1 :green_heart: | javac | 0m 32s | hbase-procedure in the patch passed. | | -0 :warning: | javac | 3m 16s | hbase-server generated 97 new + 96 unchanged - 97 fixed = 193 total (was 193) | | -0 :warning: | javac | 0m 47s | hbase-mapreduce generated 1 new + 197 unchanged - 1 fixed = 198 total (was 198) | | -0 :warning: | javac | 0m 30s | hbase-testing-util generated 100 new + 1 unchanged - 0 fixed = 101 total (was 1) | | +1 :green_heart: | javac | 0m 56s | hbase-thrift in the patch passed. | | +1 :green_heart: | javac | 0m 23s | hbase-shell in the patch passed. | | +1 :green_heart: | javac | 0m 31s | hbase-endpoint in the patch passed. | | +1 :green_heart: | javac | 0m 36s | hbase-backup in the patch passed. | | +1 :green_heart: | javac | 0m 39s | hbase-it in the patch passed. | | +1 :green_heart: | javac | 0m 39s | hbase-rest in the patch passed. | | +1 :green_heart: | javac | 0m 35s | hbase-examples in the patch passed. | | +1 :green_heart: | javac | 0m 14s | hbase-shaded-testing-util-tester in the patch passed. | | +1 :green_heart: | javac | 0m 25s | hbase-client-project in the patch passed. | | +1 :green_heart: | javac | 0m 24s | hbase-shaded-client-project in the patch passed. | | -0 :warning: | checkstyle | 1m 35s | hbase-server: The patch generated 13 new + 1445 unchanged - 43 fixed = 1458 total (was 1488) | | -0 :warning: | checkstyle | 0m 18s | hbase-testing-util: The patch generated 174 new + 0 unchanged - 0 fixed = 174 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 20m 0s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 17m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 3m 31s | The patch does not generate ASF License warnings. | | | | 124m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/9/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3478 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux 95a515dd5188 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 83d1bf1667 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/9/artifact/yetus-general-check/output/diff-compile-javac-hbase-zookeeper.txt | | javac |
[GitHub] [hbase] Apache-HBase commented on pull request #3478: HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related …
Apache-HBase commented on pull request #3478: URL: https://github.com/apache/hbase/pull/3478#issuecomment-882070085 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 21s | Docker mode activated. | | -0 :warning: | yetus | 0m 11s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | compile | 8m 11s | master passed | | +1 :green_heart: | shadedjars | 9m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 5m 58s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | compile | 8m 10s | the patch passed | | +1 :green_heart: | javac | 8m 10s | the patch passed | | +1 :green_heart: | shadedjars | 9m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 6m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 50s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 26s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 46s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 0m 36s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 12m 7s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 0m 49s | hbase-http in the patch passed. | | +1 :green_heart: | unit | 1m 41s | hbase-asyncfs in the patch passed. | | +1 :green_heart: | unit | 2m 10s | hbase-procedure in the patch passed. | | +1 :green_heart: | unit | 212m 12s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 14m 22s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 1m 35s | hbase-testing-util in the patch passed. | | +1 :green_heart: | unit | 9m 9s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 4s | hbase-shell in the patch passed. | | +1 :green_heart: | unit | 3m 18s | hbase-endpoint in the patch passed. | | +1 :green_heart: | unit | 13m 0s | hbase-backup in the patch passed. | | +1 :green_heart: | unit | 1m 8s | hbase-it in the patch passed. | | +1 :green_heart: | unit | 5m 0s | hbase-rest in the patch passed. | | +1 :green_heart: | unit | 2m 18s | hbase-examples in the patch passed. | | +1 :green_heart: | unit | 1m 30s | hbase-shaded-testing-util-tester in the patch passed. | | +1 :green_heart: | unit | 1m 11s | hbase-client-project in the patch passed. | | +1 :green_heart: | unit | 1m 8s | hbase-shaded-client-project in the patch passed. | | | | 358m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3478 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d31a460201bc 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 83d1bf1667 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/testReport/ | | Max. process+thread count | 3213 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded/hbase-shaded-testing-util-tester hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please
[GitHub] [hbase] Apache-HBase commented on pull request #3478: HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related …
Apache-HBase commented on pull request #3478: URL: https://github.com/apache/hbase/pull/3478#issuecomment-882057696 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 12s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 45s | master passed | | +1 :green_heart: | compile | 9m 22s | master passed | | +1 :green_heart: | shadedjars | 8m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 7m 27s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 14s | the patch passed | | +1 :green_heart: | compile | 9m 16s | the patch passed | | +1 :green_heart: | javac | 9m 16s | the patch passed | | +1 :green_heart: | shadedjars | 8m 17s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 7m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 52s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 23s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 46s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 0m 29s | hbase-replication in the patch passed. | | +1 :green_heart: | unit | 7m 42s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 0m 45s | hbase-http in the patch passed. | | +1 :green_heart: | unit | 1m 25s | hbase-asyncfs in the patch passed. | | +1 :green_heart: | unit | 1m 45s | hbase-procedure in the patch passed. | | +1 :green_heart: | unit | 137m 57s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 10m 19s | hbase-mapreduce in the patch passed. | | +1 :green_heart: | unit | 1m 37s | hbase-testing-util in the patch passed. | | +1 :green_heart: | unit | 6m 31s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 7m 6s | hbase-shell in the patch passed. | | +1 :green_heart: | unit | 2m 55s | hbase-endpoint in the patch passed. | | +1 :green_heart: | unit | 8m 58s | hbase-backup in the patch passed. | | +1 :green_heart: | unit | 1m 12s | hbase-it in the patch passed. | | +1 :green_heart: | unit | 3m 44s | hbase-rest in the patch passed. | | +1 :green_heart: | unit | 1m 48s | hbase-examples in the patch passed. | | +1 :green_heart: | unit | 1m 23s | hbase-shaded-testing-util-tester in the patch passed. | | +1 :green_heart: | unit | 1m 10s | hbase-client-project in the patch passed. | | +1 :green_heart: | unit | 1m 8s | hbase-shaded-client-project in the patch passed. | | | | 271m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3478 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ed881b40ea4c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 83d1bf1667 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/testReport/ | | Max. process+thread count | 5015 (vs. ulimit of 3) | | modules | C: hbase-common hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded/hbase-shaded-testing-util-tester hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log
[GitHub] [hbase] Apache-HBase commented on pull request #3478: HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related …
Apache-HBase commented on pull request #3478: URL: https://github.com/apache/hbase/pull/3478#issuecomment-882040377 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 23s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 8s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 0s | master passed | | +1 :green_heart: | compile | 14m 14s | master passed | | +1 :green_heart: | checkstyle | 7m 27s | master passed | | +1 :green_heart: | spotbugs | 13m 36s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 6s | the patch passed | | +1 :green_heart: | compile | 14m 31s | the patch passed | | -0 :warning: | javac | 0m 46s | hbase-common generated 5 new + 155 unchanged - 4 fixed = 160 total (was 159) | | -0 :warning: | javac | 0m 27s | hbase-zookeeper generated 1 new + 96 unchanged - 1 fixed = 97 total (was 97) | | -0 :warning: | javac | 3m 21s | hbase-server generated 97 new + 96 unchanged - 97 fixed = 193 total (was 193) | | -0 :warning: | javac | 0m 46s | hbase-mapreduce generated 1 new + 197 unchanged - 1 fixed = 198 total (was 198) | | -0 :warning: | javac | 0m 30s | hbase-testing-util generated 100 new + 1 unchanged - 0 fixed = 101 total (was 1) | | -0 :warning: | checkstyle | 1m 37s | hbase-server: The patch generated 92 new + 1460 unchanged - 28 fixed = 1552 total (was 1488) | | -0 :warning: | checkstyle | 0m 18s | hbase-testing-util: The patch generated 174 new + 0 unchanged - 0 fixed = 174 total (was 0) | | -0 :warning: | checkstyle | 0m 19s | hbase-it: The patch generated 2 new + 119 unchanged - 0 fixed = 121 total (was 119) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 22m 49s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 18m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 3m 31s | The patch does not generate ASF License warnings. | | | | 129m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3478 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux 385d216c7b8c 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 83d1bf1667 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-general-check/output/diff-compile-javac-hbase-common.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-general-check/output/diff-compile-javac-hbase-zookeeper.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-general-check/output/diff-compile-javac-hbase-mapreduce.txt | | javac | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-general-check/output/diff-compile-javac-hbase-testing-util.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-general-check/output/diff-checkstyle-hbase-testing-util.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3478/8/artifact/yetus-general-check/output/diff-checkstyle-hbase-it.txt | | Max. process+thread count | 87 (vs. ulimit of 3)
[GitHub] [hbase] Apache9 commented on pull request #3478: HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related …
Apache9 commented on pull request #3478:
URL: https://github.com/apache/hbase/pull/3478#issuecomment-882027035
Thank you @saintstack . Finally I think I found a way to do renaming instead
of moving. This is the release note of this issue. PTAL. Shout if you do not
think it is good enough.
> Copy HBaseCommonTestingUtility, HBaseZKTestingUtility,
HBaseTestingUtility, HBaseCluster, MiniHBaseCluster, StartMiniClusterOption to
hbase-testing-util, and mark them as Deprecated. They will be removed in 4.0.0.
End users should use TestingHBaseCluster for writing UTs in the future.
>
> And in hbase-server and related modules, these classes are renamed.
> HBaseCommonTestingUtility -> HBaseCommonTestingUtil
> HBaseZKTestingUtility -> HBaseZKTestingUtil
> HBaseTestingUtility -> HBaseTestingUtil
> HBaseCluster -> HBaseClusterInterface
> MiniHBaseCluster -> SingleProcessHBaseCluster
> StartMiniClusterOption -> StartTestingClusterOption
> And all these classes are marked as IA.LimitedPrivate("Phoenix"), and
IS.Evolving. Except Phoenix, other end users should not use them any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-26081) Copy HBTU to hbase-testing-util, rename the HBTU related classes in hbase-server and mark them as IA.LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-26081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26081: -- Description: See the discussions in the parent issue on why we want to do this. This is the final decision: https://issues.apache.org/jira/browse/HBASE-13126?focusedCommentId=17358108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17358108 > Copy HBTU to hbase-testing-util, rename the HBTU related classes in > hbase-server and mark them as IA.LimitedPrivate > --- > > Key: HBASE-26081 > URL: https://issues.apache.org/jira/browse/HBASE-26081 > Project: HBase > Issue Type: Sub-task > Components: API, test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-2 > > > See the discussions in the parent issue on why we want to do this. > This is the final decision: > https://issues.apache.org/jira/browse/HBASE-13126?focusedCommentId=17358108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17358108 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26081) Copy HBTU to hbase-testing-util, rename the HBTU related classes in hbase-server and mark them as IA.LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-26081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26081: -- Description: See the discussions in the parent issue on why we want to do this. This is the final decision: https://issues.apache.org/jira/browse/HBASE-13126?focusedCommentId=17358108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17358108 Maybe a possible way is that, we mark HBTU as deprecated and introduce another class for end users to use. And on the next major release, we just introduce a another class in hbase-server for our internal usage, and copy the whole HBTU related classes to hbase-testing-util module, and keep them there for a whole major release, and then we remove them when the next major release is out. So end users will have plenty of time to change their code. was: See the discussions in the parent issue on why we want to do this. This is the final decision: https://issues.apache.org/jira/browse/HBASE-13126?focusedCommentId=17358108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17358108 > Copy HBTU to hbase-testing-util, rename the HBTU related classes in > hbase-server and mark them as IA.LimitedPrivate > --- > > Key: HBASE-26081 > URL: https://issues.apache.org/jira/browse/HBASE-26081 > Project: HBase > Issue Type: Sub-task > Components: API, test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-2 > > > See the discussions in the parent issue on why we want to do this. > This is the final decision: > https://issues.apache.org/jira/browse/HBASE-13126?focusedCommentId=17358108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17358108 > Maybe a possible way is that, we mark HBTU as deprecated and introduce > another class for end users to use. And on the next major release, we just > introduce a another class in hbase-server for our internal usage, and copy > the whole HBTU related classes to hbase-testing-util module, and keep them > there for a whole major release, and then we remove them when the next major > release is out. So end users will have plenty of time to change their code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26081) Copy HBTU to hbase-testing-util, rename the HBTU related classes in hbase-server and mark them as IA.LimitedPrivate
[
https://issues.apache.org/jira/browse/HBASE-26081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-26081:
--
Hadoop Flags: Incompatible change,Reviewed
Release Note:
Copy HBaseCommonTestingUtility, HBaseZKTestingUtility, HBaseTestingUtility,
HBaseCluster, MiniHBaseCluster, StartMiniClusterOption to hbase-testing-util,
and mark them as Deprecated. They will be removed in 4.0.0. End users should
use TestingHBaseCluster for writing UTs in the future.
And in hbase-server and related modules, these classes are renamed.
HBaseCommonTestingUtility -> HBaseCommonTestingUtil
HBaseZKTestingUtility -> HBaseZKTestingUtil
HBaseTestingUtility -> HBaseTestingUtil
HBaseCluster -> HBaseClusterInterface
MiniHBaseCluster -> SingleProcessHBaseCluster
StartMiniClusterOption -> StartTestingClusterOption
And all these classes are marked as IA.LimitedPrivate("Phoenix"), and
IS.Evolving. Except Phoenix, other end users should not use them any more.
> Copy HBTU to hbase-testing-util, rename the HBTU related classes in
> hbase-server and mark them as IA.LimitedPrivate
> ---
>
> Key: HBASE-26081
> URL: https://issues.apache.org/jira/browse/HBASE-26081
> Project: HBase
> Issue Type: Sub-task
> Components: API, test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-26081) Copy HBTU to hbase-testing-util, rename the HBTU related classes in hbase-server and mark them as IA.LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-26081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-26081: - Assignee: Duo Zhang > Copy HBTU to hbase-testing-util, rename the HBTU related classes in > hbase-server and mark them as IA.LimitedPrivate > --- > > Key: HBASE-26081 > URL: https://issues.apache.org/jira/browse/HBASE-26081 > Project: HBase > Issue Type: Sub-task > Components: API, test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-26081) Copy HBTU to hbase-testing-util, rename the HBTU related classes in hbase-server and mark them as IA.LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-26081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-26081 started by Duo Zhang. - > Copy HBTU to hbase-testing-util, rename the HBTU related classes in > hbase-server and mark them as IA.LimitedPrivate > --- > > Key: HBASE-26081 > URL: https://issues.apache.org/jira/browse/HBASE-26081 > Project: HBase > Issue Type: Sub-task > Components: API, test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on pull request #3492: HBASE-25986 set default value of normalization enabled from hbase site
virajjasani commented on pull request #3492: URL: https://github.com/apache/hbase/pull/3492#issuecomment-882024930 Checkstyles and whitespace, once fixed, will merge all PRs of HBASE-25986 together. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
