[GitHub] [hbase] bsglz commented on a change in pull request #1737: HBASE-24382 Flush partial stores of region filtered by seqId when arc…
bsglz commented on a change in pull request #1737:
URL: https://github.com/apache/hbase/pull/1737#discussion_r432269012
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -639,26 +641,30 @@ public int getNumLogFiles() {
* If the number of un-archived WAL files ('live' WALs) is greater than
maximum allowed,
* check the first (oldest) WAL, and return those regions which should be
flushed so that
* it can be let-go/'archived'.
- * @return regions (encodedRegionNames) to flush in order to archive oldest
WAL file.
+ * @return stores of regions (encodedRegionNames) to flush in order to
archive oldest WAL file.
*/
- byte[][] findRegionsToForceFlush() throws IOException {
-byte[][] regions = null;
+ Map> findRegionsToForceFlush() throws IOException {
+Map> regions = null;
int logCount = getNumRolledLogFiles();
if (logCount > this.maxLogs && logCount > 0) {
Map.Entry firstWALEntry =
this.walFile2Props.firstEntry();
regions =
this.sequenceIdAccounting.findLower(firstWALEntry.getValue().encodedName2HighestSequenceId);
}
if (regions != null) {
- StringBuilder sb = new StringBuilder();
- for (int i = 0; i < regions.length; i++) {
-if (i > 0) {
- sb.append(", ");
+ List listForPrint = new ArrayList();
+ for (Map.Entry> r : regions.entrySet()) {
+StringBuffer families = new StringBuffer();
+for (int i = 0; i < r.getValue().size(); i++) {
+ if (i > 0) {
+families.append("|");
+ }
+ families.append(Bytes.toString(r.getValue().get(i)));
}
-sb.append(Bytes.toStringBinary(regions[i]));
+listForPrint.add(Bytes.toStringBinary(r.getKey()) + "[" +
families.toString() + "]");
}
LOG.info("Too many WALs; count=" + logCount + ", max=" + this.maxLogs +
-"; forcing flush of " + regions.length + " regions(s): " +
sb.toString());
+"; forcing flush partial stores of " + regions.size() + " region(s): "
+ StringUtils.join(",", listForPrint));
Review comment:
Will fix.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -639,26 +641,30 @@ public int getNumLogFiles() {
* If the number of un-archived WAL files ('live' WALs) is greater than
maximum allowed,
* check the first (oldest) WAL, and return those regions which should be
flushed so that
* it can be let-go/'archived'.
- * @return regions (encodedRegionNames) to flush in order to archive oldest
WAL file.
+ * @return stores of regions (encodedRegionNames) to flush in order to
archive oldest WAL file.
*/
- byte[][] findRegionsToForceFlush() throws IOException {
-byte[][] regions = null;
+ Map> findRegionsToForceFlush() throws IOException {
+Map> regions = null;
int logCount = getNumRolledLogFiles();
if (logCount > this.maxLogs && logCount > 0) {
Map.Entry firstWALEntry =
this.walFile2Props.firstEntry();
regions =
this.sequenceIdAccounting.findLower(firstWALEntry.getValue().encodedName2HighestSequenceId);
}
if (regions != null) {
- StringBuilder sb = new StringBuilder();
- for (int i = 0; i < regions.length; i++) {
-if (i > 0) {
- sb.append(", ");
+ List listForPrint = new ArrayList();
+ for (Map.Entry> r : regions.entrySet()) {
+StringBuffer families = new StringBuffer();
+for (int i = 0; i < r.getValue().size(); i++) {
+ if (i > 0) {
+families.append("|");
Review comment:
Will fix
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #1737: HBASE-24382 Flush partial stores of region filtered by seqId when arc…
bsglz commented on a change in pull request #1737:
URL: https://github.com/apache/hbase/pull/1737#discussion_r432268692
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
##
@@ -639,26 +641,30 @@ public int getNumLogFiles() {
* If the number of un-archived WAL files ('live' WALs) is greater than
maximum allowed,
* check the first (oldest) WAL, and return those regions which should be
flushed so that
* it can be let-go/'archived'.
- * @return regions (encodedRegionNames) to flush in order to archive oldest
WAL file.
+ * @return stores of regions (encodedRegionNames) to flush in order to
archive oldest WAL file.
*/
- byte[][] findRegionsToForceFlush() throws IOException {
-byte[][] regions = null;
+ Map> findRegionsToForceFlush() throws IOException {
+Map> regions = null;
int logCount = getNumRolledLogFiles();
if (logCount > this.maxLogs && logCount > 0) {
Map.Entry firstWALEntry =
this.walFile2Props.firstEntry();
regions =
this.sequenceIdAccounting.findLower(firstWALEntry.getValue().encodedName2HighestSequenceId);
}
if (regions != null) {
- StringBuilder sb = new StringBuilder();
- for (int i = 0; i < regions.length; i++) {
-if (i > 0) {
- sb.append(", ");
+ List listForPrint = new ArrayList();
+ for (Map.Entry> r : regions.entrySet()) {
+StringBuffer families = new StringBuffer();
+for (int i = 0; i < r.getValue().size(); i++) {
+ if (i > 0) {
+families.append("|");
Review comment:
will fix
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #1737: HBASE-24382 Flush partial stores of region filtered by seqId when arc…
bsglz commented on a change in pull request #1737:
URL: https://github.com/apache/hbase/pull/1737#discussion_r432267965
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
##
@@ -459,13 +459,18 @@ private FlushType isAboveLowWaterMark() {
}
@Override
- public boolean requestFlush(HRegion r, boolean forceFlushAllStores,
- FlushLifeCycleTracker tracker) {
+ public boolean requestFlush(HRegion r, boolean
forceFlushAllStores,FlushLifeCycleTracker tracker) {
Review comment:
Ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #1737: HBASE-24382 Flush partial stores of region filtered by seqId when arc…
bsglz commented on a change in pull request #1737:
URL: https://github.com/apache/hbase/pull/1737#discussion_r432267596
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
##
@@ -2444,8 +2451,16 @@ public FlushResultImpl flushcache(boolean
forceFlushAllStores, boolean writeFlus
}
try {
-Collection specificStoresToFlush =
+Collection specificStoresToFlush = null;
+if (!forceFlushAllStores && families != null) {
+ specificStoresToFlush = new ArrayList<>();
+ for (byte[] family : families) {
+specificStoresToFlush.add(stores.get(family));
+ }
+}else{
Review comment:
Sorry for the careless.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-24389) Introduce new master rpc methods to locate meta region through root region
[ https://issues.apache.org/jira/browse/HBASE-24389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-24389: -- Summary: Introduce new master rpc methods to locate meta region through root region (was: Introduce a new master rpc service to locate meta region through root region) > Introduce new master rpc methods to locate meta region through root region > -- > > Key: HBASE-24389 > URL: https://issues.apache.org/jira/browse/HBASE-24389 > Project: HBase > Issue Type: Sub-task > Components: master, meta >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > And also reimlement the client logic to use the new api to locate meta > region. In general, this will remove the assumption at client side that there > is only one meta region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-24390) Remove RegionInfoBuilder.FIRST_META_REGIONINFO
[ https://issues.apache.org/jira/browse/HBASE-24390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24390 started by Duo Zhang. - > Remove RegionInfoBuilder.FIRST_META_REGIONINFO > -- > > Key: HBASE-24390 > URL: https://issues.apache.org/jira/browse/HBASE-24390 > Project: HBase > Issue Type: Sub-task > Components: meta >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > This is very strong assumption that there is only one meta region, we should > cleanup all its usage and remove it from our code base. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24390) Remove RegionInfoBuilder.FIRST_META_REGIONINFO
[ https://issues.apache.org/jira/browse/HBASE-24390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-24390: - Assignee: Duo Zhang > Remove RegionInfoBuilder.FIRST_META_REGIONINFO > -- > > Key: HBASE-24390 > URL: https://issues.apache.org/jira/browse/HBASE-24390 > Project: HBase > Issue Type: Sub-task > Components: meta >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > This is very strong assumption that there is only one meta region, we should > cleanup all its usage and remove it from our code base. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1801: [HBASE-24441]CacheConfig details logged at Store open is not really u…
Apache-HBase commented on pull request #1801: URL: https://github.com/apache/hbase/pull/1801#issuecomment-635745648 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | checkstyle | 1m 12s | master passed | | +1 :green_heart: | spotbugs | 2m 7s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 22s | the patch passed | | +1 :green_heart: | checkstyle | 1m 5s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 28s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 44s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 35m 19s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1801/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1801 | | JIRA Issue | HBASE-24441 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 5de23a81255e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / fe1fc25fba | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1801/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24468) Add region info when log meessages in HStore.
song XinCun created HBASE-24468:
---
Summary: Add region info when log meessages in HStore.
Key: HBASE-24468
URL: https://issues.apache.org/jira/browse/HBASE-24468
Project: HBase
Issue Type: Improvement
Components: logging, regionserver
Affects Versions: 3.0.0-alpha-1
Reporter: song XinCun
Assignee: song XinCun
Region info message is need to added when HStore is created
{code:java}
2020-05-28 21:39:55,515 INFO [StoreOpener-84d6181342585332d77c9e4ee2f9ef19-1]
regionserver.HStore: Store=cf, memstore type=DefaultMemStore,
storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50,
encoding=DIFF, compression=SNAPPY{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24453) BackPort HBASE-20220 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-24453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119255#comment-17119255 ] Reid Chan commented on HBASE-24453: --- Thanks for the contribution. > BackPort HBASE-20220 to branch-1 > > > Key: HBASE-24453 > URL: https://issues.apache.org/jira/browse/HBASE-24453 > Project: HBase > Issue Type: Bug > Components: rsgroup >Reporter: wenbang >Assignee: wenbang >Priority: Minor > Fix For: 1.7.0 > > > Backport Jira : HBASE-20220 to branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24441) CacheConfig details logged at Store open is not really useful
[
https://issues.apache.org/jira/browse/HBASE-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
song XinCun updated HBASE-24441:
Affects Version/s: (was: 2.2.5)
3.0.0-alpha-1
> CacheConfig details logged at Store open is not really useful
> -
>
> Key: HBASE-24441
> URL: https://issues.apache.org/jira/browse/HBASE-24441
> Project: HBase
> Issue Type: Improvement
> Components: logging, regionserver
>Affects Versions: 3.0.0-alpha-1
>Reporter: Anoop Sam John
>Assignee: song XinCun
>Priority: Minor
>
> CacheConfig constructor is logging 'this' object at INFO level. This log
> comes during Store open(As CacheConfig instance for that store is created).
> As the log is at CacheConfig only, we don't get to know this is for which
> region:store. So not really useful log.
> {code}
> blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@7bc02941,
> cacheDataOnRead=true, cacheDataOnWrite=true, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> {code}
> Also during every compaction also this logs keeps coming. This is because
> during compaction we create new CacheConfig based on the HStore level
> CacheConfig object. We can avoid this log with every compaction happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24441) CacheConfig details logged at Store open is not really useful
[
https://issues.apache.org/jira/browse/HBASE-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
song XinCun updated HBASE-24441:
Affects Version/s: 2.2.5
> CacheConfig details logged at Store open is not really useful
> -
>
> Key: HBASE-24441
> URL: https://issues.apache.org/jira/browse/HBASE-24441
> Project: HBase
> Issue Type: Improvement
> Components: logging, regionserver
>Affects Versions: 2.2.5
>Reporter: Anoop Sam John
>Assignee: song XinCun
>Priority: Minor
>
> CacheConfig constructor is logging 'this' object at INFO level. This log
> comes during Store open(As CacheConfig instance for that store is created).
> As the log is at CacheConfig only, we don't get to know this is for which
> region:store. So not really useful log.
> {code}
> blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@7bc02941,
> cacheDataOnRead=true, cacheDataOnWrite=true, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> {code}
> Also during every compaction also this logs keeps coming. This is because
> during compaction we create new CacheConfig based on the HStore level
> CacheConfig object. We can avoid this log with every compaction happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24441) CacheConfig details logged at Store open is not really useful
[
https://issues.apache.org/jira/browse/HBASE-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
song XinCun updated HBASE-24441:
Component/s: regionserver
logging
> CacheConfig details logged at Store open is not really useful
> -
>
> Key: HBASE-24441
> URL: https://issues.apache.org/jira/browse/HBASE-24441
> Project: HBase
> Issue Type: Improvement
> Components: logging, regionserver
>Reporter: Anoop Sam John
>Assignee: song XinCun
>Priority: Minor
>
> CacheConfig constructor is logging 'this' object at INFO level. This log
> comes during Store open(As CacheConfig instance for that store is created).
> As the log is at CacheConfig only, we don't get to know this is for which
> region:store. So not really useful log.
> {code}
> blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@7bc02941,
> cacheDataOnRead=true, cacheDataOnWrite=true, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> {code}
> Also during every compaction also this logs keeps coming. This is because
> during compaction we create new CacheConfig based on the HStore level
> CacheConfig object. We can avoid this log with every compaction happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24441) CacheConfig details logged at Store open is not really useful
[
https://issues.apache.org/jira/browse/HBASE-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119247#comment-17119247
]
song XinCun commented on HBASE-24441:
-
Log after patch would look like:
{code:java}
2020-05-28 21:35:46,842 INFO [StoreOpener-3e735faebf71bb030f365fac2427f7bd-1]
hfile.CacheConfig: Created cacheConfig: cacheDataOnRead=true,
cacheDataOnWrite=true, cacheIndexesOnWrite=true, cacheBloomsOnWrite=true,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false for
family {NAME => 'cf', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false',
NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE',
CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'DIFF', TTL =>
'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW',
CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE =>
'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'SNAPPY',
BLOCKCACHE => 'true', BLOCKSIZE => '65536'} in region
table_100B_10,user5246,1590642270538.3e735faebf71bb030f365fac2427f7bd. with
blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@47c0732b
{code}
The region info which contains table and region name is added to the log.
> CacheConfig details logged at Store open is not really useful
> -
>
> Key: HBASE-24441
> URL: https://issues.apache.org/jira/browse/HBASE-24441
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: song XinCun
>Priority: Minor
>
> CacheConfig constructor is logging 'this' object at INFO level. This log
> comes during Store open(As CacheConfig instance for that store is created).
> As the log is at CacheConfig only, we don't get to know this is for which
> region:store. So not really useful log.
> {code}
> blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@7bc02941,
> cacheDataOnRead=true, cacheDataOnWrite=true, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> {code}
> Also during every compaction also this logs keeps coming. This is because
> during compaction we create new CacheConfig based on the HStore level
> CacheConfig object. We can avoid this log with every compaction happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24453) BackPort HBASE-20220 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-24453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan resolved HBASE-24453. --- Hadoop Flags: Reviewed Resolution: Fixed > BackPort HBASE-20220 to branch-1 > > > Key: HBASE-24453 > URL: https://issues.apache.org/jira/browse/HBASE-24453 > Project: HBase > Issue Type: Bug >Reporter: wenbang >Assignee: wenbang >Priority: Minor > Fix For: 1.7.0 > > > Backport Jira : HBASE-20220 to branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24453) BackPort HBASE-20220 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-24453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-24453: -- Component/s: rsgroup > BackPort HBASE-20220 to branch-1 > > > Key: HBASE-24453 > URL: https://issues.apache.org/jira/browse/HBASE-24453 > Project: HBase > Issue Type: Bug > Components: rsgroup >Reporter: wenbang >Assignee: wenbang >Priority: Minor > Fix For: 1.7.0 > > > Backport Jira : HBASE-20220 to branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24453) BackPort HBASE-20220 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-24453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-24453: -- Fix Version/s: 1.7.0 > BackPort HBASE-20220 to branch-1 > > > Key: HBASE-24453 > URL: https://issues.apache.org/jira/browse/HBASE-24453 > Project: HBase > Issue Type: Bug >Reporter: wenbang >Assignee: wenbang >Priority: Minor > Fix For: 1.7.0 > > > Backport Jira : HBASE-20220 to branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Reidddddd merged pull request #1798: HBASE-24453 BackPort HBASE-20220 to branch-1
Reidd merged pull request #1798: URL: https://github.com/apache/hbase/pull/1798 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #1797: HBASE-24451 Remove the HasThread because the related bug had been fix…
bsglz commented on a change in pull request #1797:
URL: https://github.com/apache/hbase/pull/1797#discussion_r432238067
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -1908,9 +1907,9 @@ protected void queueAssign(final RegionStateNode
regionNode) {
}
private void startAssignmentThread() {
-// Get Server Thread name. Sometimes the Server is mocked so may not
implement HasThread.
+// Get Server Thread name. Sometimes the Server is mocked so may not
implement Thread.
// For example, in tests.
-String name = master instanceof HasThread? ((HasThread)master).getName():
+String name = master instanceof Thread? ((Thread)master).getName():
Review comment:
Ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #1797: HBASE-24451 Remove the HasThread because the related bug had been fix…
bsglz commented on a change in pull request #1797:
URL: https://github.com/apache/hbase/pull/1797#discussion_r432238002
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
##
@@ -122,9 +121,9 @@ public SplitLogManager(MasterServices master, Configuration
conf)
throws IOException {
this.server = master;
this.conf = conf;
-// Get Server Thread name. Sometimes the Server is mocked so may not
implement HasThread.
+// Get Server Thread name. Sometimes the Server is mocked so may not
implement Thread.
// For example, in tests.
-String name = master instanceof HasThread? ((HasThread)master).getName():
+String name = master instanceof Thread? ((Thread)master).getName():
Review comment:
Seems still needed, since the master here maybe a mocked one.
eg: DummyMasterServices in TestSplitLogManager, it extends
MockNoopMasterServices which implement MasterServices, but does not extends
Thread.
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1800: HBASE-22287 inifinite retries on failed server in RSProcedureDispatcher
Apache-HBase commented on pull request #1800: URL: https://github.com/apache/hbase/pull/1800#issuecomment-635735653 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 47s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 53s | branch-2 passed | | +1 :green_heart: | compile | 0m 56s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 0s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 22s | the patch passed | | +1 :green_heart: | compile | 0m 54s | the patch passed | | +1 :green_heart: | javac | 0m 54s | the patch passed | | +1 :green_heart: | shadedjars | 5m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 151m 40s | hbase-server in the patch passed. | | | | 175m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1800 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 40f73a10bb48 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 60c125367b | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/testReport/ | | Max. process+thread count | 3312 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] songxincun opened a new pull request #1801: [HBASE-24441]CacheConfig details logged at Store open is not really u…
songxincun opened a new pull request #1801: URL: https://github.com/apache/hbase/pull/1801 …seful This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1800: HBASE-22287 inifinite retries on failed server in RSProcedureDispatcher
Apache-HBase commented on pull request #1800: URL: https://github.com/apache/hbase/pull/1800#issuecomment-635733901 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 31s | branch-2 passed | | +1 :green_heart: | compile | 1m 7s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 58s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in branch-2 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 6m 43s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 50s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 141m 14s | hbase-server in the patch passed. | | | | 168m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1800 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 30df66b963b1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 60c125367b | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/testReport/ | | Max. process+thread count | 3471 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-23340) hmaster /hbase/replication/rs session expired (hbase replication default value is true, we don't use ) causes logcleaner can not clean oldWALs, which resulits in old
[ https://issues.apache.org/jira/browse/HBASE-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119233#comment-17119233 ] Chao Wang commented on HBASE-23340: --- hello , I find this issue in my environment .I think you can add trying for zk.this is not a simple try, and then close zk which session is expired, new a zk for trying. > hmaster /hbase/replication/rs session expired (hbase replication default > value is true, we don't use ) causes logcleaner can not clean oldWALs, which > resulits in oldWALs too large (more than 2TB) > - > > Key: HBASE-23340 > URL: https://issues.apache.org/jira/browse/HBASE-23340 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: jackylau >Assignee: jackylau >Priority: Major > Fix For: master > > Attachments: Snipaste_2019-11-21_10-39-25.png, > Snipaste_2019-11-21_14-10-36.png > > > hmaster /hbase/replication/rs session expired (hbase replication default > value is true, we don't use ) causes logcleaner can not clean oldWALs, which > resulits in oldWALs too large (more than 2TB). > !Snipaste_2019-11-21_10-39-25.png! > > !Snipaste_2019-11-21_14-10-36.png! > > we can solve it by following : > 1) increase the session timeout(but i think it is not a good idea. because we > do not know how long to set is suitable) > 2) close the hbase replication. It is not a good idea too, when our user uses > this feature > 3) we need add retry times, for example when it has already happened three > times, we set the ReplicationLogCleaner and SnapShotCleaner stop > that is all my ideas, i do not konw it is suitable, If it is suitable, could > i commit a PR? > Does anynode have a good idea. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache-HBase commented on pull request #1774: URL: https://github.com/apache/hbase/pull/1774#issuecomment-635731093 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 1s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 35s | master passed | | +1 :green_heart: | checkstyle | 2m 6s | master passed | | +1 :green_heart: | spotbugs | 6m 31s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | +1 :green_heart: | checkstyle | 0m 10s | The patch passed checkstyle in hbase-protocol-shaded | | +1 :green_heart: | checkstyle | 0m 33s | hbase-client: The patch generated 0 new + 242 unchanged - 2 fixed = 242 total (was 244) | | +1 :green_heart: | checkstyle | 0m 13s | The patch passed checkstyle in hbase-zookeeper | | +1 :green_heart: | checkstyle | 1m 12s | hbase-server: The patch generated 0 new + 453 unchanged - 5 fixed = 453 total (was 458) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 10m 53s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 2m 15s | the patch passed | | +1 :green_heart: | spotbugs | 7m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 47s | The patch does not generate ASF License warnings. | | | | 48m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1774 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool | | uname | Linux 9a50a9bd3b5f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / fe1fc25fba | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/10/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24438) Stale ServerCrashProcedure task in HBase Master UI
[
https://issues.apache.org/jira/browse/HBASE-24438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119226#comment-17119226
]
Andrey Elenskiy commented on HBASE-24438:
-
The only way I was able to reproduce this is by restarting master right after
it handled SCP for a regionserver. Looking at the code I've found this function
in ServerCrashProcedure.java that handles setting of the task:
{code:java}
void updateProgress(boolean updateState) {
String msg = "Processing ServerCrashProcedure of " + serverName;
if (status == null) {
status = TaskMonitor.get().createStatus(msg);
return;
}
if (currentRunningState == ServerCrashState.SERVER_CRASH_FINISH) {
status.markComplete(msg + " done");
return;
}
if (updateState) {
currentRunningState = getCurrentState();
}
int childrenLatch = getChildrenLatch();
status.setStatus(msg + " current State " + currentRunningState
+ (childrenLatch > 0 ? "; remaining num of running child procedures = "
+ childrenLatch
: ""));
}
{code}
Given that the "status" part of the MonitoredTask says "null" (case see that UI
just shows "since" for Status), it means that updateProgress was called only
once. Then looking at the places where the updateProgress can be called are:
* executeFromState() called by ProcedureExecutor and cannot be the one as SCP
is not listed in Procedures & Locks tab and never actually "Started"
* TransitRegionStateProcedure.confirmOpened calls
ServerCrashProcedure.updateProgress and also cannot be the one as SCP is not in
procedures list
* deserializeStateData() is called when the procedure WAL is de-serialized and
is likely responsible for this "stale" task
The procedure never materializes because it has actually been completed by
previous master, so it just gets de-serialized and thrown away by the new
master. Does that sounds like a possible case?
It looks like that side effect was indeed added to deserializeStateData by
https://issues.apache.org/jira/browse/HBASE-21647 but unfortunately I don't see
any comments explaining the reason. Any ideas?
> Stale ServerCrashProcedure task in HBase Master UI
> --
>
> Key: HBASE-24438
> URL: https://issues.apache.org/jira/browse/HBASE-24438
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 2.2.4
> Environment: HBase 2.2.4
> HDFS 3.1.3 with erasure coding enabled
> Kubernetes
>Reporter: Andrey Elenskiy
>Priority: Major
>
> Tasks section (show non-RPC Tasks) in HBase Master UI has stale entries with
> ServerCrashProcedure after master failover. The procedures have finished with
> SUCCESS on a previously active HBase master and aren't showing in "Procedures
> & Locks".
> Based on the logs, both of those regionserver were carrying hbase:meta (logs
> are sorted newest first grepped for those specific servers that have stale
> ServerCrashProcedures):
> {noformat}
> 2020-05-21 19:04:09,176 INFO [KeepAlivePEWorker-28]
> procedure2.ProcedureExecutor: Finished pid=38, state=SUCCESS;
> ServerCrashProcedure
> server=regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347,
> splitWal=true, meta=true in 2.5290sec
> 2020-05-21 19:04:08,962 INFO [KeepAlivePEWorker-28]
> procedure.ServerCrashProcedure: removed crashed server
> regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347 after
> splitting done
> 2020-05-21 19:04:08,747 INFO [KeepAlivePEWorker-28] master.SplitLogManager:
> dead splitlog workers
> [regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347]
> 2020-05-21 19:04:08,746 INFO [KeepAlivePEWorker-28] master.MasterWalManager:
> Log dir for server
> regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347 does not
> exist
> 2020-05-21 19:04:08,636 INFO [KeepAlivePEWorker-28]
> procedure.ServerCrashProcedure:
> regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347 had 0 regions
> 2020-05-21 19:04:08,529 INFO [KeepAlivePEWorker-28]
> procedure.ServerCrashProcedure: pid=38,
> state=RUNNABLE:SERVER_CRASH_ASSIGN_META, locked=true; ServerCrashProcedure
> server=regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347,
> splitWal=true, meta=true found RIT pid=20, ppid=18,
> state=RUNNABLE:REGION_STATE_TRANSITION_OPEN, locked=true;
> TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN;
> rit=ABNORMALLY_CLOSED,
> location=regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347,
> table=hbase:meta, region=1588230740
> 2020-05-21 19:04:08,422 INFO [KeepAlivePEWorker-28] master.SplitLogManager:
> Finished splitting (more than or equal to) 0 (0 bytes) in 0 log files in
> [hdfs://aeris/hbase/WALs/regionserver-1.hbase.hbase.svc.cluster.local,16020,1590087787347-splitting]
> in 0ms
[GitHub] [hbase] bsglz commented on pull request #1799: HBASE-24455 Correct the doc of "On the number of column families"
bsglz commented on pull request #1799: URL: https://github.com/apache/hbase/pull/1799#issuecomment-635723950 > Should we clearly say, then, that, although triggered at table/region level compactions happen per store? IMO, it is unnessary since folks can get it in other chapters, WDYT? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ddupg commented on a change in pull request #1782: HBASE-24431 RSGroupInfo add configuration map to store something extra
ddupg commented on a change in pull request #1782:
URL: https://github.com/apache/hbase/pull/1782#discussion_r432225834
##
File path:
hbase-protocol-shaded/src/main/protobuf/server/rsgroup/RSGroupAdmin.proto
##
@@ -183,4 +191,7 @@ service RSGroupAdminService {
rpc RenameRSGroup(RenameRSGroupRequest)
returns (RenameRSGroupResponse);
+
Review comment:
Didn't notice RSGroupAdminServiceImpl was deprecated, let me remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24436) The store file open and close thread pool should be shared at the region level
[ https://issues.apache.org/jira/browse/HBASE-24436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119205#comment-17119205 ] Junhong Xu commented on HBASE-24436: bq. And how much slower open a store which has 20 storefiles VS open a store which has 1 storefile? Give some metric or data to show the benefit of this improvement? I am preparing for the data.Although we have reached no agreement on the solution itself so far, the data is very important > The store file open and close thread pool should be shared at the region level > -- > > Key: HBASE-24436 > URL: https://issues.apache.org/jira/browse/HBASE-24436 > Project: HBase > Issue Type: Improvement >Reporter: Junhong Xu >Assignee: Junhong Xu >Priority: Minor > > For now, we provide threads per column family evenly in general, but there > are some cases that some column families have much more store files than > others( maybe that's the life, right? ). So in that case, some Stores have > beed done quickly while others are struggling.We should share the thread pool > at the region level in case of data skew. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #1753: HBASE-24408 Introduce a general 'local region' to store data on master
Apache9 commented on a change in pull request #1753:
URL: https://github.com/apache/hbase/pull/1753#discussion_r432215118
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/store/LocalStore.java
##
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.store;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.Server;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.regionserver.HRegion.FlushResult;
+import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Used for storing data at master side. The data will be stored in a {@link
LocalRegion}.
+ */
+@InterfaceAudience.Private
+public final class LocalStore {
+
+ // Use the character $ to let the log cleaner know that this is not the
normal wal file.
+ public static final String ARCHIVED_WAL_SUFFIX = "$masterlocalwal$";
+
+ public static final String ARCHIVED_HFILE_SUFFIX = "$masterlocalhfile$";
+
+ private static final String MAX_WALS_KEY =
"hbase.master.store.region.maxwals";
+
+ private static final int DEFAULT_MAX_WALS = 10;
+
+ public static final String USE_HSYNC_KEY =
"hbase.master.store.region.wal.hsync";
+
+ public static final String MASTER_STORE_DIR = "MasterData";
+
+ private static final String FLUSH_SIZE_KEY =
"hbase.master.store.region.flush.size";
+
+ private static final long DEFAULT_FLUSH_SIZE =
TableDescriptorBuilder.DEFAULT_MEMSTORE_FLUSH_SIZE;
+
+ private static final String FLUSH_PER_CHANGES_KEY =
"hbase.master.store.region.flush.per.changes";
+
+ private static final long DEFAULT_FLUSH_PER_CHANGES = 1_000_000;
+
+ private static final String FLUSH_INTERVAL_MS_KEY =
"hbase.master.store.region.flush.interval.ms";
+
+ // default to flush every 15 minutes, for safety
+ private static final long DEFAULT_FLUSH_INTERVAL_MS =
TimeUnit.MINUTES.toMillis(15);
+
+ private static final String COMPACT_MIN_KEY =
"hbase.master.store.region.compact.min";
+
+ private static final int DEFAULT_COMPACT_MIN = 4;
+
+ private static final String ROLL_PERIOD_MS_KEY =
"hbase.master.store.region.walroll.period.ms";
+
+ private static final long DEFAULT_ROLL_PERIOD_MS =
TimeUnit.MINUTES.toMillis(15);
+
+ private static final String RING_BUFFER_SLOT_COUNT =
"hbase.master.store.ringbuffer.slot.count";
+
+ private static final int DEFAULT_RING_BUFFER_SLOT_COUNT = 128;
+
+ public static final TableName TABLE_NAME = TableName.valueOf("master:store");
+
+ public static final byte[] PROC_FAMILY = Bytes.toBytes("proc");
+
+ private static final TableDescriptor TABLE_DESC =
TableDescriptorBuilder.newBuilder(TABLE_NAME)
+.setColumnFamily(ColumnFamilyDescriptorBuilder.of(PROC_FAMILY)).build();
Review comment:
Use their own column family. The problems of introduing more local
regions are:
1. Each region would have its own fail recovery logic, which means we need
to do lease recovery multiple times when starting master.
2. Maybe we need to duplicate all the configuations here multiple times for
supporting different type of local regions, which is a pain for operators.
3. Maybe we need to introduce more types of HFileCleaner and WALCleaner,
which is also a pain.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sguggilam commented on a change in pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
sguggilam commented on a change in pull request #1755:
URL: https://github.com/apache/hbase/pull/1755#discussion_r432207528
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
##
@@ -2079,16 +2085,29 @@ private void unassign(final HRegionInfo region,
}
if (logRetries) {
- LOG.info("Server " + server + " returned " + t + " for "
-+ region.getRegionNameAsString() + ", try=" + i
-+ " of " + this.maximumAttempts, t);
+ LOG.info("Server " + server + " returned " + t + " for " +
region.getRegionNameAsString()
+ + ", try=" + i + " of " + this.maximumAttempts,
+t);
// Presume retry or server will expire.
}
}
}
-// Run out of attempts
-if (state != null) {
- regionStates.updateRegionState(region, State.FAILED_CLOSE);
+
+long sleepTime = backoffPolicy.getBackoffTime(retryConfig,
Review comment:
The current code does the sleep in the current thread where it is
handling the region already in transition kind of case where there is a max
wait time and hence they just sleep for every 100ms and keep retrying before
the max ttime ends. In our case we definitely know that we don't want to retry
before a configured backoff time, so we can as well schedule it later
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sguggilam commented on a change in pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
sguggilam commented on a change in pull request #1755:
URL: https://github.com/apache/hbase/pull/1755#discussion_r432203290
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
##
@@ -2079,16 +2085,29 @@ private void unassign(final HRegionInfo region,
}
if (logRetries) {
- LOG.info("Server " + server + " returned " + t + " for "
-+ region.getRegionNameAsString() + ", try=" + i
-+ " of " + this.maximumAttempts, t);
+ LOG.info("Server " + server + " returned " + t + " for " +
region.getRegionNameAsString()
+ + ", try=" + i + " of " + this.maximumAttempts,
+t);
// Presume retry or server will expire.
}
}
}
-// Run out of attempts
-if (state != null) {
- regionStates.updateRegionState(region, State.FAILED_CLOSE);
+
+long sleepTime = backoffPolicy.getBackoffTime(retryConfig,
Review comment:
There are two reasons for this:
1. It also deals with the case where the sendRegionClose() itself returns
false without getting into the catch block though I am not very sure of the
case where it would reach there. the method doc says the method returns false
if it doesn't acknowledge the request
2. The exponential backoff is configurable and can even be 5 minutes , so
there is no point holding the thread for that amount of time and rather have
it deal asynchronously.
I cannot think of any state machine issue as even if just sleep the current
thread , the state would remain the same and the method as such is not
returning anything and anyone else would just check the state change made by
the method either synchronously or asynchronously. Rather we would delegate it
to a new thread and not block the current thread.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1800: HBASE-22287 inifinite retries on failed server in RSProcedureDispatcher
Apache-HBase commented on pull request #1800: URL: https://github.com/apache/hbase/pull/1800#issuecomment-635697137 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 9s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 0s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 11s | the patch passed | | -0 :warning: | checkstyle | 1m 6s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 43s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 33m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1800 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 0a0763638910 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 60c125367b | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1800/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sguggilam commented on a change in pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
sguggilam commented on a change in pull request #1755:
URL: https://github.com/apache/hbase/pull/1755#discussion_r432203290
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
##
@@ -2079,16 +2085,29 @@ private void unassign(final HRegionInfo region,
}
if (logRetries) {
- LOG.info("Server " + server + " returned " + t + " for "
-+ region.getRegionNameAsString() + ", try=" + i
-+ " of " + this.maximumAttempts, t);
+ LOG.info("Server " + server + " returned " + t + " for " +
region.getRegionNameAsString()
+ + ", try=" + i + " of " + this.maximumAttempts,
+t);
// Presume retry or server will expire.
}
}
}
-// Run out of attempts
-if (state != null) {
- regionStates.updateRegionState(region, State.FAILED_CLOSE);
+
+long sleepTime = backoffPolicy.getBackoffTime(retryConfig,
Review comment:
There are two reasons for this:
1. It also deals with the case where the sendRegionClose() itself returns
false without getting into the catch block though I am not very sure of the
case where it would reach there. the method doc says the method returns false
if it doesn't acknowledge the request
2. The current sleepTime in the else of catch block is static at 100ms. The
exponential backoff is configurable and can even be 5 minutes , so there is no
point holding the thread for that amount of time and rather have it deal
asynchronously.
I cannot think of any state machine issue as even if just sleep the current
thread , the state would remain the same and the method as such is not
returning anything and anyone else would just check the state change made by
the method either synchronously or asynchronously. Rather we would delegate it
to a new thread and not block the current thread.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24458) release scripts using docker on mac os should specify needed disk consistency
[ https://issues.apache.org/jira/browse/HBASE-24458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119191#comment-17119191 ] Sean Busbey commented on HBASE-24458: - Here's the docs https://docs.docker.com/docker-for-mac/osxfs-caching/ I think we should assume a single instance and that we only really need to look at the stuff written from the host after the container is done, which sounds like using "delegate" consistency If that doesn't speed up enough we can look at running inside a named volume and rsync to the bind mount at the end, but that will get dicey if we have ugly container deaths. > release scripts using docker on mac os should specify needed disk consistency > - > > Key: HBASE-24458 > URL: https://issues.apache.org/jira/browse/HBASE-24458 > Project: HBase > Issue Type: Improvement >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > reading through existing docs and write ups for Docker Desktop (the dominant > docker option for running on OS X machines), it looks like the > host-to-container mapping get significant perf gains if we can relax from > strict container-host consistency on the filesystem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24467) Backport HBASE-23963: Split TestFromClientSide; it takes too long to complete timing out
[ https://issues.apache.org/jira/browse/HBASE-24467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-24467: -- Assignee: Guanghao Zhang > Backport HBASE-23963: Split TestFromClientSide; it takes too long to complete > timing out > > > Key: HBASE-24467 > URL: https://issues.apache.org/jira/browse/HBASE-24467 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.5 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24389) Introduce a new master rpc service to locate meta region through root region
[
https://issues.apache.org/jira/browse/HBASE-24389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119187#comment-17119187
]
Duo Zhang commented on HBASE-24389:
---
{quote}
We already have client-side Interfaces – Registry or RegionLocator. Discussion
of why these put aside?
{quote}
I'v replied why I do not want to add this to the Registry above, and for
RegionLocator, it is still there and is also used for locating meta. I've also
relied at other places as well, on the PR and also on the design doc. PTAL.
{quote}
I hear the bit about not wanting to expose an API that would be hard to
implement in different contexts but the new API is not how we do location query
up to this. New API seems to dictate a Region-at-a-time; will this work if the
RootTable has your 10k entries? What you thinking?
{quote}
I do not get your point sir, the new API is for dealing with 10k entries in
root table. We always locate a region at a time in the client code, I do not
think it is a good idea to always fetch 10k entries everytime when you only
need one of them? Other things such as prefetching and caching are controlled
by the locator related classes. We have a getAllMetaRegionLocations method, you
are free to call this method in background to pull all the entries to client,
but I do not think this is necessary. In our largest cluster which has about
20k regions, there are thousands of tables but usually a user will only contact
to a small set of tables, and typically under the same namespace, which means,
they will only need to access a single meta region.
Thanks.
> Introduce a new master rpc service to locate meta region through root region
>
>
> Key: HBASE-24389
> URL: https://issues.apache.org/jira/browse/HBASE-24389
> Project: HBase
> Issue Type: Sub-task
> Components: master, meta
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> And also reimlement the client logic to use the new api to locate meta
> region. In general, this will remove the assumption at client side that there
> is only one meta region.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24467) Backport HBASE-23963: Split TestFromClientSide; it takes too long to complete timing out
Guanghao Zhang created HBASE-24467: -- Summary: Backport HBASE-23963: Split TestFromClientSide; it takes too long to complete timing out Key: HBASE-24467 URL: https://issues.apache.org/jira/browse/HBASE-24467 Project: HBase Issue Type: Improvement Affects Versions: 2.2.5 Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24436) The store file open and close thread pool should be shared at the region level
[
https://issues.apache.org/jira/browse/HBASE-24436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119183#comment-17119183
]
Guanghao Zhang commented on HBASE-24436:
{quote}How long open or close only one storefile take?
{quote}
And how much slower open a store which has 20 storefiles VS open a store which
has 1 storefile? Give some metric or data to show the benefit of this
improvement?
> The store file open and close thread pool should be shared at the region level
> --
>
> Key: HBASE-24436
> URL: https://issues.apache.org/jira/browse/HBASE-24436
> Project: HBase
> Issue Type: Improvement
>Reporter: Junhong Xu
>Assignee: Junhong Xu
>Priority: Minor
>
> For now, we provide threads per column family evenly in general, but there
> are some cases that some column families have much more store files than
> others( maybe that's the life, right? ). So in that case, some Stores have
> beed done quickly while others are struggling.We should share the thread pool
> at the region level in case of data skew.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24389) Introduce a new master rpc service to locate meta region through root region
[ https://issues.apache.org/jira/browse/HBASE-24389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119182#comment-17119182 ] Michael Stack commented on HBASE-24389: --- I had a few quests above [~zhangduo]? We already have client-side Interfaces -- Registry or RegionLocator. Discussion of why these put aside? I hear the bit about not wanting to expose an API that would be hard to implement in different contexts but the new API is not how we do location query up to this. New API seems to dictate a Region-at-a-time; will this work if the RootTable has your 10k entries? What you thinking? > Introduce a new master rpc service to locate meta region through root region > > > Key: HBASE-24389 > URL: https://issues.apache.org/jira/browse/HBASE-24389 > Project: HBase > Issue Type: Sub-task > Components: master, meta >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > And also reimlement the client logic to use the new api to locate meta > region. In general, this will remove the assumption at client side that there > is only one meta region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #1800: HBASE-22287 inifinite retries on failed server in RSProcedureDispatcher
Apache9 commented on pull request #1800: URL: https://github.com/apache/hbase/pull/1800#issuecomment-635690562 Maybe better to change the title? It does not solve the infinite retries but to make the retry less aggressive. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#issuecomment-635688358
> Interesting. Wonder why the change was made.
Looks like it was slipped in with
https://issues.apache.org/jira/browse/HBASE-15065. There's some interesting
comments as well from the original author in regard to the plan order, the move
from a single global, optimized plan to the current list of actions ("plans").
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-22287) inifinite retries on failed server in RSProcedureDispatcher
[
https://issues.apache.org/jira/browse/HBASE-22287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119181#comment-17119181
]
Michael Stack commented on HBASE-22287:
---
Put up a patch to add some backoff so we don't fill logs.
If we get into this situation, we don't want to break off retrying. Conditions
need to change first: e.g. ServerCrashProcedure is usual way of how we break
off retrying. In the case cited above, SCP had a hole so the rpc kept on. Bug.
Shouldn't have happened.
> inifinite retries on failed server in RSProcedureDispatcher
> ---
>
> Key: HBASE-22287
> URL: https://issues.apache.org/jira/browse/HBASE-22287
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> We observed this recently on some cluster, I'm still investigating the root
> cause however seems like the retries should have special handling for this
> exception; and separately probably a cap on number of retries
> {noformat}
> 2019-04-20 04:24:27,093 WARN [RSProcedureDispatcher-pool4-t1285]
> procedure.RSProcedureDispatcher: request to server ,17020,1555742560432
> failed due to java.io.IOException: Call to :17020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: :17020, try=26603, retrying...
> {noformat}
> The corresponding worker is stuck
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache9 commented on a change in pull request #1774:
URL: https://github.com/apache/hbase/pull/1774#discussion_r432194231
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
##
@@ -1403,6 +1409,21 @@ private static void deleteFromMetaTable(final Connection
connection, final List<
}
}
+ public static Delete removeRegionReplica(byte[] metaRow, int
replicaIndexToDeleteFrom,
+int numReplicasToRemove) {
+int absoluteIndex = replicaIndexToDeleteFrom + numReplicasToRemove;
+long now = EnvironmentEdgeManager.currentTime();
+Delete deleteReplicaLocations = new Delete(metaRow);
+for (int i = replicaIndexToDeleteFrom; i < absoluteIndex; i++) {
+ deleteReplicaLocations.addColumns(getCatalogFamily(),
getServerColumn(i), now);
+ deleteReplicaLocations.addColumns(getCatalogFamily(),
getSeqNumColumn(i), now);
+ deleteReplicaLocations.addColumns(getCatalogFamily(),
getStartCodeColumn(i), now);
+ deleteReplicaLocations.addColumns(getCatalogFamily(),
getServerNameColumn(i), now);
+ deleteReplicaLocations.addColumns(getCatalogFamily(),
getRegionStateColumn(i), now);
+}
+return deleteReplicaLocations;
+ }
+
Review comment:
No. This is for supportting MasterMetaBootstrap. We do not touch this
class in the previous PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache9 commented on a change in pull request #1774:
URL: https://github.com/apache/hbase/pull/1774#discussion_r432194015
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
##
@@ -2185,4 +2206,27 @@ private static Put addSequenceNum(Put p, long
openSeqNum, int replicaId) throws
.setValue(Bytes.toBytes(openSeqNum))
.build());
}
+
+ private static byte[] buildLocateRegionStartRow(TableName tableName, byte[]
row,
+RegionLocateType locateType) {
+if (locateType.equals(RegionLocateType.BEFORE)) {
+ if (Bytes.equals(row, HConstants.EMPTY_END_ROW)) {
+byte[] binaryTableName = tableName.getName();
+return Arrays.copyOf(binaryTableName, binaryTableName.length + 1);
+ } else {
+return createRegionName(tableName, row, ZEROES, false);
+ }
+} else {
+ return createRegionName(tableName, row, NINES, false);
+}
+ }
+
+ public static Scan createLocateRegionScan(TableName tableName, byte[] row,
+RegionLocateType locateType, int prefetchLimit) {
Review comment:
This is for abstraction, at this layer we do not care whether you are
doing scan or what ever else, The only thing I care is how you want to locate a
region through the given row, i.e, the direction.
And we have this logic, at least for two years after we introduced async
client. If you are talking about the code before async client, the locating
logic is placed everywhere in code, which is not good.
For supporting reverse scan, we have special logic to locate to the previous
region in this class
https://github.com/apache/hbase/blob/branch-2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedScannerCallable.java
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack opened a new pull request #1800: HBASE-22287 inifinite retries on failed server in RSProcedureDispatcher
saintstack opened a new pull request #1800: URL: https://github.com/apache/hbase/pull/1800 Add some backoff. Does not break off retrying. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24347) Hadoop2 profiles are both active when pre-commit PR builds run
[
https://issues.apache.org/jira/browse/HBASE-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119180#comment-17119180
]
Josh Elser commented on HBASE-24347:
We don't have {{jenkins_precommit_github_yetus.sh}} in branch-2.2
(HBASE-23767), so, unless I hear otherwise, I'll be pushing this to branch-2.3
and branch-2 tomorrow.
Then, we can figure out what's going on with branch-2.2 (which I assume is
really busted for testing PR's right now...)
> Hadoop2 profiles are both active when pre-commit PR builds run
> --
>
> Key: HBASE-24347
> URL: https://issues.apache.org/jira/browse/HBASE-24347
> Project: HBase
> Issue Type: Sub-task
> Components: build
>Reporter: Michael Stack
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 2.2.6
>
> Attachments: HBASE-24280.001.branch-2.3.patch,
> HBASE-24280.001.branch-2.patch
>
>
> We need the magic done in the parent out in our precommit builds too. See how
> https://github.com/apache/hbase/pull/1664 fails in hbase-rest w/ complaint
> about jersey; this is a symptom of double hadoop2+hadoop3 profile activation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r432193207
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitPlanFirstComparator.java
##
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.normalizer;
+
+import java.util.Comparator;
+
+/**
+ * Comparator class that gives higher priority to {@link
SplitNormalizationPlan}.
+ */
+class SplitPlanFirstComparator implements Comparator {
Review comment:
Reading through the old patches, i think this is an appendix we can
safely discard.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
bharathv commented on a change in pull request #1755:
URL: https://github.com/apache/hbase/pull/1755#discussion_r432192279
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
##
@@ -2079,16 +2085,29 @@ private void unassign(final HRegionInfo region,
}
if (logRetries) {
- LOG.info("Server " + server + " returned " + t + " for "
-+ region.getRegionNameAsString() + ", try=" + i
-+ " of " + this.maximumAttempts, t);
+ LOG.info("Server " + server + " returned " + t + " for " +
region.getRegionNameAsString()
+ + ", try=" + i + " of " + this.maximumAttempts,
+t);
// Presume retry or server will expire.
}
}
}
-// Run out of attempts
-if (state != null) {
- regionStates.updateRegionState(region, State.FAILED_CLOSE);
+
+long sleepTime = backoffPolicy.getBackoffTime(retryConfig,
Review comment:
Right, so my point is this. If there is an exception in
sendRegionClose() (say ServerTooBusy), we enter the catch() block in L2024, we
don't match the if / else if in L2031 (failed server) / 2038 (region not
serving) and that leaves us with just looping for "maximumAttempts" quickly in
succession without a backoff (this is the problem we are trying to address). My
point was why not do something like this.
```
if (t instanceof RegionServerAbortedException || t instanceof
RegionServerStoppedException
|| t instanceof RegionServerStoppedException
|| t instanceof ServerNotRunningYetException) {
..
} else if ((t instanceof FailedServerException)) {
.
} else {
// Handle any other remote exception
// use your exponential backoff algorithm...
}
```
Basically we have the loop for maxAttempts just that it is not using an
exponential backoff. Just plug it in an else {} block and we are good? What is
advantage of doing it with a delayed callable, doesn't that just complicate the
state machine with async requests and book-keeping or is there something that
I'm missing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] joshelser commented on pull request #1648: HBASE-8458 Support for batch version of checkAndMutate()
joshelser commented on pull request #1648: URL: https://github.com/apache/hbase/pull/1648#issuecomment-635684687 Sorry Toshi! Rough week :) > It looks like there were no test failures in the last QA but it reported the test failure. It seems like something errors other than test failures happened... Looks like the Java process up and died. Can't tell if it was due to code you have changed though https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/3/PR_20JDK8_20Hadoop2_20Check_20Report/ Make sure to take a look at the Hadoop3 report and make sure you aren't introducing more javadoc warnings. I glanced at the one and it was the same number of warnings. A couple of more comments, but overall looks good. @Apache9 do you have interest in giving more feedback here? @busbey on your radar for 3.0.0? @saintstack FYI just as an all-around nice guy ;) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #1648: HBASE-8458 Support for batch version of checkAndMutate()
joshelser commented on a change in pull request #1648:
URL: https://github.com/apache/hbase/pull/1648#discussion_r432189368
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CheckAndMutate.java
##
@@ -0,0 +1,384 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.NavigableMap;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilder;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.io.TimeRange;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.apache.yetus.audience.InterfaceStability;
+import org.apache.hbase.thirdparty.com.google.common.base.Preconditions;
+
+/**
+ * Used to perform CheckAndMutate operations. Currently {@link Put}, {@link
Delete}
+ * and {@link RowMutations} are supported.
+ *
+ * Use the builder class to instantiate a CheckAndMutate object.
+ * This builder class is fluent style APIs, the code are like:
+ *
+ *
+ * // A CheckAndMutate operation where do the specified action if the column
(specified by the
+ * // family and the qualifier) of the row equals to the specified value
+ * CheckAndMutate checkAndMutate = CheckAndMutate.builder(row)
+ * .ifEquals(family, qualifier, value)
+ * .build(put);
+ *
+ * // A CheckAndMutate operation where do the specified action if the column
(specified by the
+ * // family and the qualifier) of the row doesn't exist
+ * CheckAndMutate checkAndMutate = CheckAndMutate.builder(row)
+ * .ifNotExists(family, qualifier)
+ * .build(put);
+ *
+ * // A CheckAndMutate operation where do the specified action if the row
matches the filter
+ * CheckAndMutate checkAndMutate = CheckAndMutate.builder(row)
+ * .ifMatches(filter)
+ * .build(delete);
+ *
+ *
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public final class CheckAndMutate extends Mutation {
+
+ /**
+ * A builder class for building a CheckAndMutate object.
+ */
+ public static final class Builder {
+private final byte[] row;
+private byte[] family;
+private byte[] qualifier;
+private CompareOperator op;
+private byte[] value;
+private Filter filter;
+private TimeRange timeRange;
+
+private Builder(byte[] row) {
+ this.row = Preconditions.checkNotNull(row, "row is null");
+}
+
+/**
+ * Check for lack of column
+ *
+ * @param family family to check
+ * @param qualifier qualifier to check
+ * @return the CheckAndMutate object
+ */
+public Builder ifNotExists(byte[] family, byte[] qualifier) {
+ return ifEquals(family, qualifier, null);
+}
+
+/**
+ * Check for equality
+ *
+ * @param family family to check
+ * @param qualifier qualifier to check
+ * @param value the expected value
+ * @return the CheckAndMutate object
+ */
+public Builder ifEquals(byte[] family, byte[] qualifier, byte[] value) {
+ return ifMatches(family, qualifier, CompareOperator.EQUAL, value);
+}
+
+/**
+ * @param family family to check
+ * @param qualifier qualifier to check
+ * @param compareOp comparison operator to use
+ * @param value the expected value
+ * @return the CheckAndMutate object
+ */
+public Builder ifMatches(byte[] family, byte[] qualifier, CompareOperator
compareOp,
+ byte[] value) {
+ this.family = Preconditions.checkNotNull(family, "family is null");
+ this.qualifier = qualifier;
+ this.op = Preconditions.checkNotNull(compareOp, "compareOp is null");
+ this.value = value;
+ return this;
+}
+
+/**
+ * @param filter filter to check
+ * @return the CheckAndMutate object
+ */
+public Builder ifMatches(Filter filter) {
+ this.filter = Preconditions.checkNotNull(filter, "filter is null");
+ return this;
+}
+
+/**
+ * @param timeRange time range to check
+ * @return the
[GitHub] [hbase] Apache9 commented on a change in pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache9 commented on a change in pull request #1774:
URL: https://github.com/apache/hbase/pull/1774#discussion_r432190127
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaTableRegionLocator.java
##
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import org.apache.hadoop.hbase.AsyncMetaTableAccessor;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.RegionLocations;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.TableNotFoundException;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The class for locating region for table other than meta.
+ */
+@InterfaceAudience.Private
+class AsyncNonMetaTableRegionLocator extends AbstractAsyncTableRegionLocator {
Review comment:
We do have these locators before this patch. Th AsyncRegionLocator is
for the general interface to communicate with upper layer, and in the past, we
have AsyncMetaRegionLocator and AsyncNonMetaRegionLocator, for dealing with
meta table and non meta table.
In this patch, the old stuff in AsyncMetaRegionLocator has all been purged,
and most of the logic in AsyncNonMetaRegionLocator have been moved to
AsyncRegionLocator and AbstractAsyncTableRegionLocator. The latter one is
created for sharing code between meta table and non meta table.
We can put all these logics in a single class, as all class inherits can
also be done by a if in the code. But I think doing this is an anti-pattern in
java, as it will introduce very big classes, which is already a big problem in
HBase(think of the old AsyncProcess class, it is a nightmare to me...).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache9 commented on a change in pull request #1774:
URL: https://github.com/apache/hbase/pull/1774#discussion_r432188121
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractAsyncTableRegionLocator.java
##
@@ -0,0 +1,308 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.client;
+
+import static org.apache.hadoop.hbase.HConstants.EMPTY_END_ROW;
+import static org.apache.hadoop.hbase.client.AsyncRegionLocatorHelper.isGood;
+import static
org.apache.hadoop.hbase.client.ConnectionUtils.createClosestRowAfter;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import org.apache.commons.lang3.ObjectUtils;
+import org.apache.hadoop.hbase.HBaseIOException;
+import org.apache.hadoop.hbase.HRegionLocation;
+import org.apache.hadoop.hbase.RegionLocations;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * The base class for locating region of a table.
+ */
+@InterfaceAudience.Private
+abstract class AbstractAsyncTableRegionLocator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(AbstractAsyncTableRegionLocator.class);
+
+ protected final AsyncConnectionImpl conn;
+
+ protected final TableName tableName;
+
+ protected final int maxConcurrent;
+
+ protected final TableRegionLocationCache cache;
+
+ protected static final class LocateRequest {
Review comment:
Not now, the locateType is introduced at the same time with async
client, for isolating all the locating related logic from upper layer. The
magic is to construct different Scan object, with modified startRow, or
include/exclude the startRow.
We have to support this, one of the most important reason is for supporting
reverse scan, where we need to locate the region before the current region, but
what we have is the start row of the current region, which is not part of the
region before, so we need to use LocateType.BEFORE to get the location.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase-native-client] joshelser commented on pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
joshelser commented on pull request #2: URL: https://github.com/apache/hbase-native-client/pull/2#issuecomment-635679333 In addition to the minicluster automation for testing, shout when you're ready just to get a "real" hbase up and running. I imagine you can figure it out, but we should be around to help as needed with the accumulo->hbase "hump". apache-hbase.slack.com is good, too, if you have a quick question. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24389) Introduce a new master rpc service to locate meta region through root region
[
https://issues.apache.org/jira/browse/HBASE-24389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119170#comment-17119170
]
Duo Zhang commented on HBASE-24389:
---
{quote}
So I think my rephrased question would be, is there a plan to load balance
these across masters to avoid hots potting, would be nice to do that in the
follow up patches.
{quote}
See HBASE-24459, I've already opened an issue. The plan is to abstract an
interface at client side which contains two methods, locateMeta and
getAllMetaRegionLocations, and use it in the AsyncMetaTableRegionLocator. The
current implementation is to ask active master, and we could also implement
something like the MasterRegistry to also ask the backup masters, and in our
deployment, we could implement a simple cache server(maybe on k8s) to let
client ask them. I think this is solution we could all be happy.
> Introduce a new master rpc service to locate meta region through root region
>
>
> Key: HBASE-24389
> URL: https://issues.apache.org/jira/browse/HBASE-24389
> Project: HBase
> Issue Type: Sub-task
> Components: master, meta
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> And also reimlement the client logic to use the new api to locate meta
> region. In general, this will remove the assumption at client side that there
> is only one meta region.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24416) RegionNormalizer spliting region should not be limited by hbase.normalizer.min.region.count
[
https://issues.apache.org/jira/browse/HBASE-24416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119171#comment-17119171
]
Hudson commented on HBASE-24416:
Results for branch branch-2.2
[build #879 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/879/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/879//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/879//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/879//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> RegionNormalizer spliting region should not be limited by
> hbase.normalizer.min.region.count
> ---
>
> Key: HBASE-24416
> URL: https://issues.apache.org/jira/browse/HBASE-24416
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.6
>
>
> In method computePlanForTable of SimpleRegionNormalizer:
> we will skip spliting region if the number of regions in the table is less
> than hbase.normalizer.min.region.count, even if there is a huge region in the
> table.
> {code:java}
> ...
> if (tableRegions == null || tableRegions.size() < minRegionCount) {
> ...
> return null;
> }
> ...
> // get region split plan
> if (splitEnabled) {
> List splitPlans = getSplitNormalizationPlan(table);
> if (splitPlans != null) {
> plans.addAll(splitPlans);
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24466) Release scripts in docker mode should be able to use a named volume for maven repo
Sean Busbey created HBASE-24466: --- Summary: Release scripts in docker mode should be able to use a named volume for maven repo Key: HBASE-24466 URL: https://issues.apache.org/jira/browse/HBASE-24466 Project: HBase Issue Type: Improvement Components: build, community Reporter: Sean Busbey Assignee: Sean Busbey Allow optional volume name to use to mount the maven repo so we get reuse of third party artifacts between runs. Forcefully remove artifacts we build from it (assuming we can't make a read-only one that's shared for third party dependencies and an isolated one that's for what we're building) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-native-client] joshelser commented on a change in pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
joshelser commented on a change in pull request #2:
URL: https://github.com/apache/hbase-native-client/pull/2#discussion_r432186426
##
File path: cmake/DownloadFolly.cmake
##
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+## Download facebook's folly library.
+## SOURCE_DIR is typically the cmake source directory
+## BINARY_DIR is the build directory, typically 'build'
+
+function(download_folly SOURCE_DIR BINARY_DIR)
+
+
+ ExternalProject_Add(
+ facebook-folly-proj
+ GIT_REPOSITORY "https://github.com/facebook/folly.git;
+ GIT_TAG "v2020.05.18.00"
+ SOURCE_DIR "${BINARY_DIR}/dependencies/facebook-folly-proj-src"
+ CMAKE_ARGS ${PASSTHROUGH_CMAKE_ARGS}
Review comment:
> My hope is to leverage an internal java project that runs a mini
cluster and relies on maven to build a jar we can run for integration tests.
This would be great.
https://github.com/apache/hbase/tree/master/hbase-testing-util should have
"enough" of what's needed to get going. Shout when you're ready for this and we
can help. The hardest part will be explicitly listing all of the test scope
dependencies, as maven won't naturally bring them in transitively (super fun --
test-scope deps have no transitive deps).
> If we end up dockerizing this on ubuntu, we may run into this problem in
which case we may have to install fmt dependency like other dependencies, just
FYI.
I think Docker is going to be a "requirement" for the rest of the HBase team
to try to be "helpful". I'm not sure if there are any frameworks out there
which would help us do the same setup+build+test across multiple OS's docker
containers. That said, having just "one" platform which works is more than we
have now :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] joshelser commented on pull request #1794: HBASE-24449: Generate version.h include file during compilation.
joshelser commented on pull request #1794: URL: https://github.com/apache/hbase/pull/1794#issuecomment-635677015 A belated +1 from me. Thanks for your help here, Bharath! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-native-client] joshelser commented on a change in pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
joshelser commented on a change in pull request #2: URL: https://github.com/apache/hbase-native-client/pull/2#discussion_r432184681 ## File path: cmake/DownloadFolly.cmake ## @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +## Download facebook's folly library. +## SOURCE_DIR is typically the cmake source directory +## BINARY_DIR is the build directory, typically 'build' + +function(download_folly SOURCE_DIR BINARY_DIR) + + + ExternalProject_Add( + facebook-folly-proj + GIT_REPOSITORY "https://github.com/facebook/folly.git; + GIT_TAG "v2020.05.18.00" Review comment: > On the other hand bumping versions after such an age gap did require some code resolutions, which I think when the are combined in this PR give perspective. From what I remembered the last time I was in this code, I assumed the version bumps were necessitated to avoid making changes for the sake of _not_ updating folly/wangle (et al) versions. Totally OK if this gets "big" just for the sake of getting back to a good position. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r432182388
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -18,126 +17,436 @@
*/
package org.apache.hadoop.hbase.master.normalizer;
+import java.io.IOException;
+import java.time.Instant;
+import java.time.Period;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
-
-import org.apache.hadoop.hbase.HBaseIOException;
+import java.util.Objects;
+import java.util.function.BooleanSupplier;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.hbase.RegionMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.Size;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.MasterSwitchType;
import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.assignment.RegionStates;
import org.apache.hadoop.hbase.master.normalizer.NormalizationPlan.PlanType;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
/**
* Simple implementation of region normalizer. Logic in use:
*
- * Get all regions of a given table
- * Get avg size S of each region (by total size of store files reported in
RegionMetrics)
- * Seek every single region one by one. If a region R0 is bigger than S *
2, it is kindly
- * requested to split. Thereon evaluate the next region R1
- * Otherwise, if R0 + R1 is smaller than S, R0 and R1 are kindly requested
to merge. Thereon
- * evaluate the next region R2
- * Otherwise, R1 is evaluated
+ * Get all regions of a given table
+ * Get avg size S of the regions in the table (by total size of store
files reported in
+ * RegionMetrics)
+ * For each region R0, if R0 is bigger than S * 2, it is kindly
requested to split.
+ * Otherwise, for the next region in the chain R1, if R0 + R1 is smaller
then S, R0 and R1
+ * are kindly requested to merge.
+ *
+ *
+ * The following parameters are configurable:
+ *
+ * Whether to split a region as part of normalization. Configuration:
+ * {@value SPLIT_ENABLED_KEY}, default: {@value
DEFAULT_SPLIT_ENABLED}.
+ * Whether to merge a region as part of normalization. Configuration:
+ * {@value MERGE_ENABLED_KEY}, default: {@value
DEFAULT_MERGE_ENABLED}.
+ * The minimum number of regions in a table to consider it for
normalization. Configuration:
+ * {@value MIN_REGION_COUNT_KEY}, default: {@value
DEFAULT_MIN_REGION_COUNT}.
+ * The minimum age for a region to be considered for a merge, in days.
Configuration:
+ * {@value MERGE_MIN_REGION_AGE_DAYS_KEY}, default:
+ * {@value DEFAULT_MERGE_MIN_REGION_AGE_DAYS}.
+ * The minimum size for a region to be considered for a merge, in whole
MBs. Configuration:
+ * {@value MERGE_MIN_REGION_SIZE_MB_KEY}, default:
+ * {@value DEFAULT_MERGE_MIN_REGION_SIZE_MB}.
*
*
- * Region sizes are coarse and approximate on the order of megabytes.
Additionally, "empty" regions
- * (less than 1MB, with the previous note) are not merged away. This is by
design to prevent
- * normalization from undoing the pre-splitting of a table.
+ * To see detailed logging of the application of these configuration values,
set the log level for
+ * this class to `TRACE`.
*/
-@InterfaceAudience.Private
-public class SimpleRegionNormalizer extends AbstractRegionNormalizer {
-
+@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.CONFIG)
+public class SimpleRegionNormalizer implements RegionNormalizer {
Review comment:
Sent a question over on the dev list.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache-HBase commented on pull request #1774: URL: https://github.com/apache/hbase/pull/1774#issuecomment-635669707 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 40s | master passed | | +1 :green_heart: | compile | 2m 27s | master passed | | +1 :green_heart: | shadedjars | 6m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 23s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | +1 :green_heart: | compile | 2m 25s | the patch passed | | +1 :green_heart: | javac | 2m 25s | the patch passed | | +1 :green_heart: | shadedjars | 6m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 46s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 4s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 42s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 193m 54s | hbase-server in the patch passed. | | | | 227m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1774 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c7aaf88b2477 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f06248ef84 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/testReport/ | | Max. process+thread count | 3534 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786: URL: https://github.com/apache/hbase/pull/1786#discussion_r432176087 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java ## @@ -18,126 +17,436 @@ */ package org.apache.hadoop.hbase.master.normalizer; +import java.io.IOException; +import java.time.Instant; +import java.time.Period; import java.util.ArrayList; import java.util.Collections; import java.util.Comparator; import java.util.List; - -import org.apache.hadoop.hbase.HBaseIOException; +import java.util.Objects; +import java.util.function.BooleanSupplier; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseInterfaceAudience; +import org.apache.hadoop.hbase.RegionMetrics; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.Size; import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.client.MasterSwitchType; import org.apache.hadoop.hbase.client.RegionInfo; +import org.apache.hadoop.hbase.client.TableDescriptor; +import org.apache.hadoop.hbase.master.MasterServices; +import org.apache.hadoop.hbase.master.RegionState; +import org.apache.hadoop.hbase.master.assignment.RegionStates; import org.apache.hadoop.hbase.master.normalizer.NormalizationPlan.PlanType; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; import org.apache.yetus.audience.InterfaceAudience; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils; /** * Simple implementation of region normalizer. Logic in use: * - * Get all regions of a given table - * Get avg size S of each region (by total size of store files reported in RegionMetrics) - * Seek every single region one by one. If a region R0 is bigger than S * 2, it is kindly - * requested to split. Thereon evaluate the next region R1 - * Otherwise, if R0 + R1 is smaller than S, R0 and R1 are kindly requested to merge. Thereon - * evaluate the next region R2 - * Otherwise, R1 is evaluated + * Get all regions of a given table + * Get avg size S of the regions in the table (by total size of store files reported in + * RegionMetrics) + * For each region R0, if R0 is bigger than S * 2, it is kindly requested to split. + * Otherwise, for the next region in the chain R1, if R0 + R1 is smaller then S, R0 and R1 + * are kindly requested to merge. Review comment: https://issues.apache.org/jira/browse/HBASE-24465 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24465) Normalizer should consider region max file size when planning merges
Nick Dimiduk created HBASE-24465:
Summary: Normalizer should consider region max file size when
planning merges
Key: HBASE-24465
URL: https://issues.apache.org/jira/browse/HBASE-24465
Project: HBase
Issue Type: Improvement
Components: master
Reporter: Nick Dimiduk
When the normalizer plans its actions, it does not consider
{{hbase.hregion.max.filesize}}. This means it could get into a merge/split loop
in collaboration with the region server: the normalizer sees two regions are
smaller than the table average, it merges them. The resulting region is larger
that this max file size, so the region server splits them. Repeat.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24464) Normalizer should have configuration for minimum region size
Nick Dimiduk created HBASE-24464:
Summary: Normalizer should have configuration for minimum region
size
Key: HBASE-24464
URL: https://issues.apache.org/jira/browse/HBASE-24464
Project: HBase
Issue Type: Improvement
Components: master
Reporter: Nick Dimiduk
Another issue raised in PR review on HBASE-24418. With the current
implementation, the lower-bound on how small the normalizer will make a region
is its lowest granularity of size measurement: 1mb. That means it will happily
balance a table out to be 1000 * 1mb regions (probably not likely, but seems
plausible). The proposal was to add configuration knob that specifies the lower
bound on region size as a guard against excessive splitting, something of an
analogue to {{hbase.hregion.max.filesize}}.
I'm not convinced this is really needed; I think the table would need to have
been crafted intentionally to trigger this behavior. However, it seems like a
reasonable guard rail to install. Discuss.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786: URL: https://github.com/apache/hbase/pull/1786#discussion_r432174796 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java ## @@ -18,126 +17,436 @@ */ package org.apache.hadoop.hbase.master.normalizer; +import java.io.IOException; +import java.time.Instant; +import java.time.Period; import java.util.ArrayList; import java.util.Collections; import java.util.Comparator; import java.util.List; - -import org.apache.hadoop.hbase.HBaseIOException; +import java.util.Objects; +import java.util.function.BooleanSupplier; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hbase.HBaseInterfaceAudience; +import org.apache.hadoop.hbase.RegionMetrics; +import org.apache.hadoop.hbase.ServerName; +import org.apache.hadoop.hbase.Size; import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.client.MasterSwitchType; import org.apache.hadoop.hbase.client.RegionInfo; +import org.apache.hadoop.hbase.client.TableDescriptor; +import org.apache.hadoop.hbase.master.MasterServices; +import org.apache.hadoop.hbase.master.RegionState; +import org.apache.hadoop.hbase.master.assignment.RegionStates; import org.apache.hadoop.hbase.master.normalizer.NormalizationPlan.PlanType; +import org.apache.hadoop.hbase.util.EnvironmentEdgeManager; import org.apache.yetus.audience.InterfaceAudience; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils; /** * Simple implementation of region normalizer. Logic in use: * - * Get all regions of a given table - * Get avg size S of each region (by total size of store files reported in RegionMetrics) - * Seek every single region one by one. If a region R0 is bigger than S * 2, it is kindly - * requested to split. Thereon evaluate the next region R1 - * Otherwise, if R0 + R1 is smaller than S, R0 and R1 are kindly requested to merge. Thereon - * evaluate the next region R2 - * Otherwise, R1 is evaluated + * Get all regions of a given table + * Get avg size S of the regions in the table (by total size of store files reported in + * RegionMetrics) + * For each region R0, if R0 is bigger than S * 2, it is kindly requested to split. Review comment: https://issues.apache.org/jira/browse/HBASE-24464 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #1761: HBASE-21406 "status 'replication'" should not show SINK if the cluste…
joshelser commented on a change in pull request #1761: URL: https://github.com/apache/hbase/pull/1761#discussion_r432173382 ## File path: hbase-shell/src/main/ruby/hbase/admin.rb ## @@ -835,12 +835,16 @@ def status(format, type) r_source_string = ' SOURCE:' r_load_sink = sl.getReplicationLoadSink next if r_load_sink.nil? + if (r_load_sink.getTimestampsOfLastAppliedOp() == r_load_sink.getTimeStampStarted()) Review comment: A comment here would also be very appreciated (e.g "If we have applied no operations since we've started replication, assume that we're not acting as a sink and don't print the normal information") ## File path: hbase-shell/src/main/ruby/hbase/admin.rb ## @@ -835,12 +835,16 @@ def status(format, type) r_source_string = ' SOURCE:' r_load_sink = sl.getReplicationLoadSink next if r_load_sink.nil? + if (r_load_sink.getTimestampsOfLastAppliedOp() == r_load_sink.getTimeStampStarted()) Review comment: nit: Can drop the wrapping parens This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119153#comment-17119153 ] Bharath Vissapragada commented on HBASE-18095: -- > We have dynamic reloading of configuration implemented in some areas. The > ability to reload the configured master addresses seems like a good place to > make use of that capability. Yes, I saw the whole ConfigurationObserver thing. For server roles, looks like it is triggered by a an updateConfiguration() RPC but in this case we need a periodic refresher that that reloads configuration on the client. > For the interface itself, should it return a Set of java.net.URL or > java.net.SocketAddress instead? Right, or a HostAndPort... I can spin up a patch unless there are any objections with this approach. > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Kyle Purtell >Assignee: Bharath Vissapragada >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch > > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24463) Allow operator to limit total normalization work per invocation
Nick Dimiduk created HBASE-24463: Summary: Allow operator to limit total normalization work per invocation Key: HBASE-24463 URL: https://issues.apache.org/jira/browse/HBASE-24463 Project: HBase Issue Type: Improvement Components: master Reporter: Nick Dimiduk During review on HBASE-24418, we observed that there's no way for an operator to limit the total amount of work a normalizer invocation will do. One suggestion was allowing settings for maximum number of plan executions per table or maximum number of plan executions per run. I've seen other systems limit the amount of CPU time permitted for a given run. At least we have a run lock that prevents multiple invocations from running concurrently, which should prevent one run from starting before the previous one finished. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r432171658
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -1911,43 +1912,51 @@ public boolean normalizeRegions() throws IOException {
return false;
}
-synchronized (this.normalizer) {
+if (!normalizationInProgressLock.tryLock()) {
// Don't run the normalizer concurrently
+ LOG.info("Normalization already in progress. Skipping request.");
+} else {
+ try {
+List allEnabledTables = new ArrayList<>(
+ tableStateManager.getTablesInStates(TableState.State.ENABLED));
+Collections.shuffle(allEnabledTables);
- List allEnabledTables = new ArrayList<>(
-this.tableStateManager.getTablesInStates(TableState.State.ENABLED));
-
- Collections.shuffle(allEnabledTables);
-
- for (TableName table : allEnabledTables) {
-TableDescriptor tblDesc = getTableDescriptors().get(table);
-if (table.isSystemTable() || (tblDesc != null &&
-!tblDesc.isNormalizationEnabled())) {
- LOG.trace("Skipping normalization for {}, as it's either system"
- + " table or doesn't have auto normalization turned on", table);
- continue;
-}
+for (TableName table : allEnabledTables) {
+ if (table.isSystemTable()) {
+continue;
+ }
+ final TableDescriptor tblDesc = getTableDescriptors().get(table);
+ if (tblDesc != null && !tblDesc.isNormalizationEnabled()) {
+LOG.debug(
+ "Skipping {} because normalization is disabled in its table
properties.", table);
+continue;
+ }
-// make one last check that the cluster isn't shutting down before
proceeding.
-if (skipRegionManagementAction("region normalizer")) {
- return false;
-}
+ // make one last check that the cluster isn't shutting down before
proceeding.
+ if (skipRegionManagementAction("region normalizer")) {
+return false;
+ }
-final List plans =
this.normalizer.computePlanForTable(table);
-if (CollectionUtils.isEmpty(plans)) {
- return true;
-}
+ final List plans =
normalizer.computePlansForTable(table);
+ if (CollectionUtils.isEmpty(plans)) {
+return true;
+ }
-try (final Admin admin =
asyncClusterConnection.toConnection().getAdmin()) {
- for (NormalizationPlan plan : plans) {
-plan.execute(admin);
-if (plan.getType() == PlanType.SPLIT) {
- splitPlanCount++;
-} else if (plan.getType() == PlanType.MERGE) {
- mergePlanCount++;
+ try (final Admin admin =
asyncClusterConnection.toConnection().getAdmin()) {
+// as of this writing, `plan.execute()` is non-blocking, so
there's no artificial rate-
+// limiting of merge requests due to this serial loop.
Review comment:
https://issues.apache.org/jira/browse/HBASE-24463
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24422) Remove hbase-native-client code from the master branch
[ https://issues.apache.org/jira/browse/HBASE-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119151#comment-17119151 ] Josh Elser commented on HBASE-24422: Sounds good to me to clean up the remaining stuff in the main tree. Thanks for your eyes, Bharath! > Remove hbase-native-client code from the master branch > -- > > Key: HBASE-24422 > URL: https://issues.apache.org/jira/browse/HBASE-24422 > Project: HBase > Issue Type: Task >Affects Versions: master >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Minor > Fix For: 3.0.0-alpha-1 > > > Per recent discussions in the slack channel, there is not much value > in having this code in the master branch. This code has it's dedicated > git repo at https://github.com/apache/hbase-native-client. > It was already removed from branch-1 (HBASE-19419) and branch-2 > (HBASE-19176) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] joshelser closed pull request #1365: HBASE-24066 Let HBase master UI act as a maven repo
joshelser closed pull request #1365: URL: https://github.com/apache/hbase/pull/1365 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24066) Expose shaded clients through WebUI as Maven repository
[ https://issues.apache.org/jira/browse/HBASE-24066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved HBASE-24066. Resolution: Incomplete Resolving this as I don't see a path forward for the change at this time. > Expose shaded clients through WebUI as Maven repository > --- > > Key: HBASE-24066 > URL: https://issues.apache.org/jira/browse/HBASE-24066 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Minor > > Props to [~busbey] for this idea. > We have a number of shaded jars which are (largely) sufficient for launching > any Java application against HBase. However, if users have multiple versions > of HBase in their organization, it might be confusing to know "which client" > do I need to use? Can we expose our shaded clients from HBase in such a way > that build tools can just ask HBase for the client jars they should use? > The idea here is that we can use embedded Jetty to "fake" out a Maven > repository that users can put in their client applications. We have no extra > burden from HBase because we already are packaging these jars. I'll link an > example Maven application which uses this "feature". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] joshelser commented on pull request #1365: HBASE-24066 Let HBase master UI act as a maven repo
joshelser commented on pull request #1365: URL: https://github.com/apache/hbase/pull/1365#issuecomment-635652513 Not seeing a path forward. Closing until we think this is worth doing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-22041) [k8s] The crashed node exists in onlineServer forever, and if it holds the meta data, master will start up hang.
[
https://issues.apache.org/jira/browse/HBASE-22041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119146#comment-17119146
]
Andrey Elenskiy commented on HBASE-22041:
-
> When did K8S register in DNS the new pod?
Given the eventually consistent nature of k8s, it's possible that the mapping
in DNS is updated after the new regionserver pod has already started.
Unfortunately, I can't verify if that's the case as DNS isn't managed by us so
I can't add extra logging there. I think for the sake of argument we can assume
that the DNS mapping is inconsistent. Although, that could be the case on any
infra as DNS can be inconsistent due to caching in multiple places
(systemd-resolved, intermediate DNS servers, java dns cache, etc) or just
operators being slow to update them.
> What happens if you run w/ -Dsun.net.inetaddr.ttl=1 instead of 10?
I was able to reproduce this issue with ttl=1 as well as ttl=0 (which I guess
means no caching).
> [k8s] The crashed node exists in onlineServer forever, and if it holds the
> meta data, master will start up hang.
>
>
> Key: HBASE-22041
> URL: https://issues.apache.org/jira/browse/HBASE-22041
> Project: HBase
> Issue Type: Bug
>Reporter: lujie
>Priority: Critical
> Attachments: bug.zip, hbasemaster.log, normal.zip
>
>
> while master fresh boot, we crash (kill- 9) the RS who hold meta. we find
> that the master startup fails and print thounds of logs like:
> {code:java}
> 2019-03-13 01:09:54,896 WARN [RSProcedureDispatcher-pool4-t1]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to java.net.ConnectException: Call to
> hadoop14/172.16.1.131:16020 failed on connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> syscall:getsockopt(..) failed: Connection refused:
> hadoop14/172.16.1.131:16020, try=0, retrying...
> 2019-03-13 01:09:55,004 WARN [RSProcedureDispatcher-pool4-t2]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=1, retrying...
> 2019-03-13 01:09:55,114 WARN [RSProcedureDispatcher-pool4-t3]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=2, retrying...
> 2019-03-13 01:09:55,219 WARN [RSProcedureDispatcher-pool4-t4]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=3, retrying...
> 2019-03-13 01:09:55,324 WARN [RSProcedureDispatcher-pool4-t5]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=4, retrying...
> 2019-03-13 01:09:55,428 WARN [RSProcedureDispatcher-pool4-t6]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=5, retrying...
> 2019-03-13 01:09:55,533 WARN [RSProcedureDispatcher-pool4-t7]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=6, retrying...
> 2019-03-13 01:09:55,638 WARN [RSProcedureDispatcher-pool4-t8]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
>
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r432168513
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -18,126 +17,436 @@
*/
package org.apache.hadoop.hbase.master.normalizer;
+import java.io.IOException;
+import java.time.Instant;
+import java.time.Period;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
-
-import org.apache.hadoop.hbase.HBaseIOException;
+import java.util.Objects;
+import java.util.function.BooleanSupplier;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.hbase.RegionMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.Size;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.MasterSwitchType;
import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.assignment.RegionStates;
import org.apache.hadoop.hbase.master.normalizer.NormalizationPlan.PlanType;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
/**
* Simple implementation of region normalizer. Logic in use:
*
- * Get all regions of a given table
- * Get avg size S of each region (by total size of store files reported in
RegionMetrics)
- * Seek every single region one by one. If a region R0 is bigger than S *
2, it is kindly
- * requested to split. Thereon evaluate the next region R1
- * Otherwise, if R0 + R1 is smaller than S, R0 and R1 are kindly requested
to merge. Thereon
- * evaluate the next region R2
- * Otherwise, R1 is evaluated
+ * Get all regions of a given table
+ * Get avg size S of the regions in the table (by total size of store
files reported in
+ * RegionMetrics)
+ * For each region R0, if R0 is bigger than S * 2, it is kindly
requested to split.
+ * Otherwise, for the next region in the chain R1, if R0 + R1 is smaller
then S, R0 and R1
+ * are kindly requested to merge.
+ *
+ *
+ * The following parameters are configurable:
+ *
+ * Whether to split a region as part of normalization. Configuration:
+ * {@value SPLIT_ENABLED_KEY}, default: {@value
DEFAULT_SPLIT_ENABLED}.
+ * Whether to merge a region as part of normalization. Configuration:
+ * {@value MERGE_ENABLED_KEY}, default: {@value
DEFAULT_MERGE_ENABLED}.
+ * The minimum number of regions in a table to consider it for
normalization. Configuration:
+ * {@value MIN_REGION_COUNT_KEY}, default: {@value
DEFAULT_MIN_REGION_COUNT}.
+ * The minimum age for a region to be considered for a merge, in days.
Configuration:
+ * {@value MERGE_MIN_REGION_AGE_DAYS_KEY}, default:
+ * {@value DEFAULT_MERGE_MIN_REGION_AGE_DAYS}.
+ * The minimum size for a region to be considered for a merge, in whole
MBs. Configuration:
+ * {@value MERGE_MIN_REGION_SIZE_MB_KEY}, default:
+ * {@value DEFAULT_MERGE_MIN_REGION_SIZE_MB}.
*
*
- * Region sizes are coarse and approximate on the order of megabytes.
Additionally, "empty" regions
- * (less than 1MB, with the previous note) are not merged away. This is by
design to prevent
- * normalization from undoing the pre-splitting of a table.
+ * To see detailed logging of the application of these configuration values,
set the log level for
+ * this class to `TRACE`.
*/
-@InterfaceAudience.Private
-public class SimpleRegionNormalizer extends AbstractRegionNormalizer {
-
+@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.CONFIG)
+public class SimpleRegionNormalizer implements RegionNormalizer {
private static final Logger LOG =
LoggerFactory.getLogger(SimpleRegionNormalizer.class);
- private static long[] skippedCount = new
long[NormalizationPlan.PlanType.values().length];
+
+ static final String SPLIT_ENABLED_KEY = "hbase.normalizer.split.enabled";
+ static final boolean DEFAULT_SPLIT_ENABLED = true;
+ static final String MERGE_ENABLED_KEY = "hbase.normalizer.merge.enabled";
+ static final boolean DEFAULT_MERGE_ENABLED = true;
+ // TODO: after HBASE-24416, `min.region.count` only applies to merge plans;
should
+ // deprecate/rename the configuration key.
+ static final String MIN_REGION_COUNT_KEY =
"hbase.normalizer.min.region.count";
+ static final int DEFAULT_MIN_REGION_COUNT = 3;
+ static final String MERGE_MIN_REGION_AGE_DAYS_KEY =
"hbase.normalizer.merge.min_region_age.days";
+ static final int DEFAULT_MERGE_MIN_REGION_AGE_DAYS = 3;
+ static final String
[jira] [Commented] (HBASE-24462) region_mover.rb NoMethodError: undefined method `tmpdir' for Dir:Class
[
https://issues.apache.org/jira/browse/HBASE-24462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119139#comment-17119139
]
Siddhi Mehta commented on HBASE-24462:
--
Adding the snippet below resolves the issue
{code:java}
require 'tmpdir'
{code}
> region_mover.rb NoMethodError: undefined method `tmpdir' for Dir:Class
> --
>
> Key: HBASE-24462
> URL: https://issues.apache.org/jira/browse/HBASE-24462
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.6.0
>Reporter: Siddhi Mehta
>Priority: Major
>
> Trying to use region_mover.rb to unloadRegion and see the following
>
> NoMethodError: undefined method `tmpdir' for Dir:Class
> getFilename at /usr/current/hbase-regionserver/bin/region_mover.rb:445
> unloadRegions at /usr/current/hbase-regionserver/bin/region_mover.rb:299
> (root) at /usr/current/hbase-regionserver/bin/region_mover.rb:525
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22287) inifinite retries on failed server in RSProcedureDispatcher
[
https://issues.apache.org/jira/browse/HBASE-22287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119138#comment-17119138
]
Michael Stack commented on HBASE-22287:
---
Here are logs showing retry 525 and 526 with 100ms in between attempts from
trace-level log attached to HBASE-22041
{code}
2020-05-21 17:29:49,267 TRACE [RSProcedureDispatcher-pool3-t44]
procedure.RSProcedureDispatcher: Building request with operations count=1
2020-05-21 17:29:49,268 DEBUG [RSProcedureDispatcher-pool3-t44]
ipc.AbstractRpcClient: Not trying to connect to
regionserver-2.hbase.hbase.svc.cluster.local/10.128.14.39:16020 this server is
in the failed servers list
2020-05-21 17:29:49,268 TRACE [RSProcedureDispatcher-pool3-t44]
ipc.AbstractRpcClient: Call: ExecuteProcedures, callTime: 0ms
2020-05-21 17:29:49,268 DEBUG [RSProcedureDispatcher-pool3-t44]
procedure.RSProcedureDispatcher: request to
regionserver-2.hbase.hbase.svc.cluster.local,16020,1590082132059 failed, try=525
org.apache.hadoop.hbase.ipc.FailedServerException: Call to
regionserver-2.hbase.hbase.svc.cluster.local/10.128.14.39:16020 failed on local
exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in
the failed servers list: regionserver-2.hbase.hbase.svc.cluster.local/
10.128.14.39:16020
at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:220)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:392)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:97)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:423)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:419)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:117)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:132)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:436)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:330)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:97)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:585)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$BlockingStub.executeProcedures(AdminProtos.java:31006)
at
org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:349)
at
org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.run(RSProcedureDispatcher.java:314)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is
in the failed servers list:
regionserver-2.hbase.hbase.svc.cluster.local/10.128.14.39:16020
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:354)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:433)
... 9 more
2020-05-21 17:29:49,268 WARN [RSProcedureDispatcher-pool3-t44]
procedure.RSProcedureDispatcher: request to server
regionserver-2.hbase.hbase.svc.cluster.local,16020,1590082132059 failed due to
org.apache.hadoop.hbase.ipc.FailedServerException: Call to
regionserver-2.hbase.hbase.svc. cluster.local/10.128.14.39:16020 failed on
local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server
is in the failed servers list:
regionserver-2.hbase.hbase.svc.cluster.local/10.128.14.39:16020, try=525,
retrying...
2020-05-21 17:29:49,368 TRACE [RSProcedureDispatcher-pool3-t45]
procedure.RSProcedureDispatcher: Building request with operations count=1
2020-05-21 17:29:49,369 DEBUG [RSProcedureDispatcher-pool3-t45]
ipc.AbstractRpcClient: Not trying to connect to
regionserver-2.hbase.hbase.svc.cluster.local/10.128.14.39:16020 this server is
in the failed servers list
2020-05-21 17:29:49,369 TRACE [RSProcedureDispatcher-pool3-t45]
ipc.AbstractRpcClient: Call: ExecuteProcedures, callTime: 1ms
2020-05-21 17:29:49,369 DEBUG [RSProcedureDispatcher-pool3-t45]
procedure.RSProcedureDispatcher: request to
regionserver-2.hbase.hbase.svc.cluster.local,16020,1590082132059 failed, try=526
org.apache.hadoop.hbase.ipc.FailedServerException: Call to
regionserver-2.hbase.hbase.svc.cluster.local/10.128.14.39:16020 failed on local
exception:
[GitHub] [hbase] Apache-HBase commented on pull request #1440: HBASE-23113: Improve and add additional Netty configuration for RPC.
Apache-HBase commented on pull request #1440: URL: https://github.com/apache/hbase/pull/1440#issuecomment-635647202 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 18s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 1s | branch-2 passed | | +1 :green_heart: | compile | 1m 23s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 29s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 0s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | shadedjars | 5m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 19s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 193m 48s | hbase-server in the patch passed. | | | | 223m 34s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1440/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1440 | | JIRA Issue | HBASE-23113 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4802ecb99503 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 60c125367b | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1440/2/testReport/ | | Max. process+thread count | 2595 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1440/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24462) region_mover.rb NoMethodError: undefined method `tmpdir' for Dir:Class
Siddhi Mehta created HBASE-24462: Summary: region_mover.rb NoMethodError: undefined method `tmpdir' for Dir:Class Key: HBASE-24462 URL: https://issues.apache.org/jira/browse/HBASE-24462 Project: HBase Issue Type: Bug Affects Versions: 1.6.0 Reporter: Siddhi Mehta Trying to use region_mover.rb to unloadRegion and see the following NoMethodError: undefined method `tmpdir' for Dir:Class getFilename at /usr/current/hbase-regionserver/bin/region_mover.rb:445 unloadRegions at /usr/current/hbase-regionserver/bin/region_mover.rb:299 (root) at /usr/current/hbase-regionserver/bin/region_mover.rb:525 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
saintstack commented on pull request #1783: URL: https://github.com/apache/hbase/pull/1783#issuecomment-635643398 Can't commit till the -1 from JIRA is raised. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-22120) Replace HTrace with OpenTracing
[ https://issues.apache.org/jira/browse/HBASE-22120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119128#comment-17119128 ] Siyao Meng commented on HBASE-22120: +1 on using OpenTelemetry instead now that it is [in beta|https://opentelemetry.io/project-status/]. Maybe just change the title? > Replace HTrace with OpenTracing > --- > > Key: HBASE-22120 > URL: https://issues.apache.org/jira/browse/HBASE-22120 > Project: HBase > Issue Type: New Feature > Components: tracing >Affects Versions: 3.0.0-alpha-1 >Reporter: Sergey Shelukhin >Assignee: Wei-Chiu Chuang >Priority: Major > > h2. Deprecate HTrace usage in HBase > * HBase 1.x (branch-1) > * Declare HTrace (htrace 3.x) deprecated in the user doc. > * HBase 2.x (branch-2) > * Declare HTrace deprecated in the user doc. Furthermore, state that it is > known not working. > * Either fix the trace context propagation bug in HBase 2.x, or backport > OpenTracing support from the master branch. I am inclined to the latter. > * HBase 3.x (master branch) > * Remove HTrace entirely. > * Add OpenTracing APIs. Potentially backport to HBase 2.4. > * Replace OpenTracing API with OpenTelemetry when the latter stabilizes. > h1. Milestones > # Doc -- deprecation notice > # Replace existing HTrace code with OpenTracing code in the master branch > (3.x) > # Java (a [poc|https://github.com/jojochuang/hbase/tree/HBASE-22120] is > currently under way) > # HBase shell and scripts (Ruby, shell script) > # Doc > # Add new trace instrumentation code for new features not instrumented by > the existing HTace code. > # Propagate the traces to other systems such as HDFS and MapReduce. > # Support other OpenTracing tracers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24449) Generate version.h include file during compilation
[
https://issues.apache.org/jira/browse/HBASE-24449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada resolved HBASE-24449.
--
Fix Version/s: 3.0.0-alpha-1
Resolution: Fixed
Not committing to other branches at this point since we are still working on
stabilizing hbase-native-client for master branch. We can commit in the future
if need be.
> Generate version.h include file during compilation
> --
>
> Key: HBASE-24449
> URL: https://issues.apache.org/jira/browse/HBASE-24449
> Project: HBase
> Issue Type: Sub-task
> Components: Client, native-client
>Affects Versions: 3.0.0-alpha-1
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Curious case of {{version.h}}
> hbase-native-client relies on this include file to compile. However that is
> not generated by the HBase repo. I see no trace of it in the git log, wonder
> how it even worked until now.
> I think it will be useful to generate it during compilation and package it
> with the release tar ball. This helps [~phrocker] in HBASE-24400.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #1783: HBASE-24436 The store file open and close thread pool should be share…
saintstack commented on a change in pull request #1783:
URL: https://github.com/apache/hbase/pull/1783#discussion_r432159021
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -590,7 +588,7 @@ void setDataBlockEncoderInTest(HFileDataBlockEncoder
blockEncoder) {
}
}
} finally {
- storeFileOpenerThreadPool.shutdownNow();
+
Review comment:
If nothing to cleanup, remove the finally block?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv merged pull request #1794: HBASE-24449: Generate version.h include file during compilation.
bharathv merged pull request #1794: URL: https://github.com/apache/hbase/pull/1794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on pull request #1794: HBASE-24449: Generate version.h include file during compilation.
bharathv commented on pull request #1794: URL: https://github.com/apache/hbase/pull/1794#issuecomment-635640305 I was letting @busbey know since he volunteered to be the RM for 3.0.0 and he wouldn't be surprised by a new directory showing up in the release artifacts. I'll go ahead and merge this now that I have your blessing. Thanks for taking at a look @saintstack. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache-HBase commented on pull request #1774: URL: https://github.com/apache/hbase/pull/1774#issuecomment-635639973 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 2m 51s | master passed | | +1 :green_heart: | shadedjars | 7m 33s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 26s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 49s | hbase-server in master failed. | | -0 :warning: | javadoc | 0m 18s | hbase-zookeeper in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 7s | the patch passed | | +1 :green_heart: | compile | 2m 50s | the patch passed | | +1 :green_heart: | javac | 2m 50s | the patch passed | | +1 :green_heart: | shadedjars | 5m 45s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 17s | hbase-zookeeper in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 58s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 10s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 45s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 124m 51s | hbase-server in the patch passed. | | | | 162m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1774 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a6d0c541668d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f06248ef84 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-zookeeper.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-zookeeper.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/testReport/ | | Max. process+thread count | 4562 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #1751: HBASE-24409: Exposing a function in HBase WALKey to set the table name
saintstack commented on pull request #1751: URL: https://github.com/apache/hbase/pull/1751#issuecomment-635639491 Please open new PR if good reason for undoing this fairly basic tenet. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #1751: HBASE-24409: Exposing a function in HBase WALKey to set the table name
saintstack commented on pull request #1751: URL: https://github.com/apache/hbase/pull/1751#issuecomment-635639242 I'm going to close this PR. System internal depends on WALKeys being immutable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack closed pull request #1751: HBASE-24409: Exposing a function in HBase WALKey to set the table name
saintstack closed pull request #1751: URL: https://github.com/apache/hbase/pull/1751 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1751: HBASE-24409: Exposing a function in HBase WALKey to set the table name
saintstack commented on a change in pull request #1751:
URL: https://github.com/apache/hbase/pull/1751#discussion_r432153953
##
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKey.java
##
@@ -57,6 +57,14 @@ default long estimatedSerializedSizeOf() {
*/
TableName getTableName();
+ /**
+ * set the table name for WALKey. This is mainly for the purpose of
+ * when we want to change the table name in the WALKey.
+ * For example:
+ * replicating across different table names in two different clusters
+ */
+ void setTableName(TableName tableName);
Review comment:
WALKey only has getters. It doesn't say it is immutable in the class
comment but it should.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r432148644
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -18,126 +17,436 @@
*/
package org.apache.hadoop.hbase.master.normalizer;
+import java.io.IOException;
+import java.time.Instant;
+import java.time.Period;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
-
-import org.apache.hadoop.hbase.HBaseIOException;
+import java.util.Objects;
+import java.util.function.BooleanSupplier;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.hbase.RegionMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.Size;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.MasterSwitchType;
import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.assignment.RegionStates;
import org.apache.hadoop.hbase.master.normalizer.NormalizationPlan.PlanType;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
/**
* Simple implementation of region normalizer. Logic in use:
*
- * Get all regions of a given table
- * Get avg size S of each region (by total size of store files reported in
RegionMetrics)
- * Seek every single region one by one. If a region R0 is bigger than S *
2, it is kindly
- * requested to split. Thereon evaluate the next region R1
- * Otherwise, if R0 + R1 is smaller than S, R0 and R1 are kindly requested
to merge. Thereon
- * evaluate the next region R2
- * Otherwise, R1 is evaluated
+ * Get all regions of a given table
+ * Get avg size S of the regions in the table (by total size of store
files reported in
+ * RegionMetrics)
+ * For each region R0, if R0 is bigger than S * 2, it is kindly
requested to split.
+ * Otherwise, for the next region in the chain R1, if R0 + R1 is smaller
then S, R0 and R1
+ * are kindly requested to merge.
+ *
+ *
+ * The following parameters are configurable:
+ *
+ * Whether to split a region as part of normalization. Configuration:
+ * {@value SPLIT_ENABLED_KEY}, default: {@value
DEFAULT_SPLIT_ENABLED}.
+ * Whether to merge a region as part of normalization. Configuration:
+ * {@value MERGE_ENABLED_KEY}, default: {@value
DEFAULT_MERGE_ENABLED}.
+ * The minimum number of regions in a table to consider it for
normalization. Configuration:
+ * {@value MIN_REGION_COUNT_KEY}, default: {@value
DEFAULT_MIN_REGION_COUNT}.
+ * The minimum age for a region to be considered for a merge, in days.
Configuration:
+ * {@value MERGE_MIN_REGION_AGE_DAYS_KEY}, default:
+ * {@value DEFAULT_MERGE_MIN_REGION_AGE_DAYS}.
+ * The minimum size for a region to be considered for a merge, in whole
MBs. Configuration:
+ * {@value MERGE_MIN_REGION_SIZE_MB_KEY}, default:
+ * {@value DEFAULT_MERGE_MIN_REGION_SIZE_MB}.
*
*
- * Region sizes are coarse and approximate on the order of megabytes.
Additionally, "empty" regions
- * (less than 1MB, with the previous note) are not merged away. This is by
design to prevent
- * normalization from undoing the pre-splitting of a table.
+ * To see detailed logging of the application of these configuration values,
set the log level for
+ * this class to `TRACE`.
*/
-@InterfaceAudience.Private
-public class SimpleRegionNormalizer extends AbstractRegionNormalizer {
-
+@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.CONFIG)
+public class SimpleRegionNormalizer implements RegionNormalizer {
private static final Logger LOG =
LoggerFactory.getLogger(SimpleRegionNormalizer.class);
- private static long[] skippedCount = new
long[NormalizationPlan.PlanType.values().length];
+
+ static final String SPLIT_ENABLED_KEY = "hbase.normalizer.split.enabled";
+ static final boolean DEFAULT_SPLIT_ENABLED = true;
+ static final String MERGE_ENABLED_KEY = "hbase.normalizer.merge.enabled";
+ static final boolean DEFAULT_MERGE_ENABLED = true;
+ // TODO: after HBASE-24416, `min.region.count` only applies to merge plans;
should
+ // deprecate/rename the configuration key.
+ static final String MIN_REGION_COUNT_KEY =
"hbase.normalizer.min.region.count";
+ static final int DEFAULT_MIN_REGION_COUNT = 3;
+ static final String MERGE_MIN_REGION_AGE_DAYS_KEY =
"hbase.normalizer.merge.min_region_age.days";
+ static final int DEFAULT_MERGE_MIN_REGION_AGE_DAYS = 3;
+ static final String
[GitHub] [hbase] Apache-HBase commented on pull request #1440: HBASE-23113: Improve and add additional Netty configuration for RPC.
Apache-HBase commented on pull request #1440: URL: https://github.com/apache/hbase/pull/1440#issuecomment-635627861 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 49s | branch-2 passed | | +1 :green_heart: | compile | 2m 0s | branch-2 passed | | +1 :green_heart: | shadedjars | 8m 0s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 36s | hbase-client in branch-2 failed. | | -0 :warning: | javadoc | 0m 49s | hbase-server in branch-2 failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 17s | the patch passed | | +1 :green_heart: | compile | 1m 56s | the patch passed | | +1 :green_heart: | javac | 1m 56s | the patch passed | | +1 :green_heart: | shadedjars | 8m 3s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 33s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 49s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 45s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 130m 35s | hbase-server in the patch passed. | | | | 171m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1440/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1440 | | JIRA Issue | HBASE-23113 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 7c521709a760 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 60c125367b | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1440/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1440/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1440/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1440/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1440/2/testReport/ | | Max. process+thread count | 3922 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1440/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24389) Introduce a new master rpc service to locate meta region through root region
[ https://issues.apache.org/jira/browse/HBASE-24389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119096#comment-17119096 ] Bharath Vissapragada commented on HBASE-24389: -- I think I see what you mean. I didn't realize you put up a patch for this already. I skimmed through it, you implemented a new meta lookup (outside of what registry provides) based on your root implementation that always goes through the active master sub (which is picked up from registry). So I think my rephrased question would be, is there a plan to load balance these across masters to avoid hots potting, would be nice to do that in the follow up patches. > Introduce a new master rpc service to locate meta region through root region > > > Key: HBASE-24389 > URL: https://issues.apache.org/jira/browse/HBASE-24389 > Project: HBase > Issue Type: Sub-task > Components: master, meta >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > And also reimlement the client logic to use the new api to locate meta > region. In general, this will remove the assumption at client side that there > is only one meta region. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #1786: HBASE-24418 Consolidate Normalizer implementations
ndimiduk commented on a change in pull request #1786:
URL: https://github.com/apache/hbase/pull/1786#discussion_r432134487
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
##
@@ -18,126 +17,436 @@
*/
package org.apache.hadoop.hbase.master.normalizer;
+import java.io.IOException;
+import java.time.Instant;
+import java.time.Period;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
-
-import org.apache.hadoop.hbase.HBaseIOException;
+import java.util.Objects;
+import java.util.function.BooleanSupplier;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseInterfaceAudience;
+import org.apache.hadoop.hbase.RegionMetrics;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.Size;
import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.MasterSwitchType;
import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.master.assignment.RegionStates;
import org.apache.hadoop.hbase.master.normalizer.NormalizationPlan.PlanType;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
/**
* Simple implementation of region normalizer. Logic in use:
*
- * Get all regions of a given table
- * Get avg size S of each region (by total size of store files reported in
RegionMetrics)
- * Seek every single region one by one. If a region R0 is bigger than S *
2, it is kindly
- * requested to split. Thereon evaluate the next region R1
- * Otherwise, if R0 + R1 is smaller than S, R0 and R1 are kindly requested
to merge. Thereon
- * evaluate the next region R2
- * Otherwise, R1 is evaluated
+ * Get all regions of a given table
+ * Get avg size S of the regions in the table (by total size of store
files reported in
+ * RegionMetrics)
+ * For each region R0, if R0 is bigger than S * 2, it is kindly
requested to split.
+ * Otherwise, for the next region in the chain R1, if R0 + R1 is smaller
then S, R0 and R1
+ * are kindly requested to merge.
+ *
+ *
+ * The following parameters are configurable:
+ *
+ * Whether to split a region as part of normalization. Configuration:
+ * {@value SPLIT_ENABLED_KEY}, default: {@value
DEFAULT_SPLIT_ENABLED}.
+ * Whether to merge a region as part of normalization. Configuration:
+ * {@value MERGE_ENABLED_KEY}, default: {@value
DEFAULT_MERGE_ENABLED}.
+ * The minimum number of regions in a table to consider it for
normalization. Configuration:
+ * {@value MIN_REGION_COUNT_KEY}, default: {@value
DEFAULT_MIN_REGION_COUNT}.
+ * The minimum age for a region to be considered for a merge, in days.
Configuration:
+ * {@value MERGE_MIN_REGION_AGE_DAYS_KEY}, default:
+ * {@value DEFAULT_MERGE_MIN_REGION_AGE_DAYS}.
+ * The minimum size for a region to be considered for a merge, in whole
MBs. Configuration:
+ * {@value MERGE_MIN_REGION_SIZE_MB_KEY}, default:
+ * {@value DEFAULT_MERGE_MIN_REGION_SIZE_MB}.
*
*
- * Region sizes are coarse and approximate on the order of megabytes.
Additionally, "empty" regions
- * (less than 1MB, with the previous note) are not merged away. This is by
design to prevent
- * normalization from undoing the pre-splitting of a table.
+ * To see detailed logging of the application of these configuration values,
set the log level for
+ * this class to `TRACE`.
*/
-@InterfaceAudience.Private
-public class SimpleRegionNormalizer extends AbstractRegionNormalizer {
-
+@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.CONFIG)
+public class SimpleRegionNormalizer implements RegionNormalizer {
private static final Logger LOG =
LoggerFactory.getLogger(SimpleRegionNormalizer.class);
- private static long[] skippedCount = new
long[NormalizationPlan.PlanType.values().length];
+
+ static final String SPLIT_ENABLED_KEY = "hbase.normalizer.split.enabled";
+ static final boolean DEFAULT_SPLIT_ENABLED = true;
+ static final String MERGE_ENABLED_KEY = "hbase.normalizer.merge.enabled";
+ static final boolean DEFAULT_MERGE_ENABLED = true;
+ // TODO: after HBASE-24416, `min.region.count` only applies to merge plans;
should
+ // deprecate/rename the configuration key.
+ static final String MIN_REGION_COUNT_KEY =
"hbase.normalizer.min.region.count";
+ static final int DEFAULT_MIN_REGION_COUNT = 3;
+ static final String MERGE_MIN_REGION_AGE_DAYS_KEY =
"hbase.normalizer.merge.min_region_age.days";
+ static final int DEFAULT_MERGE_MIN_REGION_AGE_DAYS = 3;
+ static final String
[GitHub] [hbase] Apache-HBase commented on pull request #1774: HBASE-24389 Introduce a new master rpc service to locate meta region through root region
Apache-HBase commented on pull request #1774: URL: https://github.com/apache/hbase/pull/1774#issuecomment-635594515 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 49s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 1s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | checkstyle | 2m 31s | master passed | | +1 :green_heart: | spotbugs | 8m 2s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 55s | the patch passed | | +1 :green_heart: | checkstyle | 0m 9s | The patch passed checkstyle in hbase-protocol-shaded | | +1 :green_heart: | checkstyle | 0m 33s | hbase-client: The patch generated 0 new + 242 unchanged - 2 fixed = 242 total (was 244) | | +1 :green_heart: | checkstyle | 0m 12s | The patch passed checkstyle in hbase-zookeeper | | +1 :green_heart: | checkstyle | 1m 29s | hbase-server: The patch generated 0 new + 453 unchanged - 5 fixed = 453 total (was 458) | | -0 :warning: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | hadoopcheck | 13m 22s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 2m 37s | the patch passed | | +1 :green_heart: | spotbugs | 8m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | | | | 57m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1774 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool | | uname | Linux 93468b9b505a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f06248ef84 | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/artifact/yetus-general-check/output/whitespace-eol.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-zookeeper hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1774/9/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24454) BucketCache disabled instantly before error duration toleration is reached due to timing issue
[
https://issues.apache.org/jira/browse/HBASE-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119059#comment-17119059
]
Jacob LeBlanc commented on HBASE-24454:
---
Sure, I'll contribute a patch. I was thinking the same thing about saving to a
local variable so we avoid the double read. ioErrorStartTime is already
volatile so read/writes are atomic. No need to add locking which is good given
the success case (writing -1) is executed so frequently.
> BucketCache disabled instantly before error duration toleration is reached
> due to timing issue
> --
>
> Key: HBASE-24454
> URL: https://issues.apache.org/jira/browse/HBASE-24454
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.10
> Environment: We saw this in HBase 1.4.10 (EMR 5.28.1), but I believe
> the newest code still has this problem.
>Reporter: Jacob LeBlanc
>Priority: Major
>
> We saw an instance where BucketCache was disabled after only two IO error
> were encountered at nearly the same time. The following shows all errors from
> a region server log for the 2020-05-26 17:00 hour. Notice that there are no
> other errors until the 17:14:50 at which time the BucketCache is disabled
> because it claims duration time has exceeded 6 ms:
> $ grep ERROR hbase-hbase-regionserver-ip-172-20-113-147.log.2020-05-26-17
> 2020-05-26 17:14:50,396 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,397 ERROR
> [regionserver/ip-172-20-113-147.us-west-2.compute.internal/172.20.113.147:16020-BucketCacheWriter-0]
> bucket.BucketCache: Failed syncing IO engine
> 2020-05-26 17:14:50,400 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,401 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: IO errors duration time has exceeded 6ms, disabling
> cache, please check your IOEngine
> The region server is very busy so it should be constantly getting successful
> reads and writes in the bucket cache (it is not as though there was some long
> ago error and then no successful IO to clear the error flag).
> We are running a busy EMR cluster backed by S3. A bucketcache getting
> disabled is a huge performance issue because all reads must go to S3.
> Looking at the code, I believe I've found a timing issue. Here is the code
> for checking and setting the ioErrorStartTime (taken from BucketCache.java):
>
> {code:java}
> /**
>* Check whether we tolerate IO error this time. If the duration of IOEngine
>* throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable
> the
>* cache
>*/
> private void checkIOErrorIsTolerated() {
> long now = EnvironmentEdgeManager.currentTime();
> if (this.ioErrorStartTime > 0) {
> if (cacheEnabled && (now - ioErrorStartTime) >
> this.ioErrorsTolerationDuration) {
> LOG.error("IO errors duration time has exceeded " +
> ioErrorsTolerationDuration +
> "ms, disabling cache, please check your IOEngine");
> disableCache();
> }
> } else {
> this.ioErrorStartTime = now;
> }
> }
> {code}
>
> And here is the code for clearing the ioErrorStartTime when a successful read
> or write is done:
> {code:java}
> if (this.ioErrorStartTime > 0) {
> ioErrorStartTime = -1;
> }
> {code}
> Notice that that if ioErrorStartTime is set to -1 after the first if
> statement in checkIOErrorIsTolerated but before (now - ioErrorStartTime),
> then (now - ioErrorStartTime) will evaluate to (now + 1) resulting in the
> code thinking it has exceeded ioErrorsTolerationDuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-24454) BucketCache disabled instantly before error duration toleration is reached due to timing issue
[
https://issues.apache.org/jira/browse/HBASE-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob LeBlanc reassigned HBASE-24454:
-
Assignee: Jacob LeBlanc
> BucketCache disabled instantly before error duration toleration is reached
> due to timing issue
> --
>
> Key: HBASE-24454
> URL: https://issues.apache.org/jira/browse/HBASE-24454
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 1.4.10
> Environment: We saw this in HBase 1.4.10 (EMR 5.28.1), but I believe
> the newest code still has this problem.
>Reporter: Jacob LeBlanc
>Assignee: Jacob LeBlanc
>Priority: Major
>
> We saw an instance where BucketCache was disabled after only two IO error
> were encountered at nearly the same time. The following shows all errors from
> a region server log for the 2020-05-26 17:00 hour. Notice that there are no
> other errors until the 17:14:50 at which time the BucketCache is disabled
> because it claims duration time has exceeded 6 ms:
> $ grep ERROR hbase-hbase-regionserver-ip-172-20-113-147.log.2020-05-26-17
> 2020-05-26 17:14:50,396 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,397 ERROR
> [regionserver/ip-172-20-113-147.us-west-2.compute.internal/172.20.113.147:16020-BucketCacheWriter-0]
> bucket.BucketCache: Failed syncing IO engine
> 2020-05-26 17:14:50,400 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: Failed reading block
> 9adfad3a603047cfa0c98b16da0b0974_13786 from bucket cache
> 2020-05-26 17:14:50,401 ERROR [hfile-prefetch-1589117924884]
> bucket.BucketCache: IO errors duration time has exceeded 6ms, disabling
> cache, please check your IOEngine
> The region server is very busy so it should be constantly getting successful
> reads and writes in the bucket cache (it is not as though there was some long
> ago error and then no successful IO to clear the error flag).
> We are running a busy EMR cluster backed by S3. A bucketcache getting
> disabled is a huge performance issue because all reads must go to S3.
> Looking at the code, I believe I've found a timing issue. Here is the code
> for checking and setting the ioErrorStartTime (taken from BucketCache.java):
>
> {code:java}
> /**
>* Check whether we tolerate IO error this time. If the duration of IOEngine
>* throwing errors exceeds ioErrorsDurationTimeTolerated, we will disable
> the
>* cache
>*/
> private void checkIOErrorIsTolerated() {
> long now = EnvironmentEdgeManager.currentTime();
> if (this.ioErrorStartTime > 0) {
> if (cacheEnabled && (now - ioErrorStartTime) >
> this.ioErrorsTolerationDuration) {
> LOG.error("IO errors duration time has exceeded " +
> ioErrorsTolerationDuration +
> "ms, disabling cache, please check your IOEngine");
> disableCache();
> }
> } else {
> this.ioErrorStartTime = now;
> }
> }
> {code}
>
> And here is the code for clearing the ioErrorStartTime when a successful read
> or write is done:
> {code:java}
> if (this.ioErrorStartTime > 0) {
> ioErrorStartTime = -1;
> }
> {code}
> Notice that that if ioErrorStartTime is set to -1 after the first if
> statement in checkIOErrorIsTolerated but before (now - ioErrorStartTime),
> then (now - ioErrorStartTime) will evaluate to (now + 1) resulting in the
> code thinking it has exceeded ioErrorsTolerationDuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack commented on pull request #1794: HBASE-24449: Generate version.h include file during compilation.
saintstack commented on pull request #1794: URL: https://github.com/apache/hbase/pull/1794#issuecomment-635587079 Shout if you need me to merge. Sounds like you'd like signoff by @busbey This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1797: HBASE-24451 Remove the HasThread because the related bug had been fix…
saintstack commented on a change in pull request #1797:
URL: https://github.com/apache/hbase/pull/1797#discussion_r432095352
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
##
@@ -122,9 +121,9 @@ public SplitLogManager(MasterServices master, Configuration
conf)
throws IOException {
this.server = master;
this.conf = conf;
-// Get Server Thread name. Sometimes the Server is mocked so may not
implement HasThread.
+// Get Server Thread name. Sometimes the Server is mocked so may not
implement Thread.
// For example, in tests.
-String name = master instanceof HasThread? ((HasThread)master).getName():
+String name = master instanceof Thread? ((Thread)master).getName():
Review comment:
This is not needed w/ this patch?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
##
@@ -1908,9 +1907,9 @@ protected void queueAssign(final RegionStateNode
regionNode) {
}
private void startAssignmentThread() {
-// Get Server Thread name. Sometimes the Server is mocked so may not
implement HasThread.
+// Get Server Thread name. Sometimes the Server is mocked so may not
implement Thread.
// For example, in tests.
-String name = master instanceof HasThread? ((HasThread)master).getName():
+String name = master instanceof Thread? ((Thread)master).getName():
Review comment:
Ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
