[hbase] branch HBASE-26233 updated (f40f268 -> f919774)
This is an automated email from the ASF dual-hosted git repository. zhangduo pushed a change to branch HBASE-26233 in repository https://gitbox.apache.org/repos/asf/hbase.git. discard f40f268 HBASE-26540 Implement a tool to verify the lag of region replication (#3950) discard a6b728a HBASE-26539 The default rpc timeout 200ms is too small for replicating meta edits (#3919) discard 44b6483 HBASE-26538 Should find a way to clear the replication queue for a legacy region_replica_replication peer (#3918) discard d6a5d34 HBASE-26486 Attach the new region replication framework design doc to git repo (#3884) discard 724735c HBASE-26478 Update ref guide about the new region replication framework (#3885) discard ac9258f HBASE-26481 Consider rolling upgrading from old region replication framework (#3880) discard 88daafe HBASE-26417 Remove NoopReplicationQueueStorage (#3882) discard fbe6665 HBASE-26456 Limit the size for one replicating (#3873) discard d841ffd HBASE-26416 Implement a new method for region replication instead of using replay (#3864) discard 47c35f0 HBASE-26457 Should not always clear all the failed replicas when getting a flush all request (#3850) discard 6febbc8 HBASE-26448 Make sure we do not flush a region too frequently (#3847) discard c894a76 HBASE-26449 The way we add or clear failedReplicas may have race (#3846) discard 282477c HBASE-26413 Limit the total size of buffered region replication entries (#3844) discard b3ac276 HBASE-26412 Handle sink failure in RegionReplicationSink (#3815) discard 5d69d1c HBASE-26407 Introduce a region replication sink for sinking WAL edits to secondary replicas directly (#3807) add ea0fe22 HBASE-26579 Set storage policy of recovered edits when hbase.wal.storage.type is configured (#3948) add d399799 HBASE-26580 The message of StoreTooBusy is confused (#3949) add 314e924 HBASE-26606 Upgrade log4j2 to 2.17.0 (#3965) add 6818ec2 HBASE-26613 The logic of the method incrementIV in Encryption class has problem (#3968) add 6aaef897 HBASE-26064 Introduce a StoreFileTracker to abstract the store file tracking logic add 43b40e9 HBASE-25988 Store the store file list by a file (#3578) add 6e05376 HBASE-26079 Use StoreFileTracker when splitting and merging (#3617) add 090b2fe HBASE-26224 Introduce a MigrationStoreFileTracker to support migrating from different store file tracker implementations (#3656) add 0ee1689 HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor when creating table (#3666) add 2052e80 HBASE-26248 Should find a suitable way to let users specify the store file tracker implementation (#3665) add 5ff0f98 HBASE-26264 Add more checks to prevent misconfiguration on store file tracker (#3681) add fc4f6d1 HBASE-26280 Use store file tracker when snapshoting (#3685) add 06db852 HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… (#3721) add e4e7cf8 HBASE-26386 Refactor StoreFileTracker implementations to expose the set method (#3774) add 08d1171 HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl (#3749) add 8bec26e HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configurations to TableDescriptor for existing tables (#3700) add a288365 HBASE-26271 Cleanup the broken store files under data directory (#3786) add d00b5fa HBASE-26454 CreateTableProcedure still relies on temp dir and renames… (#3845) add 771e552 HBASE-26286: Add support for specifying store file tracker when restoring or cloning snapshot add f16b7b1 HBASE-26265 Update ref guide to mention the new store file tracker im… (#3942) add bf258cd HBASE-26598 Fix excessive connections in MajorCompactor (#3961) add dbdef59 HBASE-26609 Round the size to MB or KB at the end of calculation in HRegionServer.createRegionLoad (#3967) add 8309f86 HBASE-26407 Introduce a region replication sink for sinking WAL edits to secondary replicas directly (#3807) add aec952e HBASE-26412 Handle sink failure in RegionReplicationSink (#3815) add b24c2e8 HBASE-26413 Limit the total size of buffered region replication entries (#3844) add e75b3a6 HBASE-26449 The way we add or clear failedReplicas may have race (#3846) add 5651572 HBASE-26448 Make sure we do not flush a region too frequently (#3847) add 8c810b3 HBASE-26457 Should not always clear all the failed replicas when getting a flush all request (#3850) add 21ed55f HBASE-26416 Implement a new method for region replication instead of using replay (#3864) add 1d9c339 HBASE-26456 Limit the size for one replicating (#3873) add e42b906 HBASE-26417 Remove NoopReplicationQueueStorage (#3882) add 02f514b HBASE-26481 Consider rolling upgrading from old region replication framework (#3880) add 3c2b4c8 HBASE-26478 Update ref guide about the new region
[hbase] 02/02: HBASE-26609 Round the size to MB or KB at the end of calculation in HRegionServer.createRegionLoad (#3967)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit c6ac99684c70d68e82efe9ce1167e8fb9cb6b79b
Author: Duo Zhang
AuthorDate: Thu Dec 23 10:37:17 2021 +0800
HBASE-26609 Round the size to MB or KB at the end of calculation in
HRegionServer.createRegionLoad (#3967)
Signed-off-by: Peter Somogyi
---
.../hadoop/hbase/regionserver/HRegionServer.java | 59 +++---
1 file changed, 40 insertions(+), 19 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
index 047345c..57ab15b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
@@ -1733,6 +1733,20 @@ public class HRegionServer extends Thread implements
return regionServerAccounting;
}
+ // Round the size with KB or MB.
+ // A trick here is that if the sizeInBytes is less than sizeUnit, we will
round the size to 1
+ // instead of 0 if it is not 0, to avoid some schedulers think the region
has no data. See
+ // HBASE-26340 for more details on why this is important.
+ private static int roundSize(long sizeInByte, int sizeUnit) {
+if (sizeInByte == 0) {
+ return 0;
+} else if (sizeInByte < sizeUnit) {
+ return 1;
+} else {
+ return (int) Math.min(sizeInByte / sizeUnit, Integer.MAX_VALUE);
+}
+ }
+
/**
* @param r Region to get RegionLoad for.
* @param regionLoadBldr the RegionLoad.Builder, can be null
@@ -1746,16 +1760,14 @@ public class HRegionServer extends Thread implements
int storefiles = 0;
int storeRefCount = 0;
int maxCompactedStoreFileRefCount = 0;
-int storeUncompressedSizeMB = 0;
-int storefileSizeMB = 0;
-long storefileSizeByte = 0L;
-int memstoreSizeMB = (int) (r.getMemStoreDataSize() / 1024 / 1024);
-long storefileIndexSizeKB = 0;
-int rootLevelIndexSizeKB = 0;
-int totalStaticIndexSizeKB = 0;
-int totalStaticBloomSizeKB = 0;
-long totalCompactingKVs = 0;
-long currentCompactedKVs = 0;
+long storeUncompressedSize = 0L;
+long storefileSize = 0L;
+long storefileIndexSize = 0L;
+long rootLevelIndexSize = 0L;
+long totalStaticIndexSize = 0L;
+long totalStaticBloomSize = 0L;
+long totalCompactingKVs = 0L;
+long currentCompactedKVs = 0L;
List storeList = r.getStores();
stores += storeList.size();
for (HStore store : storeList) {
@@ -1765,22 +1777,30 @@ public class HRegionServer extends Thread implements
int currentMaxCompactedStoreFileRefCount =
store.getMaxCompactedStoreFileRefCount();
maxCompactedStoreFileRefCount = Math.max(maxCompactedStoreFileRefCount,
currentMaxCompactedStoreFileRefCount);
- storeUncompressedSizeMB += (int) (store.getStoreSizeUncompressed() /
1024 / 1024);
- storefileSizeByte += store.getStorefilesSize();
+ storeUncompressedSize += store.getStoreSizeUncompressed();
+ storefileSize += store.getStorefilesSize();
//TODO: storefileIndexSizeKB is same with rootLevelIndexSizeKB?
- storefileIndexSizeKB += store.getStorefilesRootLevelIndexSize() / 1024;
+ storefileIndexSize += store.getStorefilesRootLevelIndexSize();
CompactionProgress progress = store.getCompactionProgress();
if (progress != null) {
totalCompactingKVs += progress.getTotalCompactingKVs();
currentCompactedKVs += progress.currentCompactedKVs;
}
- rootLevelIndexSizeKB += (int) (store.getStorefilesRootLevelIndexSize() /
1024);
- totalStaticIndexSizeKB += (int) (store.getTotalStaticIndexSize() / 1024);
- totalStaticBloomSizeKB += (int) (store.getTotalStaticBloomSize() / 1024);
+ rootLevelIndexSize += store.getStorefilesRootLevelIndexSize();
+ totalStaticIndexSize += store.getTotalStaticIndexSize();
+ totalStaticBloomSize += store.getTotalStaticBloomSize();
}
-//HBASE-26340 Fix false "0" size under 1MB
-storefileSizeMB = storefileSizeByte > 0 && storefileSizeByte <= 1024 * 1024
- ? 1 : (int) storefileSizeByte / 1024 / 1024;
+
+int unitMB = 1024 * 1024;
+int unitKB = 1024;
+
+int memstoreSizeMB = roundSize(r.getMemStoreDataSize(), unitMB);
+int storeUncompressedSizeMB = roundSize(storeUncompressedSize, unitMB);
+int storefileSizeMB = roundSize(storefileSize, unitMB);
+int storefileIndexSizeKB = roundSize(storefileIndexSize, unitKB);
+int rootLevelIndexSizeKB = roundSize(rootLevelIndexSize, unitKB);
+int totalStaticIndexSizeKB = roundSize(totalStaticIndexSize, unitKB);
+int totalStaticBloomSizeKB = roundSize(totalStaticBloomSize, unitKB);
HDFSBlocksDistribution hdfsBd = r.getHDFSBlocksDistribution();
[hbase] branch branch-2 updated (c6aac61 -> c6ac996)
This is an automated email from the ASF dual-hosted git repository. zhangduo pushed a change to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hbase.git. from c6aac61 HBASE-26613 The logic of the method incrementIV in Encryption class has problem (#3968) new 55ddcba HBASE-26598 Fix excessive connections in MajorCompactor (#3961) new c6ac996 HBASE-26609 Round the size to MB or KB at the end of calculation in HRegionServer.createRegionLoad (#3967) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../hadoop/hbase/regionserver/HRegionServer.java | 59 +++--- .../util/compaction/MajorCompactionRequest.java| 53 +-- .../util/compaction/MajorCompactionTTLRequest.java | 33 ++-- .../hbase/util/compaction/MajorCompactor.java | 6 +-- .../hbase/util/compaction/MajorCompactorTTL.java | 4 +- .../compaction/TestMajorCompactionRequest.java | 20 +++- .../compaction/TestMajorCompactionTTLRequest.java | 18 +++ 7 files changed, 97 insertions(+), 96 deletions(-)
[hbase] 01/02: HBASE-26598 Fix excessive connections in MajorCompactor (#3961)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 55ddcbaaf5e682c96d97347314ef6d1a00d0e4bf
Author: Samir Khan
AuthorDate: Wed Dec 22 20:29:48 2021 -0600
HBASE-26598 Fix excessive connections in MajorCompactor (#3961)
Signed-off-by: Duo Zhang
---
.../util/compaction/MajorCompactionRequest.java| 53 ++
.../util/compaction/MajorCompactionTTLRequest.java | 33 ++
.../hbase/util/compaction/MajorCompactor.java | 6 +--
.../hbase/util/compaction/MajorCompactorTTL.java | 4 +-
.../compaction/TestMajorCompactionRequest.java | 20
.../compaction/TestMajorCompactionTTLRequest.java | 18 +++-
6 files changed, 57 insertions(+), 77 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
index 22ec6cb..f765b35 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
@@ -22,13 +22,11 @@ import java.util.Collection;
import java.util.List;
import java.util.Optional;
import java.util.Set;
-import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
-import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.RegionInfo;
import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
@@ -45,26 +43,26 @@ class MajorCompactionRequest {
private static final Logger LOG =
LoggerFactory.getLogger(MajorCompactionRequest.class);
- protected final Configuration configuration;
+ protected final Connection connection;
protected final RegionInfo region;
private Set stores;
- MajorCompactionRequest(Configuration configuration, RegionInfo region) {
-this.configuration = configuration;
+ MajorCompactionRequest(Connection connection, RegionInfo region) {
+this.connection = connection;
this.region = region;
}
- MajorCompactionRequest(Configuration configuration, RegionInfo region,
+ MajorCompactionRequest(Connection connection, RegionInfo region,
Set stores) {
-this(configuration, region);
+this(connection, region);
this.stores = stores;
}
- static Optional newRequest(Configuration
configuration, RegionInfo info,
+ static Optional newRequest(Connection connection,
RegionInfo info,
Set stores, long timestamp) throws IOException {
MajorCompactionRequest request =
-new MajorCompactionRequest(configuration, info, stores);
-return request.createRequest(configuration, stores, timestamp);
+new MajorCompactionRequest(connection, info, stores);
+return request.createRequest(connection, stores, timestamp);
}
RegionInfo getRegion() {
@@ -79,28 +77,26 @@ class MajorCompactionRequest {
this.stores = stores;
}
- Optional createRequest(Configuration configuration,
+ Optional createRequest(Connection connection,
Set stores, long timestamp) throws IOException {
Set familiesToCompact = getStoresRequiringCompaction(stores,
timestamp);
MajorCompactionRequest request = null;
if (!familiesToCompact.isEmpty()) {
- request = new MajorCompactionRequest(configuration, region,
familiesToCompact);
+ request = new MajorCompactionRequest(connection, region,
familiesToCompact);
}
return Optional.ofNullable(request);
}
Set getStoresRequiringCompaction(Set requestedStores, long
timestamp)
throws IOException {
-try(Connection connection = getConnection(configuration)) {
- HRegionFileSystem fileSystem = getFileSystem(connection);
- Set familiesToCompact = Sets.newHashSet();
- for (String family : requestedStores) {
-if (shouldCFBeCompacted(fileSystem, family, timestamp)) {
- familiesToCompact.add(family);
-}
+HRegionFileSystem fileSystem = getFileSystem();
+Set familiesToCompact = Sets.newHashSet();
+for (String family : requestedStores) {
+ if (shouldCFBeCompacted(fileSystem, family, timestamp)) {
+familiesToCompact.add(family);
}
- return familiesToCompact;
}
+return familiesToCompact;
}
boolean shouldCFBeCompacted(HRegionFileSystem fileSystem, String family,
long ts)
@@ -142,10 +138,6 @@ class MajorCompactionRequest {
return false;
}
- Connection getConnection(Configuration configuration) throws IOException {
-return
[hbase] branch branch-2.5 updated (f239886 -> 51d38b5)
This is an automated email from the ASF dual-hosted git repository. zhangduo pushed a change to branch branch-2.5 in repository https://gitbox.apache.org/repos/asf/hbase.git. from f239886 HBASE-26613 The logic of the method incrementIV in Encryption class has problem (#3968) new a57f497 HBASE-26598 Fix excessive connections in MajorCompactor (#3961) new 51d38b5 HBASE-26609 Round the size to MB or KB at the end of calculation in HRegionServer.createRegionLoad (#3967) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../hadoop/hbase/regionserver/HRegionServer.java | 59 +++--- .../util/compaction/MajorCompactionRequest.java| 53 +-- .../util/compaction/MajorCompactionTTLRequest.java | 33 ++-- .../hbase/util/compaction/MajorCompactor.java | 6 +-- .../hbase/util/compaction/MajorCompactorTTL.java | 4 +- .../compaction/TestMajorCompactionRequest.java | 20 +++- .../compaction/TestMajorCompactionTTLRequest.java | 18 +++ 7 files changed, 97 insertions(+), 96 deletions(-)
[hbase] 01/02: HBASE-26598 Fix excessive connections in MajorCompactor (#3961)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch branch-2.5
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit a57f4976c1e32f711ff103456938cc8c2d4f7027
Author: Samir Khan
AuthorDate: Wed Dec 22 20:29:48 2021 -0600
HBASE-26598 Fix excessive connections in MajorCompactor (#3961)
Signed-off-by: Duo Zhang
---
.../util/compaction/MajorCompactionRequest.java| 53 ++
.../util/compaction/MajorCompactionTTLRequest.java | 33 ++
.../hbase/util/compaction/MajorCompactor.java | 6 +--
.../hbase/util/compaction/MajorCompactorTTL.java | 4 +-
.../compaction/TestMajorCompactionRequest.java | 20
.../compaction/TestMajorCompactionTTLRequest.java | 18 +++-
6 files changed, 57 insertions(+), 77 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
index 22ec6cb..f765b35 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
@@ -22,13 +22,11 @@ import java.util.Collection;
import java.util.List;
import java.util.Optional;
import java.util.Set;
-import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
-import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.RegionInfo;
import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
@@ -45,26 +43,26 @@ class MajorCompactionRequest {
private static final Logger LOG =
LoggerFactory.getLogger(MajorCompactionRequest.class);
- protected final Configuration configuration;
+ protected final Connection connection;
protected final RegionInfo region;
private Set stores;
- MajorCompactionRequest(Configuration configuration, RegionInfo region) {
-this.configuration = configuration;
+ MajorCompactionRequest(Connection connection, RegionInfo region) {
+this.connection = connection;
this.region = region;
}
- MajorCompactionRequest(Configuration configuration, RegionInfo region,
+ MajorCompactionRequest(Connection connection, RegionInfo region,
Set stores) {
-this(configuration, region);
+this(connection, region);
this.stores = stores;
}
- static Optional newRequest(Configuration
configuration, RegionInfo info,
+ static Optional newRequest(Connection connection,
RegionInfo info,
Set stores, long timestamp) throws IOException {
MajorCompactionRequest request =
-new MajorCompactionRequest(configuration, info, stores);
-return request.createRequest(configuration, stores, timestamp);
+new MajorCompactionRequest(connection, info, stores);
+return request.createRequest(connection, stores, timestamp);
}
RegionInfo getRegion() {
@@ -79,28 +77,26 @@ class MajorCompactionRequest {
this.stores = stores;
}
- Optional createRequest(Configuration configuration,
+ Optional createRequest(Connection connection,
Set stores, long timestamp) throws IOException {
Set familiesToCompact = getStoresRequiringCompaction(stores,
timestamp);
MajorCompactionRequest request = null;
if (!familiesToCompact.isEmpty()) {
- request = new MajorCompactionRequest(configuration, region,
familiesToCompact);
+ request = new MajorCompactionRequest(connection, region,
familiesToCompact);
}
return Optional.ofNullable(request);
}
Set getStoresRequiringCompaction(Set requestedStores, long
timestamp)
throws IOException {
-try(Connection connection = getConnection(configuration)) {
- HRegionFileSystem fileSystem = getFileSystem(connection);
- Set familiesToCompact = Sets.newHashSet();
- for (String family : requestedStores) {
-if (shouldCFBeCompacted(fileSystem, family, timestamp)) {
- familiesToCompact.add(family);
-}
+HRegionFileSystem fileSystem = getFileSystem();
+Set familiesToCompact = Sets.newHashSet();
+for (String family : requestedStores) {
+ if (shouldCFBeCompacted(fileSystem, family, timestamp)) {
+familiesToCompact.add(family);
}
- return familiesToCompact;
}
+return familiesToCompact;
}
boolean shouldCFBeCompacted(HRegionFileSystem fileSystem, String family,
long ts)
@@ -142,10 +138,6 @@ class MajorCompactionRequest {
return false;
}
- Connection getConnection(Configuration configuration) throws IOException {
-return
[hbase] 02/02: HBASE-26609 Round the size to MB or KB at the end of calculation in HRegionServer.createRegionLoad (#3967)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch branch-2.5
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 51d38b5e4f6f9343ced064dec3ac161fc8864422
Author: Duo Zhang
AuthorDate: Thu Dec 23 10:37:17 2021 +0800
HBASE-26609 Round the size to MB or KB at the end of calculation in
HRegionServer.createRegionLoad (#3967)
Signed-off-by: Peter Somogyi
---
.../hadoop/hbase/regionserver/HRegionServer.java | 59 +++---
1 file changed, 40 insertions(+), 19 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
index 047345c..57ab15b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
@@ -1733,6 +1733,20 @@ public class HRegionServer extends Thread implements
return regionServerAccounting;
}
+ // Round the size with KB or MB.
+ // A trick here is that if the sizeInBytes is less than sizeUnit, we will
round the size to 1
+ // instead of 0 if it is not 0, to avoid some schedulers think the region
has no data. See
+ // HBASE-26340 for more details on why this is important.
+ private static int roundSize(long sizeInByte, int sizeUnit) {
+if (sizeInByte == 0) {
+ return 0;
+} else if (sizeInByte < sizeUnit) {
+ return 1;
+} else {
+ return (int) Math.min(sizeInByte / sizeUnit, Integer.MAX_VALUE);
+}
+ }
+
/**
* @param r Region to get RegionLoad for.
* @param regionLoadBldr the RegionLoad.Builder, can be null
@@ -1746,16 +1760,14 @@ public class HRegionServer extends Thread implements
int storefiles = 0;
int storeRefCount = 0;
int maxCompactedStoreFileRefCount = 0;
-int storeUncompressedSizeMB = 0;
-int storefileSizeMB = 0;
-long storefileSizeByte = 0L;
-int memstoreSizeMB = (int) (r.getMemStoreDataSize() / 1024 / 1024);
-long storefileIndexSizeKB = 0;
-int rootLevelIndexSizeKB = 0;
-int totalStaticIndexSizeKB = 0;
-int totalStaticBloomSizeKB = 0;
-long totalCompactingKVs = 0;
-long currentCompactedKVs = 0;
+long storeUncompressedSize = 0L;
+long storefileSize = 0L;
+long storefileIndexSize = 0L;
+long rootLevelIndexSize = 0L;
+long totalStaticIndexSize = 0L;
+long totalStaticBloomSize = 0L;
+long totalCompactingKVs = 0L;
+long currentCompactedKVs = 0L;
List storeList = r.getStores();
stores += storeList.size();
for (HStore store : storeList) {
@@ -1765,22 +1777,30 @@ public class HRegionServer extends Thread implements
int currentMaxCompactedStoreFileRefCount =
store.getMaxCompactedStoreFileRefCount();
maxCompactedStoreFileRefCount = Math.max(maxCompactedStoreFileRefCount,
currentMaxCompactedStoreFileRefCount);
- storeUncompressedSizeMB += (int) (store.getStoreSizeUncompressed() /
1024 / 1024);
- storefileSizeByte += store.getStorefilesSize();
+ storeUncompressedSize += store.getStoreSizeUncompressed();
+ storefileSize += store.getStorefilesSize();
//TODO: storefileIndexSizeKB is same with rootLevelIndexSizeKB?
- storefileIndexSizeKB += store.getStorefilesRootLevelIndexSize() / 1024;
+ storefileIndexSize += store.getStorefilesRootLevelIndexSize();
CompactionProgress progress = store.getCompactionProgress();
if (progress != null) {
totalCompactingKVs += progress.getTotalCompactingKVs();
currentCompactedKVs += progress.currentCompactedKVs;
}
- rootLevelIndexSizeKB += (int) (store.getStorefilesRootLevelIndexSize() /
1024);
- totalStaticIndexSizeKB += (int) (store.getTotalStaticIndexSize() / 1024);
- totalStaticBloomSizeKB += (int) (store.getTotalStaticBloomSize() / 1024);
+ rootLevelIndexSize += store.getStorefilesRootLevelIndexSize();
+ totalStaticIndexSize += store.getTotalStaticIndexSize();
+ totalStaticBloomSize += store.getTotalStaticBloomSize();
}
-//HBASE-26340 Fix false "0" size under 1MB
-storefileSizeMB = storefileSizeByte > 0 && storefileSizeByte <= 1024 * 1024
- ? 1 : (int) storefileSizeByte / 1024 / 1024;
+
+int unitMB = 1024 * 1024;
+int unitKB = 1024;
+
+int memstoreSizeMB = roundSize(r.getMemStoreDataSize(), unitMB);
+int storeUncompressedSizeMB = roundSize(storeUncompressedSize, unitMB);
+int storefileSizeMB = roundSize(storefileSize, unitMB);
+int storefileIndexSizeKB = roundSize(storefileIndexSize, unitKB);
+int rootLevelIndexSizeKB = roundSize(rootLevelIndexSize, unitKB);
+int totalStaticIndexSizeKB = roundSize(totalStaticIndexSize, unitKB);
+int totalStaticBloomSizeKB = roundSize(totalStaticBloomSize, unitKB);
HDFSBlocksDistribution hdfsBd = r.getHDFSBlocksDistribution();
[hbase] 02/02: HBASE-26609 Round the size to MB or KB at the end of calculation in HRegionServer.createRegionLoad (#3967)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 7ac7d014e221a1b4a08b37e7888c74650b9c85c9
Author: Duo Zhang
AuthorDate: Thu Dec 23 10:37:17 2021 +0800
HBASE-26609 Round the size to MB or KB at the end of calculation in
HRegionServer.createRegionLoad (#3967)
Signed-off-by: Peter Somogyi
---
.../hadoop/hbase/regionserver/HRegionServer.java | 59 +++---
1 file changed, 40 insertions(+), 19 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
index 97f56bd..8a38135 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
@@ -1708,6 +1708,20 @@ public class HRegionServer extends Thread implements
return regionServerAccounting;
}
+ // Round the size with KB or MB.
+ // A trick here is that if the sizeInBytes is less than sizeUnit, we will
round the size to 1
+ // instead of 0 if it is not 0, to avoid some schedulers think the region
has no data. See
+ // HBASE-26340 for more details on why this is important.
+ private static int roundSize(long sizeInByte, int sizeUnit) {
+if (sizeInByte == 0) {
+ return 0;
+} else if (sizeInByte < sizeUnit) {
+ return 1;
+} else {
+ return (int) Math.min(sizeInByte / sizeUnit, Integer.MAX_VALUE);
+}
+ }
+
/**
* @param r Region to get RegionLoad for.
* @param regionLoadBldr the RegionLoad.Builder, can be null
@@ -1721,16 +1735,14 @@ public class HRegionServer extends Thread implements
int storefiles = 0;
int storeRefCount = 0;
int maxCompactedStoreFileRefCount = 0;
-int storeUncompressedSizeMB = 0;
-int storefileSizeMB = 0;
-long storefileSizeByte = 0L;
-int memstoreSizeMB = (int) (r.getMemStoreDataSize() / 1024 / 1024);
-long storefileIndexSizeKB = 0;
-int rootLevelIndexSizeKB = 0;
-int totalStaticIndexSizeKB = 0;
-int totalStaticBloomSizeKB = 0;
-long totalCompactingKVs = 0;
-long currentCompactedKVs = 0;
+long storeUncompressedSize = 0L;
+long storefileSize = 0L;
+long storefileIndexSize = 0L;
+long rootLevelIndexSize = 0L;
+long totalStaticIndexSize = 0L;
+long totalStaticBloomSize = 0L;
+long totalCompactingKVs = 0L;
+long currentCompactedKVs = 0L;
List storeList = r.getStores();
stores += storeList.size();
for (HStore store : storeList) {
@@ -1740,22 +1752,30 @@ public class HRegionServer extends Thread implements
int currentMaxCompactedStoreFileRefCount =
store.getMaxCompactedStoreFileRefCount();
maxCompactedStoreFileRefCount = Math.max(maxCompactedStoreFileRefCount,
currentMaxCompactedStoreFileRefCount);
- storeUncompressedSizeMB += (int) (store.getStoreSizeUncompressed() /
1024 / 1024);
- storefileSizeByte += store.getStorefilesSize();
+ storeUncompressedSize += store.getStoreSizeUncompressed();
+ storefileSize += store.getStorefilesSize();
//TODO: storefileIndexSizeKB is same with rootLevelIndexSizeKB?
- storefileIndexSizeKB += store.getStorefilesRootLevelIndexSize() / 1024;
+ storefileIndexSize += store.getStorefilesRootLevelIndexSize();
CompactionProgress progress = store.getCompactionProgress();
if (progress != null) {
totalCompactingKVs += progress.getTotalCompactingKVs();
currentCompactedKVs += progress.currentCompactedKVs;
}
- rootLevelIndexSizeKB += (int) (store.getStorefilesRootLevelIndexSize() /
1024);
- totalStaticIndexSizeKB += (int) (store.getTotalStaticIndexSize() / 1024);
- totalStaticBloomSizeKB += (int) (store.getTotalStaticBloomSize() / 1024);
+ rootLevelIndexSize += store.getStorefilesRootLevelIndexSize();
+ totalStaticIndexSize += store.getTotalStaticIndexSize();
+ totalStaticBloomSize += store.getTotalStaticBloomSize();
}
-//HBASE-26340 Fix false "0" size under 1MB
-storefileSizeMB = storefileSizeByte > 0 && storefileSizeByte <= 1024 * 1024
- ? 1 : (int) storefileSizeByte / 1024 / 1024;
+
+int unitMB = 1024 * 1024;
+int unitKB = 1024;
+
+int memstoreSizeMB = roundSize(r.getMemStoreDataSize(), unitMB);
+int storeUncompressedSizeMB = roundSize(storeUncompressedSize, unitMB);
+int storefileSizeMB = roundSize(storefileSize, unitMB);
+int storefileIndexSizeKB = roundSize(storefileIndexSize, unitKB);
+int rootLevelIndexSizeKB = roundSize(rootLevelIndexSize, unitKB);
+int totalStaticIndexSizeKB = roundSize(totalStaticIndexSize, unitKB);
+int totalStaticBloomSizeKB = roundSize(totalStaticBloomSize, unitKB);
HDFSBlocksDistribution hdfsBd = r.getHDFSBlocksDistribution();
[hbase] branch branch-2.4 updated (647184e -> 7ac7d01)
This is an automated email from the ASF dual-hosted git repository. zhangduo pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/hbase.git. from 647184e HBASE-26613 The logic of the method incrementIV in Encryption class has problem (#3968) new 8e56696 HBASE-26598 Fix excessive connections in MajorCompactor (#3961) new 7ac7d01 HBASE-26609 Round the size to MB or KB at the end of calculation in HRegionServer.createRegionLoad (#3967) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../hadoop/hbase/regionserver/HRegionServer.java | 59 +++--- .../util/compaction/MajorCompactionRequest.java| 53 +-- .../util/compaction/MajorCompactionTTLRequest.java | 33 ++-- .../hbase/util/compaction/MajorCompactor.java | 6 +-- .../hbase/util/compaction/MajorCompactorTTL.java | 4 +- .../compaction/TestMajorCompactionRequest.java | 20 +++- .../compaction/TestMajorCompactionTTLRequest.java | 18 +++ 7 files changed, 97 insertions(+), 96 deletions(-)
[hbase] 01/02: HBASE-26598 Fix excessive connections in MajorCompactor (#3961)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 8e56696748ac9c818c5500e4d3704124fee8f3cb
Author: Samir Khan
AuthorDate: Wed Dec 22 20:29:48 2021 -0600
HBASE-26598 Fix excessive connections in MajorCompactor (#3961)
Signed-off-by: Duo Zhang
---
.../util/compaction/MajorCompactionRequest.java| 53 ++
.../util/compaction/MajorCompactionTTLRequest.java | 33 ++
.../hbase/util/compaction/MajorCompactor.java | 6 +--
.../hbase/util/compaction/MajorCompactorTTL.java | 4 +-
.../compaction/TestMajorCompactionRequest.java | 20
.../compaction/TestMajorCompactionTTLRequest.java | 18 +++-
6 files changed, 57 insertions(+), 77 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
index 22ec6cb..f765b35 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/util/compaction/MajorCompactionRequest.java
@@ -22,13 +22,11 @@ import java.util.Collection;
import java.util.List;
import java.util.Optional;
import java.util.Set;
-import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
-import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.RegionInfo;
import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
@@ -45,26 +43,26 @@ class MajorCompactionRequest {
private static final Logger LOG =
LoggerFactory.getLogger(MajorCompactionRequest.class);
- protected final Configuration configuration;
+ protected final Connection connection;
protected final RegionInfo region;
private Set stores;
- MajorCompactionRequest(Configuration configuration, RegionInfo region) {
-this.configuration = configuration;
+ MajorCompactionRequest(Connection connection, RegionInfo region) {
+this.connection = connection;
this.region = region;
}
- MajorCompactionRequest(Configuration configuration, RegionInfo region,
+ MajorCompactionRequest(Connection connection, RegionInfo region,
Set stores) {
-this(configuration, region);
+this(connection, region);
this.stores = stores;
}
- static Optional newRequest(Configuration
configuration, RegionInfo info,
+ static Optional newRequest(Connection connection,
RegionInfo info,
Set stores, long timestamp) throws IOException {
MajorCompactionRequest request =
-new MajorCompactionRequest(configuration, info, stores);
-return request.createRequest(configuration, stores, timestamp);
+new MajorCompactionRequest(connection, info, stores);
+return request.createRequest(connection, stores, timestamp);
}
RegionInfo getRegion() {
@@ -79,28 +77,26 @@ class MajorCompactionRequest {
this.stores = stores;
}
- Optional createRequest(Configuration configuration,
+ Optional createRequest(Connection connection,
Set stores, long timestamp) throws IOException {
Set familiesToCompact = getStoresRequiringCompaction(stores,
timestamp);
MajorCompactionRequest request = null;
if (!familiesToCompact.isEmpty()) {
- request = new MajorCompactionRequest(configuration, region,
familiesToCompact);
+ request = new MajorCompactionRequest(connection, region,
familiesToCompact);
}
return Optional.ofNullable(request);
}
Set getStoresRequiringCompaction(Set requestedStores, long
timestamp)
throws IOException {
-try(Connection connection = getConnection(configuration)) {
- HRegionFileSystem fileSystem = getFileSystem(connection);
- Set familiesToCompact = Sets.newHashSet();
- for (String family : requestedStores) {
-if (shouldCFBeCompacted(fileSystem, family, timestamp)) {
- familiesToCompact.add(family);
-}
+HRegionFileSystem fileSystem = getFileSystem();
+Set familiesToCompact = Sets.newHashSet();
+for (String family : requestedStores) {
+ if (shouldCFBeCompacted(fileSystem, family, timestamp)) {
+familiesToCompact.add(family);
}
- return familiesToCompact;
}
+return familiesToCompact;
}
boolean shouldCFBeCompacted(HRegionFileSystem fileSystem, String family,
long ts)
@@ -142,10 +138,6 @@ class MajorCompactionRequest {
return false;
}
- Connection getConnection(Configuration configuration) throws IOException {
-return
[hbase] branch master updated (bf258cd -> dbdef59)
This is an automated email from the ASF dual-hosted git repository. zhangduo pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from bf258cd HBASE-26598 Fix excessive connections in MajorCompactor (#3961) add dbdef59 HBASE-26609 Round the size to MB or KB at the end of calculation in HRegionServer.createRegionLoad (#3967) No new revisions were added by this update. Summary of changes: .../hadoop/hbase/regionserver/HRegionServer.java | 59 +++--- 1 file changed, 40 insertions(+), 19 deletions(-)
[hbase] branch master updated (f16b7b1 -> bf258cd)
This is an automated email from the ASF dual-hosted git repository. zhangduo pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from f16b7b1 HBASE-26265 Update ref guide to mention the new store file tracker im… (#3942) add bf258cd HBASE-26598 Fix excessive connections in MajorCompactor (#3961) No new revisions were added by this update. Summary of changes: .../util/compaction/MajorCompactionRequest.java| 53 ++ .../util/compaction/MajorCompactionTTLRequest.java | 33 ++ .../hbase/util/compaction/MajorCompactor.java | 6 +-- .../hbase/util/compaction/MajorCompactorTTL.java | 4 +- .../compaction/TestMajorCompactionRequest.java | 20 .../compaction/TestMajorCompactionTTLRequest.java | 18 +++- 6 files changed, 57 insertions(+), 77 deletions(-)
[hbase-site] branch asf-site updated: Published site at 6818ec23b87117d32d2e1ba7f8354961ba1eb7c4.
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hbase-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 4553583 Published site at 6818ec23b87117d32d2e1ba7f8354961ba1eb7c4. 4553583 is described below commit 4553583de6ae33520da55cc2587e5cd54f050492 Author: jenkins AuthorDate: Wed Dec 22 20:17:53 2021 + Published site at 6818ec23b87117d32d2e1ba7f8354961ba1eb7c4. --- acid-semantics.html| 2 +- apache_hbase_reference_guide.pdf | 4 +- .../apache/hadoop/hbase/io/crypto/Encryption.html | 2 +- .../hadoop/hbase/io/crypto/Encryption.Context.html | 63 +++--- .../apache/hadoop/hbase/io/crypto/Encryption.html | 63 +++--- book.html | 2 +- bulk-loads.html| 2 +- checkstyle-aggregate.html | 14 ++-- coc.html | 2 +- dependencies.html | 2 +- dependency-convergence.html| 2 +- dependency-info.html | 2 +- dependency-management.html | 2 +- .../apache/hadoop/hbase/io/crypto/Encryption.html | 2 +- .../hadoop/hbase/io/crypto/Encryption.Context.html | 63 +++--- .../apache/hadoop/hbase/io/crypto/Encryption.html | 63 +++--- downloads.html | 2 +- export_control.html| 2 +- index.html | 2 +- issue-management.html | 2 +- licenses.html | 2 +- mailing-lists.html | 2 +- metrics.html | 2 +- old_news.html | 2 +- plugin-management.html | 2 +- plugins.html | 2 +- poweredbyhbase.html| 2 +- project-info.html | 2 +- project-reports.html | 2 +- pseudo-distributed.html| 2 +- replication.html | 2 +- resources.html | 2 +- scm.html | 2 +- sponsors.html | 2 +- summary.html | 2 +- supportingprojects.html| 2 +- team.html | 2 +- testdevapidocs/index-all.html | 2 + .../apache/hadoop/hbase/backup/package-tree.html | 2 +- .../hadoop/hbase/io/crypto/TestEncryption.html | 25 -- .../apache/hadoop/hbase/io/hfile/package-tree.html | 2 +- .../org/apache/hadoop/hbase/package-tree.html | 16 ++-- .../hadoop/hbase/procedure2/package-tree.html | 2 +- .../hadoop/hbase/regionserver/package-tree.html| 6 +- .../org/apache/hadoop/hbase/test/package-tree.html | 4 +- .../org/apache/hadoop/hbase/wal/package-tree.html | 4 +- .../hadoop/hbase/io/crypto/TestEncryption.html | 96 +- 47 files changed, 257 insertions(+), 234 deletions(-) diff --git a/acid-semantics.html b/acid-semantics.html index d1d1cb1..ab3f14e 100644 --- a/acid-semantics.html +++ b/acid-semantics.html @@ -455,7 +455,7 @@ Copyright 20072021 https://www.apache.org/;>The Apache Software Foundation. -All rights reserved.Last Published: 2021-12-19 +All rights reserved.Last Published: 2021-12-22 http://maven.apache.org/; title="Built by Maven" class="poweredBy"> diff --git a/apache_hbase_reference_guide.pdf b/apache_hbase_reference_guide.pdf index 8b64334..e087c6a 100644 --- a/apache_hbase_reference_guide.pdf +++ b/apache_hbase_reference_guide.pdf @@ -5,8 +5,8 @@ /Author (Apache HBase Team) /Creator (Asciidoctor PDF 1.5.3, based on Prawn 2.2.2) /Producer (Apache HBase Team) -/ModDate (D:20211219200510+00'00') -/CreationDate (D:20211219201531+00'00') +/ModDate (D:2021100529+00'00') +/CreationDate (D:2021101552+00'00') >> endobj 2 0 obj diff --git a/apidocs/org/apache/hadoop/hbase/io/crypto/Encryption.html b/apidocs/org/apache/hadoop/hbase/io/crypto/Encryption.html index 5a506b1..d679206 100644 --- a/apidocs/org/apache/hadoop/hbase/io/crypto/Encryption.html +++ b/apidocs/org/apache/hadoop/hbase/io/crypto/Encryption.html @@ -1106,7 +1106,7 @@ extends https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html hashWithAlg -public staticbyte[]hashWithAlg(https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true; title="class or interface in
[hbase-site] branch asf-site updated: INFRA-10751 Empty commit
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hbase-site.git The following commit(s) were added to refs/heads/asf-site by this push: new a96a707 INFRA-10751 Empty commit a96a707 is described below commit a96a707f64825d8a46f0e2035c36c0eaac52abd5 Author: jenkins AuthorDate: Wed Dec 22 20:18:08 2021 + INFRA-10751 Empty commit
[hbase] 15/16: HBASE-26286: Add support for specifying store file tracker when restoring or cloning snapshot
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 771e552cf4a6436176de06eb033c6de6ee545c71
Author: BukrosSzabolcs
AuthorDate: Wed Dec 15 20:09:03 2021 -0500
HBASE-26286: Add support for specifying store file tracker when restoring
or cloning snapshot
Closes #3851
Signed-off-by: Duo Zhang
Signed-off-by: Josh Elser
---
.../java/org/apache/hadoop/hbase/client/Admin.java | 44 --
.../hadoop/hbase/client/AdminOverAsyncAdmin.java | 7 ++-
.../org/apache/hadoop/hbase/client/AsyncAdmin.java | 14 -
.../hadoop/hbase/client/AsyncHBaseAdmin.java | 6 +-
.../hbase/client/ColumnFamilyDescriptor.java | 5 ++
.../client/ColumnFamilyDescriptorBuilder.java | 6 ++
.../hadoop/hbase/client/RawAsyncHBaseAdmin.java| 24 +---
.../src/main/protobuf/server/master/Master.proto | 1 +
.../protobuf/server/master/MasterProcedure.proto | 1 +
.../org/apache/hadoop/hbase/master/HMaster.java| 25
.../hadoop/hbase/master/MasterRpcServices.java | 2 +-
.../master/procedure/CloneSnapshotProcedure.java | 54 +++-
.../master/procedure/RestoreSnapshotProcedure.java | 7 ++-
.../hbase/master/snapshot/SnapshotManager.java | 27 +---
.../storefiletracker/StoreFileTrackerFactory.java | 39 +++-
.../hbase/snapshot/RestoreSnapshotHelper.java | 11 ++--
.../TestCloneSnapshotFromClientCustomSFT.java | 71 ++
.../storefiletracker/TestStoreFileTracker.java | 2 -
.../TestStoreFileTrackerFactory.java | 52
.../hbase/rsgroup/VerifyingRSGroupAdmin.java | 5 +-
hbase-shell/src/main/ruby/hbase/admin.rb | 4 +-
hbase-shell/src/main/ruby/hbase_constants.rb | 1 +
.../src/main/ruby/shell/commands/clone_snapshot.rb | 6 +-
.../hadoop/hbase/thrift2/client/ThriftAdmin.java | 4 +-
24 files changed, 357 insertions(+), 61 deletions(-)
diff --git
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
index 48893cc..6c36660 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
@@ -69,6 +69,7 @@ import org.apache.hadoop.hbase.util.Pair;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.yetus.audience.InterfaceStability;
/**
* The administrative API for HBase. Obtain an instance from {@link
Connection#getAdmin()} and
@@ -1620,7 +1621,7 @@ public interface Admin extends Abortable, Closeable {
* @throws IllegalArgumentException if the restore request is formatted
incorrectly
*/
void restoreSnapshot(String snapshotName, boolean takeFailSafeSnapshot,
boolean restoreAcl)
- throws IOException, RestoreSnapshotException;
+throws IOException, RestoreSnapshotException;
/**
* Create a new table by cloning the snapshot content.
@@ -1633,7 +1634,25 @@ public interface Admin extends Abortable, Closeable {
*/
default void cloneSnapshot(String snapshotName, TableName tableName)
throws IOException, TableExistsException, RestoreSnapshotException {
-cloneSnapshot(snapshotName, tableName, false);
+cloneSnapshot(snapshotName, tableName, false, null);
+ }
+
+ /**
+ * Create a new table by cloning the snapshot content.
+ * @param snapshotName name of the snapshot to be cloned
+ * @param tableName name of the table where the snapshot will be restored
+ * @param restoreAcl true to clone acl into newly created table
+ * @param customSFT specify the StoreFileTracker used for the table
+ * @throws IOException if a remote or network exception occurs
+ * @throws TableExistsException if table to be created already exists
+ * @throws RestoreSnapshotException if snapshot failed to be cloned
+ * @throws IllegalArgumentException if the specified table has not a valid
name
+ */
+ default void cloneSnapshot(String snapshotName, TableName tableName, boolean
restoreAcl,
+String customSFT)
+throws IOException, TableExistsException, RestoreSnapshotException {
+get(cloneSnapshotAsync(snapshotName, tableName, restoreAcl, customSFT),
getSyncWaitTimeout(),
+ TimeUnit.MILLISECONDS);
}
/**
@@ -1680,8 +1699,25 @@ public interface Admin extends Abortable, Closeable {
* @throws RestoreSnapshotException if snapshot failed to be cloned
* @throws IllegalArgumentException if the specified table has not a valid
name
*/
- Future cloneSnapshotAsync(String snapshotName, TableName tableName,
boolean restoreAcl)
- throws IOException, TableExistsException, RestoreSnapshotException;
+ default Future cloneSnapshotAsync(String snapshotName, TableName
tableName,
+
[hbase] 16/16: HBASE-26265 Update ref guide to mention the new store file tracker im… (#3942)
This is an automated email from the ASF dual-hosted git repository. elserj pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git commit f16b7b1bfa49a68029c8ec7c1882ff4bd5536593 Author: Wellington Ramos Chevreuil AuthorDate: Thu Dec 16 21:07:38 2021 + HBASE-26265 Update ref guide to mention the new store file tracker im… (#3942) --- .../asciidoc/_chapters/store_file_tracking.adoc| 145 + src/main/asciidoc/book.adoc| 1 + 2 files changed, 146 insertions(+) diff --git a/src/main/asciidoc/_chapters/store_file_tracking.adoc b/src/main/asciidoc/_chapters/store_file_tracking.adoc new file mode 100644 index 000..74d802f --- /dev/null +++ b/src/main/asciidoc/_chapters/store_file_tracking.adoc @@ -0,0 +1,145 @@ + +/** + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +[[storefiletracking]] += Store File Tracking +:doctype: book +:numbered: +:toc: left +:icons: font +:experimental: + +== Overview + +This feature introduces an abstraction layer to track store files still used/needed by store +engines, allowing for plugging different approaches of identifying store +files required by the given store. + +Historically, HBase internals have relied on creating hfiles on temporary directories first, renaming +those files to the actual store directory at operation commit time. That's a simple and convenient +way to separate transient from already finalised files that are ready to serve client reads with data. +This approach works well with strong consistent file systems, but with the popularity of less consistent +file systems, mainly Object Store which can be used like file systems, dependency on atomic rename operations starts to introduce +performance penalties. The Amazon S3 Object Store, in particular, has been the most affected deployment, +due to its lack of atomic renames. The HBase community temporarily bypassed this problem by building a distributed locking layer called HBOSS, +to guarantee atomicity of operations against S3. + +With *Store File Tracking*, decision on where to originally create new hfiles and how to proceed upon +commit is delegated to the specific Store File Tracking implementation. +The implementation can be set at the HBase service leve in *hbase-site.xml* or at the +Table or Column Family via the TableDescriptor configuration. + +NOTE: When the store file tracking implementation is specified in *hbase_site.xml*, this configuration is also propagated into a tables configuration +at table creation time. This is to avoid dangerous configuration mismatches between processes, which +could potentially lead to data loss. + +== Available Implementations + +Store File Tracking initial version provides three builtin implementations: + +* DEFAULT +* FILE +* MIGRATION + +### DEFAULT + +As per the name, this is the Store File Tracking implementation used by default when no explicit +configuration has been defined. The DEFAULT tracker implements the standard approach using temporary +directories and renames. This is how all previous (implicit) implementation that HBase used to track store files. + +### FILE + +A file tracker implementation that creates new files straight in the store directory, avoiding the +need for rename operations. It keeps a list of committed hfiles in memory, backed by meta files, in +each store directory. Whenever a new hfile is committed, the list of _tracked files_ in the given +store is updated and a new meta file is written with this list contents, discarding the previous +meta file now containing an out dated list. + +### MIGRATION + +A special implementation to be used when swapping between Store File Tracking implementations on +pre-existing tables that already contain data, and therefore, files being tracked under an specific +logic. + +== Usage + +For fresh deployments that don't yet contain any user data, *FILE* implementation can be just set as +value for *hbase.store.file-tracker.impl* property in global *hbase-site.xml* configuration, prior +to the first hbase start. Omitting this property sets the *DEFAULT* implementation. + +For clusters with data that are upgraded to
[hbase] 14/16: HBASE-26454 CreateTableProcedure still relies on temp dir and renames… (#3845)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit d00b5faade4e40f2127fe2b18bf97dae69ce33f6
Author: Wellington Ramos Chevreuil
AuthorDate: Fri Nov 19 12:16:29 2021 +
HBASE-26454 CreateTableProcedure still relies on temp dir and renames…
(#3845)
Signed-off-by: Duo Zhang
---
.../master/procedure/CreateTableProcedure.java | 30 ++-
.../master/procedure/DeleteTableProcedure.java | 95 +++---
.../access/SnapshotScannerHDFSAclHelper.java | 4 +-
.../hadoop/hbase/master/TestMasterFileSystem.java | 29 ++-
.../master/procedure/TestDeleteTableProcedure.java | 66 ---
5 files changed, 43 insertions(+), 181 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
index 0a6a469..d77b95f 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
@@ -23,7 +23,6 @@ import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.function.Supplier;
-import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.DoNotRetryIOException;
import org.apache.hadoop.hbase.HBaseIOException;
@@ -336,41 +335,22 @@ public class CreateTableProcedure
final TableDescriptor tableDescriptor, List newRegions,
final CreateHdfsRegions hdfsRegionHandler) throws IOException {
final MasterFileSystem mfs = env.getMasterServices().getMasterFileSystem();
-final Path tempdir = mfs.getTempDir();
// 1. Create Table Descriptor
// using a copy of descriptor, table will be created enabling first
-final Path tempTableDir = CommonFSUtils.getTableDir(tempdir,
tableDescriptor.getTableName());
+final Path tableDir = CommonFSUtils.getTableDir(mfs.getRootDir(),
+ tableDescriptor.getTableName());
((FSTableDescriptors)(env.getMasterServices().getTableDescriptors()))
-.createTableDescriptorForTableDirectory(tempTableDir, tableDescriptor,
false);
+.createTableDescriptorForTableDirectory(
+ tableDir, tableDescriptor, false);
// 2. Create Regions
-newRegions = hdfsRegionHandler.createHdfsRegions(env, tempdir,
+newRegions = hdfsRegionHandler.createHdfsRegions(env, mfs.getRootDir(),
tableDescriptor.getTableName(), newRegions);
-// 3. Move Table temp directory to the hbase root location
-moveTempDirectoryToHBaseRoot(env, tableDescriptor, tempTableDir);
-
return newRegions;
}
- protected static void moveTempDirectoryToHBaseRoot(
-final MasterProcedureEnv env,
-final TableDescriptor tableDescriptor,
-final Path tempTableDir) throws IOException {
-final MasterFileSystem mfs = env.getMasterServices().getMasterFileSystem();
-final Path tableDir =
- CommonFSUtils.getTableDir(mfs.getRootDir(),
tableDescriptor.getTableName());
-FileSystem fs = mfs.getFileSystem();
-if (!fs.delete(tableDir, true) && fs.exists(tableDir)) {
- throw new IOException("Couldn't delete " + tableDir);
-}
-if (!fs.rename(tempTableDir, tableDir)) {
- throw new IOException("Unable to move table from temp=" + tempTableDir +
-" to hbase root=" + tableDir);
-}
- }
-
protected static List addTableToMeta(final MasterProcedureEnv
env,
final TableDescriptor tableDescriptor, final List regions)
throws IOException {
assert (regions != null && regions.size() > 0) : "expected at least 1
region, got " + regions;
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
index 80dddc7..297efc2 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
@@ -20,10 +20,8 @@ package org.apache.hadoop.hbase.master.procedure;
import java.io.IOException;
import java.util.ArrayList;
-import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
-import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.MetaTableAccessor;
@@ -277,82 +275,49 @@ public class DeleteTableProcedure
final boolean archive) throws IOException {
final MasterFileSystem mfs = env.getMasterServices().getMasterFileSystem();
final FileSystem fs = mfs.getFileSystem();
-final Path tempdir = mfs.getTempDir();
final Path tableDir =
[hbase] 09/16: HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… (#3721)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 06db852aa009240f8baa4a79e7f688e82a17b8d0
Author: Wellington Ramos Chevreuil
AuthorDate: Wed Oct 13 15:48:13 2021 +0100
HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker…
(#3721)
Signed-off-by: Duo Zhang
Signed-off-by: Josh Elser
---
.../hbase/master/procedure/CreateTableProcedure.java | 6 ++
.../storefiletracker/FileBasedStoreFileTracker.java | 9 -
.../storefiletracker/MigrationStoreFileTracker.java | 12 +---
.../storefiletracker/StoreFileTracker.java | 5 +++--
.../storefiletracker/StoreFileTrackerBase.java | 9 -
.../storefiletracker/StoreFileTrackerFactory.java| 20 ++--
.../master/procedure/TestCreateTableProcedure.java | 16
.../storefiletracker/TestStoreFileTracker.java | 2 +-
8 files changed, 49 insertions(+), 30 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
index ee8e51f..0a6a469 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
@@ -34,7 +34,6 @@ import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.RegionInfo;
import org.apache.hadoop.hbase.client.RegionReplicaUtil;
import org.apache.hadoop.hbase.client.TableDescriptor;
-import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.client.TableState;
import org.apache.hadoop.hbase.master.MasterCoprocessorHost;
import org.apache.hadoop.hbase.master.MasterFileSystem;
@@ -290,9 +289,8 @@ public class CreateTableProcedure
(newRegions != null ? newRegions.size() : 0));
}
-TableDescriptorBuilder builder =
TableDescriptorBuilder.newBuilder(tableDescriptor);
-StoreFileTrackerFactory.persistTrackerConfig(env.getMasterConfiguration(),
builder);
-tableDescriptor = builder.build();
+tableDescriptor =
StoreFileTrackerFactory.updateWithTrackerConfigs(env.getMasterConfiguration(),
+ tableDescriptor);
final MasterCoprocessorHost cpHost = env.getMasterCoprocessorHost();
if (cpHost != null) {
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
index c370b87..4da7911 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
@@ -56,7 +56,14 @@ class FileBasedStoreFileTracker extends StoreFileTrackerBase
{
public FileBasedStoreFileTracker(Configuration conf, boolean
isPrimaryReplica, StoreContext ctx) {
super(conf, isPrimaryReplica, ctx);
-backedFile = new StoreFileListFile(ctx);
+//CreateTableProcedure needs to instantiate the configured SFT impl, in
order to update table
+//descriptors with the SFT impl specific configs. By the time this
happens, the table has no
+//regions nor stores yet, so it can't create a proper StoreContext.
+if (ctx != null) {
+ backedFile = new StoreFileListFile(ctx);
+} else {
+ backedFile = null;
+}
}
@Override
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
index 1946d4b..230c1ec 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
@@ -21,6 +21,7 @@ import java.io.IOException;
import java.util.Collection;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.procedure2.util.StringUtils;
import org.apache.hadoop.hbase.regionserver.StoreContext;
@@ -88,17 +89,6 @@ class MigrationStoreFileTracker extends StoreFileTrackerBase
{
"Should not call this method on " + getClass().getSimpleName());
}
- @Override
- public void persistConfiguration(TableDescriptorBuilder builder) {
-super.persistConfiguration(builder);
-if (StringUtils.isEmpty(builder.getValue(SRC_IMPL))) {
-
[hbase] 13/16: HBASE-26271 Cleanup the broken store files under data directory (#3786)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit a288365f929f46295dff88608f3a1660b9e98a7c
Author: BukrosSzabolcs
AuthorDate: Tue Nov 9 17:19:00 2021 +0100
HBASE-26271 Cleanup the broken store files under data directory (#3786)
Signed-off-by: Duo Zhang
Signed-off-by: Josh Elser
Signed-off-by: Wellington Ramos Chevreuil
---
.../hadoop/hbase/mob/DefaultMobStoreCompactor.java | 6 +-
.../regionserver/AbstractMultiFileWriter.java | 6 +-
.../hbase/regionserver/BrokenStoreFileCleaner.java | 202 ++
.../regionserver/DateTieredMultiFileWriter.java| 2 +-
.../hbase/regionserver/HRegionFileSystem.java | 2 +-
.../hadoop/hbase/regionserver/HRegionServer.java | 27 +++
.../apache/hadoop/hbase/regionserver/HStore.java | 6 +
.../hadoop/hbase/regionserver/StoreEngine.java | 21 ++
.../hbase/regionserver/StripeMultiFileWriter.java | 2 +-
.../compactions/AbstractMultiOutputCompactor.java | 4 +-
.../hbase/regionserver/compactions/Compactor.java | 45 -
.../compactions/DateTieredCompactor.java | 6 +-
.../regionserver/compactions/DefaultCompactor.java | 9 +-
.../regionserver/compactions/StripeCompactor.java | 2 +-
.../FileBasedStoreFileTracker.java | 2 +-
.../MigrationStoreFileTracker.java | 2 +-
.../storefiletracker/StoreFileTracker.java | 6 +
.../storefiletracker/StoreFileTrackerBase.java | 6 -
.../hbase/snapshot/RestoreSnapshotHelper.java | 2 +-
.../hadoop/hbase/mob/FaultyMobStoreCompactor.java | 3 +-
.../regionserver/TestBrokenStoreFileCleaner.java | 225 +
.../hbase/regionserver/TestCompactorMemLeak.java | 4 +-
.../storefiletracker/TestStoreFileTracker.java | 1 -
23 files changed, 556 insertions(+), 35 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
index 01fe000..15f0a73 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
@@ -38,7 +38,6 @@ import org.apache.hadoop.hbase.DoNotRetryIOException;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.PrivateCellUtil;
import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.regionserver.CellSink;
import org.apache.hadoop.hbase.regionserver.HMobStore;
import org.apache.hadoop.hbase.regionserver.HStore;
import org.apache.hadoop.hbase.regionserver.HStoreFile;
@@ -286,7 +285,6 @@ public class DefaultMobStoreCompactor extends
DefaultCompactor {
*
* @param fd File details
* @param scanner Where to read from.
- * @param writer Where to write to.
* @param smallestReadPoint Smallest read point.
* @param cleanSeqId When true, remove seqId(used to be mvcc) value which is
<= smallestReadPoint
* @param throughputController The compaction throughput controller.
@@ -295,7 +293,7 @@ public class DefaultMobStoreCompactor extends
DefaultCompactor {
* @return Whether compaction ended; false if it was interrupted for any
reason.
*/
@Override
- protected boolean performCompaction(FileDetails fd, InternalScanner scanner,
CellSink writer,
+ protected boolean performCompaction(FileDetails fd, InternalScanner scanner,
long smallestReadPoint, boolean cleanSeqId, ThroughputController
throughputController,
boolean major, int numofFilesToCompact) throws IOException {
long bytesWrittenProgressForLog = 0;
@@ -665,7 +663,7 @@ public class DefaultMobStoreCompactor extends
DefaultCompactor {
@Override
- protected List commitWriter(StoreFileWriter writer, FileDetails fd,
+ protected List commitWriter(FileDetails fd,
CompactionRequestImpl request) throws IOException {
List newFiles = Lists.newArrayList(writer.getPath());
writer.appendMetadata(fd.maxSeqId, request.isAllFiles(),
request.getFiles());
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMultiFileWriter.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMultiFileWriter.java
index f250304..82c3867 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMultiFileWriter.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMultiFileWriter.java
@@ -110,7 +110,11 @@ public abstract class AbstractMultiFileWriter implements
CellSink, ShipperListen
return paths;
}
- protected abstract Collection writers();
+ /**
+ * Returns all writers. This is used to prevent deleting currently writen
storefiles
+ * during cleanup.
+ */
+ public abstract Collection writers();
[hbase] 11/16: HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl (#3749)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 08d117197cb9de241414b1a22a037d106f7687f2
Author: Wellington Ramos Chevreuil
AuthorDate: Fri Oct 22 16:56:15 2021 +0100
HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl
(#3749)
Signed-off-by: Duo Zhang
---
.../java/org/apache/hadoop/hbase/io/HFileLink.java | 63 ++
.../master/procedure/CloneSnapshotProcedure.java | 52 +++-
.../storefiletracker/StoreFileTrackerBase.java | 1 +
.../hbase/snapshot/RestoreSnapshotHelper.java | 95 ++
.../TestCloneSnapshotProcedureFileBasedSFT.java| 42 ++
5 files changed, 157 insertions(+), 96 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java
index 74836ce..fbed724 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java
@@ -201,7 +201,6 @@ public class HFileLink extends FileLink {
return isHFileLink(path.getName());
}
-
/**
* @param fileName File name to check.
* @return True if the path is a HFileLink.
@@ -322,10 +321,10 @@ public class HFileLink extends FileLink {
* @param dstFamilyPath - Destination path (table/region/cf/)
* @param hfileRegionInfo - Linked HFile Region Info
* @param hfileName - Linked HFile name
- * @return true if the file is created, otherwise the file exists.
- * @throws IOException on file or parent directory creation failure
+ * @return the file link name.
+ * @throws IOException on file or parent directory creation failure.
*/
- public static boolean create(final Configuration conf, final FileSystem fs,
+ public static String create(final Configuration conf, final FileSystem fs,
final Path dstFamilyPath, final RegionInfo hfileRegionInfo,
final String hfileName) throws IOException {
return create(conf, fs, dstFamilyPath, hfileRegionInfo, hfileName, true);
@@ -343,10 +342,10 @@ public class HFileLink extends FileLink {
* @param hfileRegionInfo - Linked HFile Region Info
* @param hfileName - Linked HFile name
* @param createBackRef - Whether back reference should be created. Defaults
to true.
- * @return true if the file is created, otherwise the file exists.
- * @throws IOException on file or parent directory creation failure
+ * @return the file link name.
+ * @throws IOException on file or parent directory creation failure.
*/
- public static boolean create(final Configuration conf, final FileSystem fs,
+ public static String create(final Configuration conf, final FileSystem fs,
final Path dstFamilyPath, final RegionInfo hfileRegionInfo,
final String hfileName, final boolean createBackRef) throws IOException {
TableName linkedTable = hfileRegionInfo.getTable();
@@ -366,17 +365,18 @@ public class HFileLink extends FileLink {
* @param linkedTable - Linked Table Name
* @param linkedRegion - Linked Region Name
* @param hfileName - Linked HFile name
- * @return true if the file is created, otherwise the file exists.
- * @throws IOException on file or parent directory creation failure
+ * @return the file link name.
+ * @throws IOException on file or parent directory creation failure.
*/
- public static boolean create(final Configuration conf, final FileSystem fs,
+ public static String create(final Configuration conf, final FileSystem fs,
final Path dstFamilyPath, final TableName linkedTable, final String
linkedRegion,
final String hfileName) throws IOException {
return create(conf, fs, dstFamilyPath, linkedTable, linkedRegion,
hfileName, true);
}
/**
- * Create a new HFileLink
+ * Create a new HFileLink. In the event of link creation failure, this
method throws an
+ * IOException, so that the calling upper laying can decide on how to
proceed with this.
*
* It also adds a back-reference to the hfile back-reference directory
* to simplify the reference-count and the cleaning process.
@@ -388,10 +388,10 @@ public class HFileLink extends FileLink {
* @param linkedRegion - Linked Region Name
* @param hfileName - Linked HFile name
* @param createBackRef - Whether back reference should be created. Defaults
to true.
- * @return true if the file is created, otherwise the file exists.
- * @throws IOException on file or parent directory creation failure
+ * @return the file link name.
+ * @throws IOException on file or parent directory creation failure.
*/
- public static boolean create(final Configuration conf, final FileSystem fs,
+ public static String create(final Configuration conf, final FileSystem fs,
final Path dstFamilyPath, final TableName
[hbase] 12/16: HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configurations to TableDescriptor for existing tables (#3700)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 8bec26ea91c08fc1e5cdac658a7ced235a97efba
Author: GeorryHuang
AuthorDate: Sat Nov 6 22:20:12 2021 +0800
HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configurations
to TableDescriptor for existing tables (#3700)
Signed-off-by: Duo Zhang
Reviewed-by: Wellington Ramos Chevreuil
---
.../org/apache/hadoop/hbase/master/HMaster.java| 6 +
.../hbase/master/migrate/RollingUpgradeChore.java | 130 +
.../MigrateStoreFileTrackerProcedure.java | 48
.../migrate/TestMigrateStoreFileTracker.java | 107 +
4 files changed, 291 insertions(+)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index 1683b16..b80a2e2 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -131,6 +131,7 @@ import
org.apache.hadoop.hbase.master.http.MasterRedirectServlet;
import org.apache.hadoop.hbase.master.http.MasterStatusServlet;
import org.apache.hadoop.hbase.master.janitor.CatalogJanitor;
import org.apache.hadoop.hbase.master.locking.LockManager;
+import org.apache.hadoop.hbase.master.migrate.RollingUpgradeChore;
import org.apache.hadoop.hbase.master.normalizer.RegionNormalizerFactory;
import org.apache.hadoop.hbase.master.normalizer.RegionNormalizerManager;
import org.apache.hadoop.hbase.master.procedure.CreateTableProcedure;
@@ -376,6 +377,7 @@ public class HMaster extends
HBaseServerBase implements Maste
private ReplicationBarrierCleaner replicationBarrierCleaner;
private MobFileCleanerChore mobFileCleanerChore;
private MobFileCompactionChore mobFileCompactionChore;
+ private RollingUpgradeChore rollingUpgradeChore;
// used to synchronize the mobCompactionStates
private final IdLock mobCompactionLock = new IdLock();
// save the information of mob compactions in tables.
@@ -1222,6 +1224,9 @@ public class HMaster extends
HBaseServerBase implements Maste
LOG.debug("Balancer post startup initialization complete, took " + (
(EnvironmentEdgeManager.currentTime() - start) / 1000) + " seconds");
}
+
+this.rollingUpgradeChore = new RollingUpgradeChore(this);
+getChoreService().scheduleChore(rollingUpgradeChore);
}
private void createMissingCFsInMetaDuringUpgrade(
@@ -1713,6 +1718,7 @@ public class HMaster extends
HBaseServerBase implements Maste
shutdownChore(snapshotCleanerChore);
shutdownChore(hbckChore);
shutdownChore(regionsRecoveryChore);
+shutdownChore(rollingUpgradeChore);
}
/**
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/migrate/RollingUpgradeChore.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/migrate/RollingUpgradeChore.java
new file mode 100644
index 000..3896b41
--- /dev/null
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/migrate/RollingUpgradeChore.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.master.migrate;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.procedure.MasterProcedureEnv;
+import org.apache.hadoop.hbase.procedure2.ProcedureExecutor;
+import
org.apache.hadoop.hbase.regionserver.storefiletracker.MigrateStoreFileTrackerProcedure;
+import
[hbase] 07/16: HBASE-26264 Add more checks to prevent misconfiguration on store file tracker (#3681)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 5ff0f98a5367326753bc0e06d8726b66d109238c
Author: Duo Zhang
AuthorDate: Wed Sep 15 23:00:03 2021 +0800
HBASE-26264 Add more checks to prevent misconfiguration on store file
tracker (#3681)
Signed-off-by: Josh Elser
---
.../assignment/MergeTableRegionsProcedure.java | 3 +-
.../assignment/SplitTableRegionProcedure.java | 3 +-
.../master/procedure/CreateTableProcedure.java | 8 +-
.../master/procedure/ModifyTableProcedure.java | 5 +
.../hbase/regionserver/HRegionFileSystem.java | 2 +-
.../MigrationStoreFileTracker.java | 8 +
.../storefiletracker/StoreFileTrackerFactory.java | 173 +--
.../TestChangeStoreFileTracker.java| 242 +
8 files changed, 422 insertions(+), 22 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
index e6bbe44..e9051da 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
@@ -614,8 +614,7 @@ public class MergeTableRegionsProcedure
String family = hcd.getNameAsString();
Configuration trackerConfig =
StoreFileTrackerFactory.mergeConfigurations(env.getMasterConfiguration(), htd,
hcd);
- StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig,
true,
-family, regionFs);
+ StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig,
family, regionFs);
final Collection storeFiles = tracker.load();
if (storeFiles != null && storeFiles.size() > 0) {
final Configuration storeConfiguration =
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
index ff16dc5..aa0c938 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
@@ -670,8 +670,7 @@ public class SplitTableRegionProcedure
String family = cfd.getNameAsString();
Configuration trackerConfig = StoreFileTrackerFactory.
mergeConfigurations(env.getMasterConfiguration(), htd,
htd.getColumnFamily(cfd.getName()));
- StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig,
true,
-family, regionFs);
+ StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig,
family, regionFs);
Collection sfis = tracker.load();
if (sfis == null) {
continue;
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
index dccea55..ee8e51f 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
@@ -277,15 +277,17 @@ public class CreateTableProcedure
MasterProcedureUtil.checkGroupNotEmpty(rsGroupInfo, forWhom);
}
+// check for store file tracker configurations
+StoreFileTrackerFactory.checkForCreateTable(env.getMasterConfiguration(),
tableDescriptor);
+
return true;
}
private void preCreate(final MasterProcedureEnv env)
throws IOException, InterruptedException {
if (!getTableName().isSystemTable()) {
- ProcedureSyncWait.getMasterQuotaManager(env)
-.checkNamespaceTableAndRegionQuota(
- getTableName(), (newRegions != null ? newRegions.size() : 0));
+
ProcedureSyncWait.getMasterQuotaManager(env).checkNamespaceTableAndRegionQuota(getTableName(),
+(newRegions != null ? newRegions.size() : 0));
}
TableDescriptorBuilder builder =
TableDescriptorBuilder.newBuilder(tableDescriptor);
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
index 247dd9c..1640644 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
@@ -38,6 +38,7 @@ import org.apache.hadoop.hbase.client.TableDescriptor;
import
[hbase] 05/16: HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor when creating table (#3666)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 0ee1689332856440334c1a0040215d5c86b9705a
Author: Wellington Ramos Chevreuil
AuthorDate: Sun Sep 12 14:14:03 2021 +0100
HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor
when creating table (#3666)
Signed-off-by: Duo Zhang
---
.../master/procedure/CreateTableProcedure.java | 7 ++
.../hbase/regionserver/HRegionFileSystem.java | 2 +-
.../MigrationStoreFileTracker.java | 8 +++
.../storefiletracker/StoreFileTracker.java | 8 +++
.../storefiletracker/StoreFileTrackerBase.java | 13 +++
.../storefiletracker/StoreFileTrackerFactory.java | 25 +++---
.../org/apache/hadoop/hbase/client/TestAdmin.java | 6 ++
.../org/apache/hadoop/hbase/client/TestAdmin3.java | 6 ++
.../hbase/client/TestAsyncTableAdminApi.java | 6 ++
.../hbase/client/TestAsyncTableAdminApi3.java | 6 ++
.../procedure/MasterProcedureTestingUtility.java | 7 ++
.../master/procedure/TestCreateTableProcedure.java | 17 +++
.../storefiletracker/TestStoreFileTracker.java | 14 +---
13 files changed, 113 insertions(+), 12 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
index 2313e70..dccea55 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.hbase.master.procedure;
+
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@@ -33,10 +34,12 @@ import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.RegionInfo;
import org.apache.hadoop.hbase.client.RegionReplicaUtil;
import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.client.TableState;
import org.apache.hadoop.hbase.master.MasterCoprocessorHost;
import org.apache.hadoop.hbase.master.MasterFileSystem;
import org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer;
+import
org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTrackerFactory;
import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.util.CommonFSUtils;
@@ -285,6 +288,10 @@ public class CreateTableProcedure
getTableName(), (newRegions != null ? newRegions.size() : 0));
}
+TableDescriptorBuilder builder =
TableDescriptorBuilder.newBuilder(tableDescriptor);
+StoreFileTrackerFactory.persistTrackerConfig(env.getMasterConfiguration(),
builder);
+tableDescriptor = builder.build();
+
final MasterCoprocessorHost cpHost = env.getMasterCoprocessorHost();
if (cpHost != null) {
final RegionInfo[] regions = newRegions == null ? null :
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
index cb30432..aa0ee27 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
@@ -610,7 +610,7 @@ public class HRegionFileSystem {
writeRegionInfoFileContent(conf, fs, regionInfoFile, regionInfoContent);
HRegionFileSystem regionFs = HRegionFileSystem.openRegionFromFileSystem(
env.getMasterConfiguration(), fs, getTableDir(), regionInfo, false);
- insertRegionFilesIntoStoreTracker(allRegionFiles, env, regionFs);
+insertRegionFilesIntoStoreTracker(allRegionFiles, env, regionFs);
}
return regionDir;
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
index e486e6d..483a240 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
@@ -21,6 +21,7 @@ import java.io.IOException;
import java.util.Collection;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.regionserver.StoreContext;
import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
[hbase] 02/16: HBASE-25988 Store the store file list by a file (#3578)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 43b40e9374f0148b5b3a89f6d65611435c5181f6
Author: Duo Zhang
AuthorDate: Thu Aug 26 18:51:12 2021 +0800
HBASE-25988 Store the store file list by a file (#3578)
Signed-off-by: Wellington Chevreuil
---
.../protobuf/server/region/StoreFileTracker.proto | 29 +++--
.../hadoop/hbase/regionserver/StoreContext.java| 5 +
.../hadoop/hbase/regionserver/StoreEngine.java | 8 +-
.../storefiletracker/DefaultStoreFileTracker.java | 5 +-
.../FileBasedStoreFileTracker.java | 142 +
.../storefiletracker/StoreFileListFile.java| 142 +
.../storefiletracker/StoreFileTrackerBase.java | 11 +-
.../storefiletracker/StoreFileTrackerFactory.java | 12 +-
.../TestRegionWithFileBasedStoreFileTracker.java | 109
9 files changed, 430 insertions(+), 33 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerFactory.java
b/hbase-protocol-shaded/src/main/protobuf/server/region/StoreFileTracker.proto
similarity index 57%
copy from
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerFactory.java
copy to
hbase-protocol-shaded/src/main/protobuf/server/region/StoreFileTracker.proto
index 4f7231b..2a269ea 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerFactory.java
+++
b/hbase-protocol-shaded/src/main/protobuf/server/region/StoreFileTracker.proto
@@ -15,21 +15,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.hadoop.hbase.regionserver.storefiletracker;
+syntax = "proto2";
+// This file contains protocol buffers that are used for store file tracker.
+package hbase.pb;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.regionserver.StoreContext;
-import org.apache.yetus.audience.InterfaceAudience;
+option java_package = "org.apache.hadoop.hbase.shaded.protobuf.generated";
+option java_outer_classname = "StoreFileTrackerProtos";
+option java_generic_services = true;
+option java_generate_equals_and_hash = true;
+option optimize_for = SPEED;
-/**
- * Factory method for creating store file tracker.
- */
-@InterfaceAudience.Private
-public final class StoreFileTrackerFactory {
+message StoreFileEntry {
+ required string name = 1;
+ required uint64 size = 2;
+}
- public static StoreFileTracker create(Configuration conf, TableName
tableName,
-boolean isPrimaryReplica, StoreContext ctx) {
-return new DefaultStoreFileTracker(conf, tableName, isPrimaryReplica, ctx);
- }
+message StoreFileList {
+ required uint64 timestamp = 1;
+ repeated StoreFileEntry store_file = 2;
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreContext.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreContext.java
index 2a9f968..588f8f4 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreContext.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreContext.java
@@ -22,6 +22,7 @@ import java.util.Collection;
import java.util.function.Supplier;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
import org.apache.hadoop.hbase.client.RegionInfo;
import org.apache.hadoop.hbase.io.HeapSize;
@@ -109,6 +110,10 @@ public final class StoreContext implements HeapSize {
return coprocessorHost;
}
+ public TableName getTableName() {
+return getRegionInfo().getTable();
+ }
+
public RegionInfo getRegionInfo() {
return regionFileSystem.getRegionInfo();
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java
index 4033c33..0486729 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java
@@ -173,9 +173,9 @@ public abstract class StoreEnginehttp://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver.storefiletracker;
+
+import java.io.IOException;
+import
[hbase] 10/16: HBASE-26386 Refactor StoreFileTracker implementations to expose the set method (#3774)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit e4e7cf80b74585e5a1904e658c269a9efa459b74
Author: Duo Zhang
AuthorDate: Thu Oct 21 10:27:45 2021 +0800
HBASE-26386 Refactor StoreFileTracker implementations to expose the set
method (#3774)
Signed-off-by: Wellington Chevreuil
---
.../regionserver/storefiletracker/DefaultStoreFileTracker.java | 2 +-
.../storefiletracker/FileBasedStoreFileTracker.java| 2 +-
.../storefiletracker/MigrationStoreFileTracker.java| 5 +
.../hbase/regionserver/storefiletracker/StoreFileTracker.java | 7 +--
.../regionserver/storefiletracker/StoreFileTrackerBase.java| 10 --
5 files changed, 8 insertions(+), 18 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
index a13b75b..b1e298d 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
@@ -62,7 +62,7 @@ class DefaultStoreFileTracker extends StoreFileTrackerBase {
}
@Override
- void set(List files) {
+ public void set(List files) {
// NOOP
}
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
index 4da7911..079b59b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
@@ -148,7 +148,7 @@ class FileBasedStoreFileTracker extends
StoreFileTrackerBase {
}
@Override
- void set(List files) throws IOException {
+ public void set(List files) throws IOException {
synchronized (storefiles) {
storefiles.clear();
StoreFileList.Builder builder = StoreFileList.newBuilder();
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
index 230c1ec..a6648f2 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
@@ -21,9 +21,6 @@ import java.io.IOException;
import java.util.Collection;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.client.TableDescriptor;
-import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
-import org.apache.hadoop.hbase.procedure2.util.StringUtils;
import org.apache.hadoop.hbase.regionserver.StoreContext;
import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
import org.apache.yetus.audience.InterfaceAudience;
@@ -84,7 +81,7 @@ class MigrationStoreFileTracker extends StoreFileTrackerBase {
}
@Override
- void set(List files) {
+ public void set(List files) {
throw new UnsupportedOperationException(
"Should not call this method on " + getClass().getSimpleName());
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
index fd8f7c9..f56a0dd 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
@@ -20,8 +20,6 @@ package org.apache.hadoop.hbase.regionserver.storefiletracker;
import java.io.IOException;
import java.util.Collection;
import java.util.List;
-
-import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.regionserver.CreateStoreFileWriterParams;
import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
@@ -70,6 +68,11 @@ public interface StoreFileTracker {
throws IOException;
/**
+ * Set the store files.
+ */
+ void set(List files) throws IOException;
+
+ /**
* Create a writer for writing new store files.
* @return Writer for a new StoreFile
*/
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerBase.java
[hbase] 08/16: HBASE-26280 Use store file tracker when snapshoting (#3685)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit fc4f6d10e3736c976c0be91e604cbc82d42acdbf
Author: Duo Zhang
AuthorDate: Fri Sep 17 09:40:44 2021 +0800
HBASE-26280 Use store file tracker when snapshoting (#3685)
Signed-off-by: Wellington Chevreuil
Reviewed-by: Josh Elser
---
.../assignment/MergeTableRegionsProcedure.java | 5 +-
.../assignment/SplitTableRegionProcedure.java | 5 +-
.../hbase/regionserver/HRegionFileSystem.java | 10 +--
.../storefiletracker/StoreFileTrackerFactory.java | 17 ++---
.../hadoop/hbase/snapshot/SnapshotManifest.java| 42 ++--
...oneSnapshotFromClientCloneLinksAfterDelete.java | 4 +-
.../hbase/client/TestMobSnapshotFromClient.java| 7 +-
.../hbase/client/TestSnapshotFromClient.java | 30 +++--
.../hadoop/hbase/regionserver/TestHStoreFile.java | 6 +-
.../hbase/snapshot/MobSnapshotTestingUtils.java| 74 +++---
.../hbase/snapshot/SnapshotTestingUtils.java | 16 ++---
11 files changed, 107 insertions(+), 109 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
index e9051da..0f41db5 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
@@ -612,9 +612,8 @@ public class MergeTableRegionsProcedure
List mergedFiles = new ArrayList<>();
for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
String family = hcd.getNameAsString();
- Configuration trackerConfig =
-
StoreFileTrackerFactory.mergeConfigurations(env.getMasterConfiguration(), htd,
hcd);
- StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig,
family, regionFs);
+ StoreFileTracker tracker =
+StoreFileTrackerFactory.create(env.getMasterConfiguration(), htd, hcd,
regionFs);
final Collection storeFiles = tracker.load();
if (storeFiles != null && storeFiles.size() > 0) {
final Configuration storeConfiguration =
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
index aa0c938..effdba4 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
@@ -668,9 +668,8 @@ public class SplitTableRegionProcedure
new HashMap>(htd.getColumnFamilyCount());
for (ColumnFamilyDescriptor cfd : htd.getColumnFamilies()) {
String family = cfd.getNameAsString();
- Configuration trackerConfig = StoreFileTrackerFactory.
-mergeConfigurations(env.getMasterConfiguration(), htd,
htd.getColumnFamily(cfd.getName()));
- StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig,
family, regionFs);
+ StoreFileTracker tracker =
+StoreFileTrackerFactory.create(env.getMasterConfiguration(), htd, cfd,
regionFs);
Collection sfis = tracker.load();
if (sfis == null) {
continue;
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
index e78d8ad..8110025 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
@@ -598,7 +598,6 @@ public class HRegionFileSystem {
* to the proper location in the filesystem.
*
* @param regionInfo daughter {@link
org.apache.hadoop.hbase.client.RegionInfo}
- * @throws IOException
*/
public Path commitDaughterRegion(final RegionInfo regionInfo, List
allRegionFiles,
MasterProcedureEnv env) throws IOException {
@@ -625,12 +624,8 @@ public class HRegionFileSystem {
Map> fileInfoMap = new HashMap<>();
for(Path file : allFiles) {
String familyName = file.getParent().getName();
- trackerMap.computeIfAbsent(familyName, t -> {
-Configuration config =
StoreFileTrackerFactory.mergeConfigurations(conf, tblDesc,
- tblDesc.getColumnFamily(Bytes.toBytes(familyName)));
-return StoreFileTrackerFactory.
- create(config, familyName, regionFs);
- });
+ trackerMap.computeIfAbsent(familyName, t ->
StoreFileTrackerFactory.create(conf, tblDesc,
+tblDesc.getColumnFamily(Bytes.toBytes(familyName)), regionFs));
[hbase] 06/16: HBASE-26248 Should find a suitable way to let users specify the store file tracker implementation (#3665)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 2052e80e5da50ec0598cc7c873de0357a4f894a5
Author: Duo Zhang
AuthorDate: Tue Sep 14 16:28:21 2021 +0800
HBASE-26248 Should find a suitable way to let users specify the store file
tracker implementation (#3665)
Signed-off-by: Wellington Chevreuil
---
.../MigrationStoreFileTracker.java | 13 ++-
.../storefiletracker/StoreFileTracker.java | 7 +-
.../storefiletracker/StoreFileTrackerBase.java | 12 +--
.../storefiletracker/StoreFileTrackerFactory.java | 97 +++---
.../org/apache/hadoop/hbase/client/TestAdmin.java | 6 +-
.../org/apache/hadoop/hbase/client/TestAdmin3.java | 6 +-
.../hbase/client/TestAsyncTableAdminApi.java | 6 +-
.../hbase/client/TestAsyncTableAdminApi3.java | 6 +-
.../procedure/MasterProcedureTestingUtility.java | 6 +-
.../master/procedure/TestCreateTableProcedure.java | 6 +-
.../regionserver/TestMergesSplitsAddToTracker.java | 4 +-
.../TestMigrationStoreFileTracker.java | 27 +++---
.../TestRegionWithFileBasedStoreFileTracker.java | 3 +-
.../TestStoreFileTrackerFactory.java | 58 +
14 files changed, 202 insertions(+), 55 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
index 483a240..3eeef90 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
@@ -22,6 +22,7 @@ import java.util.Collection;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.procedure2.util.StringUtils;
import org.apache.hadoop.hbase.regionserver.StoreContext;
import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
import org.apache.yetus.audience.InterfaceAudience;
@@ -44,8 +45,8 @@ class MigrationStoreFileTracker extends StoreFileTrackerBase {
public MigrationStoreFileTracker(Configuration conf, boolean
isPrimaryReplica, StoreContext ctx) {
super(conf, isPrimaryReplica, ctx);
-this.src = StoreFileTrackerFactory.create(conf, SRC_IMPL,
isPrimaryReplica, ctx);
-this.dst = StoreFileTrackerFactory.create(conf, DST_IMPL,
isPrimaryReplica, ctx);
+this.src = StoreFileTrackerFactory.createForMigration(conf, SRC_IMPL,
isPrimaryReplica, ctx);
+this.dst = StoreFileTrackerFactory.createForMigration(conf, DST_IMPL,
isPrimaryReplica, ctx);
Preconditions.checkArgument(!src.getClass().equals(dst.getClass()),
"src and dst is the same: %s", src.getClass());
}
@@ -90,7 +91,11 @@ class MigrationStoreFileTracker extends StoreFileTrackerBase
{
@Override
public void persistConfiguration(TableDescriptorBuilder builder) {
super.persistConfiguration(builder);
-builder.setValue(SRC_IMPL, src.getClass().getName());
-builder.setValue(DST_IMPL, dst.getClass().getName());
+if (StringUtils.isEmpty(builder.getValue(SRC_IMPL))) {
+ builder.setValue(SRC_IMPL, src.getTrackerName());
+}
+if (StringUtils.isEmpty(builder.getValue(DST_IMPL))) {
+ builder.setValue(DST_IMPL, dst.getTrackerName());
+}
}
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
index 81fa1a9..59fe7ef 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
@@ -75,7 +75,12 @@ public interface StoreFileTracker {
StoreFileWriter createWriter(CreateStoreFileWriterParams params) throws
IOException;
/**
- * Saves StoreFileTracker implementations specific configs into the table
descriptors.
+ * Saves StoreFileTracker implementations specific configurations into the
table descriptors.
+ *
+ * This is used to avoid accidentally data loss when changing the cluster
level store file tracker
+ * implementation, and also possible misconfiguration between master and
region servers.
+ *
+ * See HBASE-26246 for more details.
* @param builder The table descriptor builder for the given table.
*/
void persistConfiguration(TableDescriptorBuilder builder);
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerBase.java
[hbase] 01/16: HBASE-26064 Introduce a StoreFileTracker to abstract the store file tracking logic
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 6aaef89789792cd7dd1483d10276273b20a35fa3
Author: Duo Zhang
AuthorDate: Thu Jul 29 18:35:19 2021 +0800
HBASE-26064 Introduce a StoreFileTracker to abstract the store file
tracking logic
Signed-off-by: Wellington Chevreuil
---
.../hadoop/hbase/mob/DefaultMobStoreCompactor.java | 22 +-
.../hadoop/hbase/mob/DefaultMobStoreFlusher.java | 4 +-
.../regionserver/CreateStoreFileWriterParams.java | 134
.../hbase/regionserver/DateTieredStoreEngine.java | 5 +-
.../hbase/regionserver/DefaultStoreEngine.java | 5 +-
.../hbase/regionserver/DefaultStoreFlusher.java| 11 +-
.../hadoop/hbase/regionserver/HMobStore.java | 3 +-
.../hbase/regionserver/HRegionFileSystem.java | 10 +-
.../apache/hadoop/hbase/regionserver/HStore.java | 729 +
.../hadoop/hbase/regionserver/StoreContext.java| 9 +
.../hadoop/hbase/regionserver/StoreEngine.java | 461 -
.../hbase/regionserver/StoreFileManager.java | 9 +
.../hadoop/hbase/regionserver/StoreFlusher.java| 9 +-
.../hadoop/hbase/regionserver/StoreUtils.java | 37 +-
.../hbase/regionserver/StripeStoreEngine.java | 9 +-
.../hbase/regionserver/StripeStoreFlusher.java | 9 +-
.../compactions/AbstractMultiOutputCompactor.java | 7 +-
.../hbase/regionserver/compactions/Compactor.java | 36 +-
.../regionserver/compactions/DefaultCompactor.java | 16 +-
.../storefiletracker/DefaultStoreFileTracker.java | 61 ++
.../storefiletracker/StoreFileTracker.java | 75 +++
.../storefiletracker/StoreFileTrackerBase.java | 178 +
.../storefiletracker/StoreFileTrackerFactory.java | 35 +
.../util/compaction/MajorCompactionRequest.java| 1 -
.../org/apache/hadoop/hbase/TestIOFencing.java | 12 +-
.../regionserver/TestCacheOnWriteInSchema.java | 6 +-
.../hbase/regionserver/TestDefaultStoreEngine.java | 5 +-
.../hadoop/hbase/regionserver/TestHRegion.java | 4 +-
.../hadoop/hbase/regionserver/TestHStore.java | 33 +-
.../TestRegionMergeTransactionOnCluster.java | 6 +-
.../regionserver/TestStoreFileRefresherChore.java | 3 +-
.../regionserver/TestStoreScannerClosure.java | 6 +-
.../hbase/regionserver/TestStripeStoreEngine.java | 2 +
.../compactions/TestDateTieredCompactor.java | 12 +-
.../compactions/TestStripeCompactionPolicy.java| 12 +-
.../compactions/TestStripeCompactor.java | 12 +-
36 files changed, 1261 insertions(+), 727 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
index c45fdff..01fe000 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
@@ -29,7 +29,6 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map.Entry;
import java.util.Optional;
-
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
@@ -144,17 +143,16 @@ public class DefaultMobStoreCompactor extends
DefaultCompactor {
};
private final CellSinkFactory writerFactory =
- new CellSinkFactory() {
-@Override
-public StoreFileWriter createWriter(InternalScanner scanner,
-
org.apache.hadoop.hbase.regionserver.compactions.Compactor.FileDetails fd,
-boolean shouldDropBehind, boolean major) throws IOException {
- // make this writer with tags always because of possible new cells
with tags.
- return store.createWriterInTmp(fd.maxKeyCount,
-major ? majorCompactionCompression : minorCompactionCompression,
-true, true, true, shouldDropBehind);
-}
- };
+new CellSinkFactory() {
+ @Override
+ public StoreFileWriter createWriter(InternalScanner scanner,
+org.apache.hadoop.hbase.regionserver.compactions.Compactor.FileDetails
fd,
+boolean shouldDropBehind, boolean major) throws IOException {
+// make this writer with tags always because of possible new cells
with tags.
+return store.getStoreEngine().createWriter(
+ createParams(fd, shouldDropBehind,
major).includeMVCCReadpoint(true).includesTag(true));
+ }
+};
public DefaultMobStoreCompactor(Configuration conf, HStore store) {
super(conf, store);
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java
index 480b85c..4a1dc7b 100644
---
[hbase] 04/16: HBASE-26224 Introduce a MigrationStoreFileTracker to support migrating from different store file tracker implementations (#3656)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 090b2fecf40491d46e0a650853266c9262bcafd1
Author: Duo Zhang
AuthorDate: Thu Sep 9 12:11:25 2021 +0800
HBASE-26224 Introduce a MigrationStoreFileTracker to support migrating from
different store file tracker implementations (#3656)
Signed-off-by: Wellington Chevreuil
---
.../storefiletracker/DefaultStoreFileTracker.java | 10 +-
.../FileBasedStoreFileTracker.java | 15 +-
.../MigrationStoreFileTracker.java | 88 ++
.../storefiletracker/StoreFileListFile.java| 6 +-
.../storefiletracker/StoreFileTrackerBase.java | 12 +-
.../storefiletracker/StoreFileTrackerFactory.java | 40 +++--
.../TestMigrationStoreFileTracker.java | 193 +
7 files changed, 343 insertions(+), 21 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
index 22e0513..a13b75b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hbase.regionserver.storefiletracker;
import java.io.IOException;
import java.util.Collection;
+import java.util.Collections;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
@@ -39,7 +40,9 @@ class DefaultStoreFileTracker extends StoreFileTrackerBase {
@Override
public List load() throws IOException {
-return
ctx.getRegionFileSystem().getStoreFiles(ctx.getFamily().getNameAsString());
+List files =
+
ctx.getRegionFileSystem().getStoreFiles(ctx.getFamily().getNameAsString());
+return files != null ? files : Collections.emptyList();
}
@Override
@@ -57,4 +60,9 @@ class DefaultStoreFileTracker extends StoreFileTrackerBase {
Collection newFiles) throws IOException {
// NOOP
}
+
+ @Override
+ void set(List files) {
+// NOOP
+ }
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
index de28b0e..c370b87 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
@@ -48,7 +48,7 @@ import
org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.
* storages.
*/
@InterfaceAudience.Private
-public class FileBasedStoreFileTracker extends StoreFileTrackerBase {
+class FileBasedStoreFileTracker extends StoreFileTrackerBase {
private final StoreFileListFile backedFile;
@@ -139,4 +139,17 @@ public class FileBasedStoreFileTracker extends
StoreFileTrackerBase {
}
}
}
+
+ @Override
+ void set(List files) throws IOException {
+synchronized (storefiles) {
+ storefiles.clear();
+ StoreFileList.Builder builder = StoreFileList.newBuilder();
+ for (StoreFileInfo info : files) {
+storefiles.put(info.getPath().getName(), info);
+builder.addStoreFile(toStoreFileEntry(info));
+ }
+ backedFile.update(builder);
+}
+ }
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
new file mode 100644
index 000..e486e6d
--- /dev/null
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package
[hbase] 03/16: HBASE-26079 Use StoreFileTracker when splitting and merging (#3617)
This is an automated email from the ASF dual-hosted git repository.
elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 6e053765e8141f6ad807325828c520b232d2600f
Author: Wellington Ramos Chevreuil
AuthorDate: Wed Sep 8 10:31:49 2021 +0100
HBASE-26079 Use StoreFileTracker when splitting and merging (#3617)
Signed-off-by: Duo Zhang
---
.../assignment/MergeTableRegionsProcedure.java | 22 +-
.../assignment/SplitTableRegionProcedure.java | 42 ++--
.../hbase/regionserver/HRegionFileSystem.java | 42 +++-
.../storefiletracker/DefaultStoreFileTracker.java | 4 +-
.../storefiletracker/StoreFileTracker.java | 1 -
.../storefiletracker/StoreFileTrackerFactory.java | 33 ++-
.../hbase/regionserver/TestDefaultStoreEngine.java | 1 +
.../regionserver/TestDirectStoreSplitsMerges.java | 32 ++-
.../hadoop/hbase/regionserver/TestHStoreFile.java | 19 +-
.../regionserver/TestMergesSplitsAddToTracker.java | 262 +
.../hbase/regionserver/TestStripeStoreEngine.java | 1 +
.../storefiletracker/TestStoreFileTracker.java}| 42 ++--
12 files changed, 434 insertions(+), 67 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
index da3d73e..e6bbe44 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
@@ -24,7 +24,6 @@ import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.stream.Stream;
-
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
@@ -56,6 +55,8 @@ import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
import org.apache.hadoop.hbase.regionserver.HStoreFile;
import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
import org.apache.hadoop.hbase.regionserver.StoreUtils;
+import org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTracker;
+import
org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTrackerFactory;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.CommonFSUtils;
import org.apache.hadoop.hbase.wal.WALSplitUtil;
@@ -587,30 +588,35 @@ public class MergeTableRegionsProcedure
final MasterFileSystem mfs = env.getMasterServices().getMasterFileSystem();
final Path tableDir = CommonFSUtils.getTableDir(mfs.getRootDir(),
regionsToMerge[0].getTable());
final FileSystem fs = mfs.getFileSystem();
-
+List mergedFiles = new ArrayList<>();
HRegionFileSystem mergeRegionFs =
HRegionFileSystem.createRegionOnFileSystem(
env.getMasterConfiguration(), fs, tableDir, mergedRegion);
for (RegionInfo ri: this.regionsToMerge) {
HRegionFileSystem regionFs = HRegionFileSystem.openRegionFromFileSystem(
env.getMasterConfiguration(), fs, tableDir, ri, false);
- mergeStoreFiles(env, regionFs, mergeRegionFs, mergedRegion);
+ mergedFiles.addAll(mergeStoreFiles(env, regionFs, mergeRegionFs,
mergedRegion));
}
assert mergeRegionFs != null;
-mergeRegionFs.commitMergedRegion();
+mergeRegionFs.commitMergedRegion(mergedFiles, env);
// Prepare to create merged regions
env.getAssignmentManager().getRegionStates().
getOrCreateRegionStateNode(mergedRegion).setState(State.MERGING_NEW);
}
- private void mergeStoreFiles(MasterProcedureEnv env, HRegionFileSystem
regionFs,
+ private List mergeStoreFiles(MasterProcedureEnv env, HRegionFileSystem
regionFs,
HRegionFileSystem mergeRegionFs, RegionInfo mergedRegion) throws
IOException {
final TableDescriptor htd = env.getMasterServices().getTableDescriptors()
.get(mergedRegion.getTable());
+List mergedFiles = new ArrayList<>();
for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
String family = hcd.getNameAsString();
- final Collection storeFiles =
regionFs.getStoreFiles(family);
+ Configuration trackerConfig =
+
StoreFileTrackerFactory.mergeConfigurations(env.getMasterConfiguration(), htd,
hcd);
+ StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig,
true,
+family, regionFs);
+ final Collection storeFiles = tracker.load();
if (storeFiles != null && storeFiles.size() > 0) {
final Configuration storeConfiguration =
StoreUtils.createStoreConfiguration(env.getMasterConfiguration(),
htd, hcd);
@@ -622,11 +628,13 @@ public class MergeTableRegionsProcedure
// is running in a regionserver's Store context, or we might not be
able
// to read the hfiles.
[hbase] branch master updated (6818ec2 -> f16b7b1)
This is an automated email from the ASF dual-hosted git repository. elserj pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 6818ec2 HBASE-26613 The logic of the method incrementIV in Encryption class has problem (#3968) new 6aaef897 HBASE-26064 Introduce a StoreFileTracker to abstract the store file tracking logic new 43b40e9 HBASE-25988 Store the store file list by a file (#3578) new 6e05376 HBASE-26079 Use StoreFileTracker when splitting and merging (#3617) new 090b2fe HBASE-26224 Introduce a MigrationStoreFileTracker to support migrating from different store file tracker implementations (#3656) new 0ee1689 HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor when creating table (#3666) new 2052e80 HBASE-26248 Should find a suitable way to let users specify the store file tracker implementation (#3665) new 5ff0f98 HBASE-26264 Add more checks to prevent misconfiguration on store file tracker (#3681) new fc4f6d1 HBASE-26280 Use store file tracker when snapshoting (#3685) new 06db852 HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… (#3721) new e4e7cf8 HBASE-26386 Refactor StoreFileTracker implementations to expose the set method (#3774) new 08d1171 HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl (#3749) new 8bec26e HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configurations to TableDescriptor for existing tables (#3700) new a288365 HBASE-26271 Cleanup the broken store files under data directory (#3786) new d00b5fa HBASE-26454 CreateTableProcedure still relies on temp dir and renames… (#3845) new 771e552 HBASE-26286: Add support for specifying store file tracker when restoring or cloning snapshot new f16b7b1 HBASE-26265 Update ref guide to mention the new store file tracker im… (#3942) The 16 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/hadoop/hbase/client/Admin.java | 44 +- .../hadoop/hbase/client/AdminOverAsyncAdmin.java | 7 +- .../org/apache/hadoop/hbase/client/AsyncAdmin.java | 14 +- .../hadoop/hbase/client/AsyncHBaseAdmin.java | 6 +- .../hbase/client/ColumnFamilyDescriptor.java | 5 + .../client/ColumnFamilyDescriptorBuilder.java | 6 + .../hadoop/hbase/client/RawAsyncHBaseAdmin.java| 24 +- .../src/main/protobuf/server/master/Master.proto | 1 + .../protobuf/server/master/MasterProcedure.proto | 1 + .../StoreFileTracker.proto}| 16 +- .../java/org/apache/hadoop/hbase/io/HFileLink.java | 63 +- .../org/apache/hadoop/hbase/master/HMaster.java| 31 +- .../hadoop/hbase/master/MasterRpcServices.java | 2 +- .../assignment/MergeTableRegionsProcedure.java | 20 +- .../assignment/SplitTableRegionProcedure.java | 40 +- .../hbase/master/migrate/RollingUpgradeChore.java | 130 .../master/procedure/CloneSnapshotProcedure.java | 106 +-- .../master/procedure/CreateTableProcedure.java | 43 +- .../master/procedure/DeleteTableProcedure.java | 95 +-- .../master/procedure/ModifyTableProcedure.java | 5 + .../master/procedure/RestoreSnapshotProcedure.java | 7 +- .../hbase/master/snapshot/SnapshotManager.java | 27 +- .../hadoop/hbase/mob/DefaultMobStoreCompactor.java | 28 +- .../hadoop/hbase/mob/DefaultMobStoreFlusher.java | 4 +- .../regionserver/AbstractMultiFileWriter.java | 6 +- .../hbase/regionserver/BrokenStoreFileCleaner.java | 202 ++ .../regionserver/CreateStoreFileWriterParams.java | 134 .../regionserver/DateTieredMultiFileWriter.java| 2 +- .../hbase/regionserver/DateTieredStoreEngine.java | 5 +- .../hbase/regionserver/DefaultStoreEngine.java | 5 +- .../hbase/regionserver/DefaultStoreFlusher.java| 11 +- .../hadoop/hbase/regionserver/HMobStore.java | 3 +- .../hbase/regionserver/HRegionFileSystem.java | 50 +- .../hadoop/hbase/regionserver/HRegionServer.java | 27 + .../apache/hadoop/hbase/regionserver/HStore.java | 735 + .../hadoop/hbase/regionserver/StoreContext.java| 14 + .../hadoop/hbase/regionserver/StoreEngine.java | 482 +- .../hbase/regionserver/StoreFileManager.java | 9 + .../hadoop/hbase/regionserver/StoreFlusher.java| 9 +- .../hadoop/hbase/regionserver/StoreUtils.java | 37 +- .../hbase/regionserver/StripeMultiFileWriter.java | 2 +- .../hbase/regionserver/StripeStoreEngine.java | 9 +- .../hbase/regionserver/StripeStoreFlusher.java | 9 +- .../compactions/AbstractMultiOutputCompactor.java | 11 +- .../hbase/regionserver/compactions/Compactor.java | 81
[hbase] branch branch-2 updated: HBASE-26613 The logic of the method incrementIV in Encryption class has problem (#3968)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new c6aac61 HBASE-26613 The logic of the method incrementIV in Encryption
class has problem (#3968)
c6aac61 is described below
commit c6aac613a0cdea255988eec8149f1ea62a2fcc37
Author: Yutong Xiao
AuthorDate: Thu Dec 23 00:00:12 2021 +0800
HBASE-26613 The logic of the method incrementIV in Encryption class has
problem (#3968)
Signed-off-by: Duo Zhang
---
.../apache/hadoop/hbase/io/crypto/Encryption.java | 21 +
.../hadoop/hbase/io/crypto/TestEncryption.java | 20
2 files changed, 29 insertions(+), 12 deletions(-)
diff --git
a/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
b/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
index 6adcae5..8077589 100644
---
a/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
+++
b/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
@@ -640,20 +640,17 @@ public final class Encryption {
}
public static void incrementIv(byte[] iv, int v) {
+// v should be > 0
int length = iv.length;
-boolean carry = true;
-// TODO: Optimize for v > 1, e.g. 16, 32
-do {
- for (int i = 0; i < length; i++) {
-if (carry) {
- iv[i] = (byte) ((iv[i] + 1) & 0xFF);
- carry = 0 == iv[i];
-} else {
- break;
-}
+int sum = 0;
+for (int i = 0; i < length; i++) {
+ if (v <= 0) {
+break;
}
- v--;
-} while (v > 0);
+ sum = v + (iv[i] & 0xFF);
+ v = sum / 256;
+ iv[i] = (byte)(sum % 256);
+}
}
/**
diff --git
a/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
b/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
index 829be39..8d850a7 100644
---
a/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
+++
b/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
@@ -89,6 +89,26 @@ public class TestEncryption {
}
}
+ @Test
+ public void testIncrementIV() {
+byte[] iv = new byte[] {1, 2, 3};
+byte[] iv_neg = new byte[] {-3, -13, 25};
+Encryption.incrementIv(iv);
+assertTrue(Bytes.equals(iv, new byte[] {2, 2, 3}));
+
+Encryption.incrementIv(iv, 255);
+assertTrue(Bytes.equals(iv, new byte[] {1, 3, 3}));
+
+Encryption.incrementIv(iv, 1024);
+assertTrue(Bytes.equals(iv, new byte[] {1, 7, 3}));
+
+Encryption.incrementIv(iv_neg);
+assertTrue(Bytes.equals(iv_neg, new byte[] {-2, -13, 25}));
+
+Encryption.incrementIv(iv_neg, 5);
+assertTrue(Bytes.equals(iv_neg, new byte[] {3, -12, 25}));
+ }
+
private void checkTransformSymmetry(byte[] keyBytes, byte[] iv, byte[]
plaintext)
throws Exception {
LOG.info("checkTransformSymmetry: AES, plaintext length = " +
plaintext.length);
[hbase] branch branch-2.5 updated: HBASE-26613 The logic of the method incrementIV in Encryption class has problem (#3968)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch branch-2.5
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.5 by this push:
new f239886 HBASE-26613 The logic of the method incrementIV in Encryption
class has problem (#3968)
f239886 is described below
commit f2398864355390a7869010b30c254f8187c26434
Author: Yutong Xiao
AuthorDate: Thu Dec 23 00:00:12 2021 +0800
HBASE-26613 The logic of the method incrementIV in Encryption class has
problem (#3968)
Signed-off-by: Duo Zhang
---
.../apache/hadoop/hbase/io/crypto/Encryption.java | 21 +
.../hadoop/hbase/io/crypto/TestEncryption.java | 20
2 files changed, 29 insertions(+), 12 deletions(-)
diff --git
a/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
b/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
index 6adcae5..8077589 100644
---
a/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
+++
b/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
@@ -640,20 +640,17 @@ public final class Encryption {
}
public static void incrementIv(byte[] iv, int v) {
+// v should be > 0
int length = iv.length;
-boolean carry = true;
-// TODO: Optimize for v > 1, e.g. 16, 32
-do {
- for (int i = 0; i < length; i++) {
-if (carry) {
- iv[i] = (byte) ((iv[i] + 1) & 0xFF);
- carry = 0 == iv[i];
-} else {
- break;
-}
+int sum = 0;
+for (int i = 0; i < length; i++) {
+ if (v <= 0) {
+break;
}
- v--;
-} while (v > 0);
+ sum = v + (iv[i] & 0xFF);
+ v = sum / 256;
+ iv[i] = (byte)(sum % 256);
+}
}
/**
diff --git
a/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
b/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
index 829be39..8d850a7 100644
---
a/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
+++
b/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
@@ -89,6 +89,26 @@ public class TestEncryption {
}
}
+ @Test
+ public void testIncrementIV() {
+byte[] iv = new byte[] {1, 2, 3};
+byte[] iv_neg = new byte[] {-3, -13, 25};
+Encryption.incrementIv(iv);
+assertTrue(Bytes.equals(iv, new byte[] {2, 2, 3}));
+
+Encryption.incrementIv(iv, 255);
+assertTrue(Bytes.equals(iv, new byte[] {1, 3, 3}));
+
+Encryption.incrementIv(iv, 1024);
+assertTrue(Bytes.equals(iv, new byte[] {1, 7, 3}));
+
+Encryption.incrementIv(iv_neg);
+assertTrue(Bytes.equals(iv_neg, new byte[] {-2, -13, 25}));
+
+Encryption.incrementIv(iv_neg, 5);
+assertTrue(Bytes.equals(iv_neg, new byte[] {3, -12, 25}));
+ }
+
private void checkTransformSymmetry(byte[] keyBytes, byte[] iv, byte[]
plaintext)
throws Exception {
LOG.info("checkTransformSymmetry: AES, plaintext length = " +
plaintext.length);
[hbase] branch branch-2.4 updated: HBASE-26613 The logic of the method incrementIV in Encryption class has problem (#3968)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 647184e HBASE-26613 The logic of the method incrementIV in Encryption
class has problem (#3968)
647184e is described below
commit 647184e617c396b5e96dd1a65ffd0eb3ec362da4
Author: Yutong Xiao
AuthorDate: Thu Dec 23 00:00:12 2021 +0800
HBASE-26613 The logic of the method incrementIV in Encryption class has
problem (#3968)
Signed-off-by: Duo Zhang
---
.../apache/hadoop/hbase/io/crypto/Encryption.java | 21 +
.../hadoop/hbase/io/crypto/TestEncryption.java | 20
2 files changed, 29 insertions(+), 12 deletions(-)
diff --git
a/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
b/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
index 6adcae5..8077589 100644
---
a/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
+++
b/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
@@ -640,20 +640,17 @@ public final class Encryption {
}
public static void incrementIv(byte[] iv, int v) {
+// v should be > 0
int length = iv.length;
-boolean carry = true;
-// TODO: Optimize for v > 1, e.g. 16, 32
-do {
- for (int i = 0; i < length; i++) {
-if (carry) {
- iv[i] = (byte) ((iv[i] + 1) & 0xFF);
- carry = 0 == iv[i];
-} else {
- break;
-}
+int sum = 0;
+for (int i = 0; i < length; i++) {
+ if (v <= 0) {
+break;
}
- v--;
-} while (v > 0);
+ sum = v + (iv[i] & 0xFF);
+ v = sum / 256;
+ iv[i] = (byte)(sum % 256);
+}
}
/**
diff --git
a/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
b/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
index 829be39..8d850a7 100644
---
a/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
+++
b/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto/TestEncryption.java
@@ -89,6 +89,26 @@ public class TestEncryption {
}
}
+ @Test
+ public void testIncrementIV() {
+byte[] iv = new byte[] {1, 2, 3};
+byte[] iv_neg = new byte[] {-3, -13, 25};
+Encryption.incrementIv(iv);
+assertTrue(Bytes.equals(iv, new byte[] {2, 2, 3}));
+
+Encryption.incrementIv(iv, 255);
+assertTrue(Bytes.equals(iv, new byte[] {1, 3, 3}));
+
+Encryption.incrementIv(iv, 1024);
+assertTrue(Bytes.equals(iv, new byte[] {1, 7, 3}));
+
+Encryption.incrementIv(iv_neg);
+assertTrue(Bytes.equals(iv_neg, new byte[] {-2, -13, 25}));
+
+Encryption.incrementIv(iv_neg, 5);
+assertTrue(Bytes.equals(iv_neg, new byte[] {3, -12, 25}));
+ }
+
private void checkTransformSymmetry(byte[] keyBytes, byte[] iv, byte[]
plaintext)
throws Exception {
LOG.info("checkTransformSymmetry: AES, plaintext length = " +
plaintext.length);
[hbase] branch master updated (314e924 -> 6818ec2)
This is an automated email from the ASF dual-hosted git repository. zhangduo pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 314e924 HBASE-26606 Upgrade log4j2 to 2.17.0 (#3965) add 6818ec2 HBASE-26613 The logic of the method incrementIV in Encryption class has problem (#3968) No new revisions were added by this update. Summary of changes: .../apache/hadoop/hbase/io/crypto/Encryption.java | 21 + .../hadoop/hbase/io/crypto/TestEncryption.java | 20 2 files changed, 29 insertions(+), 12 deletions(-)
