date:20211217

[hbase] branch HBASE-26233 updated (1c8d939 -> f40f268)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a change to branch HBASE-26233
in repository https://gitbox.apache.org/repos/asf/hbase.git.


 discard 1c8d939  HBASE-26540 Implement a tool to verify the lag of region 
replication (#3950)
omit 5c18b67  HBASE-26539 The default rpc timeout 200ms is too small for 
replicating meta edits (#3919)
omit df2e913  HBASE-26538 Should find a way to clear the replication queue 
for a legacy region_replica_replication peer (#3918)
omit 998fc08  HBASE-26486 Attach the new region replication framework 
design doc to git repo (#3884)
omit 102511e  HBASE-26478 Update ref guide about the new region replication 
framework (#3885)
omit 41c0608  HBASE-26481 Consider rolling upgrading from old region 
replication framework (#3880)
omit 269fa7b  HBASE-26417 Remove NoopReplicationQueueStorage (#3882)
omit 7c2827b  HBASE-26456 Limit the size for one replicating (#3873)
omit 4e56220  HBASE-26416 Implement a new method for region replication 
instead of using replay (#3864)
omit 99f9c55  HBASE-26457 Should not always clear all the failed replicas 
when getting a flush all request (#3850)
omit bda617f  HBASE-26448 Make sure we do not flush a region too frequently 
(#3847)
omit 5eaea68  HBASE-26449 The way we add or clear failedReplicas may have 
race (#3846)
omit ee1604b  HBASE-26413 Limit the total size of buffered region 
replication entries (#3844)
omit c9004d4  HBASE-26412 Handle sink failure in RegionReplicationSink 
(#3815)
omit 150889e  HBASE-26407 Introduce a region replication sink for sinking 
WAL edits to secondary replicas directly (#3807)
 add 0f94ab5  HBASE-26572 Upgrade to log4j 2.16.0 (#3943)
 add a36d41a  HBASE-26556 IT and Chaos Monkey improvements (#3932)
 add 8f5a12f  HBASE-26472 Adhere to semantic conventions regarding table 
data operations
 add c93e457  HBASE-26542 Apply a `package` to test protobuf files
 add 006433b  HBASE-26542 Apply a `package` to test protobuf files 
(addendum)
 add 139f085  HBASE-26523 Upgrade hbase-thirdparty dependency to 4.0.0 
(#3910)
 add e598f2c  Revert "HBASE-26523 Upgrade hbase-thirdparty dependency to 
4.0.0 (#3910)"
 add a0acf8c  HBASE-26601 maven-gpg-plugin failing with "Inappropriate 
ioctl for device"
 add 77b6b4d  HBASE-26340 TableSplit returns false size under 1MB (#3872)
 add da616c0  HBASE-26488 Memory leak when MemStore retry flushing (#3899)
 add 5d69d1c  HBASE-26407 Introduce a region replication sink for sinking 
WAL edits to secondary replicas directly (#3807)
 add b3ac276  HBASE-26412 Handle sink failure in RegionReplicationSink 
(#3815)
 add 282477c  HBASE-26413 Limit the total size of buffered region 
replication entries (#3844)
 add c894a76  HBASE-26449 The way we add or clear failedReplicas may have 
race (#3846)
 add 6febbc8  HBASE-26448 Make sure we do not flush a region too frequently 
(#3847)
 add 47c35f0  HBASE-26457 Should not always clear all the failed replicas 
when getting a flush all request (#3850)
 add d841ffd  HBASE-26416 Implement a new method for region replication 
instead of using replay (#3864)
 add fbe6665  HBASE-26456 Limit the size for one replicating (#3873)
 add 88daafe  HBASE-26417 Remove NoopReplicationQueueStorage (#3882)
 add ac9258f  HBASE-26481 Consider rolling upgrading from old region 
replication framework (#3880)
 add 724735c  HBASE-26478 Update ref guide about the new region replication 
framework (#3885)
 add d6a5d34  HBASE-26486 Attach the new region replication framework 
design doc to git repo (#3884)
 add 44b6483  HBASE-26538 Should find a way to clear the replication queue 
for a legacy region_replica_replication peer (#3918)
 add a6b728a  HBASE-26539 The default rpc timeout 200ms is too small for 
replicating meta edits (#3919)
 add f40f268  HBASE-26540 Implement a tool to verify the lag of region 
replication (#3950)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (1c8d939)
\
 N -- N -- N   refs/heads/HBASE-26233 (f40f268)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 bin/hbase-config.cmd   |   4 -
 bin/hbase-config.sh|   6 +-
 hbase-client/pom.xml   |   5 +

[hbase] branch branch-2 updated (29e3743 -> 2d5d5b2)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a change to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git.


from 29e3743  HBASE-26601 maven-gpg-plugin failing with "Inappropriate 
ioctl for device"
 add 471e398  HBASE-26340 TableSplit returns false size under 1MB (#3872)
 add 2d5d5b2  HBASE-26488 Memory leak when MemStore retry flushing (#3899)

No new revisions were added by this update.

Summary of changes:
 .../hadoop/hbase/regionserver/HRegionServer.java   |   6 +-
 .../apache/hadoop/hbase/regionserver/HStore.java   |   5 -
 .../hbase/regionserver/ImmutableSegment.java   |   4 +
 .../hbase/regionserver/MemStoreSnapshot.java   |  42 ---
 .../hadoop/hbase/regionserver/TestHStore.java  | 132 +++--
 5 files changed, 153 insertions(+), 36 deletions(-)

[hbase] 01/02: HBASE-26340 TableSplit returns false size under 1MB (#3872)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch branch-2.5
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit d072a2e694127476e19192bfe470902d0e418175
Author: Norbert Kalmar 
AuthorDate: Sat Dec 18 04:58:16 2021 +0100

HBASE-26340 TableSplit returns false size under 1MB (#3872)

Signed-off-by: Peter Somogyi 
---
 .../java/org/apache/hadoop/hbase/regionserver/HRegionServer.java| 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
index 41d13b7..047345c 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
@@ -1748,6 +1748,7 @@ public class HRegionServer extends Thread implements
 int maxCompactedStoreFileRefCount = 0;
 int storeUncompressedSizeMB = 0;
 int storefileSizeMB = 0;
+long storefileSizeByte = 0L;
 int memstoreSizeMB = (int) (r.getMemStoreDataSize() / 1024 / 1024);
 long storefileIndexSizeKB = 0;
 int rootLevelIndexSizeKB = 0;
@@ -1765,7 +1766,7 @@ public class HRegionServer extends Thread implements
   maxCompactedStoreFileRefCount = Math.max(maxCompactedStoreFileRefCount,
 currentMaxCompactedStoreFileRefCount);
   storeUncompressedSizeMB += (int) (store.getStoreSizeUncompressed() / 
1024 / 1024);
-  storefileSizeMB += (int) (store.getStorefilesSize() / 1024 / 1024);
+  storefileSizeByte += store.getStorefilesSize();
   //TODO: storefileIndexSizeKB is same with rootLevelIndexSizeKB?
   storefileIndexSizeKB += store.getStorefilesRootLevelIndexSize() / 1024;
   CompactionProgress progress = store.getCompactionProgress();
@@ -1777,6 +1778,9 @@ public class HRegionServer extends Thread implements
   totalStaticIndexSizeKB += (int) (store.getTotalStaticIndexSize() / 1024);
   totalStaticBloomSizeKB += (int) (store.getTotalStaticBloomSize() / 1024);
 }
+//HBASE-26340 Fix false "0" size under 1MB
+storefileSizeMB = storefileSizeByte > 0 && storefileSizeByte <= 1024 * 1024
+   ? 1 : (int) storefileSizeByte / 1024 / 1024;
 
 HDFSBlocksDistribution hdfsBd = r.getHDFSBlocksDistribution();
 float dataLocality = 
hdfsBd.getBlockLocalityIndex(serverName.getHostname());

[hbase] 02/02: HBASE-26488 Memory leak when MemStore retry flushing (#3899)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch branch-2.5
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 482ef6904e0b4076b17d0694f96e8350259b523f
Author: chenglei 
AuthorDate: Sat Dec 18 12:17:19 2021 +0800

HBASE-26488 Memory leak when MemStore retry flushing (#3899)

Signed-off-by: Duo Zhang 
---
 .../apache/hadoop/hbase/regionserver/HStore.java   |   5 -
 .../hbase/regionserver/ImmutableSegment.java   |   4 +
 .../hbase/regionserver/MemStoreSnapshot.java   |  42 ---
 .../hadoop/hbase/regionserver/TestHStore.java  | 132 +++--
 4 files changed, 148 insertions(+), 35 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
index 8937d34..0213827 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
@@ -2404,7 +2404,6 @@ public class HStore implements Store, HeapSize, 
StoreConfigInformation,
   long snapshotId = -1; // -1 means do not drop
   if (dropMemstoreSnapshot && snapshot != null) {
 snapshotId = snapshot.getId();
-snapshot.close();
   }
   HStore.this.updateStorefiles(storeFiles, snapshotId);
 }
@@ -2415,10 +2414,6 @@ public class HStore implements Store, HeapSize, 
StoreConfigInformation,
 @Override
 public void abort() throws IOException {
   if (snapshot != null) {
-//We need to close the snapshot when aborting, otherwise, the segment 
scanner
-//won't be closed. If we are using MSLAB, the chunk referenced by 
those scanners
-//can't be released, thus memory leak
-snapshot.close();
 HStore.this.updateStorefiles(Collections.emptyList(), 
snapshot.getId());
   }
 }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
index 8c426bc..972684f 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
@@ -85,6 +85,10 @@ public abstract class ImmutableSegment extends Segment {
 return res;
   }
 
+  /**
+   * We create a new {@link SnapshotSegmentScanner} to increase the reference 
count of
+   * {@link MemStoreLABImpl} used by this segment.
+   */
   List getSnapshotScanners() {
 return Collections.singletonList(new SnapshotSegmentScanner(this));
   }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
index 3b34828..07eb64d 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
@@ -17,31 +17,38 @@
  */
 package org.apache.hadoop.hbase.regionserver;
 
+import java.util.List;
 import org.apache.yetus.audience.InterfaceAudience;
 
-import java.io.Closeable;
-import java.util.List;
+
 /**
- * Holds details of the snapshot taken on a MemStore. Details include the 
snapshot's identifier,
- * count of cells in it and total memory size occupied by all the cells, 
timestamp information of
- * all the cells and a scanner to read all cells in it.
+ * {@link MemStoreSnapshot} is a Context Object to hold details of the 
snapshot taken on a MemStore.
+ * Details include the snapshot's identifier, count of cells in it and total 
memory size occupied by
+ * all the cells, timestamp information of all the cells and the snapshot 
immutableSegment.
+ * 
+ * NOTE:Every time when {@link MemStoreSnapshot#getScanners} is called, we 
create new
+ * {@link SnapshotSegmentScanner}s on the {@link 
MemStoreSnapshot#snapshotImmutableSegment},and
+ * {@link Segment#incScannerCount} is invoked in the {@link 
SnapshotSegmentScanner} ctor to increase
+ * the reference count of {@link MemStoreLAB} which used by
+ * {@link MemStoreSnapshot#snapshotImmutableSegment}, so after we finish using 
these scanners, we
+ * must call their close method to invoke {@link Segment#decScannerCount}.
  */
 @InterfaceAudience.Private
-public class MemStoreSnapshot implements Closeable {
+public class MemStoreSnapshot {
   private final long id;
   private final int cellsCount;
   private final MemStoreSize memStoreSize;
   private final TimeRangeTracker timeRangeTracker;
-  private final List scanners;
   private final boolean tagsPresent;
+  private final ImmutableSegment snapshotImmutableSegment;
 
   public MemStoreSnapshot(long id, ImmutableSegment snapshot) {
 this.id = id;
 this.cellsCount = snapshot.getCellsCount();

[hbase] branch branch-2.5 updated (5d56c80 -> 482ef69)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a change to branch branch-2.5
in repository https://gitbox.apache.org/repos/asf/hbase.git.


from 5d56c80  HBASE-26601 maven-gpg-plugin failing with "Inappropriate 
ioctl for device"
 new d072a2e  HBASE-26340 TableSplit returns false size under 1MB (#3872)
 new 482ef69  HBASE-26488 Memory leak when MemStore retry flushing (#3899)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../hadoop/hbase/regionserver/HRegionServer.java   |   6 +-
 .../apache/hadoop/hbase/regionserver/HStore.java   |   5 -
 .../hbase/regionserver/ImmutableSegment.java   |   4 +
 .../hbase/regionserver/MemStoreSnapshot.java   |  42 ---
 .../hadoop/hbase/regionserver/TestHStore.java  | 132 +++--
 5 files changed, 153 insertions(+), 36 deletions(-)

[hbase] 01/02: HBASE-26340 TableSplit returns false size under 1MB (#3872)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 8fa8344a8c0c72d167ce92b5f44f88beb43f325e
Author: Norbert Kalmar 
AuthorDate: Sat Dec 18 04:58:16 2021 +0100

HBASE-26340 TableSplit returns false size under 1MB (#3872)

Signed-off-by: Peter Somogyi 
---
 .../java/org/apache/hadoop/hbase/regionserver/HRegionServer.java| 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
index 05ce419..97f56bd 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
@@ -1723,6 +1723,7 @@ public class HRegionServer extends Thread implements
 int maxCompactedStoreFileRefCount = 0;
 int storeUncompressedSizeMB = 0;
 int storefileSizeMB = 0;
+long storefileSizeByte = 0L;
 int memstoreSizeMB = (int) (r.getMemStoreDataSize() / 1024 / 1024);
 long storefileIndexSizeKB = 0;
 int rootLevelIndexSizeKB = 0;
@@ -1740,7 +1741,7 @@ public class HRegionServer extends Thread implements
   maxCompactedStoreFileRefCount = Math.max(maxCompactedStoreFileRefCount,
 currentMaxCompactedStoreFileRefCount);
   storeUncompressedSizeMB += (int) (store.getStoreSizeUncompressed() / 
1024 / 1024);
-  storefileSizeMB += (int) (store.getStorefilesSize() / 1024 / 1024);
+  storefileSizeByte += store.getStorefilesSize();
   //TODO: storefileIndexSizeKB is same with rootLevelIndexSizeKB?
   storefileIndexSizeKB += store.getStorefilesRootLevelIndexSize() / 1024;
   CompactionProgress progress = store.getCompactionProgress();
@@ -1752,6 +1753,9 @@ public class HRegionServer extends Thread implements
   totalStaticIndexSizeKB += (int) (store.getTotalStaticIndexSize() / 1024);
   totalStaticBloomSizeKB += (int) (store.getTotalStaticBloomSize() / 1024);
 }
+//HBASE-26340 Fix false "0" size under 1MB
+storefileSizeMB = storefileSizeByte > 0 && storefileSizeByte <= 1024 * 1024
+   ? 1 : (int) storefileSizeByte / 1024 / 1024;
 
 HDFSBlocksDistribution hdfsBd = r.getHDFSBlocksDistribution();
 float dataLocality = 
hdfsBd.getBlockLocalityIndex(serverName.getHostname());

[hbase] branch branch-2.4 updated (6449087 -> e20b2f3)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a change to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git.


from 6449087  Preparing development version 2.4.10-SNAPSHOT
 new 8fa8344  HBASE-26340 TableSplit returns false size under 1MB (#3872)
 new e20b2f3  HBASE-26488 Memory leak when MemStore retry flushing (#3899)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../hadoop/hbase/regionserver/HRegionServer.java   |   6 +-
 .../apache/hadoop/hbase/regionserver/HStore.java   |   5 -
 .../hbase/regionserver/ImmutableSegment.java   |   4 +
 .../hbase/regionserver/MemStoreSnapshot.java   |  42 ---
 .../hadoop/hbase/regionserver/TestHStore.java  | 132 +++--
 5 files changed, 153 insertions(+), 36 deletions(-)

[hbase] 02/02: HBASE-26488 Memory leak when MemStore retry flushing (#3899)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit e20b2f3df5bb85c1290053b43f4d29e8b29092f1
Author: chenglei 
AuthorDate: Sat Dec 18 12:17:19 2021 +0800

HBASE-26488 Memory leak when MemStore retry flushing (#3899)

Signed-off-by: Duo Zhang 
---
 .../apache/hadoop/hbase/regionserver/HStore.java   |   5 -
 .../hbase/regionserver/ImmutableSegment.java   |   4 +
 .../hbase/regionserver/MemStoreSnapshot.java   |  42 ---
 .../hadoop/hbase/regionserver/TestHStore.java  | 132 +++--
 4 files changed, 148 insertions(+), 35 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
index fe2e635..e44893a 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
@@ -2504,7 +2504,6 @@ public class HStore implements Store, HeapSize, 
StoreConfigInformation,
   long snapshotId = -1; // -1 means do not drop
   if (dropMemstoreSnapshot && snapshot != null) {
 snapshotId = snapshot.getId();
-snapshot.close();
   }
   HStore.this.updateStorefiles(storeFiles, snapshotId);
 }
@@ -2515,10 +2514,6 @@ public class HStore implements Store, HeapSize, 
StoreConfigInformation,
 @Override
 public void abort() throws IOException {
   if (snapshot != null) {
-//We need to close the snapshot when aborting, otherwise, the segment 
scanner
-//won't be closed. If we are using MSLAB, the chunk referenced by 
those scanners
-//can't be released, thus memory leak
-snapshot.close();
 HStore.this.updateStorefiles(Collections.emptyList(), 
snapshot.getId());
   }
 }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
index 8c426bc..972684f 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
@@ -85,6 +85,10 @@ public abstract class ImmutableSegment extends Segment {
 return res;
   }
 
+  /**
+   * We create a new {@link SnapshotSegmentScanner} to increase the reference 
count of
+   * {@link MemStoreLABImpl} used by this segment.
+   */
   List getSnapshotScanners() {
 return Collections.singletonList(new SnapshotSegmentScanner(this));
   }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
index 3b34828..07eb64d 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
@@ -17,31 +17,38 @@
  */
 package org.apache.hadoop.hbase.regionserver;
 
+import java.util.List;
 import org.apache.yetus.audience.InterfaceAudience;
 
-import java.io.Closeable;
-import java.util.List;
+
 /**
- * Holds details of the snapshot taken on a MemStore. Details include the 
snapshot's identifier,
- * count of cells in it and total memory size occupied by all the cells, 
timestamp information of
- * all the cells and a scanner to read all cells in it.
+ * {@link MemStoreSnapshot} is a Context Object to hold details of the 
snapshot taken on a MemStore.
+ * Details include the snapshot's identifier, count of cells in it and total 
memory size occupied by
+ * all the cells, timestamp information of all the cells and the snapshot 
immutableSegment.
+ * 
+ * NOTE:Every time when {@link MemStoreSnapshot#getScanners} is called, we 
create new
+ * {@link SnapshotSegmentScanner}s on the {@link 
MemStoreSnapshot#snapshotImmutableSegment},and
+ * {@link Segment#incScannerCount} is invoked in the {@link 
SnapshotSegmentScanner} ctor to increase
+ * the reference count of {@link MemStoreLAB} which used by
+ * {@link MemStoreSnapshot#snapshotImmutableSegment}, so after we finish using 
these scanners, we
+ * must call their close method to invoke {@link Segment#decScannerCount}.
  */
 @InterfaceAudience.Private
-public class MemStoreSnapshot implements Closeable {
+public class MemStoreSnapshot {
   private final long id;
   private final int cellsCount;
   private final MemStoreSize memStoreSize;
   private final TimeRangeTracker timeRangeTracker;
-  private final List scanners;
   private final boolean tagsPresent;
+  private final ImmutableSegment snapshotImmutableSegment;
 
   public MemStoreSnapshot(long id, ImmutableSegment snapshot) {
 this.id = id;
 this.cellsCount = snapshot.getCellsCount();

[hbase] branch master updated: HBASE-26488 Memory leak when MemStore retry flushing (#3899)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git


The following commit(s) were added to refs/heads/master by this push:
 new da616c0  HBASE-26488 Memory leak when MemStore retry flushing (#3899)
da616c0 is described below

commit da616c00c73977067236ba1d60277586c3625691
Author: chenglei 
AuthorDate: Sat Dec 18 12:17:19 2021 +0800

HBASE-26488 Memory leak when MemStore retry flushing (#3899)

Signed-off-by: Duo Zhang 
---
 .../apache/hadoop/hbase/regionserver/HStore.java   |   5 -
 .../hbase/regionserver/ImmutableSegment.java   |   4 +
 .../hbase/regionserver/MemStoreSnapshot.java   |  42 ---
 .../hadoop/hbase/regionserver/TestHStore.java  | 132 +++--
 4 files changed, 148 insertions(+), 35 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
index 0ee7b57..1555bbe 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
@@ -2404,7 +2404,6 @@ public class HStore implements Store, HeapSize, 
StoreConfigInformation,
   long snapshotId = -1; // -1 means do not drop
   if (dropMemstoreSnapshot && snapshot != null) {
 snapshotId = snapshot.getId();
-snapshot.close();
   }
   HStore.this.updateStorefiles(storeFiles, snapshotId);
 }
@@ -2415,10 +2414,6 @@ public class HStore implements Store, HeapSize, 
StoreConfigInformation,
 @Override
 public void abort() throws IOException {
   if (snapshot != null) {
-//We need to close the snapshot when aborting, otherwise, the segment 
scanner
-//won't be closed. If we are using MSLAB, the chunk referenced by 
those scanners
-//can't be released, thus memory leak
-snapshot.close();
 HStore.this.updateStorefiles(Collections.emptyList(), 
snapshot.getId());
   }
 }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
index 8c426bc..972684f 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
@@ -85,6 +85,10 @@ public abstract class ImmutableSegment extends Segment {
 return res;
   }
 
+  /**
+   * We create a new {@link SnapshotSegmentScanner} to increase the reference 
count of
+   * {@link MemStoreLABImpl} used by this segment.
+   */
   List getSnapshotScanners() {
 return Collections.singletonList(new SnapshotSegmentScanner(this));
   }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
index 3b34828..07eb64d 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
@@ -17,31 +17,38 @@
  */
 package org.apache.hadoop.hbase.regionserver;
 
+import java.util.List;
 import org.apache.yetus.audience.InterfaceAudience;
 
-import java.io.Closeable;
-import java.util.List;
+
 /**
- * Holds details of the snapshot taken on a MemStore. Details include the 
snapshot's identifier,
- * count of cells in it and total memory size occupied by all the cells, 
timestamp information of
- * all the cells and a scanner to read all cells in it.
+ * {@link MemStoreSnapshot} is a Context Object to hold details of the 
snapshot taken on a MemStore.
+ * Details include the snapshot's identifier, count of cells in it and total 
memory size occupied by
+ * all the cells, timestamp information of all the cells and the snapshot 
immutableSegment.
+ * 
+ * NOTE:Every time when {@link MemStoreSnapshot#getScanners} is called, we 
create new
+ * {@link SnapshotSegmentScanner}s on the {@link 
MemStoreSnapshot#snapshotImmutableSegment},and
+ * {@link Segment#incScannerCount} is invoked in the {@link 
SnapshotSegmentScanner} ctor to increase
+ * the reference count of {@link MemStoreLAB} which used by
+ * {@link MemStoreSnapshot#snapshotImmutableSegment}, so after we finish using 
these scanners, we
+ * must call their close method to invoke {@link Segment#decScannerCount}.
  */
 @InterfaceAudience.Private
-public class MemStoreSnapshot implements Closeable {
+public class MemStoreSnapshot {
   private final long id;
   private final int cellsCount;
   private final MemStoreSize memStoreSize;
   private final TimeRangeTracker timeRangeTracker;
-  private final List scanners;
   private final boolean tagsPresent;
+  private final ImmutableSegment

[hbase] branch master updated: HBASE-26340 TableSplit returns false size under 1MB (#3872)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git


The following commit(s) were added to refs/heads/master by this push:
 new 77b6b4d  HBASE-26340 TableSplit returns false size under 1MB (#3872)
77b6b4d is described below

commit 77b6b4dee390c49fb49257f1500ec216fe620819
Author: Norbert Kalmar 
AuthorDate: Sat Dec 18 04:58:16 2021 +0100

HBASE-26340 TableSplit returns false size under 1MB (#3872)

Signed-off-by: Peter Somogyi 
---
 .../java/org/apache/hadoop/hbase/regionserver/HRegionServer.java| 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
index 4bf2d9c..ef41589 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
@@ -1431,6 +1431,7 @@ public class HRegionServer extends 
HBaseServerBase
 int maxCompactedStoreFileRefCount = 0;
 int storeUncompressedSizeMB = 0;
 int storefileSizeMB = 0;
+long storefileSizeByte = 0L;
 int memstoreSizeMB = (int) (r.getMemStoreDataSize() / 1024 / 1024);
 long storefileIndexSizeKB = 0;
 int rootLevelIndexSizeKB = 0;
@@ -1448,7 +1449,7 @@ public class HRegionServer extends 
HBaseServerBase
   maxCompactedStoreFileRefCount = Math.max(maxCompactedStoreFileRefCount,
 currentMaxCompactedStoreFileRefCount);
   storeUncompressedSizeMB += (int) (store.getStoreSizeUncompressed() / 
1024 / 1024);
-  storefileSizeMB += (int) (store.getStorefilesSize() / 1024 / 1024);
+  storefileSizeByte += store.getStorefilesSize();
   //TODO: storefileIndexSizeKB is same with rootLevelIndexSizeKB?
   storefileIndexSizeKB += store.getStorefilesRootLevelIndexSize() / 1024;
   CompactionProgress progress = store.getCompactionProgress();
@@ -1460,6 +1461,9 @@ public class HRegionServer extends 
HBaseServerBase
   totalStaticIndexSizeKB += (int) (store.getTotalStaticIndexSize() / 1024);
   totalStaticBloomSizeKB += (int) (store.getTotalStaticBloomSize() / 1024);
 }
+//HBASE-26340 Fix false "0" size under 1MB
+storefileSizeMB = storefileSizeByte > 0 && storefileSizeByte <= 1024 * 1024
+   ? 1 : (int) storefileSizeByte / 1024 / 1024;
 
 HDFSBlocksDistribution hdfsBd = r.getHDFSBlocksDistribution();
 float dataLocality = 
hdfsBd.getBlockLocalityIndex(serverName.getHostname());

svn commit: r51562 [4/4] - /dev/hbase/2.4.9RC0/

2021-12-17 Thread apurtell

Added: dev/hbase/2.4.9RC0/api_compare_2.4.8_to_2.4.9RC0.html
==
--- dev/hbase/2.4.9RC0/api_compare_2.4.8_to_2.4.9RC0.html (added)
+++ dev/hbase/2.4.9RC0/api_compare_2.4.8_to_2.4.9RC0.html Sat Dec 18 03:35:45 
2021
@@ -0,0 +1,481 @@
+
+
+http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd;>
+http://www.w3.org/1999/xhtml; xml:lang="en" lang="en">
+
+
+
+
+
+hbase: rel/2.4.8 to 2.4.9RC0 compatibility report
+
+body {
+font-family:Arial, sans-serif;
+background-color:White;
+color:Black;
+}
+hr {
+color:Black;
+background-color:Black;
+height:1px;
+border:0;
+}
+h1 {
+margin-bottom:0px;
+padding-bottom:0px;
+font-size:1.625em;
+}
+h2 {
+margin-bottom:0px;
+padding-bottom:0px;
+font-size:1.25em;
+white-space:nowrap;
+}
+div.symbols {
+color:#003E69;
+}
+div.symbols i {
+color:Brown;
+}
+span.section {
+font-weight:bold;
+cursor:pointer;
+color:#003E69;
+white-space:nowrap;
+margin-left:0.3125em;
+}
+span:hover.section {
+color:#336699;
+}
+span.sect_aff {
+cursor:pointer;
+padding-left:1.55em;
+font-size:0.875em;
+color:#cc3300;
+}
+span.ext {
+font-weight:normal;
+}
+span.jar {
+color:#cc3300;
+font-size:0.875em;
+font-weight:bold;
+}
+div.jar_list {
+padding-left:0.4em;
+font-size:0.94em;
+}
+span.pkg_t {
+color:#408080;
+font-size:0.875em;
+}
+span.pkg {
+color:#408080;
+font-size:0.875em;
+font-weight:bold;
+}
+span.cname {
+color:Green;
+font-size:0.875em;
+font-weight:bold;
+}
+span.iname_b {
+font-weight:bold;
+}
+span.iname_a {
+color:#33;
+font-weight:bold;
+font-size:0.94em;
+}
+span.sym_p {
+font-weight:normal;
+white-space:normal;
+}
+span.sym_pd {
+white-space:normal;
+}
+span.sym_p span, span.sym_pd span {
+white-space:nowrap;
+}
+span.attr {
+color:Black;
+font-weight:normal;
+}
+span.deprecated {
+color:Red;
+font-weight:bold;
+font-family:Monaco, monospace;
+}
+div.affect {
+padding-left:1em;
+padding-bottom:10px;
+font-size:0.87em;
+font-style:italic;
+line-height:0.9em;
+}
+div.affected {
+padding-left:2em;
+padding-top:10px;
+}
+table.ptable {
+border-collapse:collapse;
+border:1px outset black;
+margin-left:0.95em;
+margin-top:3px;
+margin-bottom:3px;
+width:56.25em;
+}
+table.ptable td {
+border:1px solid Gray;
+padding:3px;
+font-size:0.875em;
+text-align:left;
+vertical-align:top;
+max-width:28em;
+word-wrap:break-word;
+}
+table.ptable th {
+background-color:#ee;
+font-weight:bold;
+color:#33;
+font-family:Verdana, Arial;
+font-size:0.875em;
+border:1px solid Gray;
+text-align:center;
+vertical-align:top;
+white-space:nowrap;
+padding:3px;
+}
+table.summary {
+border-collapse:collapse;
+border:1px outset black;
+}
+table.summary th {
+background-color:#ee;
+font-weight:normal;
+text-align:left;
+font-size:0.94em;
+white-space:nowrap;
+border:1px inset Gray;
+padding:3px;
+}
+table.summary td {
+text-align:right;
+white-space:nowrap;
+border:1px inset Gray;
+padding:3px 5px 3px 10px;
+}
+span.mngl {
+padding-left:1em;
+font-size:0.875em;
+cursor:text;
+color:#44;
+font-weight:bold;
+}
+span.pleft {
+padding-left:2.5em;
+}
+span.color_p {
+font-style:italic;
+color:Brown;
+}
+span.param {
+font-style:italic;
+}
+span.focus_p {
+font-style:italic;
+background-color:#DCDCDC;
+}
+span.ttype {
+font-weight:normal;
+}
+span.nowrap {
+white-space:nowrap;
+}
+span.value {
+white-space:nowrap;
+font-weight:bold;
+}
+.passed {
+background-color:#CCFFCC;
+font-weight:normal;
+}
+.warning {
+background-color:#F4F4AF;
+font-weight:normal;
+}
+.failed {
+background-color:#FF;
+font-weight:normal;
+}
+.new {
+background-color:#C6DEFF;
+font-weight:normal;
+}
+
+.compatible {
+background-color:#CCFFCC;
+font-weight:normal;
+}
+.almost_compatible {
+background-color:#FFDAA3;
+font-weight:normal;
+}
+.incompatible {
+background-color:#FF;
+font-weight:normal;
+}
+.gray {
+background-color:#DCDCDC;
+font-weight:normal;
+}
+
+.top_ref {
+font-size:0.69em;
+}
+.footer {
+font-size:0.8125em;
+}
+.tabset {
+float:left;
+}
+a.tab {
+border:1px solid Black;
+float:left;
+margin:0px 5px -1px 0px;
+padding:3px 5px 3px 5px;
+position:relative;
+font-size:0.875em;
+background-color:#DDD;
+text-decoration:none;
+color:Black;
+}
+a.disabled:hover
+{
+color:Black;
+background:#EEE;
+}
+a.active:hover
+{
+color:Black;
+background:White;
+}
+a.active {
+border-bottom-color:White;
+background-color:White;
+}
+div.tab {
+border-top:1px solid

svn commit: r51562 [2/4] - /dev/hbase/2.4.9RC0/

2021-12-17 Thread apurtell



Added: dev/hbase/2.4.9RC0/CHANGES.md
==
--- dev/hbase/2.4.9RC0/CHANGES.md (added)
+++ dev/hbase/2.4.9RC0/CHANGES.md Sat Dec 18 03:35:45 2021
@@ -0,0 +1,1008 @@
+# HBASE Changelog
+
+
+## Release 2.4.9 - 2021-12-23
+
+
+
+### IMPROVEMENTS:
+
+| JIRA | Summary | Priority | Component |
+|: |: | :--- |: |
+| [HBASE-26601](https://issues.apache.org/jira/browse/HBASE-26601) | 
maven-gpg-plugin failing with "Inappropriate ioctl for device" |  Major | build 
|
+| [HBASE-26556](https://issues.apache.org/jira/browse/HBASE-26556) | IT and 
Chaos Monkey improvements |  Minor | integration tests |
+| [HBASE-26525](https://issues.apache.org/jira/browse/HBASE-26525) | Use 
unique thread name for group WALs |  Major | wal |
+| [HBASE-26517](https://issues.apache.org/jira/browse/HBASE-26517) | Add auth 
method information to AccessChecker audit log |  Trivial | security |
+| [HBASE-26512](https://issues.apache.org/jira/browse/HBASE-26512) | Make 
timestamp format configurable in HBase shell scan output |  Major | shell |
+| [HBASE-26485](https://issues.apache.org/jira/browse/HBASE-26485) | Introduce 
a method to clean restore directory after Snapshot Scan |  Minor | snapshots |
+| [HBASE-26475](https://issues.apache.org/jira/browse/HBASE-26475) | The flush 
and compact methods in HTU should skip processing secondary replicas |  Major | 
test |
+| [HBASE-26267](https://issues.apache.org/jira/browse/HBASE-26267) | Master 
initialization fails if Master Region WAL dir is missing |  Major | master |
+| [HBASE-26337](https://issues.apache.org/jira/browse/HBASE-26337) | 
Optimization for weighted random generators |  Major | Balancer |
+| [HBASE-26309](https://issues.apache.org/jira/browse/HBASE-26309) | Balancer 
tends to move regions to the server at the end of list |  Major | Balancer |
+
+
+### BUG FIXES:
+
+| JIRA | Summary | Priority | Component |
+|: |: | :--- |: |
+| [HBASE-26541](https://issues.apache.org/jira/browse/HBASE-26541) | 
hbase-protocol-shaded not buildable on M1 MacOSX |  Major | . |
+| [HBASE-26527](https://issues.apache.org/jira/browse/HBASE-26527) | 
ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewKeyValue() |  Major | 
wal |
+| [HBASE-26462](https://issues.apache.org/jira/browse/HBASE-26462) | Should 
persist restoreAcl flag in the procedure state for CloneSnapshotProcedure and 
RestoreSnapshotProcedure |  Critical | proc-v2, snapshots |
+| [HBASE-26533](https://issues.apache.org/jira/browse/HBASE-26533) | 
KeyValueScanner might not be properly closed when using 
InternalScan.checkOnlyMemStore() |  Minor | . |
+| [HBASE-26482](https://issues.apache.org/jira/browse/HBASE-26482) | HMaster 
may clean wals that is replicating in rare cases |  Critical | Replication |
+| [HBASE-26468](https://issues.apache.org/jira/browse/HBASE-26468) | Region 
Server doesn't exit cleanly incase it crashes. |  Major | regionserver |
+| [HBASE-25905](https://issues.apache.org/jira/browse/HBASE-25905) | Shutdown 
of WAL stuck at waitForSafePoint |  Blocker | regionserver, wal |
+| [HBASE-26450](https://issues.apache.org/jira/browse/HBASE-26450) | Server 
configuration will overwrite HStore configuration after using shell command 
'update\_config' |  Minor | Compaction, conf, regionserver |
+| [HBASE-26476](https://issues.apache.org/jira/browse/HBASE-26476) | Make 
DefaultMemStore extensible for HStore.memstore |  Major | regionserver |
+| [HBASE-26465](https://issues.apache.org/jira/browse/HBASE-26465) | 
MemStoreLAB may be released early when its SegmentScanner is scanning |  
Critical | regionserver |
+| [HBASE-26467](https://issues.apache.org/jira/browse/HBASE-26467) | Wrong 
Cell Generated by MemStoreLABImpl.forceCopyOfBigCellInto when Cell size bigger 
than data chunk size |  Critical | in-memory-compaction |
+| [HBASE-26463](https://issues.apache.org/jira/browse/HBASE-26463) | 
Unreadable table names after HBASE-24605 |  Trivial | UI |
+| [HBASE-26438](https://issues.apache.org/jira/browse/HBASE-26438) | Fix  
flaky test TestHStore.testCompactingMemStoreCellExceedInmemoryFlushSize |  
Major | test |
+| [HBASE-26311](https://issues.apache.org/jira/browse/HBASE-26311) | Balancer 
gets stuck in cohosted replica distribution |  Major | Balancer |
+| [HBASE-26384](https://issues.apache.org/jira/browse/HBASE-26384) | Segment 
already flushed to hfile may still be remained in CompactingMemStore |  Major | 
in-memory-compaction |
+| [HBASE-26410](https://issues.apache.org/jira/browse/HBASE-26410) | Fix HBase 
TestCanaryTool for Java17 |  Major | java |
+| [HBASE-26429](https://issues.apache.org/jira/browse/HBASE-26429) | 
HeapMemoryManager fails memstore flushes with NPE if enabled |  Major | 
Operability, regionserver |
+| [HBASE-25322](https://issues.apache.org/jira/browse/HBASE-25322) | Redundant 
Reference file in bottom region of split |  Minor | . |
+|

svn commit: r51562 [3/4] - /dev/hbase/2.4.9RC0/

2021-12-17 Thread apurtell

Added: dev/hbase/2.4.9RC0/RELEASENOTES.md
==
--- dev/hbase/2.4.9RC0/RELEASENOTES.md (added)
+++ dev/hbase/2.4.9RC0/RELEASENOTES.md Sat Dec 18 03:35:45 2021
@@ -0,0 +1,19913 @@
+# RELEASENOTES
+
+
+# HBASE  2.4.9 Release Notes
+
+These release notes cover new developer and user-facing incompatibilities, 
important issues, features, and major improvements.
+
+
+---
+
+* [HBASE-26542](https://issues.apache.org/jira/browse/HBASE-26542) | *Minor* | 
**Apply a \`package\` to test protobuf files**
+
+The protobuf structures used in test are all now scoped by the package name 
\`hbase.test.pb\`.
+
+
+---
+
+* [HBASE-26512](https://issues.apache.org/jira/browse/HBASE-26512) | *Major* | 
**Make timestamp format configurable in HBase shell scan output**
+
+HBASE-23930 changed the formatting of the timestamp attribute on each Cell as 
displayed by the HBase shell to be formatted as an ISO-8601 string rather that 
milliseconds since the epoch. Some users may have logic which expects the 
timestamp to be displayed as milliseconds since the epoch. This change 
introduces the configuration property hbase.shell.timestamp.format.epoch which 
controls whether the shell will print an ISO-8601 formatted timestamp (the 
default "false") or milliseconds since the epoch ("true").
+
+
+
+# HBASE  2.4.8 Release Notes
+
+These release notes cover new developer and user-facing incompatibilities, 
important issues, features, and major improvements.
+
+
+---
+
+* [HBASE-26362](https://issues.apache.org/jira/browse/HBASE-26362) | *Major* | 
**Upload mvn site artifacts for nightly build to nightlies**
+
+Now we will upload the site artifacts to nightlies for nightly build as well 
as pre commit build.
+
+
+---
+
+* [HBASE-26329](https://issues.apache.org/jira/browse/HBASE-26329) | *Major* | 
**Upgrade commons-io to 2.11.0**
+
+Upgraded commons-io to 2.11.0.
+
+
+---
+
+* [HBASE-26186](https://issues.apache.org/jira/browse/HBASE-26186) | *Major* | 
**jenkins script for caching artifacts should verify cached file before relying 
on it**
+
+Add a '--verify-tar-gz' option to cache-apache-project-artifact.sh for 
verifying whether the cached file can be parsed as a gzipped tarball.
+Use this option in our nightly job to avoid failures on broken cached hadoop 
tarballs.
+
+
+---
+
+* [HBASE-26339](https://issues.apache.org/jira/browse/HBASE-26339) | *Major* | 
**SshPublisher will skip uploading artifacts if the build is failure**
+
+Now we will mark build as unstable instead of failure when the yetus script 
returns error. This is used to solve the problem that the SshPublisher jenkins 
plugin will skip uploading artifacts if the build is marked as failure. In 
fact, the test output will be more important when there are UT failures.
+
+
+
+# HBASE  2.4.7 Release Notes
+
+These release notes cover new developer and user-facing incompatibilities, 
important issues, features, and major improvements.
+
+
+---
+
+* [HBASE-26274](https://issues.apache.org/jira/browse/HBASE-26274) | *Major* | 
**Create an option to reintroduce BlockCache to mapreduce job**
+
+Introduce \`hfile.onheap.block.cache.fixed.size\` and default to disable. When 
using ClientSideRegionScanner, it will be enabled with a fixed size for caching 
INDEX/LEAF\_INDEX block when a client, e.g. snapshot scanner, scans the entire 
HFile and does not need to seek/reseek to index block multiple times.
+
+
+---
+
+* [HBASE-26270](https://issues.apache.org/jira/browse/HBASE-26270) | *Minor* | 
**Provide getConfiguration method for Region and Store interface**
+
+Provide 'getReadOnlyConfiguration' method for Store and Region interface
+
+
+---
+
+* [HBASE-26273](https://issues.apache.org/jira/browse/HBASE-26273) | *Major* | 
**TableSnapshotInputFormat/TableSnapshotInputFormatImpl should use 
ReadType.STREAM for scanning HFiles**
+
+HBase's MapReduce API which can operate over HBase snapshots will now default 
to using ReadType.STREAM instead of ReadType.DEFAULT (which is PREAD) as a 
result of this change. HBase developers expect that STREAM will perform 
significantly better for average Snapshot-based batch jobs. Users can restore 
the previous functionality (using PREAD) by updating their code to explicitly 
set a value of \`ReadType.PREAD\` on the \`Scan\` object they provide to 
TableSnapshotInputFormat, or by setting the configuration property 
"hbase.TableSnapshotInputFormat.scanner.readtype" to "PREAD" in hbase-site.xml.
+
+
+---
+
+* [HBASE-26276](https://issues.apache.org/jira/browse/HBASE-26276) | *Major* | 
**Allow HashTable/SyncTable to perform rawScan when comparing cells**
+
+Added --rawScan option to HashTable job, which allows HashTable/SyncTable to 
perform raw scans. If this property is omitted, it defaults to false. When used 
together with --versions set to a high value, SyncTable will fabricate delete 
markers to all old versions still hanging (not cleaned yet by major 
compaction), avoiding

svn commit: r51562 [1/4] - /dev/hbase/2.4.9RC0/

2021-12-17 Thread apurtell

Author: apurtell
Date: Sat Dec 18 03:35:45 2021
New Revision: 51562

Log:
Stage HBase 2.4.9RC0

Added:
dev/hbase/2.4.9RC0/
dev/hbase/2.4.9RC0/CHANGES.md
dev/hbase/2.4.9RC0/RELEASENOTES.md
dev/hbase/2.4.9RC0/api_compare_2.4.8_to_2.4.9RC0.html
dev/hbase/2.4.9RC0/hbase-2.4.9-bin.tar.gz   (with props)
dev/hbase/2.4.9RC0/hbase-2.4.9-bin.tar.gz.asc
dev/hbase/2.4.9RC0/hbase-2.4.9-bin.tar.gz.sha512
dev/hbase/2.4.9RC0/hbase-2.4.9-client-bin.tar.gz   (with props)
dev/hbase/2.4.9RC0/hbase-2.4.9-client-bin.tar.gz.asc
dev/hbase/2.4.9RC0/hbase-2.4.9-client-bin.tar.gz.sha512
dev/hbase/2.4.9RC0/hbase-2.4.9-src.tar.gz   (with props)
dev/hbase/2.4.9RC0/hbase-2.4.9-src.tar.gz.asc
dev/hbase/2.4.9RC0/hbase-2.4.9-src.tar.gz.sha512

[hbase] annotated tag 2.4.9RC0 created (now 863ef34)

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a change to annotated tag 2.4.9RC0
in repository https://gitbox.apache.org/repos/asf/hbase.git.


  at 863ef34  (tag)
 tagging c49f7f63fca144765bf7c2da41791769286dfccc (commit)
 replaces rel/2.4.8
  by Andrew Purtell
  on Fri Dec 17 18:52:42 2021 -0800

- Log -
2.4.9RC0
---

This annotated tag includes the following new commits:

 new c49f7f6  Preparing hbase release 2.4.9RC0; tagging and updates to 
CHANGES.md and RELEASENOTES.md

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[hbase] branch branch-2.4 updated (d1b4530 -> 6449087)

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a change to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git.


from d1b4530  HBASE-26601 maven-gpg-plugin failing with "Inappropriate 
ioctl for device"
 new c49f7f6  Preparing hbase release 2.4.9RC0; tagging and updates to 
CHANGES.md and RELEASENOTES.md
 new 6449087  Preparing development version 2.4.10-SNAPSHOT

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.md | 74 ++
 RELEASENOTES.md| 20 ++
 hbase-annotations/pom.xml  |  2 +-
 hbase-archetypes/hbase-archetype-builder/pom.xml   |  2 +-
 hbase-archetypes/hbase-client-project/pom.xml  |  2 +-
 .../hbase-shaded-client-project/pom.xml|  2 +-
 hbase-archetypes/pom.xml   |  2 +-
 hbase-assembly/pom.xml |  2 +-
 hbase-asyncfs/pom.xml  |  2 +-
 hbase-build-configuration/pom.xml  |  2 +-
 hbase-checkstyle/pom.xml   |  4 +-
 hbase-client/pom.xml   |  2 +-
 hbase-common/pom.xml   |  2 +-
 hbase-endpoint/pom.xml |  2 +-
 hbase-examples/pom.xml |  2 +-
 hbase-external-blockcache/pom.xml  |  2 +-
 hbase-hadoop-compat/pom.xml|  2 +-
 hbase-hadoop2-compat/pom.xml   |  2 +-
 hbase-hbtop/pom.xml|  2 +-
 hbase-http/pom.xml |  2 +-
 hbase-it/pom.xml   |  2 +-
 hbase-logging/pom.xml  |  2 +-
 hbase-mapreduce/pom.xml|  2 +-
 hbase-metrics-api/pom.xml  |  2 +-
 hbase-metrics/pom.xml  |  2 +-
 hbase-procedure/pom.xml|  2 +-
 hbase-protocol-shaded/pom.xml  |  2 +-
 hbase-protocol/pom.xml |  2 +-
 hbase-replication/pom.xml  |  2 +-
 hbase-resource-bundle/pom.xml  |  2 +-
 hbase-rest/pom.xml |  2 +-
 hbase-rsgroup/pom.xml  |  2 +-
 hbase-server/pom.xml   |  2 +-
 hbase-shaded/hbase-shaded-check-invariants/pom.xml |  2 +-
 .../hbase-shaded-client-byo-hadoop/pom.xml |  2 +-
 hbase-shaded/hbase-shaded-client/pom.xml   |  2 +-
 hbase-shaded/hbase-shaded-mapreduce/pom.xml|  2 +-
 .../hbase-shaded-testing-util-tester/pom.xml   |  2 +-
 hbase-shaded/hbase-shaded-testing-util/pom.xml |  2 +-
 .../pom.xml|  2 +-
 hbase-shaded/pom.xml   |  2 +-
 hbase-shell/pom.xml|  2 +-
 hbase-testing-util/pom.xml |  2 +-
 hbase-thrift/pom.xml   |  2 +-
 hbase-zookeeper/pom.xml|  2 +-
 pom.xml|  2 +-
 46 files changed, 139 insertions(+), 45 deletions(-)

[hbase] 02/02: Preparing development version 2.4.10-SNAPSHOT

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 6449087228cb941182a44f574fe240ea399feb79
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 18:52:26 2021 -0800

Preparing development version 2.4.10-SNAPSHOT

Signed-off-by: Andrew Purtell 
---
 hbase-annotations/pom.xml  | 2 +-
 hbase-archetypes/hbase-archetype-builder/pom.xml   | 2 +-
 hbase-archetypes/hbase-client-project/pom.xml  | 2 +-
 hbase-archetypes/hbase-shaded-client-project/pom.xml   | 2 +-
 hbase-archetypes/pom.xml   | 2 +-
 hbase-assembly/pom.xml | 2 +-
 hbase-asyncfs/pom.xml  | 2 +-
 hbase-build-configuration/pom.xml  | 2 +-
 hbase-checkstyle/pom.xml   | 4 ++--
 hbase-client/pom.xml   | 2 +-
 hbase-common/pom.xml   | 2 +-
 hbase-endpoint/pom.xml | 2 +-
 hbase-examples/pom.xml | 2 +-
 hbase-external-blockcache/pom.xml  | 2 +-
 hbase-hadoop-compat/pom.xml| 2 +-
 hbase-hadoop2-compat/pom.xml   | 2 +-
 hbase-hbtop/pom.xml| 2 +-
 hbase-http/pom.xml | 2 +-
 hbase-it/pom.xml   | 2 +-
 hbase-logging/pom.xml  | 2 +-
 hbase-mapreduce/pom.xml| 2 +-
 hbase-metrics-api/pom.xml  | 2 +-
 hbase-metrics/pom.xml  | 2 +-
 hbase-procedure/pom.xml| 2 +-
 hbase-protocol-shaded/pom.xml  | 2 +-
 hbase-protocol/pom.xml | 2 +-
 hbase-replication/pom.xml  | 2 +-
 hbase-resource-bundle/pom.xml  | 2 +-
 hbase-rest/pom.xml | 2 +-
 hbase-rsgroup/pom.xml  | 2 +-
 hbase-server/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-check-invariants/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-client/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-mapreduce/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-testing-util-tester/pom.xml  | 2 +-
 hbase-shaded/hbase-shaded-testing-util/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-with-hadoop-check-invariants/pom.xml | 2 +-
 hbase-shaded/pom.xml   | 2 +-
 hbase-shell/pom.xml| 2 +-
 hbase-testing-util/pom.xml | 2 +-
 hbase-thrift/pom.xml   | 2 +-
 hbase-zookeeper/pom.xml| 2 +-
 pom.xml| 2 +-
 44 files changed, 45 insertions(+), 45 deletions(-)

diff --git a/hbase-annotations/pom.xml b/hbase-annotations/pom.xml
index dfad925..9719017 100644
--- a/hbase-annotations/pom.xml
+++ b/hbase-annotations/pom.xml
@@ -23,7 +23,7 @@
   
 hbase
 org.apache.hbase
-2.4.9
+2.4.10-SNAPSHOT
 ..
   
 
diff --git a/hbase-archetypes/hbase-archetype-builder/pom.xml 
b/hbase-archetypes/hbase-archetype-builder/pom.xml
index 367a53b..4d4a987 100644
--- a/hbase-archetypes/hbase-archetype-builder/pom.xml
+++ b/hbase-archetypes/hbase-archetype-builder/pom.xml
@@ -25,7 +25,7 @@
   
 hbase-archetypes
 org.apache.hbase
-2.4.9
+2.4.10-SNAPSHOT
 ..
   
 
diff --git a/hbase-archetypes/hbase-client-project/pom.xml 
b/hbase-archetypes/hbase-client-project/pom.xml
index 01b5a5d..0a0f5bb 100644
--- a/hbase-archetypes/hbase-client-project/pom.xml
+++ b/hbase-archetypes/hbase-client-project/pom.xml
@@ -26,7 +26,7 @@
   
 hbase-archetypes
 org.apache.hbase
-2.4.9
+2.4.10-SNAPSHOT
 ..
   
   hbase-client-project
diff --git a/hbase-archetypes/hbase-shaded-client-project/pom.xml 
b/hbase-archetypes/hbase-shaded-client-project/pom.xml
index c0f443c..78533c8 100644
--- a/hbase-archetypes/hbase-shaded-client-project/pom.xml
+++ b/hbase-archetypes/hbase-shaded-client-project/pom.xml
@@ -26,7 +26,7 @@
   
 hbase-archetypes
 org.apache.hbase
-2.4.9
+2.4.10-SNAPSHOT
 ..
   
   hbase-shaded-client-project
diff --git

[hbase] 01/02: Preparing hbase release 2.4.9RC0; tagging and updates to CHANGES.md and RELEASENOTES.md

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit c49f7f63fca144765bf7c2da41791769286dfccc
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 18:51:23 2021 -0800

Preparing hbase release 2.4.9RC0; tagging and updates to CHANGES.md and 
RELEASENOTES.md

Signed-off-by: Andrew Purtell 
---
 CHANGES.md | 74 ++
 RELEASENOTES.md| 20 ++
 hbase-annotations/pom.xml  |  2 +-
 hbase-archetypes/hbase-archetype-builder/pom.xml   |  2 +-
 hbase-archetypes/hbase-client-project/pom.xml  |  2 +-
 .../hbase-shaded-client-project/pom.xml|  2 +-
 hbase-archetypes/pom.xml   |  2 +-
 hbase-assembly/pom.xml |  2 +-
 hbase-asyncfs/pom.xml  |  2 +-
 hbase-build-configuration/pom.xml  |  2 +-
 hbase-checkstyle/pom.xml   |  4 +-
 hbase-client/pom.xml   |  2 +-
 hbase-common/pom.xml   |  2 +-
 hbase-endpoint/pom.xml |  2 +-
 hbase-examples/pom.xml |  2 +-
 hbase-external-blockcache/pom.xml  |  2 +-
 hbase-hadoop-compat/pom.xml|  2 +-
 hbase-hadoop2-compat/pom.xml   |  2 +-
 hbase-hbtop/pom.xml|  2 +-
 hbase-http/pom.xml |  2 +-
 hbase-it/pom.xml   |  2 +-
 hbase-logging/pom.xml  |  2 +-
 hbase-mapreduce/pom.xml|  2 +-
 hbase-metrics-api/pom.xml  |  2 +-
 hbase-metrics/pom.xml  |  2 +-
 hbase-procedure/pom.xml|  2 +-
 hbase-protocol-shaded/pom.xml  |  2 +-
 hbase-protocol/pom.xml |  2 +-
 hbase-replication/pom.xml  |  2 +-
 hbase-resource-bundle/pom.xml  |  2 +-
 hbase-rest/pom.xml |  2 +-
 hbase-rsgroup/pom.xml  |  2 +-
 hbase-server/pom.xml   |  2 +-
 hbase-shaded/hbase-shaded-check-invariants/pom.xml |  2 +-
 .../hbase-shaded-client-byo-hadoop/pom.xml |  2 +-
 hbase-shaded/hbase-shaded-client/pom.xml   |  2 +-
 hbase-shaded/hbase-shaded-mapreduce/pom.xml|  2 +-
 .../hbase-shaded-testing-util-tester/pom.xml   |  2 +-
 hbase-shaded/hbase-shaded-testing-util/pom.xml |  2 +-
 .../pom.xml|  2 +-
 hbase-shaded/pom.xml   |  2 +-
 hbase-shell/pom.xml|  2 +-
 hbase-testing-util/pom.xml |  2 +-
 hbase-thrift/pom.xml   |  2 +-
 hbase-zookeeper/pom.xml|  2 +-
 pom.xml|  2 +-
 46 files changed, 139 insertions(+), 45 deletions(-)

diff --git a/CHANGES.md b/CHANGES.md
index 056387a..687425d 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -20,6 +20,80 @@
 # Be careful doing manual edits in this file. Do not change format
 # of release header or remove the below marker. This file is generated.
 # DO NOT REMOVE THIS MARKER; FOR INTERPOLATING CHANGES!-->
+## Release 2.4.9 - 2021-12-23
+
+
+
+### IMPROVEMENTS:
+
+| JIRA | Summary | Priority | Component |
+|: |: | :--- |: |
+| [HBASE-26601](https://issues.apache.org/jira/browse/HBASE-26601) | 
maven-gpg-plugin failing with "Inappropriate ioctl for device" |  Major | build 
|
+| [HBASE-26556](https://issues.apache.org/jira/browse/HBASE-26556) | IT and 
Chaos Monkey improvements |  Minor | integration tests |
+| [HBASE-26525](https://issues.apache.org/jira/browse/HBASE-26525) | Use 
unique thread name for group WALs |  Major | wal |
+| [HBASE-26517](https://issues.apache.org/jira/browse/HBASE-26517) | Add auth 
method information to AccessChecker audit log |  Trivial | security |
+| [HBASE-26512](https://issues.apache.org/jira/browse/HBASE-26512) | Make 
timestamp format configurable in HBase shell scan output |  Major | shell |
+| [HBASE-26485](https://issues.apache.org/jira/browse/HBASE-26485) | Introduce 
a method to clean restore directory after Snapshot Scan |  Minor | snapshots |
+| [HBASE-26475](https://issues.apache.org/jira/browse/HBASE-26475) | The flush 
and compact methods in HTU should skip processing secondary replicas |  Major | 
test |
+| [HBASE-26267](https://issues.apache.org/jira/browse/HBASE-26267) | Master 
initialization fails if Master Region WAL dir is missing |  Major | master |
+| [HBASE-26337](https://issues.apache.org/jira/browse/HBASE-26337) |

[hbase] branch branch-2.5 updated: HBASE-26601 maven-gpg-plugin failing with "Inappropriate ioctl for device"

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2.5
in repository https://gitbox.apache.org/repos/asf/hbase.git


The following commit(s) were added to refs/heads/branch-2.5 by this push:
 new 5d56c80  HBASE-26601 maven-gpg-plugin failing with "Inappropriate 
ioctl for device"
5d56c80 is described below

commit 5d56c8005b3f236591629da208c7d50b90f9c1a4
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 17:58:17 2021 -0800

HBASE-26601 maven-gpg-plugin failing with "Inappropriate ioctl for device"

Signed-off-by: Andrew Purtell 
---
 pom.xml | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/pom.xml b/pom.xml
index c706281..fc239af 100755
--- a/pom.xml
+++ b/pom.xml
@@ -865,6 +865,11 @@
 
   
 
+
+  org.apache.maven.plugins
+  maven-gpg-plugin
+  ${maven.gpg.version}
+
   
 
 
@@ -1525,6 +1530,7 @@
 3.3.0
 3.1.0
 2.10
+3.0.1
 3.2.0
 1.1.0
 
3.1.2

[hbase] branch master updated: HBASE-26601 maven-gpg-plugin failing with "Inappropriate ioctl for device"

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git


The following commit(s) were added to refs/heads/master by this push:
 new a0acf8c  HBASE-26601 maven-gpg-plugin failing with "Inappropriate 
ioctl for device"
a0acf8c is described below

commit a0acf8cb421f5abcc41d52f22d32713952b0b229
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 17:58:17 2021 -0800

HBASE-26601 maven-gpg-plugin failing with "Inappropriate ioctl for device"

Signed-off-by: Andrew Purtell 
---
 pom.xml | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/pom.xml b/pom.xml
index 87448a9..ebb0f42 100755
--- a/pom.xml
+++ b/pom.xml
@@ -1075,6 +1075,11 @@
 
   
 
+
+  org.apache.maven.plugins
+  maven-gpg-plugin
+  ${maven.gpg.version}
+
   
 
 
@@ -1782,6 +1787,7 @@
 3.3.0
 3.1.0
 2.10
+3.0.1
 3.2.0
 1.1.0
 
3.1.2

[hbase] branch branch-2.4 updated: HBASE-26601 maven-gpg-plugin failing with "Inappropriate ioctl for device"

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git


The following commit(s) were added to refs/heads/branch-2.4 by this push:
 new d1b4530  HBASE-26601 maven-gpg-plugin failing with "Inappropriate 
ioctl for device"
d1b4530 is described below

commit d1b4530a155c8aa29383935f55e21ccd7a8e227d
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 17:58:17 2021 -0800

HBASE-26601 maven-gpg-plugin failing with "Inappropriate ioctl for device"

Signed-off-by: Andrew Purtell 
---
 pom.xml | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/pom.xml b/pom.xml
index f5002ff..0c29121 100755
--- a/pom.xml
+++ b/pom.xml
@@ -862,6 +862,11 @@
 
   
 
+
+  org.apache.maven.plugins
+  maven-gpg-plugin
+  ${maven.gpg.version}
+
   
 
 
@@ -1501,6 +1506,7 @@
 3.3.0
 3.1.0
 2.10
+3.0.1
 3.2.0
 1.1.0
 
3.1.2

[hbase] branch branch-2 updated: HBASE-26601 maven-gpg-plugin failing with "Inappropriate ioctl for device"

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git


The following commit(s) were added to refs/heads/branch-2 by this push:
 new 29e3743  HBASE-26601 maven-gpg-plugin failing with "Inappropriate 
ioctl for device"
29e3743 is described below

commit 29e37439f790c33949fd26a75dff4aa9ebddc024
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 17:58:17 2021 -0800

HBASE-26601 maven-gpg-plugin failing with "Inappropriate ioctl for device"

Signed-off-by: Andrew Purtell 
---
 pom.xml | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/pom.xml b/pom.xml
index d11f5b4..d0775e7 100755
--- a/pom.xml
+++ b/pom.xml
@@ -865,6 +865,11 @@
 
   
 
+
+  org.apache.maven.plugins
+  maven-gpg-plugin
+  ${maven.gpg.version}
+
   
 
 
@@ -1525,6 +1530,7 @@
 3.3.0
 3.1.0
 2.10
+3.0.1
 3.2.0
 1.1.0
 
3.1.2

[hbase] 01/03: Revert "HBASE-26490 Add builder for class ReplicationLoadSink (#3883)"

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 823b8fd667f618d029bcf0d282230bb716745314
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 18:34:09 2021 -0800

Revert "HBASE-26490 Add builder for class ReplicationLoadSink (#3883)"

This reverts commit 02fa0903a206bbcc42b7daaadb66e428e0a4184e.
---
 .../hbase/replication/ReplicationLoadSink.java | 44 ++
 .../hadoop/hbase/shaded/protobuf/ProtobufUtil.java | 10 ++---
 2 files changed, 7 insertions(+), 47 deletions(-)

diff --git 
a/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationLoadSink.java
 
b/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationLoadSink.java
index 7551560..f9afddd 100644
--- 
a/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationLoadSink.java
+++ 
b/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationLoadSink.java
@@ -22,7 +22,9 @@ public class ReplicationLoadSink {
   private final long timestampStarted;
   private final long totalOpsProcessed;
 
-  private ReplicationLoadSink(long age, long timestamp, long timestampStarted,
+  // TODO: add the builder for this class
+  @InterfaceAudience.Private
+  public ReplicationLoadSink(long age, long timestamp, long timestampStarted,
   long totalOpsProcessed) {
 this.ageOfLastAppliedOp = age;
 this.timestampsOfLastAppliedOp = timestamp;
@@ -54,44 +56,4 @@ public class ReplicationLoadSink {
   public long getTotalOpsProcessed() {
 return totalOpsProcessed;
   }
-
-  @InterfaceAudience.Private
-  public static ReplicationLoadSinkBuilder newBuilder() {
-return new ReplicationLoadSinkBuilder();
-  }
-
-  @InterfaceAudience.Private
-  public static final class ReplicationLoadSinkBuilder {
-private long ageOfLastAppliedOp;
-private long timestampsOfLastAppliedOp;
-private long timestampStarted;
-private long totalOpsProcessed;
-
-private ReplicationLoadSinkBuilder() {}
-
-public ReplicationLoadSinkBuilder setAgeOfLastAppliedOp(long 
ageOfLastAppliedOp) {
-  this.ageOfLastAppliedOp = ageOfLastAppliedOp;
-  return this;
-}
-
-public ReplicationLoadSinkBuilder setTimestampsOfLastAppliedOp(long 
timestampsOfLastAppliedOp) {
-  this.timestampsOfLastAppliedOp = timestampsOfLastAppliedOp;
-  return this;
-}
-
-public ReplicationLoadSinkBuilder setTimestampStarted(long 
timestampStarted) {
-  this.timestampStarted = timestampStarted;
-  return this;
-}
-
-public ReplicationLoadSinkBuilder setTotalOpsProcessed(long 
totalOpsProcessed) {
-  this.totalOpsProcessed = totalOpsProcessed;
-  return this;
-}
-
-public ReplicationLoadSink build() {
-  return new ReplicationLoadSink(ageOfLastAppliedOp, 
timestampsOfLastAppliedOp,
-timestampStarted, totalOpsProcessed);
-}
-  }
 }
diff --git 
a/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
 
b/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
index 9485809..c2544f6 100644
--- 
a/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
+++ 
b/hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
@@ -2796,12 +2796,10 @@ public final class ProtobufUtil {
 
   public static ReplicationLoadSink toReplicationLoadSink(
   ClusterStatusProtos.ReplicationLoadSink rls) {
-ReplicationLoadSink.ReplicationLoadSinkBuilder builder = 
ReplicationLoadSink.newBuilder();
-builder.setAgeOfLastAppliedOp(rls.getAgeOfLastAppliedOp()).
-  setTimestampsOfLastAppliedOp(rls.getTimeStampsOfLastAppliedOp()).
-  setTimestampStarted(rls.hasTimestampStarted()? 
rls.getTimestampStarted(): -1L).
-  setTotalOpsProcessed(rls.hasTotalOpsProcessed()? 
rls.getTotalOpsProcessed(): -1L);
-return builder.build();
+return new ReplicationLoadSink(rls.getAgeOfLastAppliedOp(),
+rls.getTimeStampsOfLastAppliedOp(),
+rls.hasTimestampStarted()? rls.getTimestampStarted(): -1L,
+rls.hasTotalOpsProcessed()? rls.getTotalOpsProcessed(): -1L);
   }
 
   public static ReplicationLoadSource toReplicationLoadSource(

[hbase] 03/03: Revert "Preparing hbase release 2.4.9RC0; tagging and updates to CHANGES.md and RELEASENOTES.md"

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 6e25a4e2f918b7e2d026a65869a305e39998f950
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 18:33:51 2021 -0800

Revert "Preparing hbase release 2.4.9RC0; tagging and updates to CHANGES.md 
and RELEASENOTES.md"

This reverts commit 53391b269c6dcc5051b21527cafb8e6d8cd2c4e9.
---
 CHANGES.md | 3 ---
 RELEASENOTES.md| 3 ---
 hbase-annotations/pom.xml  | 2 +-
 hbase-archetypes/hbase-archetype-builder/pom.xml   | 2 +-
 hbase-archetypes/hbase-client-project/pom.xml  | 2 +-
 hbase-archetypes/hbase-shaded-client-project/pom.xml   | 2 +-
 hbase-archetypes/pom.xml   | 2 +-
 hbase-assembly/pom.xml | 2 +-
 hbase-asyncfs/pom.xml  | 2 +-
 hbase-build-configuration/pom.xml  | 2 +-
 hbase-checkstyle/pom.xml   | 4 ++--
 hbase-client/pom.xml   | 2 +-
 hbase-common/pom.xml   | 2 +-
 hbase-endpoint/pom.xml | 2 +-
 hbase-examples/pom.xml | 2 +-
 hbase-external-blockcache/pom.xml  | 2 +-
 hbase-hadoop-compat/pom.xml| 2 +-
 hbase-hadoop2-compat/pom.xml   | 2 +-
 hbase-hbtop/pom.xml| 2 +-
 hbase-http/pom.xml | 2 +-
 hbase-it/pom.xml   | 2 +-
 hbase-logging/pom.xml  | 2 +-
 hbase-mapreduce/pom.xml| 2 +-
 hbase-metrics-api/pom.xml  | 2 +-
 hbase-metrics/pom.xml  | 2 +-
 hbase-procedure/pom.xml| 2 +-
 hbase-protocol-shaded/pom.xml  | 2 +-
 hbase-protocol/pom.xml | 2 +-
 hbase-replication/pom.xml  | 2 +-
 hbase-resource-bundle/pom.xml  | 2 +-
 hbase-rest/pom.xml | 2 +-
 hbase-rsgroup/pom.xml  | 2 +-
 hbase-server/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-check-invariants/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-client/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-mapreduce/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-testing-util-tester/pom.xml  | 2 +-
 hbase-shaded/hbase-shaded-testing-util/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-with-hadoop-check-invariants/pom.xml | 2 +-
 hbase-shaded/pom.xml   | 2 +-
 hbase-shell/pom.xml| 2 +-
 hbase-testing-util/pom.xml | 2 +-
 hbase-thrift/pom.xml   | 2 +-
 hbase-zookeeper/pom.xml| 2 +-
 pom.xml| 2 +-
 46 files changed, 45 insertions(+), 51 deletions(-)

diff --git a/CHANGES.md b/CHANGES.md
index 6e7af1e..056387a 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -20,9 +20,6 @@
 # Be careful doing manual edits in this file. Do not change format
 # of release header or remove the below marker. This file is generated.
 # DO NOT REMOVE THIS MARKER; FOR INTERPOLATING CHANGES!-->
-## Release 2.4.9 - Unreleased (as of Fri Dec 17 17:04:11 PST 2021)
-No changes
-
 ## Release 2.4.8 - 2021-11-05
 
 
diff --git a/RELEASENOTES.md b/RELEASENOTES.md
index c1b26b2..afedffe 100644
--- a/RELEASENOTES.md
+++ b/RELEASENOTES.md
@@ -20,9 +20,6 @@
 # Be careful doing manual edits in this file. Do not change format
 # of release header or remove the below marker. This file is generated.
 # DO NOT REMOVE THIS MARKER; FOR INTERPOLATING CHANGES!-->
-# hbase 2.4.9 Release Notes
-No changes
-
 # HBASE  2.4.8 Release Notes
 
 These release notes cover new developer and user-facing incompatibilities, 
important issues, features, and major improvements.
diff --git a/hbase-annotations/pom.xml b/hbase-annotations/pom.xml
index dfad925..8b3ef93 100644
--- a/hbase-annotations/pom.xml
+++ b/hbase-annotations/pom.xml
@@ -23,7 +23,7 @@
   
 hbase
 org.apache.hbase
-

[hbase] 02/03: Revert "Preparing development version 2.4.10-SNAPSHOT"

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 2059cda314141ed67f8f6ae31fff4362ae3c1bbb
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 18:33:50 2021 -0800

Revert "Preparing development version 2.4.10-SNAPSHOT"

This reverts commit deb570b2e935e0ceb60e6c312a778c24277eb36a.
---
 hbase-annotations/pom.xml  | 2 +-
 hbase-archetypes/hbase-archetype-builder/pom.xml   | 2 +-
 hbase-archetypes/hbase-client-project/pom.xml  | 2 +-
 hbase-archetypes/hbase-shaded-client-project/pom.xml   | 2 +-
 hbase-archetypes/pom.xml   | 2 +-
 hbase-assembly/pom.xml | 2 +-
 hbase-asyncfs/pom.xml  | 2 +-
 hbase-build-configuration/pom.xml  | 2 +-
 hbase-checkstyle/pom.xml   | 4 ++--
 hbase-client/pom.xml   | 2 +-
 hbase-common/pom.xml   | 2 +-
 hbase-endpoint/pom.xml | 2 +-
 hbase-examples/pom.xml | 2 +-
 hbase-external-blockcache/pom.xml  | 2 +-
 hbase-hadoop-compat/pom.xml| 2 +-
 hbase-hadoop2-compat/pom.xml   | 2 +-
 hbase-hbtop/pom.xml| 2 +-
 hbase-http/pom.xml | 2 +-
 hbase-it/pom.xml   | 2 +-
 hbase-logging/pom.xml  | 2 +-
 hbase-mapreduce/pom.xml| 2 +-
 hbase-metrics-api/pom.xml  | 2 +-
 hbase-metrics/pom.xml  | 2 +-
 hbase-procedure/pom.xml| 2 +-
 hbase-protocol-shaded/pom.xml  | 2 +-
 hbase-protocol/pom.xml | 2 +-
 hbase-replication/pom.xml  | 2 +-
 hbase-resource-bundle/pom.xml  | 2 +-
 hbase-rest/pom.xml | 2 +-
 hbase-rsgroup/pom.xml  | 2 +-
 hbase-server/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-check-invariants/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-client/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-mapreduce/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-testing-util-tester/pom.xml  | 2 +-
 hbase-shaded/hbase-shaded-testing-util/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-with-hadoop-check-invariants/pom.xml | 2 +-
 hbase-shaded/pom.xml   | 2 +-
 hbase-shell/pom.xml| 2 +-
 hbase-testing-util/pom.xml | 2 +-
 hbase-thrift/pom.xml   | 2 +-
 hbase-zookeeper/pom.xml| 2 +-
 pom.xml| 2 +-
 44 files changed, 45 insertions(+), 45 deletions(-)

diff --git a/hbase-annotations/pom.xml b/hbase-annotations/pom.xml
index 9719017..dfad925 100644
--- a/hbase-annotations/pom.xml
+++ b/hbase-annotations/pom.xml
@@ -23,7 +23,7 @@
   
 hbase
 org.apache.hbase
-2.4.10-SNAPSHOT
+2.4.9
 ..
   
 
diff --git a/hbase-archetypes/hbase-archetype-builder/pom.xml 
b/hbase-archetypes/hbase-archetype-builder/pom.xml
index 4d4a987..367a53b 100644
--- a/hbase-archetypes/hbase-archetype-builder/pom.xml
+++ b/hbase-archetypes/hbase-archetype-builder/pom.xml
@@ -25,7 +25,7 @@
   
 hbase-archetypes
 org.apache.hbase
-2.4.10-SNAPSHOT
+2.4.9
 ..
   
 
diff --git a/hbase-archetypes/hbase-client-project/pom.xml 
b/hbase-archetypes/hbase-client-project/pom.xml
index 0a0f5bb..01b5a5d 100644
--- a/hbase-archetypes/hbase-client-project/pom.xml
+++ b/hbase-archetypes/hbase-client-project/pom.xml
@@ -26,7 +26,7 @@
   
 hbase-archetypes
 org.apache.hbase
-2.4.10-SNAPSHOT
+2.4.9
 ..
   
   hbase-client-project
diff --git a/hbase-archetypes/hbase-shaded-client-project/pom.xml 
b/hbase-archetypes/hbase-shaded-client-project/pom.xml
index 78533c8..c0f443c 100644
--- a/hbase-archetypes/hbase-shaded-client-project/pom.xml
+++ b/hbase-archetypes/hbase-shaded-client-project/pom.xml
@@ -26,7 +26,7 @@
   
 hbase-archetypes
 org.apache.hbase
-2.4.10-SNAPSHOT
+2.4.9
 ..

[hbase] branch branch-2.4 updated (deb570b -> 6e25a4e)

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a change to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git.


from deb570b  Preparing development version 2.4.10-SNAPSHOT
 new 823b8fd  Revert "HBASE-26490 Add builder for class ReplicationLoadSink 
(#3883)"
 new 2059cda  Revert "Preparing development version 2.4.10-SNAPSHOT"
 new 6e25a4e  Revert "Preparing hbase release 2.4.9RC0; tagging and updates 
to CHANGES.md and RELEASENOTES.md"

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.md |  3 --
 RELEASENOTES.md|  3 --
 hbase-annotations/pom.xml  |  2 +-
 hbase-archetypes/hbase-archetype-builder/pom.xml   |  2 +-
 hbase-archetypes/hbase-client-project/pom.xml  |  2 +-
 .../hbase-shaded-client-project/pom.xml|  2 +-
 hbase-archetypes/pom.xml   |  2 +-
 hbase-assembly/pom.xml |  2 +-
 hbase-asyncfs/pom.xml  |  2 +-
 hbase-build-configuration/pom.xml  |  2 +-
 hbase-checkstyle/pom.xml   |  4 +-
 hbase-client/pom.xml   |  2 +-
 .../hbase/replication/ReplicationLoadSink.java | 44 ++
 .../hadoop/hbase/shaded/protobuf/ProtobufUtil.java | 10 ++---
 hbase-common/pom.xml   |  2 +-
 hbase-endpoint/pom.xml |  2 +-
 hbase-examples/pom.xml |  2 +-
 hbase-external-blockcache/pom.xml  |  2 +-
 hbase-hadoop-compat/pom.xml|  2 +-
 hbase-hadoop2-compat/pom.xml   |  2 +-
 hbase-hbtop/pom.xml|  2 +-
 hbase-http/pom.xml |  2 +-
 hbase-it/pom.xml   |  2 +-
 hbase-logging/pom.xml  |  2 +-
 hbase-mapreduce/pom.xml|  2 +-
 hbase-metrics-api/pom.xml  |  2 +-
 hbase-metrics/pom.xml  |  2 +-
 hbase-procedure/pom.xml|  2 +-
 hbase-protocol-shaded/pom.xml  |  2 +-
 hbase-protocol/pom.xml |  2 +-
 hbase-replication/pom.xml  |  2 +-
 hbase-resource-bundle/pom.xml  |  2 +-
 hbase-rest/pom.xml |  2 +-
 hbase-rsgroup/pom.xml  |  2 +-
 hbase-server/pom.xml   |  2 +-
 hbase-shaded/hbase-shaded-check-invariants/pom.xml |  2 +-
 .../hbase-shaded-client-byo-hadoop/pom.xml |  2 +-
 hbase-shaded/hbase-shaded-client/pom.xml   |  2 +-
 hbase-shaded/hbase-shaded-mapreduce/pom.xml|  2 +-
 .../hbase-shaded-testing-util-tester/pom.xml   |  2 +-
 hbase-shaded/hbase-shaded-testing-util/pom.xml |  2 +-
 .../pom.xml|  2 +-
 hbase-shaded/pom.xml   |  2 +-
 hbase-shell/pom.xml|  2 +-
 hbase-testing-util/pom.xml |  2 +-
 hbase-thrift/pom.xml   |  2 +-
 hbase-zookeeper/pom.xml|  2 +-
 pom.xml|  2 +-
 48 files changed, 52 insertions(+), 98 deletions(-)

[hbase] annotated tag 2.4.9RC0 deleted (was 5a024fd)

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a change to annotated tag 2.4.9RC0
in repository https://gitbox.apache.org/repos/asf/hbase.git.


*** WARNING: tag 2.4.9RC0 was deleted! ***

   tag was  5a024fd

The revisions that were on this annotated tag are still contained in
other references; therefore, this change does not discard any commits
from the repository.

[hbase] branch branch-2.4 updated (4d43e4c -> deb570b)

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a change to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git.


from 4d43e4c  HBASE-26541 hbase-shaded-protobuf not buildable on M1 MacOSX
 add 53391b2  Preparing hbase release 2.4.9RC0; tagging and updates to 
CHANGES.md and RELEASENOTES.md
 new deb570b  Preparing development version 2.4.10-SNAPSHOT

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.md | 3 +++
 RELEASENOTES.md| 3 +++
 hbase-annotations/pom.xml  | 2 +-
 hbase-archetypes/hbase-archetype-builder/pom.xml   | 2 +-
 hbase-archetypes/hbase-client-project/pom.xml  | 2 +-
 hbase-archetypes/hbase-shaded-client-project/pom.xml   | 2 +-
 hbase-archetypes/pom.xml   | 2 +-
 hbase-assembly/pom.xml | 2 +-
 hbase-asyncfs/pom.xml  | 2 +-
 hbase-build-configuration/pom.xml  | 2 +-
 hbase-checkstyle/pom.xml   | 4 ++--
 hbase-client/pom.xml   | 2 +-
 hbase-common/pom.xml   | 2 +-
 hbase-endpoint/pom.xml | 2 +-
 hbase-examples/pom.xml | 2 +-
 hbase-external-blockcache/pom.xml  | 2 +-
 hbase-hadoop-compat/pom.xml| 2 +-
 hbase-hadoop2-compat/pom.xml   | 2 +-
 hbase-hbtop/pom.xml| 2 +-
 hbase-http/pom.xml | 2 +-
 hbase-it/pom.xml   | 2 +-
 hbase-logging/pom.xml  | 2 +-
 hbase-mapreduce/pom.xml| 2 +-
 hbase-metrics-api/pom.xml  | 2 +-
 hbase-metrics/pom.xml  | 2 +-
 hbase-procedure/pom.xml| 2 +-
 hbase-protocol-shaded/pom.xml  | 2 +-
 hbase-protocol/pom.xml | 2 +-
 hbase-replication/pom.xml  | 2 +-
 hbase-resource-bundle/pom.xml  | 2 +-
 hbase-rest/pom.xml | 2 +-
 hbase-rsgroup/pom.xml  | 2 +-
 hbase-server/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-check-invariants/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-client/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-mapreduce/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-testing-util-tester/pom.xml  | 2 +-
 hbase-shaded/hbase-shaded-testing-util/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-with-hadoop-check-invariants/pom.xml | 2 +-
 hbase-shaded/pom.xml   | 2 +-
 hbase-shell/pom.xml| 2 +-
 hbase-testing-util/pom.xml | 2 +-
 hbase-thrift/pom.xml   | 2 +-
 hbase-zookeeper/pom.xml| 2 +-
 pom.xml| 2 +-
 46 files changed, 51 insertions(+), 45 deletions(-)

[hbase] 01/01: Preparing development version 2.4.10-SNAPSHOT

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit deb570b2e935e0ceb60e6c312a778c24277eb36a
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 17:04:46 2021 -0800

Preparing development version 2.4.10-SNAPSHOT

Signed-off-by: Andrew Purtell 
---
 hbase-annotations/pom.xml  | 2 +-
 hbase-archetypes/hbase-archetype-builder/pom.xml   | 2 +-
 hbase-archetypes/hbase-client-project/pom.xml  | 2 +-
 hbase-archetypes/hbase-shaded-client-project/pom.xml   | 2 +-
 hbase-archetypes/pom.xml   | 2 +-
 hbase-assembly/pom.xml | 2 +-
 hbase-asyncfs/pom.xml  | 2 +-
 hbase-build-configuration/pom.xml  | 2 +-
 hbase-checkstyle/pom.xml   | 4 ++--
 hbase-client/pom.xml   | 2 +-
 hbase-common/pom.xml   | 2 +-
 hbase-endpoint/pom.xml | 2 +-
 hbase-examples/pom.xml | 2 +-
 hbase-external-blockcache/pom.xml  | 2 +-
 hbase-hadoop-compat/pom.xml| 2 +-
 hbase-hadoop2-compat/pom.xml   | 2 +-
 hbase-hbtop/pom.xml| 2 +-
 hbase-http/pom.xml | 2 +-
 hbase-it/pom.xml   | 2 +-
 hbase-logging/pom.xml  | 2 +-
 hbase-mapreduce/pom.xml| 2 +-
 hbase-metrics-api/pom.xml  | 2 +-
 hbase-metrics/pom.xml  | 2 +-
 hbase-procedure/pom.xml| 2 +-
 hbase-protocol-shaded/pom.xml  | 2 +-
 hbase-protocol/pom.xml | 2 +-
 hbase-replication/pom.xml  | 2 +-
 hbase-resource-bundle/pom.xml  | 2 +-
 hbase-rest/pom.xml | 2 +-
 hbase-rsgroup/pom.xml  | 2 +-
 hbase-server/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-check-invariants/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-client/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-mapreduce/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-testing-util-tester/pom.xml  | 2 +-
 hbase-shaded/hbase-shaded-testing-util/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-with-hadoop-check-invariants/pom.xml | 2 +-
 hbase-shaded/pom.xml   | 2 +-
 hbase-shell/pom.xml| 2 +-
 hbase-testing-util/pom.xml | 2 +-
 hbase-thrift/pom.xml   | 2 +-
 hbase-zookeeper/pom.xml| 2 +-
 pom.xml| 2 +-
 44 files changed, 45 insertions(+), 45 deletions(-)

diff --git a/hbase-annotations/pom.xml b/hbase-annotations/pom.xml
index dfad925..9719017 100644
--- a/hbase-annotations/pom.xml
+++ b/hbase-annotations/pom.xml
@@ -23,7 +23,7 @@
   
 hbase
 org.apache.hbase
-2.4.9
+2.4.10-SNAPSHOT
 ..
   
 
diff --git a/hbase-archetypes/hbase-archetype-builder/pom.xml 
b/hbase-archetypes/hbase-archetype-builder/pom.xml
index 367a53b..4d4a987 100644
--- a/hbase-archetypes/hbase-archetype-builder/pom.xml
+++ b/hbase-archetypes/hbase-archetype-builder/pom.xml
@@ -25,7 +25,7 @@
   
 hbase-archetypes
 org.apache.hbase
-2.4.9
+2.4.10-SNAPSHOT
 ..
   
 
diff --git a/hbase-archetypes/hbase-client-project/pom.xml 
b/hbase-archetypes/hbase-client-project/pom.xml
index 01b5a5d..0a0f5bb 100644
--- a/hbase-archetypes/hbase-client-project/pom.xml
+++ b/hbase-archetypes/hbase-client-project/pom.xml
@@ -26,7 +26,7 @@
   
 hbase-archetypes
 org.apache.hbase
-2.4.9
+2.4.10-SNAPSHOT
 ..
   
   hbase-client-project
diff --git a/hbase-archetypes/hbase-shaded-client-project/pom.xml 
b/hbase-archetypes/hbase-shaded-client-project/pom.xml
index c0f443c..78533c8 100644
--- a/hbase-archetypes/hbase-shaded-client-project/pom.xml
+++ b/hbase-archetypes/hbase-shaded-client-project/pom.xml
@@ -26,7 +26,7 @@
   
 hbase-archetypes
 org.apache.hbase
-2.4.9
+2.4.10-SNAPSHOT
 ..
   
   hbase-shaded-client-project
diff --git

[hbase] 01/01: Preparing hbase release 2.4.9RC0; tagging and updates to CHANGES.md and RELEASENOTES.md

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to annotated tag 2.4.9RC0
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 53391b269c6dcc5051b21527cafb8e6d8cd2c4e9
Author: Andrew Purtell 
AuthorDate: Fri Dec 17 17:04:42 2021 -0800

Preparing hbase release 2.4.9RC0; tagging and updates to CHANGES.md and 
RELEASENOTES.md

Signed-off-by: Andrew Purtell 
---
 CHANGES.md | 3 +++
 RELEASENOTES.md| 3 +++
 hbase-annotations/pom.xml  | 2 +-
 hbase-archetypes/hbase-archetype-builder/pom.xml   | 2 +-
 hbase-archetypes/hbase-client-project/pom.xml  | 2 +-
 hbase-archetypes/hbase-shaded-client-project/pom.xml   | 2 +-
 hbase-archetypes/pom.xml   | 2 +-
 hbase-assembly/pom.xml | 2 +-
 hbase-asyncfs/pom.xml  | 2 +-
 hbase-build-configuration/pom.xml  | 2 +-
 hbase-checkstyle/pom.xml   | 4 ++--
 hbase-client/pom.xml   | 2 +-
 hbase-common/pom.xml   | 2 +-
 hbase-endpoint/pom.xml | 2 +-
 hbase-examples/pom.xml | 2 +-
 hbase-external-blockcache/pom.xml  | 2 +-
 hbase-hadoop-compat/pom.xml| 2 +-
 hbase-hadoop2-compat/pom.xml   | 2 +-
 hbase-hbtop/pom.xml| 2 +-
 hbase-http/pom.xml | 2 +-
 hbase-it/pom.xml   | 2 +-
 hbase-logging/pom.xml  | 2 +-
 hbase-mapreduce/pom.xml| 2 +-
 hbase-metrics-api/pom.xml  | 2 +-
 hbase-metrics/pom.xml  | 2 +-
 hbase-procedure/pom.xml| 2 +-
 hbase-protocol-shaded/pom.xml  | 2 +-
 hbase-protocol/pom.xml | 2 +-
 hbase-replication/pom.xml  | 2 +-
 hbase-resource-bundle/pom.xml  | 2 +-
 hbase-rest/pom.xml | 2 +-
 hbase-rsgroup/pom.xml  | 2 +-
 hbase-server/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-check-invariants/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-client/pom.xml   | 2 +-
 hbase-shaded/hbase-shaded-mapreduce/pom.xml| 2 +-
 hbase-shaded/hbase-shaded-testing-util-tester/pom.xml  | 2 +-
 hbase-shaded/hbase-shaded-testing-util/pom.xml | 2 +-
 hbase-shaded/hbase-shaded-with-hadoop-check-invariants/pom.xml | 2 +-
 hbase-shaded/pom.xml   | 2 +-
 hbase-shell/pom.xml| 2 +-
 hbase-testing-util/pom.xml | 2 +-
 hbase-thrift/pom.xml   | 2 +-
 hbase-zookeeper/pom.xml| 2 +-
 pom.xml| 2 +-
 46 files changed, 51 insertions(+), 45 deletions(-)

diff --git a/CHANGES.md b/CHANGES.md
index 056387a..6e7af1e 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -20,6 +20,9 @@
 # Be careful doing manual edits in this file. Do not change format
 # of release header or remove the below marker. This file is generated.
 # DO NOT REMOVE THIS MARKER; FOR INTERPOLATING CHANGES!-->
+## Release 2.4.9 - Unreleased (as of Fri Dec 17 17:04:11 PST 2021)
+No changes
+
 ## Release 2.4.8 - 2021-11-05
 
 
diff --git a/RELEASENOTES.md b/RELEASENOTES.md
index afedffe..c1b26b2 100644
--- a/RELEASENOTES.md
+++ b/RELEASENOTES.md
@@ -20,6 +20,9 @@
 # Be careful doing manual edits in this file. Do not change format
 # of release header or remove the below marker. This file is generated.
 # DO NOT REMOVE THIS MARKER; FOR INTERPOLATING CHANGES!-->
+# hbase 2.4.9 Release Notes
+No changes
+
 # HBASE  2.4.8 Release Notes
 
 These release notes cover new developer and user-facing incompatibilities, 
important issues, features, and major improvements.
diff --git a/hbase-annotations/pom.xml b/hbase-annotations/pom.xml
index 8b3ef93..dfad925 100644
--- a/hbase-annotations/pom.xml
+++ b/hbase-annotations/pom.xml
@@ -23,7 +23,7 @@
   
 hbase
 org.apache.hbase
-2.4.9-SNAPSHOT
+2.4.9
 ..

[hbase] annotated tag 2.4.9RC0 created (now 5a024fd)

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a change to annotated tag 2.4.9RC0
in repository https://gitbox.apache.org/repos/asf/hbase.git.


  at 5a024fd  (tag)
 tagging 53391b269c6dcc5051b21527cafb8e6d8cd2c4e9 (commit)
 replaces rel/2.4.8
  by Andrew Purtell
  on Fri Dec 17 17:04:42 2021 -0800

- Log -
Via create-release
-BEGIN PGP SIGNATURE-

iQIzBAABCAAdFiEEUPHou3xnqxS9/AohhZd1TdU2XM0FAmG9M6oACgkQhZd1TdU2
XM12+hAAhpRy3LcqfTXoD9AyfGKHy0JRall0PQ4n/9pkYe/bJGnqYEfhd+AJgJc2
f4zI+ObbCsn7jyQW9s585Els1ZY1ki/BxL/RopJg0KEOQxG41dOeqsXUM/E3lVNv
kQNNsnKSK5yW6eqRErtQYIQRfhVXZF28pCJyuVDuxt7MpbHytG4TxsSlo8lWzmCT
511zF8/dUnjihmBG7OvDNEqwe5BUR3fk6tVt3G6+rL8bS8Xye1aNHgzErGH+9b9a
MA0L8GZwa6e1+HU3aDIIHg8g0dKFEWS6m0Txg2SKRYIcvfMrXdiUc1YGIQ/UsFUH
fBkZ1S3A4g/9yyuAZ6wYMGvukqxvCnn8//iexE1Eu8jvaFSj6+z7qbSNCO8efmXp
x6lW+tzl4SG0ZcHk5UZgixI7u9+9Fc+e50sZgD6P5hwufz8n7U6LCDuFjabY5xy0
nFAaGMc13v3vChLeuBAKY8AQmzsFoKwWcl7oMSFv4lWurGt31FHZTGUzBD8JNR5a
ulDhX0sf0HikXkSkNxsA9E6FBAaHYDhZ43XEXiQvD60/5drw8Yx2DbxPXBaaaYj9
LAq8RR/5pFXKSxZphe2ZKO9BNHUOOBMwsJBUTBq5XiuzHyRU0q+YcOpAkwicK6Yd
S5zJPa44FMFe7tAZ4YYM5SRBKN92On7j7nR9itu1j/6Mt+J4fg8=
=oZci
-END PGP SIGNATURE-
---

This annotated tag includes the following new commits:

 new 53391b2  Preparing hbase release 2.4.9RC0; tagging and updates to 
CHANGES.md and RELEASENOTES.md

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[hbase] branch branch-2.4 updated: HBASE-26541 hbase-shaded-protobuf not buildable on M1 MacOSX

2021-12-17 Thread apurtell

This is an automated email from the ASF dual-hosted git repository.

apurtell pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/hbase.git


The following commit(s) were added to refs/heads/branch-2.4 by this push:
 new 4d43e4c  HBASE-26541 hbase-shaded-protobuf not buildable on M1 MacOSX
4d43e4c is described below

commit 4d43e4cc945be5ced9f96cc1ae78001b8192b563
Author: Andrew Purtell 
AuthorDate: Mon Dec 6 11:07:24 2021 -0800

HBASE-26541 hbase-shaded-protobuf not buildable on M1 MacOSX

Signed-off-by: Andrew Purtell 
---
 hbase-protocol-shaded/pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/hbase-protocol-shaded/pom.xml b/hbase-protocol-shaded/pom.xml
index ca77bcd..354e997 100644
--- a/hbase-protocol-shaded/pom.xml
+++ b/hbase-protocol-shaded/pom.xml
@@ -33,8 +33,8 @@
 true
 
-3.11.4
+-->
+3.17.3

[hbase-site] branch asf-site updated: INFRA-10751 Empty commit

2021-12-17 Thread git-site-role

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hbase-site.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new cef3bca  INFRA-10751 Empty commit
cef3bca is described below

commit cef3bcaea6021938446be36714aaa568dc12ab76
Author: jenkins 
AuthorDate: Fri Dec 17 20:18:12 2021 +

INFRA-10751 Empty commit

[hbase-site] branch asf-site updated: Published site at e598f2c663e932f273121243f08f7e765d8c4d58.

2021-12-17 Thread git-site-role

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hbase-site.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 7707438  Published site at e598f2c663e932f273121243f08f7e765d8c4d58.
7707438 is described below

commit 77074385447228cae775aa3cb2b29cb1cc74509b
Author: jenkins 
AuthorDate: Fri Dec 17 20:18:00 2021 +

Published site at e598f2c663e932f273121243f08f7e765d8c4d58.
---
 acid-semantics.html|  2 +-
 apache_hbase_reference_guide.pdf   |  4 ++--
 book.html  |  2 +-
 bulk-loads.html|  2 +-
 checkstyle-aggregate.html  | 14 +++---
 coc.html   |  2 +-
 dependencies.html  |  2 +-
 dependency-convergence.html|  2 +-
 dependency-info.html   |  2 +-
 dependency-management.html |  2 +-
 downloads.html |  2 +-
 export_control.html|  2 +-
 index.html |  2 +-
 issue-management.html  |  2 +-
 licenses.html  |  2 +-
 mailing-lists.html |  2 +-
 metrics.html   |  2 +-
 old_news.html  |  2 +-
 plugin-management.html |  2 +-
 plugins.html   |  2 +-
 poweredbyhbase.html|  2 +-
 project-info.html  |  2 +-
 project-reports.html   |  2 +-
 pseudo-distributed.html|  2 +-
 replication.html   |  2 +-
 resources.html |  2 +-
 scm.html   |  2 +-
 sponsors.html  |  2 +-
 summary.html   |  2 +-
 supportingprojects.html|  2 +-
 team.html  |  2 +-
 .../hadoop/hbase/ipc/TestRpcServerSlowConnectionSetup.html |  2 +-
 32 files changed, 39 insertions(+), 39 deletions(-)

diff --git a/acid-semantics.html b/acid-semantics.html
index fb9b219..60c2afd 100644
--- a/acid-semantics.html
+++ b/acid-semantics.html
@@ -455,7 +455,7 @@
 
 Copyright 20072021
 https://www.apache.org/;>The Apache Software Foundation.
-All rights reserved.Last 
Published: 2021-12-16
+All rights reserved.Last 
Published: 2021-12-17
 
 
 http://maven.apache.org/; title="Built by Maven" class="poweredBy">
diff --git a/apache_hbase_reference_guide.pdf b/apache_hbase_reference_guide.pdf
index 8283829..d109e18 100644
--- a/apache_hbase_reference_guide.pdf
+++ b/apache_hbase_reference_guide.pdf
@@ -5,8 +5,8 @@
 /Author (Apache HBase Team)
 /Creator (Asciidoctor PDF 1.5.3, based on Prawn 2.2.2)
 /Producer (Apache HBase Team)
-/ModDate (D:20211216200607+00'00')
-/CreationDate (D:20211216201635+00'00')
+/ModDate (D:20211217200535+00'00')
+/CreationDate (D:20211217201558+00'00')
 >>
 endobj
 2 0 obj
diff --git a/book.html b/book.html
index 5a64751..215927c 100644
--- a/book.html
+++ b/book.html
@@ -46245,7 +46245,7 @@ 
org/apache/hadoop/hbase/security/access/AccessControlClient.revoke:(Lorg/apache/
 
 
 Version 3.0.0-alpha-2
-Last updated 2021-12-16 20:06:07 UTC
+Last updated 2021-12-17 20:05:35 UTC
 
 
 
diff --git a/bulk-loads.html b/bulk-loads.html
index c66d4a0..f48a1a5 100644
--- a/bulk-loads.html
+++ b/bulk-loads.html
@@ -160,7 +160,7 @@
 
 Copyright 20072021
 https://www.apache.org/;>The Apache Software Foundation.
-All rights reserved.Last 
Published: 2021-12-16
+All rights reserved.Last 
Published: 2021-12-17
 
 
 http://maven.apache.org/; title="Built by Maven" class="poweredBy">
diff --git a/checkstyle-aggregate.html b/checkstyle-aggregate.html
index 4d08b6c..b159c6d 100644
--- a/checkstyle-aggregate.html
+++ b/checkstyle-aggregate.html
@@ -7367,7 +7367,7 @@
 
 annotation
 http://checkstyle.sourceforge.net/config_annotation.html#MissingDeprecated;>MissingDeprecated
-8
+6
 Error
 
 blocks
@@ -7466,7 +7466,7 @@
 http://checkstyle.sourceforge.net/config_javadoc.html#JavadocTagContinuationIndentation;>JavadocTagContinuationIndentation
 
 offset: 2
-514
+516
 Error
 
 
@@

svn commit: r51544 - in /dev/hbase/hbase-thirdparty-4.0.1RC0: ./ CHANGES.md RELEASENOTES.md hbase-thirdparty-4.0.1-src.tar.gz hbase-thirdparty-4.0.1-src.tar.gz.asc hbase-thirdparty-4.0.1-src.tar.gz.sh

2021-12-17 Thread zhangduo

Author: zhangduo
Date: Fri Dec 17 14:18:09 2021
New Revision: 51544

Log:
Apache hbase-thirdparty 4.0.1RC0

Added:
dev/hbase/hbase-thirdparty-4.0.1RC0/
dev/hbase/hbase-thirdparty-4.0.1RC0/CHANGES.md
dev/hbase/hbase-thirdparty-4.0.1RC0/RELEASENOTES.md
dev/hbase/hbase-thirdparty-4.0.1RC0/hbase-thirdparty-4.0.1-src.tar.gz   
(with props)
dev/hbase/hbase-thirdparty-4.0.1RC0/hbase-thirdparty-4.0.1-src.tar.gz.asc
dev/hbase/hbase-thirdparty-4.0.1RC0/hbase-thirdparty-4.0.1-src.tar.gz.sha512

Added: dev/hbase/hbase-thirdparty-4.0.1RC0/CHANGES.md
==
--- dev/hbase/hbase-thirdparty-4.0.1RC0/CHANGES.md (added)
+++ dev/hbase/hbase-thirdparty-4.0.1RC0/CHANGES.md Fri Dec 17 14:18:09 2021
@@ -0,0 +1,266 @@
+# HBASE Changelog
+
+# HBASE Changelog
+
+## Release thirdparty-4.0.1 - Unreleased (as of 2021-12-17)
+
+
+
+### SUB-TASKS:
+
+| JIRA | Summary | Priority | Component |
+|: |: | :--- |: |
+| [HBASE-26593](https://issues.apache.org/jira/browse/HBASE-26593) | 
[hbase-thirdparty] Set version as 4.0.1 in prep for first RC |  Major | build, 
pom, thirdparty |
+| [HBASE-26594](https://issues.apache.org/jira/browse/HBASE-26594) | 
[hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 4.0.1 |  Major | 
thirdparty |
+| [HBASE-26592](https://issues.apache.org/jira/browse/HBASE-26592) | Fix the 
broken shaded protobuf module |  Blocker | hbase-thirdparty, Protobufs |
+
+
+## Release thirdparty-4.0.0 - Unreleased (as of 2021-11-30)
+
+
+
+### BUG FIXES:
+
+| JIRA | Summary | Priority | Component |
+|: |: | :--- |: |
+| [HBASE-26496](https://issues.apache.org/jira/browse/HBASE-26496) | 
[hbase-thirdparty] Exclude the original protobuf-java jar when shading |  Major 
| hbase-thirdparty |
+
+
+### SUB-TASKS:
+
+| JIRA | Summary | Priority | Component |
+|: |: | :--- |: |
+| [HBASE-26515](https://issues.apache.org/jira/browse/HBASE-26515) | 
[hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 4.0.0 |  Major | 
thirdparty |
+| [HBASE-26514](https://issues.apache.org/jira/browse/HBASE-26514) | 
[hbase-thirdparty] Set version as 4.0.0 in prep for first RC |  Major | build, 
pom, thirdparty |
+| [HBASE-25868](https://issues.apache.org/jira/browse/HBASE-25868) | 
[hbase-thirdparty] Shade jackson-jaxrs-json-provider for use with shaded jersey 
|  Major | hbase-thirdparty |
+| [HBASE-25863](https://issues.apache.org/jira/browse/HBASE-25863) | 
[hbase-thirdparty] Shade javax.ws.rs package for use with shaded Jersey |  
Major | thirdparty |
+| [HBASE-26506](https://issues.apache.org/jira/browse/HBASE-26506) | 
[hbase-thirdparty] Bump jersey version to 2.35 |  Major | hbase-thirdparty |
+| [HBASE-26505](https://issues.apache.org/jira/browse/HBASE-26505) | 
[hbase-thirdparty] Bump commons-cli version to 1.5.0 |  Major | 
hbase-thirdparty |
+| [HBASE-26504](https://issues.apache.org/jira/browse/HBASE-26504) | 
[hbase-thirdparty] Bump maven plugin versions |  Major | build, 
hbase-thirdparty, pom |
+| [HBASE-26503](https://issues.apache.org/jira/browse/HBASE-26503) | 
[hbase-thirdparty] Bump guava version to 31.0.1-jre |  Major | hbase-thirdparty 
|
+| [HBASE-26502](https://issues.apache.org/jira/browse/HBASE-26502) | 
[hbase-thirdparty] Bump gson version to 2.8.9 |  Major | hbase-thirdparty |
+| [HBASE-26501](https://issues.apache.org/jira/browse/HBASE-26501) | 
[hbase-thirdparty] Bump jetty version to 9.4.44.v20210927 |  Major | 
hbase-thirdparty |
+| [HBASE-26499](https://issues.apache.org/jira/browse/HBASE-26499) | 
[hbase-thirdparty] Bump netty version to 4.1.70.Final |  Major | 
hbase-thirdparty |
+| [HBASE-26509](https://issues.apache.org/jira/browse/HBASE-26509) | 
[hbase-thirdparty] Bump version to 4.0.0-SNAPSHOT on master branch |  Major | 
build, hbase-thirdparty, pom |
+| [HBASE-26500](https://issues.apache.org/jira/browse/HBASE-26500) | 
[hbase-thirdparty] Bump protobuf version to 3.19.1 |  Major | hbase-thirdparty |
+
+
+## Release thirdparty-3.5.1 - Unreleased (as of 2021-06-02)
+
+
+
+### BUG FIXES:
+
+| JIRA | Summary | Priority | Component |
+|: |: | :--- |: |
+| [HBASE-25959](https://issues.apache.org/jira/browse/HBASE-25959) | Relocate 
libnetty\_transport\_native\_epoll\_aarch\_64.so in hbase-thirdparty |  Minor | 
hbase-thirdparty |
+| [HBASE-25746](https://issues.apache.org/jira/browse/HBASE-25746) | 
[hbase-thirdparty] Update jetty to \>= 9.4.39 due to CVE-2021-28165 |  Major | 
thirdparty |
+| [HBASE-25728](https://issues.apache.org/jira/browse/HBASE-25728) | 
[hbase-thirdparty] Upgrade Netty library to \>= 4.1.60 due to security 
vulnerability CVE-2021-21295 |  Major | thirdparty |
+
+
+### SUB-TASKS:
+
+| JIRA | Summary | Priority | Component |
+|: |: | :--- |: |
+| [HBASE-25946](https://issues.apache.org/jira/browse/HBASE-25946) | 
[hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.5.1 |  Major | 
thirdparty |
+|

[hbase-thirdparty] annotated tag 4.0.1RC0 created (now 050a914)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a change to annotated tag 4.0.1RC0
in repository https://gitbox.apache.org/repos/asf/hbase-thirdparty.git.


  at 050a914  (tag)
 tagging 15fc52e69a8646f0a7fcaa27a2a8c4104269b125 (commit)
 replaces rel/4.0.0
  by Duo Zhang
  on Fri Dec 17 22:14:43 2021 +0800

- Log -
Tag the first release candidate for 4.0.1
---

No new revisions were added by this update.

[hbase-thirdparty] branch master updated (775a3bc -> 23da379)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hbase-thirdparty.git.


from 775a3bc  HBASE-26592 Fix the broken shaded protobuf module (#70)
 add 23da379  HBASE-26593 [hbase-thirdparty] Set version as 4.0.1 in prep 
for first RC (#71)

No new revisions were added by this update.

Summary of changes:
 hbase-noop-htrace/pom.xml| 2 +-
 hbase-shaded-gson/pom.xml| 2 +-
 hbase-shaded-jackson-jaxrs-json-provider/pom.xml | 2 +-
 hbase-shaded-jersey/pom.xml  | 2 +-
 hbase-shaded-jetty/pom.xml   | 2 +-
 hbase-shaded-miscellaneous/pom.xml   | 2 +-
 hbase-shaded-netty/pom.xml   | 2 +-
 hbase-shaded-protobuf/pom.xml| 2 +-
 pom.xml  | 2 +-
 9 files changed, 9 insertions(+), 9 deletions(-)

[hbase-thirdparty] branch master updated: HBASE-26594 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 4.0.1 (#72)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase-thirdparty.git


The following commit(s) were added to refs/heads/master by this push:
 new 15fc52e  HBASE-26594 [hbase-thirdparty] Generate CHANGES.md and 
RELEASENOTES.md for 4.0.1 (#72)
15fc52e is described below

commit 15fc52e69a8646f0a7fcaa27a2a8c4104269b125
Author: Duo Zhang 
AuthorDate: Fri Dec 17 22:12:19 2021 +0800

HBASE-26594 [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 
4.0.1 (#72)

Signed-off-by: Yi Mei 
---
 CHANGES.md  | 13 +
 RELEASENOTES.md | 13 +
 2 files changed, 26 insertions(+)

diff --git a/CHANGES.md b/CHANGES.md
index 346762a..ef3c6ff 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -34,6 +34,19 @@ last argument is the version to search JIRA with.
 DO NOT REMOVE THIS MARKER; FOR INTERPOLATING CHANGES!-->
 # HBASE Changelog
 
+## Release thirdparty-4.0.1 - Unreleased (as of 2021-12-17)
+
+
+
+### SUB-TASKS:
+
+| JIRA | Summary | Priority | Component |
+|: |: | :--- |: |
+| [HBASE-26593](https://issues.apache.org/jira/browse/HBASE-26593) | 
[hbase-thirdparty] Set version as 4.0.1 in prep for first RC |  Major | build, 
pom, thirdparty |
+| [HBASE-26594](https://issues.apache.org/jira/browse/HBASE-26594) | 
[hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 4.0.1 |  Major | 
thirdparty |
+| [HBASE-26592](https://issues.apache.org/jira/browse/HBASE-26592) | Fix the 
broken shaded protobuf module |  Blocker | hbase-thirdparty, Protobufs |
+
+
 ## Release thirdparty-4.0.0 - Unreleased (as of 2021-11-30)
 
 
diff --git a/RELEASENOTES.md b/RELEASENOTES.md
index f2fe892..1351957 100644
--- a/RELEASENOTES.md
+++ b/RELEASENOTES.md
@@ -20,6 +20,19 @@
 # Be careful doing manual edits in this file. Do not change format
 # of release header or remove the below marker. This file is generated.
 # DO NOT REMOVE THIS MARKER; FOR INTERPOLATING CHANGES!-->
+# HBASE  thirdparty-4.0.1 Release Notes
+
+These release notes cover new developer and user-facing incompatibilities, 
important issues, features, and major improvements.
+
+
+---
+
+* [HBASE-26592](https://issues.apache.org/jira/browse/HBASE-26592) | *Blocker* 
| **Fix the broken shaded protobuf module**
+
+Remove protobuf-java dependency from hbase-shaded-protobuf module, as we will 
unpack its source and patch it and then recompile from source. Also add a 
resources section to make sure that we include the proto files in the final jar.
+
+
+
 # HBASE  thirdparty-4.0.0 Release Notes
 
 These release notes cover new developer and user-facing incompatibilities, 
important issues, features, and major improvements.

[hbase] 15/16: HBASE-26286: Add support for specifying store file tracker when restoring or cloning snapshot

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit b64da0670bd9deeb013053e8e46932496abeb312
Author: BukrosSzabolcs 
AuthorDate: Wed Dec 15 20:09:03 2021 -0500

HBASE-26286: Add support for specifying store file tracker when restoring 
or cloning snapshot

Closes #3851

Signed-off-by: Duo Zhang 
Signed-off-by: Josh Elser 
---
 .../java/org/apache/hadoop/hbase/client/Admin.java | 44 --
 .../hadoop/hbase/client/AdminOverAsyncAdmin.java   |  7 ++-
 .../org/apache/hadoop/hbase/client/AsyncAdmin.java | 14 -
 .../hadoop/hbase/client/AsyncHBaseAdmin.java   |  6 +-
 .../hbase/client/ColumnFamilyDescriptor.java   |  5 ++
 .../client/ColumnFamilyDescriptorBuilder.java  |  6 ++
 .../hadoop/hbase/client/RawAsyncHBaseAdmin.java| 24 +---
 .../src/main/protobuf/server/master/Master.proto   |  1 +
 .../protobuf/server/master/MasterProcedure.proto   |  1 +
 .../org/apache/hadoop/hbase/master/HMaster.java| 25 
 .../hadoop/hbase/master/MasterRpcServices.java |  2 +-
 .../master/procedure/CloneSnapshotProcedure.java   | 54 +++-
 .../master/procedure/RestoreSnapshotProcedure.java |  7 ++-
 .../hbase/master/snapshot/SnapshotManager.java | 27 +---
 .../storefiletracker/StoreFileTrackerFactory.java  | 39 +++-
 .../hbase/snapshot/RestoreSnapshotHelper.java  | 11 ++--
 .../TestCloneSnapshotFromClientCustomSFT.java  | 71 ++
 .../storefiletracker/TestStoreFileTracker.java |  2 -
 .../TestStoreFileTrackerFactory.java   | 52 
 .../hbase/rsgroup/VerifyingRSGroupAdmin.java   |  5 +-
 hbase-shell/src/main/ruby/hbase/admin.rb   |  4 +-
 hbase-shell/src/main/ruby/hbase_constants.rb   |  1 +
 .../src/main/ruby/shell/commands/clone_snapshot.rb |  6 +-
 .../hadoop/hbase/thrift2/client/ThriftAdmin.java   |  4 +-
 24 files changed, 357 insertions(+), 61 deletions(-)

diff --git 
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java 
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
index 48893cc..6c36660 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
@@ -69,6 +69,7 @@ import org.apache.hadoop.hbase.util.Pair;
 import org.apache.yetus.audience.InterfaceAudience;
 
 import org.apache.hbase.thirdparty.com.google.common.collect.ImmutableList;
+import org.apache.yetus.audience.InterfaceStability;
 
 /**
  * The administrative API for HBase. Obtain an instance from {@link 
Connection#getAdmin()} and
@@ -1620,7 +1621,7 @@ public interface Admin extends Abortable, Closeable {
* @throws IllegalArgumentException if the restore request is formatted 
incorrectly
*/
   void restoreSnapshot(String snapshotName, boolean takeFailSafeSnapshot, 
boolean restoreAcl)
-  throws IOException, RestoreSnapshotException;
+throws IOException, RestoreSnapshotException;
 
   /**
* Create a new table by cloning the snapshot content.
@@ -1633,7 +1634,25 @@ public interface Admin extends Abortable, Closeable {
*/
   default void cloneSnapshot(String snapshotName, TableName tableName)
   throws IOException, TableExistsException, RestoreSnapshotException {
-cloneSnapshot(snapshotName, tableName, false);
+cloneSnapshot(snapshotName, tableName, false, null);
+  }
+
+  /**
+   * Create a new table by cloning the snapshot content.
+   * @param snapshotName name of the snapshot to be cloned
+   * @param tableName name of the table where the snapshot will be restored
+   * @param restoreAcl true to clone acl into newly created table
+   * @param customSFT specify the StoreFileTracker used for the table
+   * @throws IOException if a remote or network exception occurs
+   * @throws TableExistsException if table to be created already exists
+   * @throws RestoreSnapshotException if snapshot failed to be cloned
+   * @throws IllegalArgumentException if the specified table has not a valid 
name
+   */
+  default void cloneSnapshot(String snapshotName, TableName tableName, boolean 
restoreAcl,
+String customSFT)
+throws IOException, TableExistsException, RestoreSnapshotException {
+get(cloneSnapshotAsync(snapshotName, tableName, restoreAcl, customSFT), 
getSyncWaitTimeout(),
+  TimeUnit.MILLISECONDS);
   }
 
   /**
@@ -1680,8 +1699,25 @@ public interface Admin extends Abortable, Closeable {
* @throws RestoreSnapshotException if snapshot failed to be cloned
* @throws IllegalArgumentException if the specified table has not a valid 
name
*/
-  Future cloneSnapshotAsync(String snapshotName, TableName tableName, 
boolean restoreAcl)
-  throws IOException, TableExistsException, RestoreSnapshotException;
+  default Future cloneSnapshotAsync(String snapshotName, TableName

[hbase] 12/16: HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configurations to TableDescriptor for existing tables (#3700)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 3dd4dc852a2085c8fdf98084f4396847a933cb66
Author: GeorryHuang 
AuthorDate: Sat Nov 6 22:20:12 2021 +0800

HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configurations 
to TableDescriptor for existing tables (#3700)

Signed-off-by: Duo Zhang 
Reviewed-by: Wellington Ramos Chevreuil 
---
 .../org/apache/hadoop/hbase/master/HMaster.java|   6 +
 .../hbase/master/migrate/RollingUpgradeChore.java  | 130 +
 .../MigrateStoreFileTrackerProcedure.java  |  48 
 .../migrate/TestMigrateStoreFileTracker.java   | 107 +
 4 files changed, 291 insertions(+)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index 1683b16..b80a2e2 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -131,6 +131,7 @@ import 
org.apache.hadoop.hbase.master.http.MasterRedirectServlet;
 import org.apache.hadoop.hbase.master.http.MasterStatusServlet;
 import org.apache.hadoop.hbase.master.janitor.CatalogJanitor;
 import org.apache.hadoop.hbase.master.locking.LockManager;
+import org.apache.hadoop.hbase.master.migrate.RollingUpgradeChore;
 import org.apache.hadoop.hbase.master.normalizer.RegionNormalizerFactory;
 import org.apache.hadoop.hbase.master.normalizer.RegionNormalizerManager;
 import org.apache.hadoop.hbase.master.procedure.CreateTableProcedure;
@@ -376,6 +377,7 @@ public class HMaster extends 
HBaseServerBase implements Maste
   private ReplicationBarrierCleaner replicationBarrierCleaner;
   private MobFileCleanerChore mobFileCleanerChore;
   private MobFileCompactionChore mobFileCompactionChore;
+  private RollingUpgradeChore rollingUpgradeChore;
   // used to synchronize the mobCompactionStates
   private final IdLock mobCompactionLock = new IdLock();
   // save the information of mob compactions in tables.
@@ -1222,6 +1224,9 @@ public class HMaster extends 
HBaseServerBase implements Maste
   LOG.debug("Balancer post startup initialization complete, took " + (
   (EnvironmentEdgeManager.currentTime() - start) / 1000) + " seconds");
 }
+
+this.rollingUpgradeChore = new RollingUpgradeChore(this);
+getChoreService().scheduleChore(rollingUpgradeChore);
   }
 
   private void createMissingCFsInMetaDuringUpgrade(
@@ -1713,6 +1718,7 @@ public class HMaster extends 
HBaseServerBase implements Maste
 shutdownChore(snapshotCleanerChore);
 shutdownChore(hbckChore);
 shutdownChore(regionsRecoveryChore);
+shutdownChore(rollingUpgradeChore);
   }
 
   /**
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/migrate/RollingUpgradeChore.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/migrate/RollingUpgradeChore.java
new file mode 100644
index 000..3896b41
--- /dev/null
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/migrate/RollingUpgradeChore.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hbase.master.migrate;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ScheduledChore;
+import org.apache.hadoop.hbase.Stoppable;
+import org.apache.hadoop.hbase.TableDescriptors;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.master.MasterServices;
+import org.apache.hadoop.hbase.master.procedure.MasterProcedureEnv;
+import org.apache.hadoop.hbase.procedure2.ProcedureExecutor;
+import 
org.apache.hadoop.hbase.regionserver.storefiletracker.MigrateStoreFileTrackerProcedure;
+import

[hbase] 11/16: HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl (#3749)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 539845fab09997dd6ab399fb719d49216480a0c3
Author: Wellington Ramos Chevreuil 
AuthorDate: Fri Oct 22 16:56:15 2021 +0100

HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl 
(#3749)

Signed-off-by: Duo Zhang 
---
 .../java/org/apache/hadoop/hbase/io/HFileLink.java | 63 ++
 .../master/procedure/CloneSnapshotProcedure.java   | 52 +++-
 .../storefiletracker/StoreFileTrackerBase.java |  1 +
 .../hbase/snapshot/RestoreSnapshotHelper.java  | 95 ++
 .../TestCloneSnapshotProcedureFileBasedSFT.java| 42 ++
 5 files changed, 157 insertions(+), 96 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java
index 74836ce..fbed724 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java
@@ -201,7 +201,6 @@ public class HFileLink extends FileLink {
 return isHFileLink(path.getName());
   }
 
-
   /**
* @param fileName File name to check.
* @return True if the path is a HFileLink.
@@ -322,10 +321,10 @@ public class HFileLink extends FileLink {
* @param dstFamilyPath - Destination path (table/region/cf/)
* @param hfileRegionInfo - Linked HFile Region Info
* @param hfileName - Linked HFile name
-   * @return true if the file is created, otherwise the file exists.
-   * @throws IOException on file or parent directory creation failure
+   * @return the file link name.
+   * @throws IOException on file or parent directory creation failure.
*/
-  public static boolean create(final Configuration conf, final FileSystem fs,
+  public static String create(final Configuration conf, final FileSystem fs,
   final Path dstFamilyPath, final RegionInfo hfileRegionInfo,
   final String hfileName) throws IOException {
 return create(conf, fs, dstFamilyPath, hfileRegionInfo, hfileName, true);
@@ -343,10 +342,10 @@ public class HFileLink extends FileLink {
* @param hfileRegionInfo - Linked HFile Region Info
* @param hfileName - Linked HFile name
* @param createBackRef - Whether back reference should be created. Defaults 
to true.
-   * @return true if the file is created, otherwise the file exists.
-   * @throws IOException on file or parent directory creation failure
+   * @return the file link name.
+   * @throws IOException on file or parent directory creation failure.
*/
-  public static boolean create(final Configuration conf, final FileSystem fs,
+  public static String create(final Configuration conf, final FileSystem fs,
   final Path dstFamilyPath, final RegionInfo hfileRegionInfo,
   final String hfileName, final boolean createBackRef) throws IOException {
 TableName linkedTable = hfileRegionInfo.getTable();
@@ -366,17 +365,18 @@ public class HFileLink extends FileLink {
* @param linkedTable - Linked Table Name
* @param linkedRegion - Linked Region Name
* @param hfileName - Linked HFile name
-   * @return true if the file is created, otherwise the file exists.
-   * @throws IOException on file or parent directory creation failure
+   * @return the file link name.
+   * @throws IOException on file or parent directory creation failure.
*/
-  public static boolean create(final Configuration conf, final FileSystem fs,
+  public static String create(final Configuration conf, final FileSystem fs,
   final Path dstFamilyPath, final TableName linkedTable, final String 
linkedRegion,
   final String hfileName) throws IOException {
 return create(conf, fs, dstFamilyPath, linkedTable, linkedRegion, 
hfileName, true);
   }
 
   /**
-   * Create a new HFileLink
+   * Create a new HFileLink. In the event of link creation failure, this 
method throws an
+   * IOException, so that the calling upper laying can decide on how to 
proceed with this.
*
* It also adds a back-reference to the hfile back-reference directory
* to simplify the reference-count and the cleaning process.
@@ -388,10 +388,10 @@ public class HFileLink extends FileLink {
* @param linkedRegion - Linked Region Name
* @param hfileName - Linked HFile name
* @param createBackRef - Whether back reference should be created. Defaults 
to true.
-   * @return true if the file is created, otherwise the file exists.
-   * @throws IOException on file or parent directory creation failure
+   * @return the file link name.
+   * @throws IOException on file or parent directory creation failure.
*/
-  public static boolean create(final Configuration conf, final FileSystem fs,
+  public static String create(final Configuration conf, final FileSystem fs,
   final Path dstFamilyPath, final

[hbase] 09/16: HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… (#3721)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 979b474a90fbef1b5fcb7c7ef122e10f0d06ae5b
Author: Wellington Ramos Chevreuil 
AuthorDate: Wed Oct 13 15:48:13 2021 +0100

HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… 
(#3721)

Signed-off-by: Duo Zhang 
Signed-off-by: Josh Elser 
---
 .../hbase/master/procedure/CreateTableProcedure.java |  6 ++
 .../storefiletracker/FileBasedStoreFileTracker.java  |  9 -
 .../storefiletracker/MigrationStoreFileTracker.java  | 12 +---
 .../storefiletracker/StoreFileTracker.java   |  5 +++--
 .../storefiletracker/StoreFileTrackerBase.java   |  9 -
 .../storefiletracker/StoreFileTrackerFactory.java| 20 ++--
 .../master/procedure/TestCreateTableProcedure.java   | 16 
 .../storefiletracker/TestStoreFileTracker.java   |  2 +-
 8 files changed, 49 insertions(+), 30 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
index ee8e51f..0a6a469 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
@@ -34,7 +34,6 @@ import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.RegionInfo;
 import org.apache.hadoop.hbase.client.RegionReplicaUtil;
 import org.apache.hadoop.hbase.client.TableDescriptor;
-import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
 import org.apache.hadoop.hbase.client.TableState;
 import org.apache.hadoop.hbase.master.MasterCoprocessorHost;
 import org.apache.hadoop.hbase.master.MasterFileSystem;
@@ -290,9 +289,8 @@ public class CreateTableProcedure
 (newRegions != null ? newRegions.size() : 0));
 }
 
-TableDescriptorBuilder builder = 
TableDescriptorBuilder.newBuilder(tableDescriptor);
-StoreFileTrackerFactory.persistTrackerConfig(env.getMasterConfiguration(), 
builder);
-tableDescriptor = builder.build();
+tableDescriptor = 
StoreFileTrackerFactory.updateWithTrackerConfigs(env.getMasterConfiguration(),
+  tableDescriptor);
 
 final MasterCoprocessorHost cpHost = env.getMasterCoprocessorHost();
 if (cpHost != null) {
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
index c370b87..4da7911 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
@@ -56,7 +56,14 @@ class FileBasedStoreFileTracker extends StoreFileTrackerBase 
{
 
   public FileBasedStoreFileTracker(Configuration conf, boolean 
isPrimaryReplica, StoreContext ctx) {
 super(conf, isPrimaryReplica, ctx);
-backedFile = new StoreFileListFile(ctx);
+//CreateTableProcedure needs to instantiate the configured SFT impl, in 
order to update table
+//descriptors with the SFT impl specific configs. By the time this 
happens, the table has no
+//regions nor stores yet, so it can't create a proper StoreContext.
+if (ctx != null) {
+  backedFile = new StoreFileListFile(ctx);
+} else {
+  backedFile = null;
+}
   }
 
   @Override
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
index 1946d4b..230c1ec 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
@@ -21,6 +21,7 @@ import java.io.IOException;
 import java.util.Collection;
 import java.util.List;
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.TableDescriptor;
 import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
 import org.apache.hadoop.hbase.procedure2.util.StringUtils;
 import org.apache.hadoop.hbase.regionserver.StoreContext;
@@ -88,17 +89,6 @@ class MigrationStoreFileTracker extends StoreFileTrackerBase 
{
   "Should not call this method on " + getClass().getSimpleName());
   }
 
-  @Override
-  public void persistConfiguration(TableDescriptorBuilder builder) {
-super.persistConfiguration(builder);
-if (StringUtils.isEmpty(builder.getValue(SRC_IMPL))) {
-

[hbase] 07/16: HBASE-26264 Add more checks to prevent misconfiguration on store file tracker (#3681)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 16d055140d77e63c39dfe79b94fe708208bce392
Author: Duo Zhang 
AuthorDate: Wed Sep 15 23:00:03 2021 +0800

HBASE-26264 Add more checks to prevent misconfiguration on store file 
tracker (#3681)

Signed-off-by: Josh Elser 
---
 .../assignment/MergeTableRegionsProcedure.java |   3 +-
 .../assignment/SplitTableRegionProcedure.java  |   3 +-
 .../master/procedure/CreateTableProcedure.java |   8 +-
 .../master/procedure/ModifyTableProcedure.java |   5 +
 .../hbase/regionserver/HRegionFileSystem.java  |   2 +-
 .../MigrationStoreFileTracker.java |   8 +
 .../storefiletracker/StoreFileTrackerFactory.java  | 173 +--
 .../TestChangeStoreFileTracker.java| 242 +
 8 files changed, 422 insertions(+), 22 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
index e6bbe44..e9051da 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
@@ -614,8 +614,7 @@ public class MergeTableRegionsProcedure
   String family = hcd.getNameAsString();
   Configuration trackerConfig =
 
StoreFileTrackerFactory.mergeConfigurations(env.getMasterConfiguration(), htd, 
hcd);
-  StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig, 
true,
-family, regionFs);
+  StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig, 
family, regionFs);
   final Collection storeFiles = tracker.load();
   if (storeFiles != null && storeFiles.size() > 0) {
 final Configuration storeConfiguration =
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
index ff16dc5..aa0c938 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
@@ -670,8 +670,7 @@ public class SplitTableRegionProcedure
   String family = cfd.getNameAsString();
   Configuration trackerConfig = StoreFileTrackerFactory.
 mergeConfigurations(env.getMasterConfiguration(), htd, 
htd.getColumnFamily(cfd.getName()));
-  StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig, 
true,
-family, regionFs);
+  StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig, 
family, regionFs);
   Collection sfis = tracker.load();
   if (sfis == null) {
 continue;
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
index dccea55..ee8e51f 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
@@ -277,15 +277,17 @@ public class CreateTableProcedure
   MasterProcedureUtil.checkGroupNotEmpty(rsGroupInfo, forWhom);
 }
 
+// check for store file tracker configurations
+StoreFileTrackerFactory.checkForCreateTable(env.getMasterConfiguration(), 
tableDescriptor);
+
 return true;
   }
 
   private void preCreate(final MasterProcedureEnv env)
   throws IOException, InterruptedException {
 if (!getTableName().isSystemTable()) {
-  ProcedureSyncWait.getMasterQuotaManager(env)
-.checkNamespaceTableAndRegionQuota(
-  getTableName(), (newRegions != null ? newRegions.size() : 0));
+  
ProcedureSyncWait.getMasterQuotaManager(env).checkNamespaceTableAndRegionQuota(getTableName(),
+(newRegions != null ? newRegions.size() : 0));
 }
 
 TableDescriptorBuilder builder = 
TableDescriptorBuilder.newBuilder(tableDescriptor);
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
index 247dd9c..1640644 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java
@@ -38,6 +38,7 @@ import org.apache.hadoop.hbase.client.TableDescriptor;
 import

[hbase] 08/16: HBASE-26280 Use store file tracker when snapshoting (#3685)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 60ceccb34fbf0c8dbc1b0eaa04e2e82f50b174be
Author: Duo Zhang 
AuthorDate: Fri Sep 17 09:40:44 2021 +0800

HBASE-26280 Use store file tracker when snapshoting (#3685)

Signed-off-by: Wellington Chevreuil 
Reviewed-by: Josh Elser 
---
 .../assignment/MergeTableRegionsProcedure.java |  5 +-
 .../assignment/SplitTableRegionProcedure.java  |  5 +-
 .../hbase/regionserver/HRegionFileSystem.java  | 10 +--
 .../storefiletracker/StoreFileTrackerFactory.java  | 17 ++---
 .../hadoop/hbase/snapshot/SnapshotManifest.java| 42 ++--
 ...oneSnapshotFromClientCloneLinksAfterDelete.java |  4 +-
 .../hbase/client/TestMobSnapshotFromClient.java|  7 +-
 .../hbase/client/TestSnapshotFromClient.java   | 30 +++--
 .../hadoop/hbase/regionserver/TestHStoreFile.java  |  6 +-
 .../hbase/snapshot/MobSnapshotTestingUtils.java| 74 +++---
 .../hbase/snapshot/SnapshotTestingUtils.java   | 16 ++---
 11 files changed, 107 insertions(+), 109 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
index e9051da..0f41db5 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
@@ -612,9 +612,8 @@ public class MergeTableRegionsProcedure
 List mergedFiles = new ArrayList<>();
 for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
   String family = hcd.getNameAsString();
-  Configuration trackerConfig =
-
StoreFileTrackerFactory.mergeConfigurations(env.getMasterConfiguration(), htd, 
hcd);
-  StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig, 
family, regionFs);
+  StoreFileTracker tracker =
+StoreFileTrackerFactory.create(env.getMasterConfiguration(), htd, hcd, 
regionFs);
   final Collection storeFiles = tracker.load();
   if (storeFiles != null && storeFiles.size() > 0) {
 final Configuration storeConfiguration =
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
index aa0c938..effdba4 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/SplitTableRegionProcedure.java
@@ -668,9 +668,8 @@ public class SplitTableRegionProcedure
 new HashMap>(htd.getColumnFamilyCount());
 for (ColumnFamilyDescriptor cfd : htd.getColumnFamilies()) {
   String family = cfd.getNameAsString();
-  Configuration trackerConfig = StoreFileTrackerFactory.
-mergeConfigurations(env.getMasterConfiguration(), htd, 
htd.getColumnFamily(cfd.getName()));
-  StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig, 
family, regionFs);
+  StoreFileTracker tracker =
+StoreFileTrackerFactory.create(env.getMasterConfiguration(), htd, cfd, 
regionFs);
   Collection sfis = tracker.load();
   if (sfis == null) {
 continue;
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
index e78d8ad..8110025 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
@@ -598,7 +598,6 @@ public class HRegionFileSystem {
* to the proper location in the filesystem.
*
* @param regionInfo daughter {@link 
org.apache.hadoop.hbase.client.RegionInfo}
-   * @throws IOException
*/
   public Path commitDaughterRegion(final RegionInfo regionInfo, List 
allRegionFiles,
   MasterProcedureEnv env) throws IOException {
@@ -625,12 +624,8 @@ public class HRegionFileSystem {
 Map> fileInfoMap = new HashMap<>();
 for(Path file : allFiles) {
   String familyName = file.getParent().getName();
-  trackerMap.computeIfAbsent(familyName, t -> {
-Configuration config = 
StoreFileTrackerFactory.mergeConfigurations(conf, tblDesc,
-  tblDesc.getColumnFamily(Bytes.toBytes(familyName)));
-return StoreFileTrackerFactory.
-  create(config, familyName, regionFs);
-  });
+  trackerMap.computeIfAbsent(familyName, t -> 
StoreFileTrackerFactory.create(conf, tblDesc,
+tblDesc.getColumnFamily(Bytes.toBytes(familyName)), regionFs));

[hbase] 16/16: HBASE-26265 Update ref guide to mention the new store file tracker im… (#3942)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit a6cc0af9a9dd2c822eb6ba15c88f998944f86e62
Author: Wellington Ramos Chevreuil 
AuthorDate: Thu Dec 16 21:07:38 2021 +

HBASE-26265 Update ref guide to mention the new store file tracker im… 
(#3942)
---
 .../asciidoc/_chapters/store_file_tracking.adoc| 145 +
 src/main/asciidoc/book.adoc|   1 +
 2 files changed, 146 insertions(+)

diff --git a/src/main/asciidoc/_chapters/store_file_tracking.adoc 
b/src/main/asciidoc/_chapters/store_file_tracking.adoc
new file mode 100644
index 000..74d802f
--- /dev/null
+++ b/src/main/asciidoc/_chapters/store_file_tracking.adoc
@@ -0,0 +1,145 @@
+
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+[[storefiletracking]]
+= Store File Tracking
+:doctype: book
+:numbered:
+:toc: left
+:icons: font
+:experimental:
+
+== Overview
+
+This feature introduces an abstraction layer to track store files still 
used/needed by store
+engines, allowing for plugging different approaches of identifying store
+files required by the given store.
+
+Historically, HBase internals have relied on creating hfiles on temporary 
directories first, renaming
+those files to the actual store directory at operation commit time. That's a 
simple and convenient
+way to separate transient from already finalised files that are ready to serve 
client reads with data.
+This approach works well with strong consistent file systems, but with the 
popularity of less consistent
+file systems, mainly Object Store which can be used like file systems, 
dependency on atomic rename operations starts to introduce
+performance penalties. The Amazon S3 Object Store, in particular, has been the 
most affected deployment,
+due to its lack of atomic renames. The HBase community temporarily bypassed 
this problem by building a distributed locking layer called HBOSS,
+to guarantee atomicity of operations against S3.
+
+With *Store File Tracking*, decision on where to originally create new hfiles 
and how to proceed upon
+commit is delegated to the specific Store File Tracking implementation.
+The implementation can be set at the HBase service leve in *hbase-site.xml* or 
at the
+Table or Column Family via the TableDescriptor configuration.
+
+NOTE: When the store file tracking implementation is specified in 
*hbase_site.xml*, this configuration is also propagated into a tables 
configuration
+at table creation time. This is to avoid dangerous configuration mismatches 
between processes, which
+could potentially lead to data loss.
+
+== Available Implementations
+
+Store File Tracking initial version provides three builtin implementations:
+
+* DEFAULT
+* FILE
+* MIGRATION
+
+### DEFAULT
+
+As per the name, this is the Store File Tracking implementation used by 
default when no explicit
+configuration has been defined. The DEFAULT tracker implements the standard 
approach using temporary
+directories and renames. This is how all previous (implicit) implementation 
that HBase used to track store files.
+
+### FILE
+
+A file tracker implementation that creates new files straight in the store 
directory, avoiding the
+need for rename operations. It keeps a list of committed hfiles in memory, 
backed by meta files, in
+each store directory. Whenever a new hfile is committed, the list of _tracked 
files_ in the given
+store is updated and a new meta file is written with this list contents, 
discarding the previous
+meta file now containing an out dated list.
+
+### MIGRATION
+
+A special implementation to be used when swapping between Store File Tracking 
implementations on
+pre-existing tables that already contain data, and therefore, files being 
tracked under an specific
+logic.
+
+== Usage
+
+For fresh deployments that don't yet contain any user data, *FILE* 
implementation can be just set as
+value for *hbase.store.file-tracker.impl* property in global *hbase-site.xml* 
configuration, prior
+to the first hbase start. Omitting this property sets the *DEFAULT* 
implementation.
+
+For clusters with data that are

[hbase] 13/16: HBASE-26271 Cleanup the broken store files under data directory (#3786)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 766d8a3f67b3529b5811d2571dc873bb0642ac26
Author: BukrosSzabolcs 
AuthorDate: Tue Nov 9 17:19:00 2021 +0100

HBASE-26271 Cleanup the broken store files under data directory (#3786)

Signed-off-by: Duo Zhang 
Signed-off-by: Josh Elser 
Signed-off-by: Wellington Ramos Chevreuil 
---
 .../hadoop/hbase/mob/DefaultMobStoreCompactor.java |   6 +-
 .../regionserver/AbstractMultiFileWriter.java  |   6 +-
 .../hbase/regionserver/BrokenStoreFileCleaner.java | 202 ++
 .../regionserver/DateTieredMultiFileWriter.java|   2 +-
 .../hbase/regionserver/HRegionFileSystem.java  |   2 +-
 .../hadoop/hbase/regionserver/HRegionServer.java   |  27 +++
 .../apache/hadoop/hbase/regionserver/HStore.java   |   6 +
 .../hadoop/hbase/regionserver/StoreEngine.java |  21 ++
 .../hbase/regionserver/StripeMultiFileWriter.java  |   2 +-
 .../compactions/AbstractMultiOutputCompactor.java  |   4 +-
 .../hbase/regionserver/compactions/Compactor.java  |  45 -
 .../compactions/DateTieredCompactor.java   |   6 +-
 .../regionserver/compactions/DefaultCompactor.java |   9 +-
 .../regionserver/compactions/StripeCompactor.java  |   2 +-
 .../FileBasedStoreFileTracker.java |   2 +-
 .../MigrationStoreFileTracker.java |   2 +-
 .../storefiletracker/StoreFileTracker.java |   6 +
 .../storefiletracker/StoreFileTrackerBase.java |   6 -
 .../hbase/snapshot/RestoreSnapshotHelper.java  |   2 +-
 .../hadoop/hbase/mob/FaultyMobStoreCompactor.java  |   3 +-
 .../regionserver/TestBrokenStoreFileCleaner.java   | 225 +
 .../hbase/regionserver/TestCompactorMemLeak.java   |   4 +-
 .../storefiletracker/TestStoreFileTracker.java |   1 -
 23 files changed, 556 insertions(+), 35 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
index 01fe000..15f0a73 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
@@ -38,7 +38,6 @@ import org.apache.hadoop.hbase.DoNotRetryIOException;
 import org.apache.hadoop.hbase.KeyValue;
 import org.apache.hadoop.hbase.PrivateCellUtil;
 import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.regionserver.CellSink;
 import org.apache.hadoop.hbase.regionserver.HMobStore;
 import org.apache.hadoop.hbase.regionserver.HStore;
 import org.apache.hadoop.hbase.regionserver.HStoreFile;
@@ -286,7 +285,6 @@ public class DefaultMobStoreCompactor extends 
DefaultCompactor {
* 
* @param fd File details
* @param scanner Where to read from.
-   * @param writer Where to write to.
* @param smallestReadPoint Smallest read point.
* @param cleanSeqId When true, remove seqId(used to be mvcc) value which is 
<= smallestReadPoint
* @param throughputController The compaction throughput controller.
@@ -295,7 +293,7 @@ public class DefaultMobStoreCompactor extends 
DefaultCompactor {
* @return Whether compaction ended; false if it was interrupted for any 
reason.
*/
   @Override
-  protected boolean performCompaction(FileDetails fd, InternalScanner scanner, 
CellSink writer,
+  protected boolean performCompaction(FileDetails fd, InternalScanner scanner,
   long smallestReadPoint, boolean cleanSeqId, ThroughputController 
throughputController,
   boolean major, int numofFilesToCompact) throws IOException {
 long bytesWrittenProgressForLog = 0;
@@ -665,7 +663,7 @@ public class DefaultMobStoreCompactor extends 
DefaultCompactor {
 
 
   @Override
-  protected List commitWriter(StoreFileWriter writer, FileDetails fd,
+  protected List commitWriter(FileDetails fd,
   CompactionRequestImpl request) throws IOException {
 List newFiles = Lists.newArrayList(writer.getPath());
 writer.appendMetadata(fd.maxSeqId, request.isAllFiles(), 
request.getFiles());
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMultiFileWriter.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMultiFileWriter.java
index f250304..82c3867 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMultiFileWriter.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMultiFileWriter.java
@@ -110,7 +110,11 @@ public abstract class AbstractMultiFileWriter implements 
CellSink, ShipperListen
 return paths;
   }
 
-  protected abstract Collection writers();
+  /**
+   * Returns all writers. This is used to prevent deleting currently writen 
storefiles
+   * during cleanup.
+   */
+  public abstract Collection

[hbase] 14/16: HBASE-26454 CreateTableProcedure still relies on temp dir and renames… (#3845)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit d00bb63d38689f6a8ea504d75a5963a3d32056bd
Author: Wellington Ramos Chevreuil 
AuthorDate: Fri Nov 19 12:16:29 2021 +

HBASE-26454 CreateTableProcedure still relies on temp dir and renames… 
(#3845)

Signed-off-by: Duo Zhang 
---
 .../master/procedure/CreateTableProcedure.java | 30 ++-
 .../master/procedure/DeleteTableProcedure.java | 95 +++---
 .../access/SnapshotScannerHDFSAclHelper.java   |  4 +-
 .../hadoop/hbase/master/TestMasterFileSystem.java  | 29 ++-
 .../master/procedure/TestDeleteTableProcedure.java | 66 ---
 5 files changed, 43 insertions(+), 181 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
index 0a6a469..d77b95f 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
@@ -23,7 +23,6 @@ import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.function.Supplier;
-import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.hbase.DoNotRetryIOException;
 import org.apache.hadoop.hbase.HBaseIOException;
@@ -336,41 +335,22 @@ public class CreateTableProcedure
   final TableDescriptor tableDescriptor, List newRegions,
   final CreateHdfsRegions hdfsRegionHandler) throws IOException {
 final MasterFileSystem mfs = env.getMasterServices().getMasterFileSystem();
-final Path tempdir = mfs.getTempDir();
 
 // 1. Create Table Descriptor
 // using a copy of descriptor, table will be created enabling first
-final Path tempTableDir = CommonFSUtils.getTableDir(tempdir, 
tableDescriptor.getTableName());
+final Path tableDir = CommonFSUtils.getTableDir(mfs.getRootDir(),
+  tableDescriptor.getTableName());
 ((FSTableDescriptors)(env.getMasterServices().getTableDescriptors()))
-.createTableDescriptorForTableDirectory(tempTableDir, tableDescriptor, 
false);
+.createTableDescriptorForTableDirectory(
+  tableDir, tableDescriptor, false);
 
 // 2. Create Regions
-newRegions = hdfsRegionHandler.createHdfsRegions(env, tempdir,
+newRegions = hdfsRegionHandler.createHdfsRegions(env, mfs.getRootDir(),
 tableDescriptor.getTableName(), newRegions);
 
-// 3. Move Table temp directory to the hbase root location
-moveTempDirectoryToHBaseRoot(env, tableDescriptor, tempTableDir);
-
 return newRegions;
   }
 
-  protected static void moveTempDirectoryToHBaseRoot(
-final MasterProcedureEnv env,
-final TableDescriptor tableDescriptor,
-final Path tempTableDir) throws IOException {
-final MasterFileSystem mfs = env.getMasterServices().getMasterFileSystem();
-final Path tableDir =
-  CommonFSUtils.getTableDir(mfs.getRootDir(), 
tableDescriptor.getTableName());
-FileSystem fs = mfs.getFileSystem();
-if (!fs.delete(tableDir, true) && fs.exists(tableDir)) {
-  throw new IOException("Couldn't delete " + tableDir);
-}
-if (!fs.rename(tempTableDir, tableDir)) {
-  throw new IOException("Unable to move table from temp=" + tempTableDir +
-" to hbase root=" + tableDir);
-}
-  }
-
   protected static List addTableToMeta(final MasterProcedureEnv 
env,
 final TableDescriptor tableDescriptor, final List regions) 
throws IOException {
 assert (regions != null && regions.size() > 0) : "expected at least 1 
region, got " + regions;
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
index 80dddc7..297efc2 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
@@ -20,10 +20,8 @@ package org.apache.hadoop.hbase.master.procedure;
 
 import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.List;
 import java.util.stream.Collectors;
-import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.hbase.MetaTableAccessor;
@@ -277,82 +275,49 @@ public class DeleteTableProcedure
   final boolean archive) throws IOException {
 final MasterFileSystem mfs = env.getMasterServices().getMasterFileSystem();
 final FileSystem fs = mfs.getFileSystem();
-final Path tempdir = mfs.getTempDir();
 
 final Path tableDir =

[hbase] 04/16: HBASE-26224 Introduce a MigrationStoreFileTracker to support migrating from different store file tracker implementations (#3656)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit f691a55dbe92721870786f0ee51ef82233d1fcb0
Author: Duo Zhang 
AuthorDate: Thu Sep 9 12:11:25 2021 +0800

HBASE-26224 Introduce a MigrationStoreFileTracker to support migrating from 
different store file tracker implementations (#3656)

Signed-off-by: Wellington Chevreuil 
---
 .../storefiletracker/DefaultStoreFileTracker.java  |  10 +-
 .../FileBasedStoreFileTracker.java |  15 +-
 .../MigrationStoreFileTracker.java |  88 ++
 .../storefiletracker/StoreFileListFile.java|   6 +-
 .../storefiletracker/StoreFileTrackerBase.java |  12 +-
 .../storefiletracker/StoreFileTrackerFactory.java  |  40 +++--
 .../TestMigrationStoreFileTracker.java | 193 +
 7 files changed, 343 insertions(+), 21 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
index 22e0513..a13b75b 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hbase.regionserver.storefiletracker;
 
 import java.io.IOException;
 import java.util.Collection;
+import java.util.Collections;
 import java.util.List;
 import org.apache.hadoop.conf.Configuration;
 
@@ -39,7 +40,9 @@ class DefaultStoreFileTracker extends StoreFileTrackerBase {
 
   @Override
   public List load() throws IOException {
-return 
ctx.getRegionFileSystem().getStoreFiles(ctx.getFamily().getNameAsString());
+List files =
+  
ctx.getRegionFileSystem().getStoreFiles(ctx.getFamily().getNameAsString());
+return files != null ? files : Collections.emptyList();
   }
 
   @Override
@@ -57,4 +60,9 @@ class DefaultStoreFileTracker extends StoreFileTrackerBase {
 Collection newFiles) throws IOException {
 // NOOP
   }
+
+  @Override
+  void set(List files) {
+// NOOP
+  }
 }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
index de28b0e..c370b87 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
@@ -48,7 +48,7 @@ import 
org.apache.hadoop.hbase.shaded.protobuf.generated.StoreFileTrackerProtos.
  * storages.
  */
 @InterfaceAudience.Private
-public class FileBasedStoreFileTracker extends StoreFileTrackerBase {
+class FileBasedStoreFileTracker extends StoreFileTrackerBase {
 
   private final StoreFileListFile backedFile;
 
@@ -139,4 +139,17 @@ public class FileBasedStoreFileTracker extends 
StoreFileTrackerBase {
   }
 }
   }
+
+  @Override
+  void set(List files) throws IOException {
+synchronized (storefiles) {
+  storefiles.clear();
+  StoreFileList.Builder builder = StoreFileList.newBuilder();
+  for (StoreFileInfo info : files) {
+storefiles.put(info.getPath().getName(), info);
+builder.addStoreFile(toStoreFileEntry(info));
+  }
+  backedFile.update(builder);
+}
+  }
 }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
new file mode 100644
index 000..e486e6d
--- /dev/null
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package

[hbase] 10/16: HBASE-26386 Refactor StoreFileTracker implementations to expose the set method (#3774)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 5af5c5ef8c67cb69e1613df3e72e33e960c946bf
Author: Duo Zhang 
AuthorDate: Thu Oct 21 10:27:45 2021 +0800

HBASE-26386 Refactor StoreFileTracker implementations to expose the set 
method (#3774)

Signed-off-by: Wellington Chevreuil 
---
 .../regionserver/storefiletracker/DefaultStoreFileTracker.java |  2 +-
 .../storefiletracker/FileBasedStoreFileTracker.java|  2 +-
 .../storefiletracker/MigrationStoreFileTracker.java|  5 +
 .../hbase/regionserver/storefiletracker/StoreFileTracker.java  |  7 +--
 .../regionserver/storefiletracker/StoreFileTrackerBase.java| 10 --
 5 files changed, 8 insertions(+), 18 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
index a13b75b..b1e298d 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
@@ -62,7 +62,7 @@ class DefaultStoreFileTracker extends StoreFileTrackerBase {
   }
 
   @Override
-  void set(List files) {
+  public void set(List files) {
 // NOOP
   }
 }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
index 4da7911..079b59b 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
@@ -148,7 +148,7 @@ class FileBasedStoreFileTracker extends 
StoreFileTrackerBase {
   }
 
   @Override
-  void set(List files) throws IOException {
+  public void set(List files) throws IOException {
 synchronized (storefiles) {
   storefiles.clear();
   StoreFileList.Builder builder = StoreFileList.newBuilder();
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
index 230c1ec..a6648f2 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
@@ -21,9 +21,6 @@ import java.io.IOException;
 import java.util.Collection;
 import java.util.List;
 import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.client.TableDescriptor;
-import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
-import org.apache.hadoop.hbase.procedure2.util.StringUtils;
 import org.apache.hadoop.hbase.regionserver.StoreContext;
 import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
 import org.apache.yetus.audience.InterfaceAudience;
@@ -84,7 +81,7 @@ class MigrationStoreFileTracker extends StoreFileTrackerBase {
   }
 
   @Override
-  void set(List files) {
+  public void set(List files) {
 throw new UnsupportedOperationException(
   "Should not call this method on " + getClass().getSimpleName());
   }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
index fd8f7c9..f56a0dd 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
@@ -20,8 +20,6 @@ package org.apache.hadoop.hbase.regionserver.storefiletracker;
 import java.io.IOException;
 import java.util.Collection;
 import java.util.List;
-
-import org.apache.hadoop.hbase.client.TableDescriptor;
 import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
 import org.apache.hadoop.hbase.regionserver.CreateStoreFileWriterParams;
 import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
@@ -70,6 +68,11 @@ public interface StoreFileTracker {
 throws IOException;
 
   /**
+   * Set the store files.
+   */
+  void set(List files) throws IOException;
+
+  /**
* Create a writer for writing new store files.
* @return Writer for a new StoreFile
*/
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerBase.java

[hbase] 02/16: HBASE-25988 Store the store file list by a file (#3578)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit c3aa48ffcaca2dca28733c8b841315a9bfd9abdc
Author: Duo Zhang 
AuthorDate: Thu Aug 26 18:51:12 2021 +0800

HBASE-25988 Store the store file list by a file (#3578)

Signed-off-by: Wellington Chevreuil 
---
 .../protobuf/server/region/StoreFileTracker.proto  |  29 +++--
 .../hadoop/hbase/regionserver/StoreContext.java|   5 +
 .../hadoop/hbase/regionserver/StoreEngine.java |   8 +-
 .../storefiletracker/DefaultStoreFileTracker.java  |   5 +-
 .../FileBasedStoreFileTracker.java | 142 +
 .../storefiletracker/StoreFileListFile.java| 142 +
 .../storefiletracker/StoreFileTrackerBase.java |  11 +-
 .../storefiletracker/StoreFileTrackerFactory.java  |  12 +-
 .../TestRegionWithFileBasedStoreFileTracker.java   | 109 
 9 files changed, 430 insertions(+), 33 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerFactory.java
 b/hbase-protocol-shaded/src/main/protobuf/server/region/StoreFileTracker.proto
similarity index 57%
copy from 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerFactory.java
copy to 
hbase-protocol-shaded/src/main/protobuf/server/region/StoreFileTracker.proto
index 4f7231b..2a269ea 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerFactory.java
+++ 
b/hbase-protocol-shaded/src/main/protobuf/server/region/StoreFileTracker.proto
@@ -15,21 +15,22 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.apache.hadoop.hbase.regionserver.storefiletracker;
+syntax = "proto2";
+// This file contains protocol buffers that are used for store file tracker.
+package hbase.pb;
 
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.regionserver.StoreContext;
-import org.apache.yetus.audience.InterfaceAudience;
+option java_package = "org.apache.hadoop.hbase.shaded.protobuf.generated";
+option java_outer_classname = "StoreFileTrackerProtos";
+option java_generic_services = true;
+option java_generate_equals_and_hash = true;
+option optimize_for = SPEED;
 
-/**
- * Factory method for creating store file tracker.
- */
-@InterfaceAudience.Private
-public final class StoreFileTrackerFactory {
+message StoreFileEntry {
+  required string name = 1;
+  required uint64 size = 2;
+}
 
-  public static StoreFileTracker create(Configuration conf, TableName 
tableName,
-boolean isPrimaryReplica, StoreContext ctx) {
-return new DefaultStoreFileTracker(conf, tableName, isPrimaryReplica, ctx);
-  }
+message StoreFileList {
+  required uint64 timestamp = 1;
+  repeated StoreFileEntry store_file = 2;
 }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreContext.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreContext.java
index 2a9f968..588f8f4 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreContext.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreContext.java
@@ -22,6 +22,7 @@ import java.util.Collection;
 import java.util.function.Supplier;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.hbase.CellComparator;
+import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
 import org.apache.hadoop.hbase.client.RegionInfo;
 import org.apache.hadoop.hbase.io.HeapSize;
@@ -109,6 +110,10 @@ public final class StoreContext implements HeapSize {
 return coprocessorHost;
   }
 
+  public TableName getTableName() {
+return getRegionInfo().getTable();
+  }
+
   public RegionInfo getRegionInfo() {
 return regionFileSystem.getRegionInfo();
   }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java
index 4033c33..0486729 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreEngine.java
@@ -173,9 +173,9 @@ public abstract class StoreEnginehttp://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver.storefiletracker;
+
+import java.io.IOException;
+import

[hbase] 06/16: HBASE-26248 Should find a suitable way to let users specify the store file tracker implementation (#3665)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 56988dd89057d1cf4cf989dd3e1c727d25b56251
Author: Duo Zhang 
AuthorDate: Tue Sep 14 16:28:21 2021 +0800

HBASE-26248 Should find a suitable way to let users specify the store file 
tracker implementation (#3665)

Signed-off-by: Wellington Chevreuil 
---
 .../MigrationStoreFileTracker.java | 13 ++-
 .../storefiletracker/StoreFileTracker.java |  7 +-
 .../storefiletracker/StoreFileTrackerBase.java | 12 +--
 .../storefiletracker/StoreFileTrackerFactory.java  | 97 +++---
 .../org/apache/hadoop/hbase/client/TestAdmin.java  |  6 +-
 .../org/apache/hadoop/hbase/client/TestAdmin3.java |  6 +-
 .../hbase/client/TestAsyncTableAdminApi.java   |  6 +-
 .../hbase/client/TestAsyncTableAdminApi3.java  |  6 +-
 .../procedure/MasterProcedureTestingUtility.java   |  6 +-
 .../master/procedure/TestCreateTableProcedure.java |  6 +-
 .../regionserver/TestMergesSplitsAddToTracker.java |  4 +-
 .../TestMigrationStoreFileTracker.java | 27 +++---
 .../TestRegionWithFileBasedStoreFileTracker.java   |  3 +-
 .../TestStoreFileTrackerFactory.java   | 58 +
 14 files changed, 202 insertions(+), 55 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
index 483a240..3eeef90 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
@@ -22,6 +22,7 @@ import java.util.Collection;
 import java.util.List;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.procedure2.util.StringUtils;
 import org.apache.hadoop.hbase.regionserver.StoreContext;
 import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
 import org.apache.yetus.audience.InterfaceAudience;
@@ -44,8 +45,8 @@ class MigrationStoreFileTracker extends StoreFileTrackerBase {
 
   public MigrationStoreFileTracker(Configuration conf, boolean 
isPrimaryReplica, StoreContext ctx) {
 super(conf, isPrimaryReplica, ctx);
-this.src = StoreFileTrackerFactory.create(conf, SRC_IMPL, 
isPrimaryReplica, ctx);
-this.dst = StoreFileTrackerFactory.create(conf, DST_IMPL, 
isPrimaryReplica, ctx);
+this.src = StoreFileTrackerFactory.createForMigration(conf, SRC_IMPL, 
isPrimaryReplica, ctx);
+this.dst = StoreFileTrackerFactory.createForMigration(conf, DST_IMPL, 
isPrimaryReplica, ctx);
 Preconditions.checkArgument(!src.getClass().equals(dst.getClass()),
   "src and dst is the same: %s", src.getClass());
   }
@@ -90,7 +91,11 @@ class MigrationStoreFileTracker extends StoreFileTrackerBase 
{
   @Override
   public void persistConfiguration(TableDescriptorBuilder builder) {
 super.persistConfiguration(builder);
-builder.setValue(SRC_IMPL, src.getClass().getName());
-builder.setValue(DST_IMPL, dst.getClass().getName());
+if (StringUtils.isEmpty(builder.getValue(SRC_IMPL))) {
+  builder.setValue(SRC_IMPL, src.getTrackerName());
+}
+if (StringUtils.isEmpty(builder.getValue(DST_IMPL))) {
+  builder.setValue(DST_IMPL, dst.getTrackerName());
+}
   }
 }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
index 81fa1a9..59fe7ef 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
@@ -75,7 +75,12 @@ public interface StoreFileTracker {
   StoreFileWriter createWriter(CreateStoreFileWriterParams params) throws 
IOException;
 
   /**
-   * Saves StoreFileTracker implementations specific configs into the table 
descriptors.
+   * Saves StoreFileTracker implementations specific configurations into the 
table descriptors.
+   * 
+   * This is used to avoid accidentally data loss when changing the cluster 
level store file tracker
+   * implementation, and also possible misconfiguration between master and 
region servers.
+   * 
+   * See HBASE-26246 for more details.
* @param builder The table descriptor builder for the given table.
*/
   void persistConfiguration(TableDescriptorBuilder builder);
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerBase.java

[hbase] 05/16: HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor when creating table (#3666)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 1ec667143822738a76fd298aeaaff48a05426ac5
Author: Wellington Ramos Chevreuil 
AuthorDate: Sun Sep 12 14:14:03 2021 +0100

HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor 
when creating table (#3666)

Signed-off-by: Duo Zhang 
---
 .../master/procedure/CreateTableProcedure.java |  7 ++
 .../hbase/regionserver/HRegionFileSystem.java  |  2 +-
 .../MigrationStoreFileTracker.java |  8 +++
 .../storefiletracker/StoreFileTracker.java |  8 +++
 .../storefiletracker/StoreFileTrackerBase.java | 13 +++
 .../storefiletracker/StoreFileTrackerFactory.java  | 25 +++---
 .../org/apache/hadoop/hbase/client/TestAdmin.java  |  6 ++
 .../org/apache/hadoop/hbase/client/TestAdmin3.java |  6 ++
 .../hbase/client/TestAsyncTableAdminApi.java   |  6 ++
 .../hbase/client/TestAsyncTableAdminApi3.java  |  6 ++
 .../procedure/MasterProcedureTestingUtility.java   |  7 ++
 .../master/procedure/TestCreateTableProcedure.java | 17 +++
 .../storefiletracker/TestStoreFileTracker.java | 14 +---
 13 files changed, 113 insertions(+), 12 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
index 2313e70..dccea55 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java
@@ -18,6 +18,7 @@
 
 package org.apache.hadoop.hbase.master.procedure;
 
+
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
@@ -33,10 +34,12 @@ import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.RegionInfo;
 import org.apache.hadoop.hbase.client.RegionReplicaUtil;
 import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
 import org.apache.hadoop.hbase.client.TableState;
 import org.apache.hadoop.hbase.master.MasterCoprocessorHost;
 import org.apache.hadoop.hbase.master.MasterFileSystem;
 import org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer;
+import 
org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTrackerFactory;
 import org.apache.hadoop.hbase.replication.ReplicationException;
 import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
 import org.apache.hadoop.hbase.util.CommonFSUtils;
@@ -285,6 +288,10 @@ public class CreateTableProcedure
   getTableName(), (newRegions != null ? newRegions.size() : 0));
 }
 
+TableDescriptorBuilder builder = 
TableDescriptorBuilder.newBuilder(tableDescriptor);
+StoreFileTrackerFactory.persistTrackerConfig(env.getMasterConfiguration(), 
builder);
+tableDescriptor = builder.build();
+
 final MasterCoprocessorHost cpHost = env.getMasterCoprocessorHost();
 if (cpHost != null) {
   final RegionInfo[] regions = newRegions == null ? null :
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
index cb30432..aa0ee27 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
@@ -610,7 +610,7 @@ public class HRegionFileSystem {
   writeRegionInfoFileContent(conf, fs, regionInfoFile, regionInfoContent);
   HRegionFileSystem regionFs = HRegionFileSystem.openRegionFromFileSystem(
 env.getMasterConfiguration(), fs, getTableDir(), regionInfo, false);
-  insertRegionFilesIntoStoreTracker(allRegionFiles, env, regionFs);
+insertRegionFilesIntoStoreTracker(allRegionFiles, env, regionFs);
 }
 return regionDir;
   }
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
index e486e6d..483a240 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
@@ -21,6 +21,7 @@ import java.io.IOException;
 import java.util.Collection;
 import java.util.List;
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
 import org.apache.hadoop.hbase.regionserver.StoreContext;
 import

[hbase] 03/16: HBASE-26079 Use StoreFileTracker when splitting and merging (#3617)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 80e417c022e66d51f96196e892e97fb323130728
Author: Wellington Ramos Chevreuil 
AuthorDate: Wed Sep 8 10:31:49 2021 +0100

HBASE-26079 Use StoreFileTracker when splitting and merging (#3617)

Signed-off-by: Duo Zhang 
---
 .../assignment/MergeTableRegionsProcedure.java |  22 +-
 .../assignment/SplitTableRegionProcedure.java  |  42 ++--
 .../hbase/regionserver/HRegionFileSystem.java  |  42 +++-
 .../storefiletracker/DefaultStoreFileTracker.java  |   4 +-
 .../storefiletracker/StoreFileTracker.java |   1 -
 .../storefiletracker/StoreFileTrackerFactory.java  |  33 ++-
 .../hbase/regionserver/TestDefaultStoreEngine.java |   1 +
 .../regionserver/TestDirectStoreSplitsMerges.java  |  32 ++-
 .../hadoop/hbase/regionserver/TestHStoreFile.java  |  19 +-
 .../regionserver/TestMergesSplitsAddToTracker.java | 262 +
 .../hbase/regionserver/TestStripeStoreEngine.java  |   1 +
 .../storefiletracker/TestStoreFileTracker.java}|  42 ++--
 12 files changed, 434 insertions(+), 67 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
index da3d73e..e6bbe44 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
@@ -24,7 +24,6 @@ import java.util.Collection;
 import java.util.Collections;
 import java.util.List;
 import java.util.stream.Stream;
-
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
@@ -56,6 +55,8 @@ import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
 import org.apache.hadoop.hbase.regionserver.HStoreFile;
 import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
 import org.apache.hadoop.hbase.regionserver.StoreUtils;
+import org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTracker;
+import 
org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTrackerFactory;
 import org.apache.hadoop.hbase.util.Bytes;
 import org.apache.hadoop.hbase.util.CommonFSUtils;
 import org.apache.hadoop.hbase.wal.WALSplitUtil;
@@ -587,30 +588,35 @@ public class MergeTableRegionsProcedure
 final MasterFileSystem mfs = env.getMasterServices().getMasterFileSystem();
 final Path tableDir = CommonFSUtils.getTableDir(mfs.getRootDir(), 
regionsToMerge[0].getTable());
 final FileSystem fs = mfs.getFileSystem();
-
+List mergedFiles = new ArrayList<>();
 HRegionFileSystem mergeRegionFs = 
HRegionFileSystem.createRegionOnFileSystem(
   env.getMasterConfiguration(), fs, tableDir, mergedRegion);
 
 for (RegionInfo ri: this.regionsToMerge) {
   HRegionFileSystem regionFs = HRegionFileSystem.openRegionFromFileSystem(
   env.getMasterConfiguration(), fs, tableDir, ri, false);
-  mergeStoreFiles(env, regionFs, mergeRegionFs, mergedRegion);
+  mergedFiles.addAll(mergeStoreFiles(env, regionFs, mergeRegionFs, 
mergedRegion));
 }
 assert mergeRegionFs != null;
-mergeRegionFs.commitMergedRegion();
+mergeRegionFs.commitMergedRegion(mergedFiles, env);
 
 // Prepare to create merged regions
 env.getAssignmentManager().getRegionStates().
 getOrCreateRegionStateNode(mergedRegion).setState(State.MERGING_NEW);
   }
 
-  private void mergeStoreFiles(MasterProcedureEnv env, HRegionFileSystem 
regionFs,
+  private List mergeStoreFiles(MasterProcedureEnv env, HRegionFileSystem 
regionFs,
 HRegionFileSystem mergeRegionFs, RegionInfo mergedRegion) throws 
IOException {
 final TableDescriptor htd = env.getMasterServices().getTableDescriptors()
   .get(mergedRegion.getTable());
+List mergedFiles = new ArrayList<>();
 for (ColumnFamilyDescriptor hcd : htd.getColumnFamilies()) {
   String family = hcd.getNameAsString();
-  final Collection storeFiles = 
regionFs.getStoreFiles(family);
+  Configuration trackerConfig =
+
StoreFileTrackerFactory.mergeConfigurations(env.getMasterConfiguration(), htd, 
hcd);
+  StoreFileTracker tracker = StoreFileTrackerFactory.create(trackerConfig, 
true,
+family, regionFs);
+  final Collection storeFiles = tracker.load();
   if (storeFiles != null && storeFiles.size() > 0) {
 final Configuration storeConfiguration =
   StoreUtils.createStoreConfiguration(env.getMasterConfiguration(), 
htd, hcd);
@@ -622,11 +628,13 @@ public class MergeTableRegionsProcedure
   // is running in a regionserver's Store context, or we might not be 
able
   // to read the hfiles.

[hbase] branch HBASE-26067 updated (1dbdefc -> a6cc0af)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a change to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git.


omit 1dbdefc  HBASE-26265 Update ref guide to mention the new store file 
tracker im… (#3942)
omit 81c9b87  HBASE-26286: Add support for specifying store file tracker 
when restoring or cloning snapshot
omit 4aa3f47  HBASE-26454 CreateTableProcedure still relies on temp dir and 
renames… (#3845)
omit 2fee980  HBASE-26271 Cleanup the broken store files under data 
directory (#3786)
omit 3f1a935  HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker 
configurations to TableDescriptor for existing tables (#3700)
omit 0fa1b88  HBASE-26328 Clone snapshot doesn't load reference files into 
FILE SFT impl (#3749)
omit 0be4424  HBASE-26386 Refactor StoreFileTracker implementations to 
expose the set method (#3774)
omit d36749c  HBASE-26326 CreateTableProcedure fails when 
FileBasedStoreFileTracker… (#3721)
omit 97c68be  HBASE-26280 Use store file tracker when snapshoting (#3685)
omit 33079f3  HBASE-26264 Add more checks to prevent misconfiguration on 
store file tracker (#3681)
omit 8af8ac9  HBASE-26248 Should find a suitable way to let users specify 
the store file tracker implementation (#3665)
omit ccf90ab  HBASE-26246 Persist the StoreFileTracker configurations to 
TableDescriptor when creating table (#3666)
omit f021793  HBASE-26224 Introduce a MigrationStoreFileTracker to support 
migrating from different store file tracker implementations (#3656)
omit 2a2b274  HBASE-26079 Use StoreFileTracker when splitting and merging 
(#3617)
omit 620366a  HBASE-25988 Store the store file list by a file (#3578)
omit 723cf07  HBASE-26064 Introduce a StoreFileTracker to abstract the 
store file tracking logic
 add 1004e43  HBASE-26529 Document HBASE-26524 to section of Dynamic 
Unloading (#3909)
 add bdbb4fa  HBASE-26525 Use unique thread name for group WALs (#3903)
 add ca3ba49  HBASE-26526 Introduce a timeout to shutdown of WAL (#3297)
 add b5cf3cd  HBASE-26541 hbase-shaded-protobuf not buildable on M1 MacOSX
 add 02c0444  HBASE-26557 log4j2 has a critical RCE vulnerability (#3933)
 add 8bca21b  HBASE-26558 Set version as 3.0.0-alpha-2 in master in prep 
for first RC of 3.0.0-alpha-2 (#3935)
 add a3ff8e4  HBASE-26557 Correct export to set log4j2.formatMsgNoLookups 
in HBASE_OPTS (#3941)
 add 136b1ea  HBASE-26554 Introduce a new parameter in jmx servlet to 
exclude the specific mbean (#3930)
 add 96c971f  HBASE-26549 hbaseprotoc plugin should initialize maven (#3927)
 add 13f3b17  HBASE-26550 Make sure the master is running normally before 
accepting a balance command (#3928)
 add 0f94ab5  HBASE-26572 Upgrade to log4j 2.16.0 (#3943)
 add a36d41a  HBASE-26556 IT and Chaos Monkey improvements (#3932)
 add 8f5a12f  HBASE-26472 Adhere to semantic conventions regarding table 
data operations
 add c93e457  HBASE-26542 Apply a `package` to test protobuf files
 add 006433b  HBASE-26542 Apply a `package` to test protobuf files 
(addendum)
 add 139f085  HBASE-26523 Upgrade hbase-thirdparty dependency to 4.0.0 
(#3910)
 add e598f2c  Revert "HBASE-26523 Upgrade hbase-thirdparty dependency to 
4.0.0 (#3910)"
 new 00b02c4  HBASE-26064 Introduce a StoreFileTracker to abstract the 
store file tracking logic
 new c3aa48f  HBASE-25988 Store the store file list by a file (#3578)
 new 80e417c  HBASE-26079 Use StoreFileTracker when splitting and merging 
(#3617)
 new f691a55  HBASE-26224 Introduce a MigrationStoreFileTracker to support 
migrating from different store file tracker implementations (#3656)
 new 1ec6671  HBASE-26246 Persist the StoreFileTracker configurations to 
TableDescriptor when creating table (#3666)
 new 56988dd  HBASE-26248 Should find a suitable way to let users specify 
the store file tracker implementation (#3665)
 new 16d0551  HBASE-26264 Add more checks to prevent misconfiguration on 
store file tracker (#3681)
 new 60ceccb  HBASE-26280 Use store file tracker when snapshoting (#3685)
 new 979b474  HBASE-26326 CreateTableProcedure fails when 
FileBasedStoreFileTracker… (#3721)
 new 5af5c5e  HBASE-26386 Refactor StoreFileTracker implementations to 
expose the set method (#3774)
 new 539845f  HBASE-26328 Clone snapshot doesn't load reference files into 
FILE SFT impl (#3749)
 new 3dd4dc8  HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker 
configurations to TableDescriptor for existing tables (#3700)
 new 766d8a3  HBASE-26271 Cleanup the broken store files under data 
directory (#3786)
 new d00bb63  HBASE-26454 CreateTableProcedure still relies on temp dir and 
renames… (#3845)
 new b64da06  HBASE-26286: Add support for specifying store file tracker 
when restoring or cloning snapshot
 new a6cc0af  HBASE-26265 Update ref guide to

[hbase] 01/16: HBASE-26064 Introduce a StoreFileTracker to abstract the store file tracking logic

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch HBASE-26067
in repository https://gitbox.apache.org/repos/asf/hbase.git

commit 00b02c40821a93afd71ba86fad3e9c336c06bf2e
Author: Duo Zhang 
AuthorDate: Thu Jul 29 18:35:19 2021 +0800

HBASE-26064 Introduce a StoreFileTracker to abstract the store file 
tracking logic

Signed-off-by: Wellington Chevreuil 
---
 .../hadoop/hbase/mob/DefaultMobStoreCompactor.java |  22 +-
 .../hadoop/hbase/mob/DefaultMobStoreFlusher.java   |   4 +-
 .../regionserver/CreateStoreFileWriterParams.java  | 134 
 .../hbase/regionserver/DateTieredStoreEngine.java  |   5 +-
 .../hbase/regionserver/DefaultStoreEngine.java |   5 +-
 .../hbase/regionserver/DefaultStoreFlusher.java|  11 +-
 .../hadoop/hbase/regionserver/HMobStore.java   |   3 +-
 .../hbase/regionserver/HRegionFileSystem.java  |  10 +-
 .../apache/hadoop/hbase/regionserver/HStore.java   | 729 +
 .../hadoop/hbase/regionserver/StoreContext.java|   9 +
 .../hadoop/hbase/regionserver/StoreEngine.java | 461 -
 .../hbase/regionserver/StoreFileManager.java   |   9 +
 .../hadoop/hbase/regionserver/StoreFlusher.java|   9 +-
 .../hadoop/hbase/regionserver/StoreUtils.java  |  37 +-
 .../hbase/regionserver/StripeStoreEngine.java  |   9 +-
 .../hbase/regionserver/StripeStoreFlusher.java |   9 +-
 .../compactions/AbstractMultiOutputCompactor.java  |   7 +-
 .../hbase/regionserver/compactions/Compactor.java  |  36 +-
 .../regionserver/compactions/DefaultCompactor.java |  16 +-
 .../storefiletracker/DefaultStoreFileTracker.java  |  61 ++
 .../storefiletracker/StoreFileTracker.java |  75 +++
 .../storefiletracker/StoreFileTrackerBase.java | 178 +
 .../storefiletracker/StoreFileTrackerFactory.java  |  35 +
 .../util/compaction/MajorCompactionRequest.java|   1 -
 .../org/apache/hadoop/hbase/TestIOFencing.java |  12 +-
 .../regionserver/TestCacheOnWriteInSchema.java |   6 +-
 .../hbase/regionserver/TestDefaultStoreEngine.java |   5 +-
 .../hadoop/hbase/regionserver/TestHRegion.java |   4 +-
 .../hadoop/hbase/regionserver/TestHStore.java  |  33 +-
 .../TestRegionMergeTransactionOnCluster.java   |   6 +-
 .../regionserver/TestStoreFileRefresherChore.java  |   3 +-
 .../regionserver/TestStoreScannerClosure.java  |   6 +-
 .../hbase/regionserver/TestStripeStoreEngine.java  |   2 +
 .../compactions/TestDateTieredCompactor.java   |  12 +-
 .../compactions/TestStripeCompactionPolicy.java|  12 +-
 .../compactions/TestStripeCompactor.java   |  12 +-
 36 files changed, 1261 insertions(+), 727 deletions(-)

diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
index c45fdff..01fe000 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
@@ -29,7 +29,6 @@ import java.util.HashMap;
 import java.util.List;
 import java.util.Map.Entry;
 import java.util.Optional;
-
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.FileSystem;
@@ -144,17 +143,16 @@ public class DefaultMobStoreCompactor extends 
DefaultCompactor {
   };
 
   private final CellSinkFactory writerFactory =
-  new CellSinkFactory() {
-@Override
-public StoreFileWriter createWriter(InternalScanner scanner,
-
org.apache.hadoop.hbase.regionserver.compactions.Compactor.FileDetails fd,
-boolean shouldDropBehind, boolean major) throws IOException {
-  // make this writer with tags always because of possible new cells 
with tags.
-  return store.createWriterInTmp(fd.maxKeyCount,
-major ? majorCompactionCompression : minorCompactionCompression,
-true, true, true, shouldDropBehind);
-}
-  };
+new CellSinkFactory() {
+  @Override
+  public StoreFileWriter createWriter(InternalScanner scanner,
+org.apache.hadoop.hbase.regionserver.compactions.Compactor.FileDetails 
fd,
+boolean shouldDropBehind, boolean major) throws IOException {
+// make this writer with tags always because of possible new cells 
with tags.
+return store.getStoreEngine().createWriter(
+  createParams(fd, shouldDropBehind, 
major).includeMVCCReadpoint(true).includesTag(true));
+  }
+};
 
   public DefaultMobStoreCompactor(Configuration conf, HStore store) {
 super(conf, store);
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java
index 480b85c..4a1dc7b 100644
---

[hbase] branch HBASE-26233 updated (5c18b67 -> 1c8d939)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a change to branch HBASE-26233
in repository https://gitbox.apache.org/repos/asf/hbase.git.


from 5c18b67  HBASE-26539 The default rpc timeout 200ms is too small for 
replicating meta edits (#3919)
 add 1c8d939  HBASE-26540 Implement a tool to verify the lag of region 
replication (#3950)

No new revisions were added by this update.

Summary of changes:
 .../hbase/RegionReplicationLagEvaluation.java  | 167 +
 .../hbase/TestRegionReplicationLagEvaluation.java  |  30 ++--
 2 files changed, 181 insertions(+), 16 deletions(-)
 create mode 100644 
hbase-server/src/test/java/org/apache/hadoop/hbase/RegionReplicationLagEvaluation.java
 copy 
hbase-examples/src/test/java/org/apache/hadoop/hbase/client/example/TestAsyncClientExample.java
 => 
hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionReplicationLagEvaluation.java
 (64%)

[hbase-thirdparty] branch branch-3 updated: HBASE-26592 Fix the broken shaded protobuf module (#70)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hbase-thirdparty.git


The following commit(s) were added to refs/heads/branch-3 by this push:
 new 0710f6b  HBASE-26592 Fix the broken shaded protobuf module (#70)
0710f6b is described below

commit 0710f6b0d2b0ec8d5b2cc263ebd01b9626520d96
Author: Duo Zhang 
AuthorDate: Fri Dec 17 21:07:06 2021 +0800

HBASE-26592 Fix the broken shaded protobuf module (#70)

Signed-off-by: Pankaj Kumar 
---
 hbase-shaded-protobuf/pom.xml | 28 +++-
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/hbase-shaded-protobuf/pom.xml b/hbase-shaded-protobuf/pom.xml
index cb0ac5d..3b46166 100644
--- a/hbase-shaded-protobuf/pom.xml
+++ b/hbase-shaded-protobuf/pom.xml
@@ -32,6 +32,15 @@
 Pulls down protobuf, patches it, compiles, and then relocates/shades.
   
   
+
+  
+src/main/java
+
+  
+  google/**
+
+  
+
 
   
 
   
 
   com.google.protobuf
@@ -148,12 +163,6 @@
 
   true
   true
-  
-
-  
-  com.google.protobuf:protobuf-java
-
-  
   
 
   com.google.protobuf
@@ -177,11 +186,4 @@
   
 
   
-  
-
-  com.google.protobuf
-  protobuf-java
-  ${protobuf.version}
-
-

[hbase-thirdparty] branch master updated: HBASE-26592 Fix the broken shaded protobuf module (#70)

2021-12-17 Thread zhangduo

This is an automated email from the ASF dual-hosted git repository.

zhangduo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase-thirdparty.git


The following commit(s) were added to refs/heads/master by this push:
 new 775a3bc  HBASE-26592 Fix the broken shaded protobuf module (#70)
775a3bc is described below

commit 775a3bccb957d14cdbcfb467ecae184178660083
Author: Duo Zhang 
AuthorDate: Fri Dec 17 21:07:06 2021 +0800

HBASE-26592 Fix the broken shaded protobuf module (#70)

Signed-off-by: Pankaj Kumar 
---
 hbase-shaded-protobuf/pom.xml | 28 +++-
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/hbase-shaded-protobuf/pom.xml b/hbase-shaded-protobuf/pom.xml
index c7afbda..1220518 100644
--- a/hbase-shaded-protobuf/pom.xml
+++ b/hbase-shaded-protobuf/pom.xml
@@ -32,6 +32,15 @@
 Pulls down protobuf, patches it, compiles, and then relocates/shades.
   
   
+
+  
+src/main/java
+
+  
+  google/**
+
+  
+
 
   
 
   
 
   com.google.protobuf
@@ -146,12 +161,6 @@
 
   true
   true
-  
-
-  
-  com.google.protobuf:protobuf-java
-
-  
   
 
   com.google.protobuf
@@ -175,11 +184,4 @@
   
 
   
-  
-
-  com.google.protobuf
-  protobuf-java
-  ${protobuf.version}
-
-

58 matches

Mail list logo