This is an automated email from the ASF dual-hosted git repository.
forwardxu pushed a change to branch release-0.12.1
in repository https://gitbox.apache.org/repos/asf/hudi.git
from 2a3b0b5af8a add DeleteFsFileProcedure
new d8043d48ea0 fix metatable hbase-site
new ebf99a7e84f [HUDI-5278] Support more conf to cluster procedure (#7304)
new 9a9fd985f2f improve BackupInvalidParquetProcedure
new 0344765bdf3 [HUDI-5671] BucketIndexPartitioner partition algorithm
skew (#7815)
new 3c4a403a0a3 [HUDI-5318] Fix partition pruning for clustering
scheduling (#7366)
new b0a0912cfaf [HUDI-5326] Fix clustering group building in
SparkSizeBasedClusteringPlanStrategy (#7372)
new 2c8586f5a3b fix enable metrics on
new 5e2db20bb20 fix Zhiyan metrics reporter
new cca17a599d6 [MINOR] improve RunClusteringProcedure with partition
selected
new 59094436c9d [HUDI-5343] HoodieFlinkStreamer supports async clustering
for append mode (#7403)
new 09df7305557 [HUDI-5515] Fix concurrency conflict in ClusteringOperator
with latency marker (#7625)
new 94ee9a133ad [HUDI-5543] Description of
clustering.plan.partition.filter.mode supports DAY_ROLLING strategy (#7656)
new 700717c7344 [HUDI-5235] Clustering target size should larger than
small file limit (#7232)
new ee779fe86fe [HUDI-5341] CleanPlanner retains earliest commits must not
be later than earliest pending commit (#7568)
new e44285447a1 improve getCommitInstantsToArchive data duplication
new c265ddcc4b5 [HUDI-5506] StreamWriteOperatorCoordinator may not
recommit with partial uncommitted write metadata event (#7611)
new 108c6afd308 [HUDI-4968] Update misleading
read.streaming.skip_compaction/skip_clustering config (#6856)
new 0f056e52e58 [HUDI-5286] UnsupportedOperationException throws when
enabling filesystem retry (#7313)
new 7588b918de2 add DropPartitionsProcedure
new 45bef56db55 [HUDI-5495] add some property to table config
The 20 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.idea/vcs.xml | 1 -
.../apache/hudi/client/HoodieTimelineArchiver.java | 54 ++--
.../apache/hudi/config/HoodieClusteringConfig.java | 56 +++-
.../hudi/table/action/clean/CleanPlanner.java | 19 +-
.../PartitionAwareClusteringPlanStrategy.java | 26 +-
.../TestPartitionAwareClusteringPlanStrategy.java | 2 +-
.../SparkSizeBasedClusteringPlanStrategy.java | 14 +-
.../SparkSortAndSizeExecutionStrategy.java | 6 +-
.../TestSparkSizeBasedClusteringPlanStrategy.java | 94 ++++++
.../common/fs/HoodieRetryWrapperFileSystem.java | 5 +
.../hudi/common/table/HoodieTableMetaClient.java | 19 ++
.../apache/hudi/common/util/ClusteringUtils.java | 37 +++
hudi-common/src/main/resources/hbase-site.xml | 2 +-
.../fs/TestFSUtilsWithRetryWrapperEnable.java | 14 +
.../hudi/common/util/TestClusteringUtils.java | 69 +++++
.../apache/hudi/configuration/FlinkOptions.java | 21 +-
.../hudi/sink/StreamWriteOperatorCoordinator.java | 8 +-
.../sink/bucket/BucketStreamWriteFunction.java | 5 +-
.../hudi/sink/clustering/ClusteringOperator.java | 63 ++--
.../sink/clustering/FlinkClusteringConfig.java | 37 ++-
.../hudi/sink/compact/FlinkCompactionConfig.java | 30 +-
.../sink/partitioner/BucketIndexPartitioner.java | 5 +-
.../apache/hudi/streamer/FlinkStreamerConfig.java | 53 +++-
.../apache/hudi/streamer/HoodieFlinkStreamer.java | 21 +-
.../java/org/apache/hudi/util/StreamerUtil.java | 57 ++--
.../sink/TestStreamWriteOperatorCoordinator.java | 16 +
.../org/apache/hudi/HoodieSparkSqlWriter.scala | 4 +-
.../procedures/BackupInvalidParquetProcedure.scala | 23 +-
.../procedures/DropPartitionsProcedure.scala | 118 +++++++
.../hudi/command/procedures/HoodieProcedures.scala | 1 +
.../procedures/RunClusteringProcedure.scala | 168 +++++++---
.../hudi/functional/TestLayoutOptimization.scala | 3 +-
.../TestBackupInvalidParquetProcedure.scala | 19 +-
.../hudi/procedure/TestClusteringProcedure.scala | 339 ++++++++++++++++++++-
.../procedure/TestDropPartitionsProcedure.scala | 67 ++++
35 files changed, 1263 insertions(+), 213 deletions(-)
create mode 100644
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/clustering/plan/strategy/TestSparkSizeBasedClusteringPlanStrategy.java
create mode 100644
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/DropPartitionsProcedure.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestDropPartitionsProcedure.scala
