lilei1128 opened a new pull request, #9219:
URL: https://github.com/apache/paimon/pull/9219
### Purpose
For non-partitioned tables with many buckets, or partitioned tables where
each partition
contains many buckets, compacting all buckets in one job may consume
excessive planning
and execution resources and eventually cause OOM errors.
This PR introduces bucket-level compaction for fixed-bucket tables in
Flink and Spark.
Users can split a large compaction into multiple jobs by selecting
individual bucket IDs
or bucket ranges.
For example:
```
CALL sys.compact(
`table` => 'database.large_bucket_table',
compact_strategy => 'full',
buckets => '0-999'
);
```
A table with many buckets can be compacted in batches such as 0-999,
1000-1999, and 2000-2999.
### Tests
-
paimon-common/src/test/java/org/apache/paimon/utils/ParameterUtilsTest.java
- testParseIntegerRanges
- testInvalidIntegerRanges
-
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/action/CompactActionITCase.java
- testCompactSpecifiedBucketRangesFromAction
-
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/procedure/CompactProcedureITCase.java
- testCompactSpecifiedBucketRanges
-
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/procedure/CompactProcedureTestBase.scala
- Paimon Procedure: compact specified bucket ranges
- Paimon Procedure: reject buckets for dynamic bucket table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
