lilei1128 opened a new pull request, #9021:
URL: https://github.com/apache/paimon/pull/9021
### Purpose
Support compaction.skip-expired-partitions in Spark compaction.
For partitioned primary-key tables using the values-time expiration
strategy,
avoid unnecessary compaction IO on partitions that are already expired and
will subsequently be deleted.
### Changes
- Filter expired partitions while Spark plans fixed/dynamic bucket
compaction tasks.
- Reuse PartitionValuesTimeExpireStrategy to keep expiration semantics
aligned with Flink.
- Apply filtering before Spark parallelizes compact tasks and reads data
files.
- Preserve existing behavior when:
- the option is disabled;
- partition.expiration-time is not configured;
- the expiration strategy is not values-time;
- the table uses an unsupported bucket mode.
- Preserve partition expiration during commit, including when
end-input.check-partition-expire=true.
- Add Spark SQL documentation for compact and compact_database.
Related PR:
https://github.com/apache/paimon/pull/7537
https://github.com/apache/paimon/pull/8973
### Tests
- Skipping expired partitions for fixed bucket primary-key tables.
- Skipping expired partitions for dynamic bucket primary-key tables.
- Default behavior when the option is disabled.
- Ignoring the option for the update-time strategy.
- Deleting skipped partitions during commit when
end-input.check-partition-expire=true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
