lilei1128 commented on PR #8973:
URL: https://github.com/apache/paimon/pull/8973#issuecomment-5166429971
> You can just `options.put(CoreOptions.PARTITION_EXPIRATION_TIME.key(),
null);`.
Thanks for the suggestion. I may not have explained the purpose of this PR
clearly enough.
Setting:
` options.put(CoreOptions.PARTITION_EXPIRATION_TIME.key(), null);`
would disable partition expiration for the compaction job. However, it
would
not prevent partitions that are already expired according to the
update-time
strategy from being scanned and compacted.
The purpose of this PR is different: **it keeps partition expiration
enabled,
but avoids unnecessary compaction IO for partitions that are already
expired
and will soon be deleted.**
For example, if a partition's last file creation time is older than
partition.expiration-time, the current update-time strategy still allows
the
dedicated compactor source to emit and compact that partition. This PR
filters
it from the compactor source, consistent with the existing behavior for the
values-time strategy.
For batch compaction, the partition metadata is loaded once. For streaming
compaction, it is refreshed when the input snapshot changes, because a new
snapshot may update a partition's last file creation time. The metadata is
then reused for all rows from the same snapshot to avoid repeated manifest
scans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
