wangxianghu opened a new pull request, #19137:
URL: https://github.com/apache/hudi/pull/19137
### Describe the issue this Pull Request addresses
KeepByTimeStrategy hard-codes the parallelism used to collect the last
commit time of candidate partitions to Math.min(partitionPaths.size(), 200).
When a Hudi table enables partition TTL for the first time, there can be a
very large number of historical partitions, and the fixed cap of 200 makes the
stats-collection phase unnecessarily slow. This PR makes that parallelism cap
configurable.
### Summary and Changelog
Users can now tune the parallelism used during partition TTL stats
collection via a new config, instead of being limited to the hard-coded value
of 200. The default remains 200, so existing behavior is unchanged.
Changes:
- HoodieTTLConfig: added config
hoodie.partition.ttl.strategy.stats.max.parallelism (constant
STATS_MAX_PARALLELISM, default 200, advanced, since 1.3.0) with documentation,
plus builder method withTTLStatsMaxParallelism(...).
- HoodieWriteConfig: added getter getPartitionTTLStatsMaxParallelism().
- KeepByTimeStrategy: replaced the hard-coded 200 with
Math.min(partitionPaths.size(),
writeConfig.getPartitionTTLStatsMaxParallelism()). The config is an upper bound
— the effective parallelism is the smaller of the candidate partition count and
the configured value.
- TestKeepByTimeStrategy: added tests covering both branches — parallelism
capped by the configured max when partitions exceed it, and bounded by the
partition count when it is smaller.
No code was copied.
### Impact
none
### Risk Level
none
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
