DeviousCardi opened a new pull request, #20383: URL: https://github.com/apache/hudi/pull/20383
### Describe the issue this Pull Request addresses Closes #20057. A file index reused across a long-lived Spark session keeps every queried partition's file slices without eviction. ### Summary and Changelog Use a weighted Caffeine cache for lazy listing. The new `hoodie.file.index.cache.max.mem` setting defaults to 500 MiB; zero or a negative value disables the bound. Evicted partitions are re-listed when needed. Queries keep complete results even when they exceed the cache budget. Listed file sizes are tracked separately so eviction does not shrink Spark's planning estimate. ### Impact The budget estimates heap from file counts and sampled full paths, including bootstrap paths. It limits cached file slices retained between queries; a single query still materializes its complete listing, so its peak memory is not bounded by this setting. Eager listing and the spillable-map path retain their current behavior. The size ledger still grows with distinct partitions until refresh, so this does not bound all index memory. No storage-format changes. ### Risk Level Medium: eviction can increase metadata listing work. All 27 focused tests pass (21 cache tests and 6 upstream timeline cases retained during the merge), covering retention, oversized queries, concurrency, refresh, size accounting, and eager listing. Checkstyle and license checks pass. Full Spark/utility suites were not run. ### Documentation Update The setting is documented in `HoodieCommonConfig`; website configuration docs need a follow-up. AI assistance: Claude Code prepared the implementation and tests. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable Written by Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
