DeviousCardi opened a new pull request, #20383: URL: https://github.com/apache/hudi/pull/20383
### Describe the issue this Pull Request addresses Closes #20057. A file index reused by a long-lived Spark session retains the file slices of every partition queried, with no eviction. ### Summary and Changelog Bound the lazy-listing cache by estimated heap size using Caffeine. The new `hoodie.file.index.cache.max.mem` setting defaults to 500 MiB; a non-positive value disables the bound. Eager listing and the existing spillable-map path keep their current retention behavior. Each query keeps a complete result even if its working set exceeds the cache budget. Track listed file sizes separately so eviction does not shrink the size estimate used by Spark's planner. ### Impact Long-lived sessions retain fewer file slices and may re-list evicted partitions. The weight is an estimate based on file counts and sampled full path lengths, including bootstrap paths. The separate size ledger still grows with distinct partitions until refresh, so this does not bound all index memory. No storage-format changes. ### Risk Level Medium: eviction changes the metadata read path and can increase listing work. Regression tests cover bounded retention, oversized queries, concurrency, refresh, size accounting, and eager-listing behavior. Focused Maven validation passes: 3 common-module tests and 18 Hadoop-backed file-index tests, with no Checkstyle or license-check violations. The full Spark/utility suites were not run. ### Documentation Update The new setting is documented in `HoodieCommonConfig`. Website configuration documentation needs a follow-up. AI assistance: Claude Code prepared the implementation and tests. Codex read the changes, corrected the config description to match the weight estimate, and prepared this PR description. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
