danny0405 opened a new pull request, #20134:
URL: https://github.com/apache/hudi/pull/20134
### Describe the issue this Pull Request addresses
Partitioned Flink record-level index cache eviction recalculates total heap
usage and restarts the LRU scan after every removal. Evicting many partitions
can therefore take quadratic time in the number of cached partitions.
Both partitioned and global record-level index backends also reload an
unused data-table timeline after each completed checkpoint, immediately before
recreating metadata readers with their own meta clients.
### Summary and Changelog
- Compute total cache heap usage once and walk the LRU map once, subtracting
each evicted cache's size before closing it.
- Preserve the protected partition, checkpoint-watermark eligibility, LRU
eviction order, and reserved space for the next cache.
- Remove redundant timeline reloads from `RecordLevelIndexBackend` and
`GlobalRecordLevelIndexBackend`; retain metadata-reader recreation.
- Add a regression test covering multiple evictions, interleaved protected
and recent partitions, stopping at the memory limit, resource closure, and
bounded heap-size reads.
No code was copied from external sources.
### Impact
Cache cleanup takes linear rather than quadratic time when evicting many
partitions. Both Flink RLI backends avoid an unused timeline refresh per
completed checkpoint. No public API, configuration, storage-format, or
compatibility changes.
### Risk Level
low
The change affects cache eviction bookkeeping and checkpoint refresh work.
Existing eviction protections remain intact, and the regression test checks
memory accounting before cache closure.
Validation: all 23 tests in `TestRecordLevelIndexBackend` and
`TestGlobalRecordLevelIndexBackend` passed, with zero Checkstyle violations.
The full test suite and performance benchmarks were not run.
```sh
mvn -nsu -pl hudi-flink-datasource/hudi-flink -am \
-Dtest=TestRecordLevelIndexBackend,TestGlobalRecordLevelIndexBackend \
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false test
```
### Documentation Update
none — internal performance improvements with no new features,
configuration, or user-facing behavior changes.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
