cshuo opened a new issue, #20025: URL: https://github.com/apache/hudi/issues/20025
### Bug Description **What happened:** `LsmFileGroupRecordIterator.initializeReaders()` accumulates opened readers in `sortedRunReaders`, but does not close them if creating or initializing a later reader fails. Previously opened base/log readers and earlier `SpillableLsmRecordIterator` instances can therefore leak their I/O resources and local spill files. The constructor assigns `readers` only after `initializeReaders()` returns. If initialization throws, the outer reader never receives the constructed iterator and cannot release those resources through its `close()` method. There is a related initialization gap in `addReader()`: it calls `SortedRunReader.advance()` before adding the reader to the list. If that first read throws, the current iterator is not in `sortedRunReaders` and also needs explicit cleanup. This was identified by code inspection and raised in [PR #20023 review comment](https://github.com/apache/hudi/pull/20023#discussion_r4070081043). That PR fixes cleanup of the individual spill iterator when its source fails to close; it does not clean up readers previously opened by `initializeReaders()`. **What you expected:** - Close all readers already initialized when initialization fails. - Close the current iterator if its first read fails before it is registered in `sortedRunReaders`. - Continue attempting cleanup if one reader's close fails, preserving the initialization exception and attaching cleanup failures as suppressed exceptions. - Remove spill files owned by the readers being cleaned up. **Failure scenario / proposed regression tests:** 1. Initialize a file group with multiple sorted runs and force at least one earlier run to spill. 2. Inject a failure while opening or initializing a later run. 3. Assert that all previously opened readers are closed and their spill files are deleted. 4. Separately inject a failure in the first `advance()` of a newly created reader and verify that it is closed even though it was not added to the list. 5. Inject a reader-close failure during cleanup and verify that the other readers are still closed and the original initialization failure is preserved. ### Environment - Hudi version: `1.3.0-SNAPSHOT`. - Component: shared LSM file-group reader in `hudi-common`. - Relevant configuration: multiple LSM sorted runs; use a low `hoodie.lsm.sort.merge.spill.threshold` to exercise spill-file cleanup. ### Follow-up Track and fix this initialization cleanup issue in a separate PR from #20023. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
