nsivabalan opened a new pull request, #20662: URL: https://github.com/apache/hudi/pull/20662
### Describe the issue this Pull Request addresses On branch-0.x, Spark snapshot reads (and CDC reads) of MERGE_ON_READ file slices with large log files can run executors out of memory. `LogFileIterator.scanLog` in `Iterators.scala` copies every merged log record into a heap `scala.collection.mutable.HashMap`, including the records `HoodieMergedLogRecordScanner` had already spilled to disk in its `ExternalSpillableMap`. Spilling therefore saves no heap. On top of that, the memory cap for the scanner comes from `getMaxCompactionMemoryInBytes(JobConf)`, which is `compaction.memory.fraction` (0.75) times `mapreduce.map.memory.mb` (MapReduce default 1024 MB). Spark jobs rarely set the MapReduce value, so the cap is about 768 MB whatever the executor heap is. **Not an issue on master (1.x):** `Iterators.scala` and these iterators no longer exist there. MOR reads go through the file group reader, whose `FileGroupRecordBuffer` keeps merged log records in an `ExternalSpillableMap` and iterates it in place (`records.iterator()`) without copying it to the heap. This fix is for branch-0.x only. ### Summary and Changelog Spark MOR snapshot and CDC reads on 0.x iterate merged log records in place instead of copying them all to the heap, and size the in-memory part from the executor memory. - `scanLog` keeps the scanner open and returns a new `LogRecords` wrapper over the scanner's record map. Iteration uses `ExternalSpillableMap#iterator`, which walks the in-memory records and then the disk map lazily. (`ExternalSpillableMap#values` would copy everything into one list, so it is avoided.) - `LogFileIterator` is now `Closeable`. The scanner and its spill files are released when the records are drained, on task completion, or when `HoodieCDCRDD` moves to the next CDC file. `close` is idempotent, and a closed iterator never touches the disk map again. - The merge memory cap is the smaller of the previous `JobConf`-based value and `IOUtils.getMaxMemoryAllowedForMerge`, which is derived from the executor memory available to one task. - New test `TestMORDataSource#testSnapshotReadWithSpilledLogRecords` (BITCASK and ROCKS_DB disk maps, INMEMORY and BLOOM index, so log-only slices and base plus log slices): forces every merged log record to spill and asserts the full merged rows of a snapshot read after inserts, updates and deletes. It runs with AVRO records only, because with the SPARK record merger, merging a delete block already fails on 0.x with an NPE on the null ordering value in `HoodieMergedLogRecordScanner#processNextDeletedRecord`. That failure is independent of this change. Verified: the new test (4 cases), `TestCDCDataFrameSuite`, `TestMORDataSourceStorage`, and the existing `TestMORDataSource` spill and payload-delete tests pass. The new test checks that lazy iteration with every record spilled returns correct results; it does not reproduce the out-of-memory failure. ### Impact Lower, executor-aware heap use for MOR snapshot and CDC reads of log files on 0.x. No public API or config change. ### Risk Level medium. The scanner now lives as long as the iterator, instead of being closed inside `scanLog`. To limit the risk, it is closed when the records are drained, on task completion, and when the CDC reader switches files. The snapshot, CDC and storage test suites pass with every record spilled on both disk map types. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
