voonhous opened a new pull request, #20545:
URL: https://github.com/apache/hudi/pull/20545

   ### Describe the issue this Pull Request addresses
   
   Closes #20543
   
   Partitioned record level index initialization unions the index records of 
each data partition one at a time. On Spark every `JavaRDD.union` adds a 
`UnionRDD` level, so the lineage handed to the bulk commit is as deep as the 
number of data partitions, and task serialization overflows the stack at 
roughly 500 partitions on a 1 MB stack.
   
   ### Summary and Changelog
   
   - `HoodieBackedTableMetadataWriter.initializeMetadataPartition` collects the 
per-partition records and unions them once.
   - New `HoodieEngineContext.union(List<HoodieData<T>>)`:
     - Spark: a single `JavaSparkContext.union`, i.e. one `UnionRDD` whose 
parents are the inputs.
     - In-memory engines (Java, Flink, local): flattened in one pass through 
the new `HoodieListData.unionAll`, since chained `Stream.concat` nests the same 
way. The result keeps the eager semantic of `emptyHoodieData()`, as the old 
fold did.
     - Other `HoodieData` types fall back to folding.
   - Tests: 
`TestRecordLevelIndex#testPartitionedRecordLevelIndexInitializationUnionDepth` 
asserts the union depth stays flat over 50 data partitions (depth 50 before 
this change), plus unit tests in `TestHoodieListData`, 
`TestHoodieLocalEngineContext` and `TestHoodieSparkEngineContext`.
   
   <details><summary>Stack thresholds before and after</summary>
   
   JDK 11 arm64, Spark 3.5, local mode, one file per partition.
   
   | `-Xss` | before: passes up to | before: fails from | after |
   |---|---|---|---|
   | 512k | 250 | 260-300 | |
   | 1m | ~450-520 | ~500-550 | 800 and 1500 pass, depth 1 |
   | 2m | ~1000 | 1100+ | |
   
   Failures were mostly driver-side (`Task serialization failed: 
java.lang.StackOverflowError` in `DAGScheduler.submitMissingTasks`), 
occasionally executor-side during task deserialization.
   
   </details>
   
   ### Impact
   
   Partitioned RLI initialization works on tables with many data partitions. 
New public method `HoodieEngineContext.union(List)`; no behavior change for 
existing callers.
   
   ### Risk Level
   
   low
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to