voonhous opened a new pull request, #20545:
URL: https://github.com/apache/hudi/pull/20545
### Describe the issue this Pull Request addresses
Closes #20543
Partitioned record level index initialization unions the index records of
each data partition one at a time. On Spark every `JavaRDD.union` adds a
`UnionRDD` level, so the lineage handed to the bulk commit is as deep as the
number of data partitions, and task serialization overflows the stack at
roughly 500 partitions on a 1 MB stack.
### Summary and Changelog
- `HoodieBackedTableMetadataWriter.initializeMetadataPartition` collects the
per-partition records and unions them once.
- New `HoodieEngineContext.union(List<HoodieData<T>>)`:
- Spark: a single `JavaSparkContext.union`, i.e. one `UnionRDD` whose
parents are the inputs.
- In-memory engines (Java, Flink, local): flattened in one pass through
the new `HoodieListData.unionAll`, since chained `Stream.concat` nests the same
way. The result keeps the eager semantic of `emptyHoodieData()`, as the old
fold did.
- Other `HoodieData` types fall back to folding.
- Tests:
`TestRecordLevelIndex#testPartitionedRecordLevelIndexInitializationUnionDepth`
asserts the union depth stays flat over 50 data partitions (depth 50 before
this change), plus unit tests in `TestHoodieListData`,
`TestHoodieLocalEngineContext` and `TestHoodieSparkEngineContext`.
<details><summary>Stack thresholds before and after</summary>
JDK 11 arm64, Spark 3.5, local mode, one file per partition.
| `-Xss` | before: passes up to | before: fails from | after |
|---|---|---|---|
| 512k | 250 | 260-300 | |
| 1m | ~450-520 | ~500-550 | 800 and 1500 pass, depth 1 |
| 2m | ~1000 | 1100+ | |
Failures were mostly driver-side (`Task serialization failed:
java.lang.StackOverflowError` in `DAGScheduler.submitMissingTasks`),
occasionally executor-side during task deserialization.
</details>
### Impact
Partitioned RLI initialization works on tables with many data partitions.
New public method `HoodieEngineContext.union(List)`; no behavior change for
existing callers.
### Risk Level
low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]