Davis-Zhang-Onehouse opened a new pull request, #20729: URL: https://github.com/apache/hudi/pull/20729
### Describe the issue this Pull Request addresses issue: #20728 Part of #20683, RFC-118 (#20684). Stacked on #20727: review only the last commit. Until the PRs below it merge, this PR also shows their commits. Prepared inputs (image and video frame payloads) can be written to an input store instead of inline, and prep work is balanced by size and type: a balancing strategy plans task groups from the declared object sizes and kinds, with its parallelism taken from the num-partitions setting, local mode, Spark dynamic allocation or the executor count. ### Summary and Changelog - Input store: prepared image and video-frame payloads can be written out of line (with markers, so a failed round rolls them back) instead of inline. - `SizeTypeAwareBalancingStrategy`: plans prep task groups from the declared object sizes and kinds. Its parallelism comes from the num-partitions setting, local mode, `spark.dynamicAllocation.maxExecutors` or `spark.executor.instances`. - The video embedding round and opt-in memory cells of the video round (`-Dvideo.memory.cells`, not run by default CI), with a large-frames fixture and its generator script. - 31 test classes (21 new). No code was copied from other projects. ### Impact Large media rounds spread their work by size and type and can keep payloads out of line. No change outside the new source. ### Risk Level medium: changes how prep work is distributed for the new source. Verified on JDK 11 / Spark 3.5 / Scala 2.12 with 245 tests (7 skipped: external model input or opt-in cells), module compilation, checkstyle and RAT. ### Documentation Update New balancing and input-store settings carry their config documentation; a website page follows once the chain lands. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
