wombatu-kun commented on code in PR #20545:
URL: https://github.com/apache/hudi/pull/20545#discussion_r4533999803
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieEngineContext.java:
##########
@@ -82,6 +82,27 @@ public <T> HoodieData<T> parallelize(List<T> data) {
public abstract <T> HoodieData<T> parallelize(List<T> data, int parallelism);
+ /**
+ * Unions all the given {@link HoodieData} into a single {@link HoodieData}.
+ *
+ * <p>The default implementation folds the inputs with {@link
HoodieData#union(HoodieData)}, which
+ * does not nest when {@link #emptyHoodieData()} is eager, since every step
materializes its result.
+ * Engines whose union nests per call should override this with a single
flat union: on Spark, a
+ * chain of a few hundred {@code UnionRDD} levels is enough to overflow the
stack during task
+ * serialization.
+ *
+ * @param dataList the {@link HoodieData} to union
+ * @param <T> type of object
+ * @return the union of all the inputs, or {@link #emptyHoodieData()} if
there is none
+ */
+ public <T> HoodieData<T> union(List<HoodieData<T>> dataList) {
Review Comment:
#17591 fixes the same partitioned RLI bootstrap overflow with the same
`union` API and is still open with an approval. Should it be closed as
superseded once this lands?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]