voonhous commented on code in PR #20545:
URL: https://github.com/apache/hudi/pull/20545#discussion_r4534780625
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieEngineContext.java:
##########
@@ -82,6 +82,27 @@ public <T> HoodieData<T> parallelize(List<T> data) {
public abstract <T> HoodieData<T> parallelize(List<T> data, int parallelism);
+ /**
+ * Unions all the given {@link HoodieData} into a single {@link HoodieData}.
+ *
+ * <p>The default implementation folds the inputs with {@link
HoodieData#union(HoodieData)}, which
+ * does not nest when {@link #emptyHoodieData()} is eager, since every step
materializes its result.
+ * Engines whose union nests per call should override this with a single
flat union: on Spark, a
+ * chain of a few hundred {@code UnionRDD} levels is enough to overflow the
stack during task
+ * serialization.
+ *
+ * @param dataList the {@link HoodieData} to union
+ * @param <T> type of object
+ * @return the union of all the inputs, or {@link #emptyHoodieData()} if
there is none
+ */
+ public <T> HoodieData<T> union(List<HoodieData<T>> dataList) {
Review Comment:
Closed #17591 as superseded: it edits the pre-#18348 init path and conflicts
with master. #20545 credits it in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]