milenkovicm commented on code in PR #2335:
URL:
https://github.com/apache/datafusion-ballista/pull/2335#discussion_r3825892589
##########
ballista/scheduler/src/state/task_builder.rs:
##########
@@ -70,6 +71,56 @@ pub fn restrict_plan_to_partitions(
restrict(plan, partitions, /* under_collect */ false)
}
+/// Merge a marked writer's sorted partitions into one before the write.
+///
+/// The task writes one file instead of one per partition, and reports it as
+/// output partition 0 — `file_id` already distinguishes producers, which is
what
+/// `GlobalPartitionMap::Collapsed` handles. The mark comes from the AQE
adapter,
+/// the only place that knows the consumer reads this back with an
+/// ordering-preserving merge.
+///
+/// Must run after [`restrict_plan_to_partitions`], which treats a
+/// `SortPreservingMergeExec` as a collapse and gives leaves below one the full
+/// upstream — merging first would make every task read the whole stage input.
+pub fn merge_task_partitions_before_write(
Review Comment:
I wonder could this be moved to AQE planner rule instead of having planning
logic at this point ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
