andygrove commented on PR #2722: URL: https://github.com/apache/datafusion-ballista/pull/2722#issuecomment-6755599303
@coderfender Added in fa5fe79ed, under both the static and the adaptive planner, with `max_partitions_per_task = 1`. One wrinkle: the job's output partitions come back in the order their tasks finished, so the concatenated stream isn't something we can check for global order. The test checks what makes it a global sort instead: every key exactly once, each partition sorted, and no two partitions' key ranges overlapping. With the planner change reverted, the static test fails with overlapping partitions. Writing it turned up something useful. Since #2682 landed, submitted plans go through the adaptive planner by default, and it pins a range repartition as an exchange. On main that job fails with `unsupported shuffle output partitioning`, and this PR's writer change is what makes it work, so the adaptive test covers that path too. I've rebased onto main as well. No perf numbers yet, I'll leave that for a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
