avantgardnerio commented on PR #26593: URL: https://github.com/apache/datafusion/pull/26593#issuecomment-6639300308
> the difference is run-time cost and DFD code, not a plan the alternative can't express I think there's an information difference too. Under the empty-partition alternative, a task advertises `Range(16, ...)`, and which partitions are empty lives only in the source's private state. Nothing in the plan says the task's keys fall within `[cut3, cut7)`, so no rule running in that task can use it. With bounds, the task's partitioning states its own key range. Bounds aren't used for optimization yet, but they're what pruning the other side of a join, tightening dynamic filters, or key statistics would need. Getting the same information under the alternative would need a way to declare empty partitions on `Partitioning`, along the lines stuhood suggested, which would be a separate API. More generally, a property the plan doesn't expose is one no rule can act on, so I'd rather the partitioning describe the task's actual key range than leave it implicit in the source. Ballista is in the same position as DFD here: each task runs its own DataFusion plan over a slice of the global partitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
