comphead commented on PR #7256: URL: https://github.com/apache/datafusion-comet/pull/7256#issuecomment-6528169673
Thanks @andygrove, all three are addressed in 8e0b8e0da1. 1. The `SortedInputExec` doc now says the planner puts it on a window's input too. The "Reporting an Output Ordering" section says that a native window declares the order of its partition keys. An operator that reports an order it cannot keep now breaks a window as well as an outer hash join, and both serdes fall back over it. 2. The window now falls back the way the outer hash join does. When `mayReportFalseOrder` with `countInPlan = false` flags the child's order, nothing native can declare it, so the window stays in Spark with the reason "the order of the window's input, which Spark relies on, cannot be checked, so the window stays in Spark". When only a Comet join in the same native plan raises the doubt, nothing is sent, and DataFusion uses the order it sees there. `a window over a cache whose order cannot be checked stays in Spark` covers the fallback. It reuses the setup of the outer-join test, with a filter that hides the first join's declaration, and it fails natively without the change. 3. Agreed, the window now declares only its partition keys. `get_partition_by_sort_exprs` is the only check either operator makes. `required_input_ordering` lists the order keys too, but only DataFusion's optimizer reads it, and Comet does not run that. Every case in the stacked tests still runs, since each window sends its own partition keys. A float order key is never declared now. A float partition key never reaches the declaration either. `NormalizeFloatingNumbers` rewrites the window's partition keys, so a cache sorted by the raw column does not satisfy the required order, and Spark plans a sort. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
