andygrove commented on issue #27427: URL: https://github.com/apache/datafusion/issues/27427#issuecomment-6875980010
### This is a regression in 56 for FFI query planners For users of a foreign `QueryPlanner`, this is a regression from 55 to 56: a query that plans and runs on 55.0.0 fails on 56 with the `SanityCheckPlan` error above. - `FFI_QueryPlanner` first shipped in 55.0.0 (#24028). - datafusion-python 55.0.0 shipped support for it (apache/datafusion-python#1677), including `test_spawning_plan_across_three_libraries`. That test passes on 55. - apache/datafusion-python#1826 only bumps DataFusion to 56. Nothing on the datafusion-python side changed, and the same test now fails during planning. It is the only failure in that PR. The gap in `FFI_ExecutionPlan` is not new: it never carried input requirements. 56 makes it visible because #26461 moved `SanityCheckPlan` out of the rule list and into `DefaultPhysicalPlanner`. On 55 the check ran as one of the host's rules over FFI, so it was just as blind as `EnsureRequirements` and passed the plan. On 55, this particular query still returned correct results, by luck. The partial aggregate produces a single partition with one row per key, so round-robin sends each key to exactly one final partition. With a multi-partition input, the same plan could split a key across partitions and return duplicate groups. So 56 turns a silent correctness hazard into a hard error, which is better behavior, but users who upgrade still see working queries start failing. The fix is contained to `datafusion-ffi`: `FFI_ExecutionPlan` gets callbacks for `input_distribution_requirements`, `required_input_ordering`, `maintains_input_order` and `benefits_from_input_partitioning`. It cherry-picks cleanly onto `branch-56` apart from the upgrade-guide file, so it is a candidate for 56.0.0 or a 56.0.x patch release. PR to follow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
