dwsmith1983 opened a new pull request, #7623: URL: https://github.com/apache/datafusion-comet/pull/7623
## Which issue does this PR close? No issue is filed for this. The `branch-1.1` backport of #6268 depends on it and should merge after it. ## Rationale for this change On `branch-1.1`, the native planner builds the `EmptyExec` for a bucket with no files from `required_schema`, which leaves out partition columns and ignores the projection order. A native operator above a partitioned, bucketed scan then fails on the empty buckets with `Column index 1 is out of bound`. `main` fixed this inside #6181 (74fdec5bcdb95f209104a2b698712f668aea4ae2, by Ping Zhang), a TopK fusion feature, so this takes only the `planner.rs` hunk and its native tests rather than the whole commit. `branch-1.2` already has it. `branch-1.0` has the same code; no backport is planned there, since nothing that needs this targets 1.0. ## What changes are included in this PR? - An empty bucket's `EmptyExec` uses the scan's projected table schema, the one a bucket with files produces. - The native planner tests from #6181 that compare the empty and nonempty scan schemas, unchanged. - A new `CometExecSuite` test that reads a partitioned table with eight buckets and two rows through a native projection. #6181's own Scala test needs TopK fusion, so this is a reduced form of it. - One comment in the hunk is reworded, since this branch has no fused operators. ## How are these changes tested? Without the fix, three of the four native tests fail, and the new Scala test fails with the `out of bound` error on Spark 3.5 and 4.1. With it they pass, as does the full `CometExecSuite` on Spark 4.1. Clippy and rustfmt are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
