comphead opened a new issue, #7619: URL: https://github.com/apache/datafusion-comet/issues/7619
### What is the problem the feature request solves? apache/datafusion#27337 (merged 2026-11-12) moved the `datafusion-spark` crate out of apache/datafusion into its own repository, [apache/datafusion-spark](https://github.com/apache/datafusion-spark), as agreed in apache/datafusion-spark#87. The crate name and version numbers stay the same. 56.x is still released from `branch-56` in apache/datafusion, and 57.0.0 will be the first release from the new repository, after DataFusion 57.0.0 (apache/datafusion-spark#56). The Spark-related issues were transferred there too. Comet uses `datafusion-spark` 55.2.0 from crates.io, so the build on `main` is unaffected. But some parts of Comet still assume the crate lives in apache/datafusion: - The roadmap, the `datafusion-comet-spark-expr` README, the top-level project proposal and the `implement-comet-expression` skill say it does. - The `wire-datafusion-function` skill looks for its sources under `datafusion/spark/src/function/` in a DataFusion clone. At DataFusion 57 or later that path does not exist, so the lookup silently finds nothing. - Three links point at moved issues: apache/datafusion#25913 (now apache/datafusion-spark#32) and apache/datafusion#25981 (now apache/datafusion-spark#34). GitHub redirects them, but the numbers changed. - From DataFusion 57, Comet can no longer take `datafusion-spark` from the same DataFusion revision as the other DataFusion crates. ### Describe the potential solution - [ ] Point the docs, skills and moved-issue links at apache/datafusion-spark. A PR follows. - [ ] DataFusion 56 (#7249): nothing to change. It pins `datafusion-spark` to the `56.0.0-rc1` tag in apache/datafusion, which is still where 56.x is released. A fix Comet needs in 56.x has to be backported to `branch-56` there. - [ ] DataFusion 57: take `datafusion-spark` from apache/datafusion-spark. That repository depends on DataFusion `branch-56` through git, so pairing it with another DataFusion revision builds two copies of DataFusion. #7618 (draft, DataFusion `main`) handles this with a `[patch]` section that points its DataFusion crates at Comet's revision through the `apache/arrow-datafusion` URL, because Cargo rejects a patch with the same URL. The patch can go once apache/datafusion-spark has a branch that builds against the same DataFusion revision (still open in apache/datafusion-spark#56), or once both are released. - [ ] The DataFusion 57 upgrade can switch to released crates only after `datafusion-spark` 57.0.0 is released too, which has its own vote after DataFusion's. - [ ] Describe the two items above in the DataFusion upgrade guide (#7250). ### Additional context - When Comet moves to `datafusion-spark` 57.0.0, recheck the workarounds for bugs that were fixed in the new repository. For example, apache/datafusion-spark#83 changed `array_contains` to compare floats the way Spark does, which Comet handles in `spark_array_contains` (#6520). - #6789 asks whether `datafusion-spark` should move to Comet. It moved to its own repository under the DataFusion PMC instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
