andygrove opened a new issue, #27103: URL: https://github.com/apache/datafusion/issues/27103
At the DataFusion community call on 2026-11-04 there was renewed interest in `datafusion-spark`. This issue lists the challenges and some options for making it reliably Spark-compatible. ## Challenges 1. **Development has stalled.** In my opinion, most DataFusion reviewers aren't interested in Spark functionality, and the people who are, in Comet and Sail, rarely review or contribute because of the challenges below. Since November 2025, 79% of `spark` PRs came from outside the Comet and Sail regulars. Comet regulars reviewed 89 of the 347 PRs, 56 of them by one reviewer, and Sail contributors reviewed 5. Comet runs about 34 Spark expressions on this crate and about 96 on its own code, including at least 21 functions this crate also implements. It would like to upstream more, such as `CAST` (#11201), but that isn't practical without real tests that ensure Spark compatibility. 2. **We don't test against Spark.** Nothing checks the expected results in `test_files/spark/*.slt` against Spark, and reviewers can't verify them without running Spark. Audits against a live Spark (#23893) keep finding divergences, including tests that assert the wrong value (#26333, #26338). I raised this risk when the crate moved here (#5600). 3. **Spark versions differ.** Behavior changes between Spark releases (ANSI mode became the default in 4.0, for example), but the crate has no notion of a Spark version (#21698) and an `.slt` query can only have one expected result (#23887). Comet supports five Spark versions (3.4 to 4.2), and even Comet runs only one of them before merge, the rest nightly or on demand. 4. **Two kinds of users, one test path.** Comet builds physical plans directly, so it needs `invoke_with_args()`. Projects that use DataFusion's planner and optimizer can use `simplify()` rewrites instead. SLT always goes through the planner and optimizer, so it tests the rewrite rather than the physical path, and functions with no physical implementation pass every test (#21959). ## Options 1. **Run Spark in this repo's CI.** #21508 was an earlier attempt. This catches divergences before merge, but a DataFusion PR could then fail a Spark test its author can't debug, and it adds a JVM and a Spark version matrix to CI whose usage we are already trying to reduce (#25148). 2. **Generate expected results with Spark and check them in per Spark version.** A script runs the `.slt` queries on each supported Spark version and records the results, and CI runs only DataFusion against them. Nobody needs Spark to debug a failure, since it is an ordinary SLT diff against a value Spark produced. Whoever adds a test has to run the script (or ask a bot to), the queries must be valid Spark SQL (DataFusion now has a Spark dialect, #22529), and passing on more than one Spark version needs #21698. 3. **Move the crate to a new `apache/datafusion-spark` repository** under the DataFusion PMC, like Comet, Ballista and `datafusion-iceberg`. Its CI can run real Spark across versions, and its contributors and reviewers would be people who can debug Spark. The pieces of core that only this crate uses would move with it: `enable_ansi_mode`, `datafusion.spark.*`, `datafusion-cli --spark` and the Spark handling in the sqllogictest runner. These can be combined, and option 2 works in either repository. Each option also needs a way to test the physical path, for example by running the Spark `.slt` files a second time with `simplify()` disabled. ## Impact on downstream projects [Comet](https://github.com/apache/datafusion-comet) and [Sail](https://github.com/lakehq/sail) depend on `datafusion-spark` (55.2.0), which today ships with each DataFusion release. Options 1 and 2 keep that. Option 3 would mean: - A separate release, with its own vote, after each DataFusion release before Comet and Sail can upgrade. In #5600 Sail said waiting on out-of-tree crates slowed its upgrades, but that this goes away if Comet and Sail maintainers help keep the crate in sync. - DataFusion API changes absorbed when the new repo upgrades, as Comet does today, rather than in the same PR. - Public APIs for code shared with `datafusion-functions` (#17964). Which options, or combination, should we pursue? Are there others? cc @alamb @comphead @mbutrovich @shehabgamin @linhr @Jefffrey @coderfender @xanderbailey -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
