I've opened a PR to remove the experimental, off-by-default, Spark compatibility feature from Ballista. It provided minimal compatibility, covering only a handful of expressions.
https://github.com/apache/datafusion-ballista/pull/2566 Having spent 6+ years working on Spark-compatible engines, I can confidently say that it is a huge undertaking that never really finishes, because Spark is always evolving. I don't think this makes sense for the Ballista project. There are already Spark accelerators such as Apache DataFusion Comet, Apache Gluten, Apache Auron, Spark RAPIDS, etc. There is also LakeSail for a pure Rust Spark alternative. I think it would be better for the Ballista project to keep evolving on its own merits rather than trying to mimic Spark. I know this is a departure from some of my early thinking on the project. Please comment on the PR if you have opinions on this! Thank, Andy. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
