milenkovicm opened a new issue, #2618: URL: https://github.com/apache/datafusion-ballista/issues/2618
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The Ballista Python client wraps DataFusion Python: it subclasses `SessionContext` and `DataFrame`. Every method that executes a plan is intercepted, the logical plan is serialized, and the plan is resubmitted through a separate Ballista-backed session. As a result: - methods the wrapper doesn't know about run locally instead of on the cluster; - every execution creates a new Ballista session; - the wrapper has to be kept in line with DataFusion Python's API, which is an ongoing maintenance cost; - users get a `DistributedDataFrame` instead of a regular `datafusion.DataFrame`. **Describe the solution you'd like** DataFusion Python 55 can install a custom query planner and extension codecs through FFI. Ballista can use this to plug into a standard `datafusion.SessionContext` with no wrapper: DataFusion plans each query as usual, and the Ballista planner sends the plan to the scheduler. Any operation that executes a plan (`collect`, `show`, `to_pandas`, `write_parquet`, `EXPLAIN ANALYZE`, ...) then runs on the cluster. Tasks: - [x] Replace the wrapper-based client with a native FFI query planner and extension codecs (`BallistaSessionContext` returns a plain `datafusion.SessionContext`, `DistributedDataFrame` is removed, Ballista settings are registered as a DataFusion config extension, upgrade guide added) #2588 - [ ] Python UDF support (scalar, aggregate, window). Python UDFs can't be serialized yet, because no extension codec handles them. - [ ] Refactor configuration so it is more ergonomic for Ballista use cases. Settings are currently split between `config` (local planning only) and `cluster_config` (sent to the scheduler), and the table of DataFusion options Ballista restricts is duplicated between Rust and Python. The goal is one clear way to configure a Ballista context, with a single source of truth for restricted options. **Describe alternatives you've considered** Keep the wrapper-based client. That leaves the problems above in place, and every DataFusion Python release needs matching wrapper changes. **Additional context** This is a breaking change for the Python client: - `DistributedDataFrame` is removed; - `BallistaSessionContext` becomes a factory function, not a class; - `explain_visual` and `collect_with_progress` move to `ballista.jupyter`. The upgrade guide covers these in detail. Known limitations, to be tracked separately: - In-memory table providers (`from_pydict`, `register_record_batches`, ...) can't be serialized. - `information_schema` scans can't cross the FFI planner boundary. - Executors don't apply `file_compression_type` when reading compressed CSV. - The CSV writer quotes every field when writing a header. - `COPY ... OPTIONS` are dropped by datafusion-proto (upstream). - `datafusion-ffi` 55.2.0 doesn't support file formats in `ForeignLogicalExtensionCodec`. The client works around this for `DataFrame.write_*`, and the workaround should be removed once it's fixed upstream. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
