milenkovicm opened a new issue, #2781: URL: https://github.com/apache/datafusion-ballista/issues/2781
The Python client (`BallistaSessionContext`) runs most of the DataFusion Python API on a Ballista cluster, but some features still fail or behave differently there. Each gap below has a strict `xfail` test in `python/python/tests/test_context.py`. When a gap is fixed, remove its `xfail` marker in the same PR and tick it off here. - [ ] **Gzip CSV/JSON files are read as plain text.** `file_compression_type` isn't applied on the executors. Tests: `test_read_csv_gzip`, `test_read_json_gzip` - [ ] **In-memory tables fail** (`from_pydict`, `from_arrow`, `register_record_batches`). No extension codec can serialize in-memory table providers (`MemTable`). Tests: `test_from_pydict`, `test_from_arrow`, `test_register_record_batches` - [ ] **`DataFrame.cache()` fails.** `BallistaCacheFactory` isn't installed on Python sessions, so `cache()` falls back to an in-memory table, which can't be serialized. Test: `test_dataframe_cache` (both `cache_noop` values) - [ ] **`NOT IN` subqueries run as a single task.** datafusion-python can't install Ballista's `NOT IN` rewrite (a logical optimizer rule), so `NOT IN` reaches the scheduler as a null-aware anti join. Test: `test_not_in_subquery_avoids_null_aware_join` - [ ] **CSV writer quotes every field when `with_header=True`.** Test: `test_write_csv_header[True]` - [ ] **`COPY ... OPTIONS` has no effect.** datafusion-proto drops the `CopyTo` options. Test: `test_copy_to` - [ ] **`information_schema` queries fail.** `information_schema` scans can't be serialized across the FFI planner boundary, so Ballista settings can't be inspected through SQL. Test: `test_ballista_config_in_information_schema` To list the current gaps: ```bash cd python pytest python/tests/test_context.py -rx ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
