milenkovicm opened a new pull request, #2832:
URL: https://github.com/apache/datafusion-ballista/pull/2832
# Which issue does this PR close?
Closes #.
# Rationale for this change
The Python package has two entry points that do the same job:
`BallistaSessionContext`, which uses the native Rust client, and
`BallistaPythonSessionContext`, which uses the Python `BallistaClient`. Users
have to pick one, even though the only real difference is which client talks to
the cluster. Planning, configuration and the returned
`datafusion.SessionContext` are the same for both.
As at the moment we are not sure which implementation will be be kept for
future this PR leaves one entry point, `BallistaSessionContext`. The client is
chosen by whether a `client` argument is passed.
# What changes are included in this PR?
- `BallistaSessionContext` has a new optional `client: BallistaClient`
argument.
- Without `client`, it uses the native Rust client, same as before.
- With `client`, the Python client handles job submission, status tracking
and Flight result fetching. `address` is ignored in this case, because the
client has its own address.
- The previous `BallistaSessionContext` is renamed to
`BallistaRustSessionContext` in `ballista/extension.py`. Both it and
`BallistaPythonSessionContext` in `ballista/client.py` are now internal and are
not exported from `ballista`.
- `BallistaPythonSessionContext` now requires `client`. It no longer builds
a default client from `address` and `config`.
- The deprecated `cluster_config` argument is handled in
`BallistaSessionContext`, so it works with both clients and the deprecation
warning is attributed to the caller.
- `ballista.client` is imported only when `client` is passed, so `import
ballista` and the native path still work without the `python-client` extra.
- Tests:
- The `ctx` fixture in `test_context.py` runs every context test against
both clients through `BallistaSessionContext`.
- `test_client.py` uses the unified context.
- New tests check that the native path is used when no client is given,
that `address` is ignored when a client is given, and that `cluster_config`
works together with a client.
- Docs: the 56.0.0 upgrade guide, the extras table in the Python quickstart,
the README (new "Python Cluster Client" section), `examples/python_client.py`
and the comment on the `pyproject.toml` extra.
# Are there any user-facing changes?
Yes, in the Python API:
- `BallistaSessionContext(..., client=BallistaClient(...))` replaces
`BallistaPythonSessionContext(client=...)`:
```python
from ballista import BallistaClient, BallistaSessionContext
ctx = BallistaSessionContext(client=BallistaClient("df://localhost:50050"))
```
- `BallistaPythonSessionContext` is no longer exported. It has not been
released yet (it was added after 55.0.0 and only appears in the unreleased
56.0.0 upgrade guide), so this does not break any released API. The 56.0.0
upgrade guide is updated to match.
- `ballista.*` options in `config` no longer configure a Python client
implicitly. They still reach the scheduler with each query. Client-level
options, such as `ballista.client.pull`, go in the `BallistaClient` `config`.
- Existing `BallistaSessionContext` calls without `client` behave exactly as
before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
