milenkovicm commented on issue #1612:
URL:
https://github.com/apache/datafusion-python/issues/1612#issuecomment-4893067192
> * From an API and packaging perspective, end users may end up with a more
fragmented experience: different APIs inconsistent with their Rust
counterparts, managing multiple Python packages (each with overlapping or
duplicated wheels), navigating documentation spread across several projects,
and manually dealing with version compatibility between those packages.
I'm not sure I understand your concern, with option 2 you'd just need to
provide a factory method which would configure `SessionContext` to work with
df-distributed / ballista, all other behaviour would be the same like df-python,
> To me, this second point is the one that should carry the most weight when
evaluating Option 1 vs. Option 2, hence the reason we I think it's important to
start constructing from end-user experience first.
from user perspective it does change much, for example
```python
from datafusion import col, lit
from datafusion import DataFrame
# we do not need datafusion context
# it will be replaced by BallistaSessionContext
# from datafusion import SessionContext
from ballista import BallistaSessionContext
# Change from:
#
# ctx = SessionContext()
#
# to:
ctx = BallistaSessionContext("df://localhost:50050")
# all other functions and functions are from
# datafusion module
ctx.sql("create external table t stored as parquet location
'./testdata/test.parquet'")
df : DataFrame = ctx.sql("select * from t limit 5")
df.show()
```
ballista support is one line change, all the api is the same, all underlying
interaction is with df-python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
