Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21939
>Do you mean we will upgrade arrow to 0.10.0 at java side, but leave the
python side as it is? So people can still use PySpark with pyarrow 0.8.0 and
python 3.4? If they go with arrow 0.10.0 and python 3.5, they can get these bug
fixes?
@cloud-fan , that is correct. This PR updates the Java artifact and since
we are not bumping up the minimum pyarrow version, there is nothing that needs
to be done in the python code. It would be best to have pyarrow 0.10.0 in our
CI, but @shaneknapp has run tests and I have also locally to be confident
enough that there are no issues using Arrow Java 0.10.0 with pyarrow 0.8.0 to
0.10.0. There were also no binary compatibility breaking changes in the Arrow
format made since 0.8.0.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
