[
https://issues.apache.org/jira/browse/ARROW-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926207#comment-16926207
]
Bryan Cutler commented on ARROW-6474:
-------------------------------------
I think this is more for the case that a user is stuck with an already released
Spark version, e.g. <= 2.4.4, and ends up installing pyarrow >= 0.15.0. The
pyarrow writers will use the new format by default, which the Arrow Java
version in Spark will be unable to handle since it's using 0.14.1. There is no
way for the user to set the option in the pyarrow writer either, so they would
have to downgrade pyarrow. I think this it's fair to say they need to stick
with pyarrow 0.14.1, but an env variable would give them a way to use the
latest release.
> [Python] Provide mechanism for python to write out old format
> -------------------------------------------------------------
>
> Key: ARROW-6474
> URL: https://issues.apache.org/jira/browse/ARROW-6474
> Project: Apache Arrow
> Issue Type: Sub-task
> Reporter: Micah Kornfield
> Priority: Blocker
> Fix For: 0.15.0
>
>
> I think this needs to be an environment variable, so it can be made to work
> with old version of the Java library pyspark integration.
>
> [~bryanc] can you check if this captures the requirements?
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
