[
https://issues.apache.org/jira/browse/ARROW-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957745#comment-16957745
]
Joris Van den Bossche edited comment on ARROW-6958 at 10/23/19 10:43 AM:
-------------------------------------------------------------------------
pyspark is not yet compatible with the latest pyarrow 0.15.0 version, see eg
https://stackoverflow.com/questions/58273063/pandasudf-and-pyarrow-0-15-0 for
an explanation and how to solve it. I suppose you are encountering the same
issue.
was (Author: jorisvandenbossche):
pyspark is not yet compatible with the latest pyarrow 0.15.0 version, see eg
https://stackoverflow.com/questions/58273063/pandasudf-and-pyarrow-0-15-0 for
an explanation
> [python] tutorial script for arrow in spark throws error
> --------------------------------------------------------
>
> Key: ARROW-6958
> URL: https://issues.apache.org/jira/browse/ARROW-6958
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java, Python
> Affects Versions: 0.15.0
> Environment: Ubuntu v 18. Cluster spun up on google dataproc - see
> startup for specs of cluster
> Reporter: Karl Svensson
> Priority: Major
> Labels: newbie
> Fix For: 0.8.0
>
> Attachments: arrow_error.txt, start-cluster-nl.ps1.txt
>
>
> Running the arrow example for pyspark ([found here
> |[https://github.com/apache/spark/blob/master/examples/src/main/python/sql/arrow.py]])
> causes a java.lang.IllegalArgumentException error. Running the same script
> with pyarrow v 0.8.0 causes the script to run correctly.
> Attached are the startup settings in google dataproc I'm using to create the
> cluster, as well as the output (with error text). It isn't immediately
> obvious to me what is causing the issue.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
