[
https://issues.apache.org/jira/browse/ARROW-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16696147#comment-16696147
]
Florian Wilhelm edited comment on ARROW-3850 at 11/22/18 5:36 PM:
------------------------------------------------------------------
According to the [PySpark 2.4 docs|http://example.com/] and
[here|https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html#supported-sql-types]
it's not supported. But to be more precise, I think only nested StructType is
not supported since the Pandas DataFrame itself seems to be converted into a
StructType which works. Only if a column itself is of type StructType it is
unsupported.
was (Author: florianwilhelm):
According to the [PySpark 2.4 docs|http://example.com/] and
[here|https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html#supported-sql-types]
it's not supported. But to more precise I think only nested StructType is not
supported since the Pandas DataFrame itself seems to be converted into a
StructType which works. Only if a column itself is of type StructType it is
unsupported.
> [Python] Support MapType and StructType for enhanced PySpark integration
> ------------------------------------------------------------------------
>
> Key: ARROW-3850
> URL: https://issues.apache.org/jira/browse/ARROW-3850
> Project: Apache Arrow
> Issue Type: New Feature
> Affects Versions: 0.11.1
> Reporter: Florian Wilhelm
> Priority: Major
> Fix For: 0.13.0
>
>
> It would be great to support MapType and StructType in Arrow so that PySpark
> can make use of it.
>
> Quite often as in my use-case in Hive table cells are also complex types
> saved. Currently it's not possible to user the new
> {{[pandas_udf|https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=explode#pyspark.sql.functions.pandas_udf]}}
> decorator which internally uses Arrow to generate a UDF an columns with
> complex types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
