ueshin commented on a change in pull request #24177: Use pandas DataFrame for
struct type argument in Scalar Pandas UDF.
URL: https://github.com/apache/spark/pull/24177#discussion_r268061647
##########
File path: python/pyspark/sql/types.py
##########
@@ -1674,6 +1674,16 @@ def from_arrow_type(at):
if types.is_timestamp(at.value_type):
raise TypeError("Unsupported type in conversion from Arrow: " +
str(at))
spark_type = ArrayType(from_arrow_type(at.value_type))
+ elif types.is_struct(at):
+ # TODO: remove version check once minimum pyarrow version is 0.11.0
+ if LooseVersion(pa.__version__) < LooseVersion("0.11.0"):
+ raise TypeError("Unsupported type in conversion from Arrow: " +
str(at) +
+ "\nPlease install pyarrow >= 0.11.0 for StructType
support.")
Review comment:
Currently only support for pyarrow >=0.11 since I couldn't find a way to
reconstruct pandas DataFrame from `pyarrow.lib.StructArray`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
