BryanCutler commented on a change in pull request #23900:
URL: https://github.com/apache/spark/pull/23900#discussion_r426887061
##########
File path: python/pyspark/sql/types.py
##########
@@ -1616,6 +1616,12 @@ def to_arrow_type(dt):
if type(dt.elementType) == TimestampType:
raise TypeError("Unsupported type in conversion to Arrow: " +
str(dt))
arrow_type = pa.list_(to_arrow_type(dt.elementType))
+ elif type(dt) == StructType:
+ if any(type(field.dataType) == StructType for field in dt):
+ raise TypeError("Nested StructType not supported in conversion to
Arrow")
Review comment:
@cfmcgrady support wasn't removed, it was never allowed to have
`ArrayType(elementType = StructType)` - I don't think there was an explicit
check before this. It might be possible to add this in the future, but it's a
little tricky to represent this in Pandas efficiently, I believe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
