zhengruifeng commented on code in PR #54143:
URL: https://github.com/apache/spark/pull/54143#discussion_r2785700422
##########
python/pyspark/sql/conversion.py:
##########
@@ -1350,8 +1374,17 @@ def _prefer_convert_numpy(
)
if df_for_struct and isinstance(spark_type, StructType):
return all(isinstance(f.dataType, supported_types) for f in
spark_type.fields)
+ elif isinstance(spark_type, supported_types):
+ return True
+ elif isinstance(spark_type, (ArrayType, MapType, StructType)):
+ # Complex types can use convert_numpy if they don't need
post-processing
+ if ndarray_as_list:
+ return False # PyArrow doesn't support ndarray_as_list
natively
+ if struct_in_pandas == "row":
+ return False # PyArrow doesn't support Row conversion natively
Review Comment:
we should eventually support all conditions in `convert_numpy`
of course, we can start with partial support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
