ueshin commented on code in PR #52140:
URL: https://github.com/apache/spark/pull/52140#discussion_r2330962366
##########
python/pyspark/sql/pandas/serializers.py:
##########
@@ -227,6 +227,68 @@ def load_stream(self, stream):
result_batches.append(batch.column(i))
yield result_batches
+ def _create_array(self, arr, arrow_type):
+ import pyarrow as pa
+
+ assert isinstance(arr, pa.Array)
+ assert isinstance(arrow_type, pa.DataType)
+ if arr.type == arrow_type:
+ return arr
+ else:
+ try:
+ # when safe is True, the cast will fail if there's a overflow
or other
+ # unsafe conversion.
+ # RecordBatch.cast(...) isn't used as minimum PyArrow version
+ # required for RecordBatch.cast(...) is v21.0.0
Review Comment:
RecordBatch.cast is available since 16.0.
https://arrow.apache.org/docs/16.0/python/generated/pyarrow.RecordBatch.html#pyarrow.RecordBatch.cast
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
