zhengruifeng commented on code in PR #53678:
URL: https://github.com/apache/spark/pull/53678#discussion_r2660911125
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -115,21 +111,15 @@ def to_arrow_type(
arrow_type = pa.float64()
elif type(dt) == DecimalType:
arrow_type = pa.decimal128(dt.precision, dt.scale)
- elif type(dt) == StringType and prefers_large_types:
- arrow_type = pa.large_string()
elif type(dt) == StringType:
- arrow_type = pa.string()
- elif type(dt) == BinaryType and prefers_large_types:
- arrow_type = pa.large_binary()
+ arrow_type = pa.large_string() if prefers_large_types else pa.string()
elif type(dt) == BinaryType:
- arrow_type = pa.binary()
+ arrow_type = pa.large_binary() if prefers_large_types else pa.binary()
elif type(dt) == DateType:
arrow_type = pa.date32()
- elif type(dt) == TimestampType and timestamp_utc:
- # Timestamps should be in UTC, JVM Arrow timestamps require a timezone
to be read
Review Comment:
And the JVM side does send the timestamps with timezone to the Python side,
see the test
https://github.com/apache/spark/blob/b4c84c917aee673d26162a92bd95afc743ea7684/python/pyspark/sql/tests/arrow/test_arrow_udf.py#L103-L160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
