luc-pimentel commented on code in PR #59326:
URL: https://github.com/apache/spark/pull/59326#discussion_r4239342359
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -58,14 +58,34 @@
from pyspark.sql.pandas._typing import DataFrameLike as PandasDataFrameLike
+def _create_array_from_pandas(
+ series: "pd.Series",
+ arrow_type: Optional["pa.DataType"],
+ *,
+ mask: Optional["pd.Series"],
+ safecheck: bool,
+) -> Union["pa.Array", "pa.ChunkedArray"]:
+ import pandas as pd
+ import pyarrow as pa
+
+ if isinstance(series.array, pd.arrays.ArrowExtensionArray):
+ # pyarrow-backed arrays can ignore the requested type or force a safe
cast.
+ # Extract first, then cast explicitly. Other dtypes already honor the
type.
+ result = pa.Array.from_pandas(series, safe=safecheck)
+ if arrow_type is not None and result.type != arrow_type:
+ result = result.cast(arrow_type, safe=safecheck)
Review Comment:
Confirmed the shift here too: a tz-naive `pd.ArrowDtype` timestamp under
`America/Sao_Paulo` comes out three hours early while the NumPy series is
right. The localization you describe is what #58765 (SPARK-59467) adds in
`_check_series_convert_timestamps_internal`, including the `ambiguous`
handling, so I'd rather not duplicate it here. Once that lands, the cast in
this helper only relabels the timezone and the instants line up. I'll keep this
PR to the type and safecheck handling.
---
Generated-by: Claude Code 2.1.296 (claude-fable-5-1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
