HyukjinKwon commented on code in PR #59326:
URL: https://github.com/apache/spark/pull/59326#discussion_r4238550089
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -58,14 +58,34 @@
from pyspark.sql.pandas._typing import DataFrameLike as PandasDataFrameLike
+def _create_array_from_pandas(
+ series: "pd.Series",
+ arrow_type: Optional["pa.DataType"],
+ *,
+ mask: Optional["pd.Series"],
+ safecheck: bool,
+) -> Union["pa.Array", "pa.ChunkedArray"]:
+ import pandas as pd
+ import pyarrow as pa
+
+ if isinstance(series.array, pd.arrays.ArrowExtensionArray):
+ # pyarrow-backed arrays can ignore the requested type or force a safe
cast.
+ # Extract first, then cast explicitly. Other dtypes already honor the
type.
+ result = pa.Array.from_pandas(series, safe=safecheck)
+ if arrow_type is not None and result.type != arrow_type:
+ result = result.cast(arrow_type, safe=safecheck)
Review Comment:
**Non-blocking (P2):** Take a tz-naive `pd.ArrowDtype` timestamp series and
a `TimestampType` field. This line casts `timestamp[ns]` to `timestamp('us',
tz=<session tz>)` by changing only the metadata, so Arrow reads the wall-clock
values as UTC. NumPy `datetime64` series get localized to the session timezone
first, in `_check_series_convert_timestamps_internal`. That function only
checks `is_datetime64_dtype` and `DatetimeTZDtype`, and both are False for
`ArrowDtype`.
Example with session timezone `America/Los_Angeles`:
`pd.Series(pd.to_datetime(['2020-01-01 10:00']),
dtype=pd.ArrowDtype(pa.timestamp('ns')))` becomes 1577872800000000 (10:00 UTC).
The same NumPy series becomes 1577901600000000 (18:00 UTC). I reproduced this
with the helper's calls on pyarrow 25.0.1 / pandas 3.0.6.
PyArrow >= 19 already gave the same result through its protocol cast before
this PR. On PyArrow 18, the supported minimum, the requested type used to be
ignored and the `timestamp[ns]` column failed downstream. Classic
`createDataFrame` and pandas UDF/UDTF results now succeed with the shifted
value instead. None of the new tests uses a timestamp target. tz-aware Arrow
timestamps and `TimestampNTZType` targets are not affected.
**Recommended change:** In the shared pandas timestamp converter, localize
tz-naive Arrow-backed timestamp series to the session (or local) timezone and
normalize them to UTC, using the same DST-ambiguity choice (standard time) and
nonexistent-time handling as the NumPy path. Leave tz-aware Arrow-backed
timestamps unchanged. Add PySpark conversion tests for Arrow-backed
TimestampType targets, and extend the PySpark migration note to cover this
behavior.
**Why this works:** Once the converter emits tz-aware UTC instants for
Arrow-backed naive input, the cast in `_create_array_from_pandas` from that
tz-aware source to `timestamp('us', tz=<session>)` only changes the timezone
label and preserves the instant. Arrow-backed and NumPy-backed inputs then
produce the same values on every PyArrow version and entry point.
**Scope:** Localize tz-naive Arrow-backed timestamp columns in the shared
pandas timestamp converter, so the shared Arrow constructor's cast preserves
instants for every pandas backing and PyArrow version.
**Compatibility:** NumPy-backed datetime64 and DatetimeTZDtype localization,
including the standard-time DST choice, stays unchanged.
**Risks:** On PyArrow >= 19 the merge target already returned shifted values
for this input, so the fix changes observable results there too. The migration
note must say so. pandas' ArrowDtype `.dt.tz_localize` does not accept
`ambiguous=False`, so the implementation must reproduce the standard-time
choice for ambiguous DST times another way.
**Constraints:** Do not change NumPy-backed `datetime64` or
`DatetimeTZDtype` handling. Keep the timestamp resolution, and do not overflow
dates outside the nanosecond range for microsecond-unit Arrow inputs.
**Success:** With a non-UTC session timezone, a tz-naive `pd.ArrowDtype`
timestamp series converted for TimestampType through
`create_arrow_array_from_pandas` and through
`PandasToArrowConversion.from_pandas` (both error-handling modes) yields the
same UTC instants as the equivalent NumPy `datetime64` series, on every
supported PyArrow version. tz-aware Arrow-backed timestamps keep their
instants, TimestampNTZType targets keep their wall-clock values, and nulls are
preserved.
**Verification:**
- **Regression:** A tz-naive Arrow-backed timestamp series converted for
TimestampType under a non-UTC session timezone produces the same epoch values
as the equivalent NumPy-backed series, through both the createDataFrame helper
and the pandas UDF output conversion.
- **Behavior:** tz-aware Arrow-backed timestamps keep their instants, and
TimestampNTZType targets keep their wall-clock values, including nulls.
- **Compatibility:** The existing NumPy-backed timestamp conversion and
SPARK-60009 requested-type, safety, offset-width, nullable-dtype, and
legacy-path conversion checks still pass.
##########
python/pyspark/sql/conversion.py:
##########
@@ -543,7 +544,9 @@ def _convert_column_legacy(
# trigger the cast fallback when arrow_cast is enabled.
try:
try:
- return pa.Array.from_pandas(series, mask=mask,
type=arrow_type, safe=safecheck)
+ return _create_array_from_pandas(
Review Comment:
Thanks, confirmed.
`test_from_pandas_arrow_dtype_requested_type_legacy_error_handling` runs the
`use_legacy_error_handling=True, arrow_cast=True` branch with a
`pd.ArrowDtype(pa.int32())` series and asserts the `int64` result type. So if
that call site went back to the raw `pa.Array.from_pandas(..., type=...)` call,
the test would fail on the PyArrow 18 job.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"inline:4238252067","thread_id":"inline:4238252067","verdict_sha256":"b5a8c6db037983c56343f6fda30b5b312ccbc91ff9dc23933c51b7143516b533"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
