luc-pimentel commented on code in PR #59326:
URL: https://github.com/apache/spark/pull/59326#discussion_r4237990630
##########
python/pyspark/sql/pandas/conversion.py:
##########
@@ -58,14 +58,33 @@
from pyspark.sql.pandas._typing import DataFrameLike as PandasDataFrameLike
+def _create_array_from_pandas(
+ series: "pd.Series",
+ arrow_type: Optional["pa.DataType"],
+ *,
+ mask: Optional["pd.Series"],
+ safecheck: bool,
+) -> Union["pa.Array", "pa.ChunkedArray"]:
+ import pyarrow as pa
+
+ if mask is None:
Review Comment:
Good catch, thanks. The branch now applies only to
`pd.arrays.ArrowExtensionArray`, so `Int64`, `Float64`, `boolean` and
`string[python]` keep the typed `from_pandas` call and mismatches raise as
before. Added `test_nullable_dtype_type_mismatch` and
`test_nullable_dtype_requested_type`, both going through
`PandasToArrowConversion.from_pandas` and `create_arrow_array_from_pandas`.
The mismatch test fails on the previous head and passes now. The full
conversion module (56 tests) passes on PyArrow 18.0.0 and 19.0.0 with pandas
2.2.3, on 25.0.1 with pandas 2.3.3, and on 25.0.1 with pandas 3.0.6. The six
local JVM checks pass on all three Arrow versions.
---
Generated-by: Claude Code 2.1.296 (claude-fable-5-1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
