HyukjinKwon opened a new pull request, #270: URL: https://github.com/apache/spark-connect-rust/pull/270
### What changes were proposed in this pull request? `createDataFrame` of a pandas DataFrame with an explicit schema (a `StructType` or a DDL string) now converts it as PySpark 4.3.0 does: with the vendored official `create_arrow_table_from_pandas` (`pyspark.sql.conversion`, unchanged), after reading the Arrow options, the session time zone and `spark.sql.execution.pandas.convertToArrowArraySafely` as PySpark does, and sends the table through the Arrow importer, which uploads it at the relation cache threshold (`create_from_pandas_arrow` in `crates/pyspark-rs/src/session.rs`, `prepare_pandas_table` in `python/pyspark/sql/_pandas_input.py`). Customized frames, arrays, dtypes and schema classes keep the row path (`accepts_pandas_table`), and the fast paths for int64, timestamp, float64, string and date columns run first as before. Without a schema, a Python int beyond int64 in an object column raises `OverflowError`, as Arrow does. ### Why are the changes needed? [SPARK-61079](https://issues.apache.org/jira/browse/SPARK-61079): the row conversion of pandas cells differed from PySpark's Arrow conversion. Against a local Spark 4.3.0 server: | Input | Current master | This change and PySpark 4.3.0 | | --- | --- | --- | | `{"f": [1, None]}` or `[1.0, nan]` into `f int` | `PySparkRuntimeError: Type mismatch in row data` | `[Row(f=1), Row(f=None)]` | | int64 `257` into `b tinyint` | `[Row(b=1)]`, silently | `PySparkValueError` (Arrow's overflow) | | a null in a non-nullable field | `PySparkRuntimeError: Failed to create Arrow batch` | `ValueError: Casting field 'a' with null values to non-nullable` | | naive `2020-07-01 12:00`, client `TZ=America/Los_Angeles`, session UTC | collects as 12:00 | collects as 05:00 (stored as 12:00 UTC); a DST gap raises as PySpark | | a timestamp with nanoseconds | truncated silently | `PySparkValueError` | | a lone surrogate into `string` | `TypeError` | `PySparkValueError` | | a `datetime` into `date` | `PySparkRuntimeError` | `date(2020, 1, 2)` | | NaN nested in arrays and maps | kept as NaN | `None` | | `2**64` in an inferred object column | a double | `OverflowError` | ### Does this PR introduce _any_ user-facing change? Yes, these inputs convert, or fail, as in PySpark. Explicit-schema pandas input reads its configs as PySpark does before converting. ### How was this patch tested? - 30 new tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow input service, for each case above, the time-zone variants and the DST gaps; existing tests that pinned the row path's errors or its config order now expect PySpark's (listed in the commits). - `test_arrow_handoff.py` (903 passed) and the pandas and createDataFrame tests of the offline suite; each case above and both time-zone variants give the same output as PySpark 4.3.0. Still different: a frame whose width differs from the schema raises `AXIS_LENGTH_MISMATCH`, where PySpark drops a column or raises `ArrowInvalid`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
