HyukjinKwon opened a new pull request, #308: URL: https://github.com/apache/spark-connect-rust/pull/308
### What changes were proposed in this pull request? [SPARK-61104](https://issues.apache.org/jira/browse/SPARK-61104): `spark.createDataFrame` of a pandas DataFrame without a schema, or with a list of names, now converts it as PySpark Connect does. This follows [SPARK-61079](https://issues.apache.org/jira/browse/SPARK-61079), which did the same for an explicit schema. Each column goes through the vendored official `create_arrow_table_from_pandas`, with the type that Arrow infers. The exceptions are datetimes, which become timestamps in the session time zone, and timedeltas, which become day-time intervals. - The table is sent without a schema, so the server infers its types from Arrow's. - `toDF` names the columns: by their labels, as strings, or by the given names, with `_n` for any columns past them. - With `spark.sql.execution.pandas.inferPandasDictAsMap`, a column of dicts is a map; otherwise it is a struct. This is `create_from_pandas_inferred` in `crates/pyspark-rs/src/session.rs`, and `prepare_inferred_pandas_table` in `python/pyspark/sql/_pandas_input.py`. The fast path for int64 columns still runs first. A customized frame, and any frame when pyarrow is not installed, keep the row path, as with an explicit schema. The check of the frame is split out of `accepts_pandas_table` as `accepts_pandas_frame`. It reuses `pyarrow_import::table_batches` of [SPARK-61103](https://issues.apache.org/jira/browse/SPARK-61103). ### Why are the changes needed? The client converted such a frame row by row, typing each column by its first value. On a local Spark 4.3.0 server, with a client in Asia/Seoul and a session in UTC, 32 of 92 cases of pandas input differed from PySpark 4.3.0: | Input | master | This change and PySpark 4.3.0 | | --- | --- | --- | | an `int8`, `int16` or `int32` column | `bigint` | `tinyint`, `smallint`, `int` | | a `float32` column | `double` | `float` | | a column of dicts | `map<string,bigint>` | `struct<k:bigint>` | | dicts of other keys | `PySparkRuntimeError` | a struct of all the keys | | a naive `2020-01-01 01:02:03` | collects as 01:02:03 | collects as 10:02:03 (01:02:03 UTC) | | an int and a float in an object column | `PySparkRuntimeError: Type mismatch in row data` | a `double` | | an int and a str in a column | `PySparkRuntimeError: Type mismatch in row data` | `PySparkValueError` of the Arrow conversion | | a timestamp with nanoseconds | truncated silently | `PySparkValueError` of the Arrow conversion | | a column of sets, periods or intervals | `TypeError` | an array, a bigint, a struct | | a `uint8`, `uint64` or `float16` column | a bigint or a double | `UNSUPPORTED_ARROWTYPE` | | a sparse column | a bigint | `PySparkTypeError` of the Arrow conversion | ### Does this PR introduce _any_ user-facing change? Yes. pandas input without a schema converts, or fails, as in PySpark 4.3.0. ### How was this patch tested? - A new live test checks the types of an `int8`, a dict and a `float32` column, and the time of a naive datetime. It passes with official PySpark 4.3.0 and fails without this change. - New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow input service, check: - the Arrow table and its names for no schema, names, and fewer names - `inferPandasDictAsMap` - the session time zone for two client time zones - the errors - Existing tests that pinned the row path for a plain frame without a schema now expect PySpark's behavior: - the IPC column names `_0`, `_1`, ..., named by `toDF`, and no schema on the relation - the configuration request first - the errors of the Arrow conversion - Four tests created such frames on a session with no server, which the configuration request now needs. They now use the mock Connect server. - The tests of the row path's converter (`py_to_value`) now build a frame of a `pd.DataFrame` subclass, which keeps that path, so they still cover it. - The mock servers now serve `spark.sql.execution.pandas.inferPandasDictAsMap`, as a real server does. - On a local server, 92 cases of pandas input without a schema give the same results as official PySpark 4.3.0. They cover each dtype, object columns of each kind of value, time zones, column labels and names, empty frames, large frames and the Arrow settings. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2068 passed, 8 skipped), and the live drop-in and Arrow suites (162 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
