HyukjinKwon opened a new pull request, #254: URL: https://github.com/apache/spark-connect-rust/pull/254
### What changes were proposed in this pull request? `createDataFrame` of local rows converts as PySpark 4.3.0 does in three cases where it differed: - An array field takes any sequence, NumPy arrays included, when its elements need no converter, as `LocalDataToArrowConversion._need_converter` decides (`crates/pyspark-rs/src/infer.rs`). Arrays of strings or binaries, which need one, still require a list or `array.array`. - A date field takes an int, a NumPy int, bool or float as days since the epoch, as pyarrow does, with pyarrow's errors for a Python bool, out-of-range values and other objects. - The Arrow schema of a nested struct numbers duplicate field names (`a, a` becomes `a_0, a_1`), as PySpark's `_dedup_names` does, while the relation's schema keeps the names (`crates/spark-connect/src/session.rs`). ### Why are the changes needed? [SPARK-61063](https://issues.apache.org/jira/browse/SPARK-61063): against a local Spark 4.3.0 server: | Input | Current master | This change and PySpark 4.3.0 | | --- | --- | --- | | `[(np.array([1, 2]),)]` into `array<bigint>` (also `array<double>`, inside a struct) | `AssertionError` | `[Row(a=[1, 2])]` | | `[(1,), (-1,)]` into `date` | `ValueError: Could not convert 1 with type int` | `1970-01-02`, `1969-12-31` | | `[(Row("a", "a")(1, 2),)]` | the server cannot read the IPC | `[Row(_1=Row(a=1, a=2))]` | ### Does this PR introduce _any_ user-facing change? Yes, these inputs convert as in PySpark. ### How was this patch tested? - New offline tests in `python/tests/test_dropin_offline.py` that check the IPC of each case against a mock server, a live test in `python/tests/test_dropin_e2e.py` for the nested duplicate names, and a unit test of the name numbering. - The offline suite (930 passed, 8 skipped), and the lib tests of `apache-spark-connect`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
