HyukjinKwon opened a new pull request, #303: URL: https://github.com/apache/spark-connect-rust/pull/303
### What changes were proposed in this pull request? [SPARK-61103](https://issues.apache.org/jira/browse/SPARK-61103): `spark.createDataFrame` now takes a numpy array as PySpark Connect's does. As PySpark Connect's, an array of 1 or 2 dimensions becomes an Arrow table with a column for each column of the array. The table is sent with the schema, if there is one, and the server casts to it. The columns are named: - by a list or tuple of names, if one is given - `value`, for a 1-dimensional array or an array of one column - `_1`, `_2`, ... otherwise It raises: - `INVALID_NDARRAY_DIMENSION` for an array of another dimension - `AXIS_LENGTH_MISMATCH` when the names, or a schema, have another number of columns An empty array is empty data, as an empty list is. The Arrow table is built by `pa.array` of each column, as in PySpark. The values, the types and the errors of pyarrow and the server are therefore the same. Examples are datetimes of `datetime64[us]` as `timestamp_ntz`, `UNSUPPORTED_ARROWTYPE` for `uint64`, and `CAST_OVERFLOW` when the server casts to a schema. ### Why are the changes needed? The client took a numpy array as rows of Python values, whose types it could not infer. `spark.createDataFrame(np.array([1, 2]))` raised `CANNOT_INFER_TYPE_FOR_FIELD`, and so did an array of 3 dimensions. ### Does this PR introduce _any_ user-facing change? Yes. `createDataFrame` of a numpy array creates the DataFrame that PySpark 4.3.0 creates. ### How was this patch tested? - A new live test checks the schema and the rows of a 1- and a 2-dimensional array, of names and of a DDL schema, and the errors. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks the Arrow data and the schema sent for each way of naming the columns, and the errors. - On a local server, 65 cases give the same results as official PySpark 4.3.0. They cover: - arrays of 0 to 3 dimensions and of each dtype, including object, masked and datetime arrays - names and struct, DDL and atomic schemas - empty arrays - The lib tests of both crates (588 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2051 passed, 12 skipped), and the live drop-in and Arrow suites (155 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
