HyukjinKwon opened a new pull request, #187: URL: https://github.com/apache/spark-connect-rust/pull/187
### What changes were proposed in this pull request? [SPARK-60981](https://issues.apache.org/jira/browse/SPARK-60981): `Column.alias(*alias, **kwargs)`, and `Column.name`, now follow PySpark's `Expression.alias` and `ColumnAlias`: - **Names:** every given name is kept, so all the columns of a generator are named. For example, `F.explode("m").alias("k", "v")` names the key and the value of a map. - **Metadata:** metadata is sent as `json.dumps(metadata)`, so its values can be of any JSON type, such as `metadata={"k": 1, "n": [1, 2]}`. - **Errors, as in PySpark:** - metadata with more than one name raises `PySparkValueError` `ONLY_ALLOWED_FOR_SINGLE_COLUMN`; - another keyword argument raises `AssertionError: Unexpected kwargs where passed: {...}`. - **Repr:** the names render joined by commas, as `ColumnAlias.__repr__` does: `a AS x,y`. In the Rust crate, `Column` gains `aliases(names)` for several names, as Scala's `as(aliases: Seq[String])`, and `Alias` gains `with_names`. ### Why are the changes needed? - Only the first name was kept, by a note that the other names "are not applied here". So `spark.sql("SELECT map('k1', 1) AS m").select(F.explode("m").alias("k", "v"))` failed on the server with `UDTF_ALIAS_NUMBER_MISMATCH`, where PySpark 4.3.0 gives the columns `k` and `v`. - Metadata took string values only, so `F.col("id").alias("x", metadata={"k": 1})` raised `TypeError: 'int' object is not an instance of 'str'`. - An unexpected keyword argument was ignored. ### Does this PR introduce _any_ user-facing change? Yes, `Column.alias` behaves as in PySpark 4.3.0. ### How was this patch tested? - A probe of 10 alias cases gives the same reprs and errors as official PySpark 4.3.0. The cases cover one, two or no names, metadata with one or two names, explicit `None` metadata, an unexpected keyword, `name` and `explode`. - Against a local Spark 4.3.0 server, the two failing cases above give the same columns, rows and metadata as official PySpark 4.3.0. - New unit tests check `Column::aliases` and the render. - A new offline test checks the reprs and errors, and a new live test checks the columns and metadata. Both also pass with official PySpark 4.3.0, and fail without this change. - The lib tests and every integration test, e2e included, against the local server (646 passed), the build of all tests with `wasm-udf`, the offline Python suites (652 passed, 8 live cases skipped), and the live drop-in and Arrow suites (64 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
