HyukjinKwon opened a new pull request, #318: URL: https://github.com/apache/spark-connect-rust/pull/318
### What changes were proposed in this pull request? [SPARK-61123](https://issues.apache.org/jira/browse/SPARK-61123): A Python `Enum` is the literal of its value, as PySpark's `enum_to_value` makes it, in `lit`, the Column operators and the functions that take a value. An `Enum` of an `Enum` unwraps fully. The conversion of a Python value to a literal now takes the value of an `Enum`. `lit`, the operators (`+`, `==`, `&`, `**`, ...), `isin`, `between`, `contains`, `like`, `bitwiseOR`, `F.pow`, `F.concat_ws`, `F.array_contains` and the other functions that take a value all use it. ### Why are the changes needed? The client raised `PySparkTypeError: [UNSUPPORTED_DATA_TYPE] Unsupported DataType 'IntEnum'` for all of them except `substr`, while PySpark takes an `Enum` as its value. Apache Spark's `test_enum_literals` checks them. ### Does this PR introduce _any_ user-facing change? Yes. `Enum` values work as literals, as with PySpark 4.3.0. ### How was this patch tested? - A new offline test checks the expressions of `Enum` operands of `lit`, the operators and functions, and an `Enum` of an `Enum`. It passes with official PySpark 4.3.0. - 22 cases give the same expressions as official PySpark 4.3.0. - Apache Spark's `test_enum_literals` of `test_parity_column.py`, both variants, pass on this client against a local server. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2168 passed, 8 skipped), and the live drop-in and Arrow suites (173 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
