HyukjinKwon opened a new pull request, #204: URL: https://github.com/apache/spark-connect-rust/pull/204
### What changes were proposed in this pull request? [SPARK-60999](https://issues.apache.org/jira/browse/SPARK-60999): the Python client now creates the UDTFs of `udtf()` and `arrow_udtf()` as PySpark's `_create_py_udtf` and `_create_pyarrow_udtf` do. - `arrow_udtf()` makes a PyArrow-native UDTF (`SQL_ARROW_UDTF`), whose `eval()` takes `pyarrow` arrays and returns `pyarrow` tables. It made an Arrow-optimized UDTF (`SQL_ARROW_TABLE_UDF`) instead, which passes Python values. - `udtf()` without `useArrow` makes an Arrow-optimized UDTF when the active session enables `spark.sql.execution.pythonUDTF.arrow.enabled`, as the 4.3.0 server does by default. Without a session, it makes a regular UDTF. - A UDTF is not deterministic unless `asDeterministic()` marks it so, in place, and the call sends that flag. It was deterministic, and `asDeterministic()` returned a copy. - `SparkSession.active()` raises `NO_ACTIVE_OR_DEFAULT_SESSION`, as PySpark's does, and so does a UDTF called without a session. It raised a plain `RuntimeError`. ### Why are the changes needed? Every `arrow_udtf` that used `pyarrow` failed, because its `eval()` was given Python values. For example, `pc.add(x, 1)` of a `PlusOne(lit(1))` returns `[Row(x=2, kind='Int32Array')]` in PySpark 4.3.0, and failed here. UDTFs also did not follow the Arrow configuration, so their eval type and determinism differed from PySpark's. ### Does this PR introduce _any_ user-facing change? Yes. `arrow_udtf()` and `udtf()` create the UDTFs of PySpark 4.3.0, and `SparkSession.active()` raises its error. ### How was this patch tested? - A new live test runs an `arrow_udtf` that uses `pyarrow`. It also passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the eval type and the determinism of the UDTFs that a mock Connect server receives: - `udtf()` follows the configuration, which the mock now serves with the server's default, and `useArrow` overrides it. - `arrow_udtf()` makes a `SQL_ARROW_UDTF`. - `asDeterministic()` marks the UDTF in place. - Only `udtf()` without `useArrow` reads the configuration. It fails without this change. - The test of a UDTF called without a session now runs in a new interpreter. It checks the eval type and the `NO_ACTIVE_OR_DEFAULT_SESSION` error. - Each offline test now starts without an active session, and the tests of the mock server have a session of it each, which is active while they run. A session that an earlier test left, of a server that does not listen, made `udtf()` wait on its configuration. - The tests of the eval types and the determinism of UDTFs now expect PySpark's. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (1158 passed, 8 skipped), and the live drop-in and Arrow suites (84 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
