HyukjinKwon opened a new pull request, #178: URL: https://github.com/apache/spark-connect-rust/pull/178
### What changes were proposed in this pull request? [SPARK-60953](https://issues.apache.org/jira/browse/SPARK-60953): `instr`, `regexp_replace` and `call_function` columns are now built and rendered as in PySpark Connect. - **`instr` and `regexp_replace`** are plain function calls (`UnresolvedFunction`), as `_invoke_function_over_columns` builds them, with their optional 4.3.0 arguments when given. Before, they were `CallFunction` expressions so that the optional arguments could be passed. - **`call_function`** still builds a `CallFunction`. It now renders as PySpark's `CallFunction.__repr__`, `CallFunction('<name>', <arg>, ...)`, instead of the Rust debug form of the expression. - **Rust API:** for the optional arguments, `spark_connect::functions` gains `instr_with_start`, `instr_with_start_and_occurrence` and `regexp_replace_with_position`. | Column | PySpark 4.3.0 and now | Before | | --- | --- | --- | | `F.instr("s", "b", occurrence=2)` | `Column<'instr(s, b, 1, 2)'>` | `Column<'CallFunction(CallFunctionWrapper { function_name: "instr", arguments: [ColumnReference(...` | | `F.regexp_replace("s", "a", "b")` | `Column<'regexp_replace(s, a, b)'>` | the same debug form | | `F.call_function("abs", "x")` | `Column<'CallFunction('abs', x)'>` | the same debug form | ### Why are the changes needed? The repr of each of these columns was the Rust debug dump of the expression. That dump also appeared in the reprs of the columns built from them and in error messages. `instr` and `regexp_replace` also sent a different expression from PySpark's. ### Does this PR introduce _any_ user-facing change? Yes, these columns render as in PySpark 4.3.0, and `instr` and `regexp_replace` send the same expressions as PySpark. Their results are unchanged. ### How was this patch tested? - A probe of 14 columns gives the same reprs as official PySpark 4.3.0: the forms of `instr` and `regexp_replace`, and `call_function` with no argument, nested, aliased and with a null literal. Before this change, all 14 differed. - Against a local Spark 4.3.0 server, the column names, values and error of a probe of these functions are the same as with official PySpark 4.3.0. - New unit tests check the builders and the render of `CallFunction`. The expression-sharing test now pins the new render; the debug form is unchanged. - A new offline test checks the reprs, and a new live test checks the column names and values. Both also pass with official PySpark 4.3.0. - The lib tests and every integration test, e2e included, against the local server (644 passed), the build of all tests with `wasm-udf`, the offline Python suites (624 passed, 8 live cases skipped), and the live drop-in and Arrow suites (58 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
