HyukjinKwon opened a new pull request, #252: URL: https://github.com/apache/spark-connect-rust/pull/252
### What changes were proposed in this pull request? [SPARK-61054](https://issues.apache.org/jira/browse/SPARK-61054): `spark.range`, `spark.tvf.range` and the table-valued functions of `spark.tvf` now take their arguments as PySpark Connect does. - `spark.range(start, end, step, numPartitions)` and `spark.tvf.range` take the `int()` of each argument, of `numPartitions` first, such as `spark.range("3")` and `spark.range(2.7)`. A value out of the range of its proto field raises the `ValueError` of protobuf, "Value out of range: ...". The client raises it when the DataFrame is made; PySpark raises it when it sends the plan. - The functions of `spark.tvf`, such as `explode`, `inline`, `json_tuple` and `stack`, take Columns or column names, as `F._to_col` does, else `NOT_EXPECTED_TYPE` of `col`. - `json_tuple` asks for fields before it takes its arguments. ### Why are the changes needed? The client raised `TypeError`s for arguments of `range` that are not ints. It took the arguments of `spark.tvf` as literals: - `spark.tvf.json_tuple(json, "a")` took the string "a" rather than the column `a` - `spark.tvf.explode([1, 2])` exploded an array of the list rather than raising - `spark.tvf.stack(2, ...)` took an int rather than raising ### Does this PR introduce _any_ user-facing change? Yes. These take the arguments of PySpark 4.3.0, and raise its errors. ### How was this patch tested? - A new live test checks the rows of `range` of a str and of floats, the errors of `range`, the column of a str argument of `json_tuple`, and the `NOT_EXPECTED_TYPE` of a list. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the `Range` and the arguments of the table-valued functions that these send to a mock Connect server, and their errors. - On a local server, 30 cases of `spark.range` and `spark.tvf.range`, and 23 of `spark.tvf`, give the rows and errors of official PySpark 4.3.0. One `TypeError` of `int()` names the class of the client's Column in full. - The lib tests of both crates (574 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1813 passed, 8 skipped), and the live drop-in and Arrow suites (112 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
