HyukjinKwon opened a new pull request, #322: URL: https://github.com/apache/spark-connect-rust/pull/322
### What changes were proposed in this pull request? [SPARK-61124](https://issues.apache.org/jira/browse/SPARK-61124): `DataFrame.nearestByJoin` checks its arguments, and takes them, as PySpark's. PySpark 4.3.0's `nearestByJoin(other, rankingExpression, numResults, mode, direction, *, joinType="inner")` checks its arguments before it builds the plan, as the Scala `Dataset.validateNearestByJoinArgs` does. It raises an `AnalysisException`: - `NEAREST_BY_JOIN.NUM_RESULTS_OUT_OF_RANGE`, for a `numResults` that is not between 1 and 100000 - `NEAREST_BY_JOIN.UNSUPPORTED_JOIN_TYPE`, for a `joinType` other than `inner`, `left` or `leftouter` (case- and underscore-insensitive) - `NEAREST_BY_JOIN.UNSUPPORTED_MODE`, for a `mode` other than `approx` or `exact` - `NEAREST_BY_JOIN.UNSUPPORTED_DIRECTION`, for a `direction` other than `distance` or `similarity` The client now does the same. `numResults`, `mode` and `direction` have no defaults, `joinType` is keyword-only, and `numResults` is taken as `int(numResults)`. ### Why are the changes needed? The client checked none of these, so an invalid argument failed only when the query ran, if at all. Its `numResults`, `mode` and `direction` defaulted to `1`, `"brute"` and `"asc"`, values that PySpark rejects. ### Does this PR introduce _any_ user-facing change? Yes. Invalid arguments raise as with PySpark 4.3.0, and the arguments without defaults in PySpark have none here. ### How was this patch tested? - A new offline test checks: - the error class and parameters of each invalid argument - valid arguments of other cases and a string `numResults` - the `TypeError` of missing arguments and of a positional `joinType` It passes with official PySpark 4.3.0. - The existing session test now passes the required arguments. - Apache Spark's `test_parity_nearest_by_join.py` passes on this client against a local server, all 14 tests. Before this change, its five `test_invalid_*` tests failed. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2179 passed, 8 skipped), and the live drop-in and Arrow suites (174 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
