HyukjinKwon opened a new pull request, #231: URL: https://github.com/apache/spark-connect-rust/pull/231
### What changes were proposed in this pull request? [SPARK-61035](https://issues.apache.org/jira/browse/SPARK-61035): `mapInPandas` and `mapInArrow` now send their resource profile, as PySpark Connect does. `df.mapInPandas(func, schema, barrier, profile)` and `df.mapInArrow(...)` send the id of the `ResourceProfile` `profile`, which `profile.id` creates on the server, as the `profile_id` of their `MapPartitions`. The stage then runs with the resources of the profile (stage-level scheduling). The client reads `profile.id` when `mapInPandas` or `mapInArrow` is called; PySpark reads it when it sends the plan. The core `MapPartitions` plan carries the id, which the new `DataFrame::map_in_pandas_with_profile` and `map_in_arrow_with_profile` take. ### Why are the changes needed? The client took `profile` and ignored it. The stage ran without the resources of the profile, and against a server that does not support the profile, as one in local mode, the action succeeded. PySpark fails it with `SparkException: TaskResourceProfiles are only supported for Standalone, Yarn and Kubernetes cluster for now when dynamic allocation is disabled.` ### Does this PR introduce _any_ user-facing change? Yes. The resource profiles of `mapInPandas` and `mapInArrow` apply, as in PySpark 4.3.0. ### How was this patch tested? - A new live test checks that a profile of task resources fails `mapInPandas` and `mapInArrow` on the local server, which does not support it, and that no profile does not. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the `profile_id` that `mapInPandas` and `mapInArrow` send to a mock Connect server. - A new lib test checks the `profile_id` of the `MapPartitions` relation. - The lib tests of both crates (569 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1800 passed, 8 skipped), and the live drop-in and Arrow suites (106 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
