This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 69962e4  [SPARK-61035][CONNECT][PYTHON] Send the resource profile of 
mapInPandas and mapInArrow in the Python client as PySpark does
69962e4 is described below

commit 69962e48fdb71ec9392e41676b3bdbe671b1aa5b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 06:17:51 2026 +0900

    [SPARK-61035][CONNECT][PYTHON] Send the resource profile of mapInPandas and 
mapInArrow in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61035](https://issues.apache.org/jira/browse/SPARK-61035): 
`mapInPandas` and `mapInArrow` now send their resource profile, as PySpark 
Connect does.
    
    `df.mapInPandas(func, schema, barrier, profile)` and `df.mapInArrow(...)` 
send the id of the `ResourceProfile` `profile`, which `profile.id` creates on 
the server, as the `profile_id` of their `MapPartitions`. The stage then runs 
with the resources of the profile (stage-level scheduling). The client reads 
`profile.id` when `mapInPandas` or `mapInArrow` is called; PySpark reads it 
when it sends the plan.
    
    The core `MapPartitions` plan carries the id, which the new 
`DataFrame::map_in_pandas_with_profile` and `map_in_arrow_with_profile` take.
    
    ### Why are the changes needed?
    
    The client took `profile` and ignored it. The stage ran without the 
resources of the profile, and against a server that does not support the 
profile, as one in local mode, the action succeeded. PySpark fails it with 
`SparkException: TaskResourceProfiles are only supported for Standalone, Yarn 
and Kubernetes cluster for now when dynamic allocation is disabled.`
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The resource profiles of `mapInPandas` and `mapInArrow` apply, as in 
PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test runs `mapInPandas` and `mapInArrow` of a profile of task 
resources. A server that does not support it, as one in local mode, rejects it; 
otherwise, as in the `local-cluster` mode of CI, the stage runs with the 
profile, of the id that the server gave it. It passes with official PySpark 
4.3.0, and fails without this change.
    - A new offline test checks the `profile_id` that `mapInPandas` and 
`mapInArrow` send to a mock Connect server.
    - A new lib test checks the `profile_id` of the `MapPartitions` relation.
    - The lib tests of both crates (569 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (1800 passed, 8 skipped), and the 
live drop-in and Arrow suites (106 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #231 from HyukjinKwon/SPARK-61035-map-profile.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs                 | 34 +++++++----
 crates/spark-connect/src/dataframe.rs              | 66 ++++++++++++++++++++--
 crates/spark-connect/src/plan.rs                   |  6 +-
 .../spark-connect/tests/coverage_plan_map_udtf.rs  |  1 +
 python/tests/test_dropin_e2e.py                    | 20 +++++++
 python/tests/test_dropin_offline.py                | 24 ++++++++
 6 files changed, 135 insertions(+), 16 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 2b87429..aa47f24 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2049,13 +2049,9 @@ impl PyDataFrame {
         Ok(format!("DataFrame[{fields}]"))
     }
 
-    /// `DataFrame.mapInPandas` — map an iterator of pandas DataFrames.
-    ///
-    /// `profile` (a ResourceProfile) is accepted for signature parity with 
reference
-    /// pyspark; the connect core does not yet attach a resource profile id to 
the
-    /// map-partitions plan, so a supplied profile is currently a no-op.
+    /// `DataFrame.mapInPandas` — map an iterator of pandas DataFrames, in a 
stage of the
+    /// resource profile `profile`, if any, as PySpark's.
     #[pyo3(name = "mapInPandas", signature = (func, schema, barrier = false, 
profile = None))]
-    #[allow(unused_variables)]
     fn map_in_pandas(
         &self,
         py: Python<'_>,
@@ -2073,13 +2069,16 @@ impl PyDataFrame {
             eval_type::SQL_MAP_PANDAS_ITER_UDF,
             &cols,
         )?;
-        Ok(PyDataFrame::new(self.dataframe.map_in_pandas(udf, barrier)))
+        let profile_id = profile_id_of(profile.as_ref())?;
+        Ok(PyDataFrame::new(
+            self.dataframe
+                .map_in_pandas_with_profile(udf, barrier, profile_id),
+        ))
     }
 
-    /// `DataFrame.mapInArrow` — map an iterator of Arrow batches. `profile` 
is accepted
-    /// for signature parity (see `mapInPandas`); it is currently a no-op.
+    /// `DataFrame.mapInArrow` — map an iterator of Arrow batches, in a stage 
of the
+    /// resource profile `profile`, if any, as PySpark's.
     #[pyo3(name = "mapInArrow", signature = (func, schema, barrier = false, 
profile = None))]
-    #[allow(unused_variables)]
     fn map_in_arrow(
         &self,
         py: Python<'_>,
@@ -2091,7 +2090,11 @@ impl PyDataFrame {
         validate_map_function(py, &func, eval_type::SQL_MAP_ARROW_ITER_UDF)?;
         let cols = py.detach(|| self.dataframe.columns()).to_pyerr()?;
         let udf = build_map_udf(py, &func, &schema, 
eval_type::SQL_MAP_ARROW_ITER_UDF, &cols)?;
-        Ok(PyDataFrame::new(self.dataframe.map_in_arrow(udf, barrier)))
+        let profile_id = profile_id_of(profile.as_ref())?;
+        Ok(PyDataFrame::new(
+            self.dataframe
+                .map_in_arrow_with_profile(udf, barrier, profile_id),
+        ))
     }
 
     /// `DataFrame.foreach`, as PySpark Connect's: a UDF over a struct of the 
columns of each
@@ -3196,6 +3199,15 @@ pub(crate) fn resolve_datatype(
     ))
 }
 
+/// The id of the resource profile `profile` of `mapInPandas` and 
`mapInArrow`, as PySpark's
+/// `MapPartitions` reads `profile.id`, which creates the profile on the 
server.
+fn profile_id_of(profile: Option<&Bound<'_, PyAny>>) -> PyResult<Option<i32>> {
+    match profile {
+        Some(profile) if !profile.is_none() => 
Ok(Some(profile.getattr("id")?.extract()?)),
+        _ => Ok(None),
+    }
+}
+
 /// Validate the function of a map UDF for its eval type with
 /// `pyspark.sql.pandas.functions._validate_vectorized_udf`, as the reference 
client does
 /// before it builds the UDF.
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index b0fd446..8d1c2ac 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -689,7 +689,19 @@ impl DataFrame {
         func: CommonInlineUserDefinedFunctionExpression,
         is_barrier: bool,
     ) -> DataFrame {
-        self.map_partitions(func, is_barrier)
+        self.map_partitions(func, is_barrier, None)
+    }
+
+    /// Map over each partition with a pandas UDF in a stage of the resource 
profile of
+    /// `profile_id`, as `DataFrame.mapInPandas(func, schema, barrier, 
profile)` does with the
+    /// id of `profile`, which [`SparkSession::build_resource_profile`] 
registers.
+    pub fn map_in_pandas_with_profile(
+        &self,
+        func: CommonInlineUserDefinedFunctionExpression,
+        is_barrier: bool,
+        profile_id: Option<i32>,
+    ) -> DataFrame {
+        self.map_partitions(func, is_barrier, profile_id)
     }
 
     /// Map over each partition with an Arrow UDF (`DataFrame.mapInArrow`).
@@ -698,7 +710,18 @@ impl DataFrame {
         func: CommonInlineUserDefinedFunctionExpression,
         is_barrier: bool,
     ) -> DataFrame {
-        self.map_partitions(func, is_barrier)
+        self.map_partitions(func, is_barrier, None)
+    }
+
+    /// Map over each partition with an Arrow UDF in a stage of the resource 
profile of
+    /// `profile_id`, as `DataFrame.mapInArrow(func, schema, barrier, 
profile)` does.
+    pub fn map_in_arrow_with_profile(
+        &self,
+        func: CommonInlineUserDefinedFunctionExpression,
+        is_barrier: bool,
+        profile_id: Option<i32>,
+    ) -> DataFrame {
+        self.map_partitions(func, is_barrier, profile_id)
     }
 
     /// Build a `MapPartitions` relation from an already-constructed UDF.
@@ -706,12 +729,14 @@ impl DataFrame {
         &self,
         func: CommonInlineUserDefinedFunctionExpression,
         is_barrier: bool,
+        profile_id: Option<i32>,
     ) -> DataFrame {
         let schema = func.python_udf.output_type.clone();
         let plan = LogicalPlan::MapPartitions {
             input: Arc::clone(&self.plan),
             func,
             is_barrier,
+            profile_id,
         };
         Self::new_with_schema(self.session.clone(), plan, Some(schema))
     }
@@ -720,14 +745,14 @@ impl DataFrame {
     ///
     /// Backed by an Arrow map partition; results are forced and discarded.
     pub fn foreach(&self, func: CommonInlineUserDefinedFunctionExpression) -> 
Result<()> {
-        let _ = self.map_partitions(func, false).collect()?;
+        let _ = self.map_partitions(func, false, None).collect()?;
         Ok(())
     }
 
     /// Apply a function to each partition for its side effects
     /// (`DataFrame.foreachPartition`).
     pub fn foreach_partition(&self, func: 
CommonInlineUserDefinedFunctionExpression) -> Result<()> {
-        let _ = self.map_partitions(func, false).collect()?;
+        let _ = self.map_partitions(func, false, None).collect()?;
         Ok(())
     }
 
@@ -4184,6 +4209,39 @@ mod plan_construction_tests {
         );
         ser(&udtf_df);
     }
+
+    #[test]
+    fn map_functions_send_the_id_of_their_resource_profile() {
+        use crate::types::DataType;
+        use crate::udf::{CommonInlineUserDefinedFunctionExpression, 
PythonUDFPayload};
+        use spark_connect_proto::relation::RelType;
+
+        let spark = session();
+        let df = spark.range(5).unwrap();
+        let udf = || {
+            CommonInlineUserDefinedFunctionExpression::new(
+                "f".to_string(),
+                true,
+                vec![],
+                PythonUDFPayload::new(DataType::Integer, 205, vec![1], 
"3.11".to_string()),
+            )
+        };
+        let sent = |mapped: DataFrame| match mapped.relation().rel_type {
+            Some(RelType::MapPartitions(map)) => (map.profile_id, 
map.is_barrier),
+            other => panic!("expected MapPartitions, got {other:?}"),
+        };
+        // As PySpark's MapPartitions, of the id of the profile, if any.
+        assert_eq!(
+            sent(df.map_in_pandas_with_profile(udf(), true, Some(7))),
+            (Some(7), Some(true))
+        );
+        assert_eq!(
+            sent(df.map_in_arrow_with_profile(udf(), false, Some(8))),
+            (Some(8), Some(false))
+        );
+        assert_eq!(sent(df.map_in_pandas(udf(), false)), (None, Some(false)));
+        assert_eq!(sent(df.map_in_arrow(udf(), true)), (None, Some(true)));
+    }
 }
 
 /// Deterministic coverage of the private Arrow-array -> `Value` converter
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index f0a700e..f90cd38 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -348,11 +348,13 @@ pub enum LogicalPlan {
         input: Arc<LogicalPlan>,
         index_columns: Vec<Expression>,
     },
-    /// MapPartitions: `DataFrame.mapInPandas` / `mapInArrow` (also backs 
foreach).
+    /// MapPartitions: `DataFrame.mapInPandas` / `mapInArrow` (also backs 
foreach), of the id
+    /// of the resource profile of its stage, if any.
     MapPartitions {
         input: Arc<LogicalPlan>,
         func: CommonInlineUserDefinedFunctionExpression,
         is_barrier: bool,
+        profile_id: Option<i32>,
     },
     /// GroupMap: `GroupedData.applyInPandas` / `applyInArrow`, and the 
stateful
     /// `transformWithState[InPandas]`.
@@ -1427,11 +1429,13 @@ impl LogicalPlan {
                 input,
                 func,
                 is_barrier,
+                profile_id,
             } => {
                 let mut mp = proto::MapPartitions::default();
                 mp.input = Some(Box::new(relation_of(input, plan_ids)));
                 mp.func = Some(func.to_proto());
                 mp.is_barrier = Some(*is_barrier);
+                mp.profile_id = *profile_id;
                 relation.rel_type = 
Some(proto::relation::RelType::MapPartitions(Box::new(mp)));
             }
             LogicalPlan::GroupMap {
diff --git a/crates/spark-connect/tests/coverage_plan_map_udtf.rs 
b/crates/spark-connect/tests/coverage_plan_map_udtf.rs
index e452eea..ce033e2 100644
--- a/crates/spark-connect/tests/coverage_plan_map_udtf.rs
+++ b/crates/spark-connect/tests/coverage_plan_map_udtf.rs
@@ -41,6 +41,7 @@ fn map_partitions_to_proto() {
         input: Arc::new(base()),
         func: udf(),
         is_barrier: true,
+        profile_id: None,
     };
     assert!(p.to_proto().rel_type.is_some());
 }
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 9709aa2..ccf7b56 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -285,6 +285,26 @@ def test_map_functions_take_a_ddl_schema(session):
         assert sorted(tuple(r) for r in result.collect()) == [(1, 1.0), (1, 
2.0), (2, 3.0)]
 
 
+def test_map_functions_run_in_the_stage_of_their_resource_profile(session):
+    # As PySpark's, mapInPandas and mapInArrow send their resource profile, 
which the server
+    # creates: a server that does not support it, as one in local mode, 
rejects it, and
+    # otherwise the stage runs with the profile, of the id that the server 
gave it.
+    from pyspark.errors import PySparkException
+    from pyspark.resource import ResourceProfileBuilder, TaskResourceRequests
+
+    for name in ("mapInPandas", "mapInArrow"):
+        profile = 
ResourceProfileBuilder().require(TaskResourceRequests().cpus(1)).build
+        try:
+            mapped = getattr(session.range(3), name)(lambda it: it, "id long", 
profile=profile)
+            rows = mapped.collect()
+        except PySparkException as error:
+            assert "ResourceProfiles are only supported" in str(error)
+        else:
+            assert sorted(r.id for r in rows) == [0, 1, 2]
+            assert isinstance(profile._id, int)
+    assert session.range(3).mapInPandas(lambda it: it, "id long", 
profile=None).count() == 3
+
+
 def test_transform_with_state_in_pandas_takes_a_ddl_output_type(session, 
tmp_path):
     import pandas as pd
     from pyspark.sql.streaming.stateful_processor import StatefulProcessor
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 297ce84..685dea5 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -882,6 +882,30 @@ def 
test_map_functions_send_the_names_of_their_functions_as_pyspark(mock_connect
     assert sent_name(stateful, 34, 3) == "with_state"
 
 
+def test_map_functions_send_their_resource_profile_as_pyspark(mock_connect):
+    # As PySpark's, mapInPandas and mapInArrow send the id of their resource 
profile, which
+    # `profile.id` creates on the server, so that their stage runs with its 
resources.
+    from pyspark.resource import ResourceProfileBuilder, TaskResourceRequests
+
+    spark, service = mock_connect
+    profile = 
ResourceProfileBuilder().require(TaskResourceRequests().cpus(1)).build
+    profile._id = 7  # As if the server had created it.
+
+    def sent_map_partitions(dataframe):
+        # ExecutePlanRequest.plan=3, Plan.root=1, Relation.map_partitions=28.
+        dataframe.collect()
+        plan = 
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+        return parse_message(parse_message(plan)[28][0])
+
+    df = spark.range(1)
+    # MapPartitions.profile_id=4.
+    for mapped in (df.mapInPandas(lambda it: it, "id long", profile=profile),
+                   df.mapInArrow(lambda it: it, "id long", True, profile)):
+        assert sent_map_partitions(mapped)[4] == [7]
+    assert 4 not in sent_map_partitions(df.mapInPandas(lambda it: it, "id 
long"))
+    assert 4 not in sent_map_partitions(df.mapInArrow(lambda it: it, "id 
long", profile=None))
+
+
 def test_udf_and_udtf_calls_pass_named_arguments_as_pyspark(mock_connect):
     # As PySpark's UserDefinedFunction and UserDefinedTableFunction: each 
keyword argument
     # of a call is a NamedArgumentExpression, after the positional arguments, 
and a bare


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to