This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 69962e4 [SPARK-61035][CONNECT][PYTHON] Send the resource profile of
mapInPandas and mapInArrow in the Python client as PySpark does
69962e4 is described below
commit 69962e48fdb71ec9392e41676b3bdbe671b1aa5b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 06:17:51 2026 +0900
[SPARK-61035][CONNECT][PYTHON] Send the resource profile of mapInPandas and
mapInArrow in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61035](https://issues.apache.org/jira/browse/SPARK-61035):
`mapInPandas` and `mapInArrow` now send their resource profile, as PySpark
Connect does.
`df.mapInPandas(func, schema, barrier, profile)` and `df.mapInArrow(...)`
send the id of the `ResourceProfile` `profile`, which `profile.id` creates on
the server, as the `profile_id` of their `MapPartitions`. The stage then runs
with the resources of the profile (stage-level scheduling). The client reads
`profile.id` when `mapInPandas` or `mapInArrow` is called; PySpark reads it
when it sends the plan.
The core `MapPartitions` plan carries the id, which the new
`DataFrame::map_in_pandas_with_profile` and `map_in_arrow_with_profile` take.
### Why are the changes needed?
The client took `profile` and ignored it. The stage ran without the
resources of the profile, and against a server that does not support the
profile, as one in local mode, the action succeeded. PySpark fails it with
`SparkException: TaskResourceProfiles are only supported for Standalone, Yarn
and Kubernetes cluster for now when dynamic allocation is disabled.`
### Does this PR introduce _any_ user-facing change?
Yes. The resource profiles of `mapInPandas` and `mapInArrow` apply, as in
PySpark 4.3.0.
### How was this patch tested?
- A new live test runs `mapInPandas` and `mapInArrow` of a profile of task
resources. A server that does not support it, as one in local mode, rejects it;
otherwise, as in the `local-cluster` mode of CI, the stage runs with the
profile, of the id that the server gave it. It passes with official PySpark
4.3.0, and fails without this change.
- A new offline test checks the `profile_id` that `mapInPandas` and
`mapInArrow` send to a mock Connect server.
- A new lib test checks the `profile_id` of the `MapPartitions` relation.
- The lib tests of both crates (569 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (1800 passed, 8 skipped), and the
live drop-in and Arrow suites (106 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #231 from HyukjinKwon/SPARK-61035-map-profile.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 34 +++++++----
crates/spark-connect/src/dataframe.rs | 66 ++++++++++++++++++++--
crates/spark-connect/src/plan.rs | 6 +-
.../spark-connect/tests/coverage_plan_map_udtf.rs | 1 +
python/tests/test_dropin_e2e.py | 20 +++++++
python/tests/test_dropin_offline.py | 24 ++++++++
6 files changed, 135 insertions(+), 16 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 2b87429..aa47f24 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2049,13 +2049,9 @@ impl PyDataFrame {
Ok(format!("DataFrame[{fields}]"))
}
- /// `DataFrame.mapInPandas` — map an iterator of pandas DataFrames.
- ///
- /// `profile` (a ResourceProfile) is accepted for signature parity with
reference
- /// pyspark; the connect core does not yet attach a resource profile id to
the
- /// map-partitions plan, so a supplied profile is currently a no-op.
+ /// `DataFrame.mapInPandas` — map an iterator of pandas DataFrames, in a
stage of the
+ /// resource profile `profile`, if any, as PySpark's.
#[pyo3(name = "mapInPandas", signature = (func, schema, barrier = false,
profile = None))]
- #[allow(unused_variables)]
fn map_in_pandas(
&self,
py: Python<'_>,
@@ -2073,13 +2069,16 @@ impl PyDataFrame {
eval_type::SQL_MAP_PANDAS_ITER_UDF,
&cols,
)?;
- Ok(PyDataFrame::new(self.dataframe.map_in_pandas(udf, barrier)))
+ let profile_id = profile_id_of(profile.as_ref())?;
+ Ok(PyDataFrame::new(
+ self.dataframe
+ .map_in_pandas_with_profile(udf, barrier, profile_id),
+ ))
}
- /// `DataFrame.mapInArrow` — map an iterator of Arrow batches. `profile`
is accepted
- /// for signature parity (see `mapInPandas`); it is currently a no-op.
+ /// `DataFrame.mapInArrow` — map an iterator of Arrow batches, in a stage
of the
+ /// resource profile `profile`, if any, as PySpark's.
#[pyo3(name = "mapInArrow", signature = (func, schema, barrier = false,
profile = None))]
- #[allow(unused_variables)]
fn map_in_arrow(
&self,
py: Python<'_>,
@@ -2091,7 +2090,11 @@ impl PyDataFrame {
validate_map_function(py, &func, eval_type::SQL_MAP_ARROW_ITER_UDF)?;
let cols = py.detach(|| self.dataframe.columns()).to_pyerr()?;
let udf = build_map_udf(py, &func, &schema,
eval_type::SQL_MAP_ARROW_ITER_UDF, &cols)?;
- Ok(PyDataFrame::new(self.dataframe.map_in_arrow(udf, barrier)))
+ let profile_id = profile_id_of(profile.as_ref())?;
+ Ok(PyDataFrame::new(
+ self.dataframe
+ .map_in_arrow_with_profile(udf, barrier, profile_id),
+ ))
}
/// `DataFrame.foreach`, as PySpark Connect's: a UDF over a struct of the
columns of each
@@ -3196,6 +3199,15 @@ pub(crate) fn resolve_datatype(
))
}
+/// The id of the resource profile `profile` of `mapInPandas` and
`mapInArrow`, as PySpark's
+/// `MapPartitions` reads `profile.id`, which creates the profile on the
server.
+fn profile_id_of(profile: Option<&Bound<'_, PyAny>>) -> PyResult<Option<i32>> {
+ match profile {
+ Some(profile) if !profile.is_none() =>
Ok(Some(profile.getattr("id")?.extract()?)),
+ _ => Ok(None),
+ }
+}
+
/// Validate the function of a map UDF for its eval type with
/// `pyspark.sql.pandas.functions._validate_vectorized_udf`, as the reference
client does
/// before it builds the UDF.
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index b0fd446..8d1c2ac 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -689,7 +689,19 @@ impl DataFrame {
func: CommonInlineUserDefinedFunctionExpression,
is_barrier: bool,
) -> DataFrame {
- self.map_partitions(func, is_barrier)
+ self.map_partitions(func, is_barrier, None)
+ }
+
+ /// Map over each partition with a pandas UDF in a stage of the resource
profile of
+ /// `profile_id`, as `DataFrame.mapInPandas(func, schema, barrier,
profile)` does with the
+ /// id of `profile`, which [`SparkSession::build_resource_profile`]
registers.
+ pub fn map_in_pandas_with_profile(
+ &self,
+ func: CommonInlineUserDefinedFunctionExpression,
+ is_barrier: bool,
+ profile_id: Option<i32>,
+ ) -> DataFrame {
+ self.map_partitions(func, is_barrier, profile_id)
}
/// Map over each partition with an Arrow UDF (`DataFrame.mapInArrow`).
@@ -698,7 +710,18 @@ impl DataFrame {
func: CommonInlineUserDefinedFunctionExpression,
is_barrier: bool,
) -> DataFrame {
- self.map_partitions(func, is_barrier)
+ self.map_partitions(func, is_barrier, None)
+ }
+
+ /// Map over each partition with an Arrow UDF in a stage of the resource
profile of
+ /// `profile_id`, as `DataFrame.mapInArrow(func, schema, barrier,
profile)` does.
+ pub fn map_in_arrow_with_profile(
+ &self,
+ func: CommonInlineUserDefinedFunctionExpression,
+ is_barrier: bool,
+ profile_id: Option<i32>,
+ ) -> DataFrame {
+ self.map_partitions(func, is_barrier, profile_id)
}
/// Build a `MapPartitions` relation from an already-constructed UDF.
@@ -706,12 +729,14 @@ impl DataFrame {
&self,
func: CommonInlineUserDefinedFunctionExpression,
is_barrier: bool,
+ profile_id: Option<i32>,
) -> DataFrame {
let schema = func.python_udf.output_type.clone();
let plan = LogicalPlan::MapPartitions {
input: Arc::clone(&self.plan),
func,
is_barrier,
+ profile_id,
};
Self::new_with_schema(self.session.clone(), plan, Some(schema))
}
@@ -720,14 +745,14 @@ impl DataFrame {
///
/// Backed by an Arrow map partition; results are forced and discarded.
pub fn foreach(&self, func: CommonInlineUserDefinedFunctionExpression) ->
Result<()> {
- let _ = self.map_partitions(func, false).collect()?;
+ let _ = self.map_partitions(func, false, None).collect()?;
Ok(())
}
/// Apply a function to each partition for its side effects
/// (`DataFrame.foreachPartition`).
pub fn foreach_partition(&self, func:
CommonInlineUserDefinedFunctionExpression) -> Result<()> {
- let _ = self.map_partitions(func, false).collect()?;
+ let _ = self.map_partitions(func, false, None).collect()?;
Ok(())
}
@@ -4184,6 +4209,39 @@ mod plan_construction_tests {
);
ser(&udtf_df);
}
+
+ #[test]
+ fn map_functions_send_the_id_of_their_resource_profile() {
+ use crate::types::DataType;
+ use crate::udf::{CommonInlineUserDefinedFunctionExpression,
PythonUDFPayload};
+ use spark_connect_proto::relation::RelType;
+
+ let spark = session();
+ let df = spark.range(5).unwrap();
+ let udf = || {
+ CommonInlineUserDefinedFunctionExpression::new(
+ "f".to_string(),
+ true,
+ vec![],
+ PythonUDFPayload::new(DataType::Integer, 205, vec![1],
"3.11".to_string()),
+ )
+ };
+ let sent = |mapped: DataFrame| match mapped.relation().rel_type {
+ Some(RelType::MapPartitions(map)) => (map.profile_id,
map.is_barrier),
+ other => panic!("expected MapPartitions, got {other:?}"),
+ };
+ // As PySpark's MapPartitions, of the id of the profile, if any.
+ assert_eq!(
+ sent(df.map_in_pandas_with_profile(udf(), true, Some(7))),
+ (Some(7), Some(true))
+ );
+ assert_eq!(
+ sent(df.map_in_arrow_with_profile(udf(), false, Some(8))),
+ (Some(8), Some(false))
+ );
+ assert_eq!(sent(df.map_in_pandas(udf(), false)), (None, Some(false)));
+ assert_eq!(sent(df.map_in_arrow(udf(), true)), (None, Some(true)));
+ }
}
/// Deterministic coverage of the private Arrow-array -> `Value` converter
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index f0a700e..f90cd38 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -348,11 +348,13 @@ pub enum LogicalPlan {
input: Arc<LogicalPlan>,
index_columns: Vec<Expression>,
},
- /// MapPartitions: `DataFrame.mapInPandas` / `mapInArrow` (also backs
foreach).
+ /// MapPartitions: `DataFrame.mapInPandas` / `mapInArrow` (also backs
foreach), of the id
+ /// of the resource profile of its stage, if any.
MapPartitions {
input: Arc<LogicalPlan>,
func: CommonInlineUserDefinedFunctionExpression,
is_barrier: bool,
+ profile_id: Option<i32>,
},
/// GroupMap: `GroupedData.applyInPandas` / `applyInArrow`, and the
stateful
/// `transformWithState[InPandas]`.
@@ -1427,11 +1429,13 @@ impl LogicalPlan {
input,
func,
is_barrier,
+ profile_id,
} => {
let mut mp = proto::MapPartitions::default();
mp.input = Some(Box::new(relation_of(input, plan_ids)));
mp.func = Some(func.to_proto());
mp.is_barrier = Some(*is_barrier);
+ mp.profile_id = *profile_id;
relation.rel_type =
Some(proto::relation::RelType::MapPartitions(Box::new(mp)));
}
LogicalPlan::GroupMap {
diff --git a/crates/spark-connect/tests/coverage_plan_map_udtf.rs
b/crates/spark-connect/tests/coverage_plan_map_udtf.rs
index e452eea..ce033e2 100644
--- a/crates/spark-connect/tests/coverage_plan_map_udtf.rs
+++ b/crates/spark-connect/tests/coverage_plan_map_udtf.rs
@@ -41,6 +41,7 @@ fn map_partitions_to_proto() {
input: Arc::new(base()),
func: udf(),
is_barrier: true,
+ profile_id: None,
};
assert!(p.to_proto().rel_type.is_some());
}
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 9709aa2..ccf7b56 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -285,6 +285,26 @@ def test_map_functions_take_a_ddl_schema(session):
assert sorted(tuple(r) for r in result.collect()) == [(1, 1.0), (1,
2.0), (2, 3.0)]
+def test_map_functions_run_in_the_stage_of_their_resource_profile(session):
+ # As PySpark's, mapInPandas and mapInArrow send their resource profile,
which the server
+ # creates: a server that does not support it, as one in local mode,
rejects it, and
+ # otherwise the stage runs with the profile, of the id that the server
gave it.
+ from pyspark.errors import PySparkException
+ from pyspark.resource import ResourceProfileBuilder, TaskResourceRequests
+
+ for name in ("mapInPandas", "mapInArrow"):
+ profile =
ResourceProfileBuilder().require(TaskResourceRequests().cpus(1)).build
+ try:
+ mapped = getattr(session.range(3), name)(lambda it: it, "id long",
profile=profile)
+ rows = mapped.collect()
+ except PySparkException as error:
+ assert "ResourceProfiles are only supported" in str(error)
+ else:
+ assert sorted(r.id for r in rows) == [0, 1, 2]
+ assert isinstance(profile._id, int)
+ assert session.range(3).mapInPandas(lambda it: it, "id long",
profile=None).count() == 3
+
+
def test_transform_with_state_in_pandas_takes_a_ddl_output_type(session,
tmp_path):
import pandas as pd
from pyspark.sql.streaming.stateful_processor import StatefulProcessor
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 297ce84..685dea5 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -882,6 +882,30 @@ def
test_map_functions_send_the_names_of_their_functions_as_pyspark(mock_connect
assert sent_name(stateful, 34, 3) == "with_state"
+def test_map_functions_send_their_resource_profile_as_pyspark(mock_connect):
+ # As PySpark's, mapInPandas and mapInArrow send the id of their resource
profile, which
+ # `profile.id` creates on the server, so that their stage runs with its
resources.
+ from pyspark.resource import ResourceProfileBuilder, TaskResourceRequests
+
+ spark, service = mock_connect
+ profile =
ResourceProfileBuilder().require(TaskResourceRequests().cpus(1)).build
+ profile._id = 7 # As if the server had created it.
+
+ def sent_map_partitions(dataframe):
+ # ExecutePlanRequest.plan=3, Plan.root=1, Relation.map_partitions=28.
+ dataframe.collect()
+ plan =
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+ return parse_message(parse_message(plan)[28][0])
+
+ df = spark.range(1)
+ # MapPartitions.profile_id=4.
+ for mapped in (df.mapInPandas(lambda it: it, "id long", profile=profile),
+ df.mapInArrow(lambda it: it, "id long", True, profile)):
+ assert sent_map_partitions(mapped)[4] == [7]
+ assert 4 not in sent_map_partitions(df.mapInPandas(lambda it: it, "id
long"))
+ assert 4 not in sent_map_partitions(df.mapInArrow(lambda it: it, "id
long", profile=None))
+
+
def test_udf_and_udtf_calls_pass_named_arguments_as_pyspark(mock_connect):
# As PySpark's UserDefinedFunction and UserDefinedTableFunction: each
keyword argument
# of a call is a NamedArgumentExpression, after the positional arguments,
and a bare
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]