This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 24bb0cfd [SPARK-61147][CONNECT][PYTHON] Refer to the column of
withMetadata and pivot through the DataFrame, so a missing column is
CANNOT_RESOLVE_DATAFRAME_COLUMN as in PySpark
24bb0cfd is described below
commit 24bb0cfd3a00090b6c76b8fc7624caeb10268cc0
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:11:09 2026 +0900
[SPARK-61147][CONNECT][PYTHON] Refer to the column of withMetadata and
pivot through the DataFrame, so a missing column is
CANNOT_RESOLVE_DATAFRAME_COLUMN as in PySpark
### What changes were proposed in this pull request?
[SPARK-61147](https://issues.apache.org/jira/browse/SPARK-61147): Refer to
the column of `withMetadata` and `pivot` through the DataFrame, so a missing
column is `CANNOT_RESOLVE_DATAFRAME_COLUMN` as in PySpark.
PySpark's `DataFrame.withMetadata` sends `self[columnName]` and
`GroupedData.pivot` sends `self._df[pivot_col]`. Both are columns bound to the
plan ID of the DataFrame. The Python client built them by name with
`functions.col`, so the server resolved them by name against the plan. A
missing column then raised `UNRESOLVED_COLUMN.WITH_SUGGESTION`.
- `LogicalPlan::WithColumnMetadata` carries the column as an expression
built by `DataFrame::col`, and its alias sends that expression.
- The PyO3 `pivot` builds its column with
`GroupedData::dataframe().col(name)`.
- Other places that PySpark builds through the DataFrame were checked:
`metadataColumn`, `df[...]`/`df.name` and the ordinals already refer to the
DataFrame, and the `agg(dict)` path uses `F.col` in PySpark too, so they are
unchanged.
Official PySpark 4.3.0 raises `CANNOT_RESOLVE_DATAFRAME_COLUMN` for
`df.withMetadata("nope", {...})` and `df.groupBy("k").pivot("nope").count()`,
the same as this change.
### Why are the changes needed?
```python
df.withMetadata("nope", {"a": "b"}).collect()
# AnalysisException: UNRESOLVED_COLUMN.WITH_SUGGESTION ...
df.groupBy("k").pivot("nope").count().collect()
# AnalysisException: UNRESOLVED_COLUMN.WITH_SUGGESTION ...
```
PySpark raises `CANNOT_RESOLVE_DATAFRAME_COLUMN` for both, so code that
catches that condition does not see these errors.
### Does this PR introduce _any_ user-facing change?
Yes. The error condition of a missing column in `withMetadata` and `pivot`
is `CANNOT_RESOLVE_DATAFRAME_COLUMN`, as in PySpark.
### How was this patch tested?
- Offline
`test_with_metadata_and_pivot_refer_to_their_column_through_the_dataframe_as_pyspark`:
the column of the `WithColumns` alias and of the pivot of an `Aggregate` is an
UnresolvedAttribute whose plan ID is that of the input.
- Live
`test_with_metadata_and_pivot_of_a_missing_column_are_dataframe_errors_as_pyspark`:
both missing-column calls raise `CANNOT_RESOLVE_DATAFRAME_COLUMN`, and an
existing column still gets its metadata. The same results with official PySpark
4.3.0.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and
the live tests of metadata, pivot and grouping (11 passed) against a local
Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics in the
changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #337 from HyukjinKwon/SPARK-61147-metadata-pivot-plan-id.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/group.rs | 4 +++-
crates/spark-connect/src/dataframe.rs | 1 +
crates/spark-connect/src/plan.rs | 14 ++++++++------
python/tests/test_dropin_e2e.py | 13 +++++++++++++
python/tests/test_dropin_offline.py | 33 +++++++++++++++++++++++++++++++++
5 files changed, 58 insertions(+), 7 deletions(-)
diff --git a/crates/pyspark-rs/src/group.rs b/crates/pyspark-rs/src/group.rs
index 1f148dc6..c98c4506 100644
--- a/crates/pyspark-rs/src/group.rs
+++ b/crates/pyspark-rs/src/group.rs
@@ -214,7 +214,9 @@ impl PyGroupedData {
Some(out)
}
};
- let pcol = spark_connect::functions::col(&name.to_cow()?);
+ // As PySpark's `self._df[pivot_col]`: a column of the DataFrame that
is grouped, by its
+ // plan ID.
+ let pcol = self.grouped_data.dataframe().col(&name.to_cow()?);
Ok(PyGroupedData::new(
self.grouped_data.pivot(pcol, vals).to_pyerr()?,
))
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 5ed29850..1f164b94 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -2054,6 +2054,7 @@ impl DataFrame {
let plan = LogicalPlan::WithColumnMetadata {
input: Arc::clone(&self.plan),
column_name: column_name.to_string(),
+ column: self.col(column_name).expression().clone(),
metadata_json: metadata_json.to_string(),
};
DataFrame::new(self.session.clone(), plan)
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 559c81b5..185cca19 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -181,6 +181,8 @@ pub enum LogicalPlan {
WithColumnMetadata {
input: Arc<LogicalPlan>,
column_name: String,
+ /// The column as a reference to the input's plan ID, as PySpark's
`self[columnName]`.
+ column: Expression,
metadata_json: String,
},
/// WithColumnsRenamed: `df.withColumnRenamed(old, new)` or
`df.withColumnsRenamed(...)`.
@@ -1048,16 +1050,16 @@ impl LogicalPlan {
LogicalPlan::WithColumnMetadata {
input,
column_name,
+ column,
metadata_json,
} => {
let mut wc = proto::WithColumns::default();
wc.input = Some(Box::new(relation_of(input, plan_ids)));
- let col_expr =
crate::expression::Expression::column(column_name.clone());
- let mut alias = proto::expression::Alias::default();
- alias.expr = Some(Box::new(col_expr.to_proto()));
- alias.name = vec![column_name.clone()];
- alias.metadata = Some(metadata_json.clone());
- wc.aliases.push(alias);
+ wc.aliases.push(proto::expression::Alias {
+ expr: Some(Box::new(column.to_proto())),
+ name: vec![column_name.clone()],
+ metadata: Some(metadata_json.clone()),
+ });
relation.rel_type =
Some(proto::relation::RelType::WithColumns(Box::new(wc)));
}
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index e70e9fbe..5972870e 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1512,6 +1512,19 @@ def
test_binary_values_of_collected_rows_follow_binary_as_bytes_as_pyspark(sessi
+def
test_with_metadata_and_pivot_of_a_missing_column_are_dataframe_errors_as_pyspark(session):
+ # As PySpark's: withMetadata and pivot refer to the column through the
DataFrame, so a
+ # missing column is CANNOT_RESOLVE_DATAFRAME_COLUMN, not UNRESOLVED_COLUMN.
+ df = session.createDataFrame([("a", 1)], "k string, v int")
+ with pytest.raises(AnalysisException) as error:
+ df.withMetadata("nope", {"a": "b"}).collect()
+ assert error.value.getCondition() == "CANNOT_RESOLVE_DATAFRAME_COLUMN"
+ with pytest.raises(AnalysisException) as error:
+ df.groupBy("k").pivot("nope").count().collect()
+ assert error.value.getCondition() == "CANNOT_RESOLVE_DATAFRAME_COLUMN"
+ assert df.withMetadata("v", {"a": "b"}).schema["v"].metadata == {"a": "b"}
+
+
def
test_internal_values_of_collected_rows_are_those_of_spark_as_pyspark(session):
# As PySpark's: the toInternal of a collected row is the internal value of
Spark that the
# server computes, the days and the microseconds since the epoch, and its
fromInternal is
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index a8d89486..2ce2d6d4 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6677,6 +6677,39 @@ def
test_sample_sort_group_by_and_drop_take_their_arguments_as_pyspark(mock_conn
assert error.value.getMessageParameters() == parameters
+def
test_with_metadata_and_pivot_refer_to_their_column_through_the_dataframe_as_pyspark(
+ mock_connect):
+ # As PySpark's withMetadata (self[columnName]) and pivot
(self._df[pivot_col]): the column
+ # is an UnresolvedAttribute with the plan ID of the DataFrame, which is
that of its input.
+ spark, service = mock_connect
+ df = spark.range(2)
+
+ def root(dataframe):
+ dataframe.collect()
+ # ExecutePlanRequest.plan=3, Plan.root=1.
+ return
parse_message(parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0])
+
+ def attribute_and_input_plan_ids(input_relation, expression):
+ # Relation.common=1, RelationCommon.plan_id=2;
Expression.unresolved_attribute=2,
+ # UnresolvedAttribute.plan_id=2.
+ return (parse_message(parse_message(input_relation)[1][0])[2][0],
+ parse_message(parse_message(expression)[2][0])[2][0])
+
+ # Relation.with_columns=23: WithColumns.input=1, aliases=2; Alias.expr=1.
+ with_columns = parse_message(root(df.withMetadata("id", {"a":
"b"}))[23][0])
+ alias = parse_message(with_columns[2][0])
+ input_plan_id, attribute_plan_id = attribute_and_input_plan_ids(
+ with_columns[1][0], alias[1][0])
+ assert attribute_plan_id == input_plan_id
+
+ # Relation.aggregate=9: Aggregate.input=1, pivot=5, Pivot.col=1.
+ aggregate = parse_message(root(df.groupBy("id").pivot("id").count())[9][0])
+ pivot = parse_message(aggregate[5][0])
+ input_plan_id, attribute_plan_id = attribute_and_input_plan_ids(
+ aggregate[1][0], pivot[1][0])
+ assert attribute_plan_id == input_plan_id
+
+
def test_ordinals_and_names_of_columns_carry_the_origin_of_col_as_pyspark():
# As PySpark's, `df[ordinal]`, and the columns of sort and groupBy of an
ordinal or a
# name, are of `F.col`, whose origin they carry. In a new interpreter,
whose first column
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]