This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 24bb0cfd [SPARK-61147][CONNECT][PYTHON] Refer to the column of 
withMetadata and pivot through the DataFrame, so a missing column is 
CANNOT_RESOLVE_DATAFRAME_COLUMN as in PySpark
24bb0cfd is described below

commit 24bb0cfd3a00090b6c76b8fc7624caeb10268cc0
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:11:09 2026 +0900

    [SPARK-61147][CONNECT][PYTHON] Refer to the column of withMetadata and 
pivot through the DataFrame, so a missing column is 
CANNOT_RESOLVE_DATAFRAME_COLUMN as in PySpark
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61147](https://issues.apache.org/jira/browse/SPARK-61147): Refer to 
the column of `withMetadata` and `pivot` through the DataFrame, so a missing 
column is `CANNOT_RESOLVE_DATAFRAME_COLUMN` as in PySpark.
    
    PySpark's `DataFrame.withMetadata` sends `self[columnName]` and 
`GroupedData.pivot` sends `self._df[pivot_col]`. Both are columns bound to the 
plan ID of the DataFrame. The Python client built them by name with 
`functions.col`, so the server resolved them by name against the plan. A 
missing column then raised `UNRESOLVED_COLUMN.WITH_SUGGESTION`.
    
    - `LogicalPlan::WithColumnMetadata` carries the column as an expression 
built by `DataFrame::col`, and its alias sends that expression.
    - The PyO3 `pivot` builds its column with 
`GroupedData::dataframe().col(name)`.
    - Other places that PySpark builds through the DataFrame were checked: 
`metadataColumn`, `df[...]`/`df.name` and the ordinals already refer to the 
DataFrame, and the `agg(dict)` path uses `F.col` in PySpark too, so they are 
unchanged.
    
    Official PySpark 4.3.0 raises `CANNOT_RESOLVE_DATAFRAME_COLUMN` for 
`df.withMetadata("nope", {...})` and `df.groupBy("k").pivot("nope").count()`, 
the same as this change.
    
    ### Why are the changes needed?
    
    ```python
    df.withMetadata("nope", {"a": "b"}).collect()
    # AnalysisException: UNRESOLVED_COLUMN.WITH_SUGGESTION ...
    df.groupBy("k").pivot("nope").count().collect()
    # AnalysisException: UNRESOLVED_COLUMN.WITH_SUGGESTION ...
    ```
    
    PySpark raises `CANNOT_RESOLVE_DATAFRAME_COLUMN` for both, so code that 
catches that condition does not see these errors.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The error condition of a missing column in `withMetadata` and `pivot` 
is `CANNOT_RESOLVE_DATAFRAME_COLUMN`, as in PySpark.
    
    ### How was this patch tested?
    
    - Offline 
`test_with_metadata_and_pivot_refer_to_their_column_through_the_dataframe_as_pyspark`:
 the column of the `WithColumns` alias and of the pivot of an `Aggregate` is an 
UnresolvedAttribute whose plan ID is that of the input.
    - Live 
`test_with_metadata_and_pivot_of_a_missing_column_are_dataframe_errors_as_pyspark`:
 both missing-column calls raise `CANNOT_RESOLVE_DATAFRAME_COLUMN`, and an 
existing column still gets its metadata. The same results with official PySpark 
4.3.0.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and 
the live tests of metadata, pivot and grouping (11 passed) against a local 
Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics in the 
changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #337 from HyukjinKwon/SPARK-61147-metadata-pivot-plan-id.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/group.rs        |  4 +++-
 crates/spark-connect/src/dataframe.rs |  1 +
 crates/spark-connect/src/plan.rs      | 14 ++++++++------
 python/tests/test_dropin_e2e.py       | 13 +++++++++++++
 python/tests/test_dropin_offline.py   | 33 +++++++++++++++++++++++++++++++++
 5 files changed, 58 insertions(+), 7 deletions(-)

diff --git a/crates/pyspark-rs/src/group.rs b/crates/pyspark-rs/src/group.rs
index 1f148dc6..c98c4506 100644
--- a/crates/pyspark-rs/src/group.rs
+++ b/crates/pyspark-rs/src/group.rs
@@ -214,7 +214,9 @@ impl PyGroupedData {
                 Some(out)
             }
         };
-        let pcol = spark_connect::functions::col(&name.to_cow()?);
+        // As PySpark's `self._df[pivot_col]`: a column of the DataFrame that 
is grouped, by its
+        // plan ID.
+        let pcol = self.grouped_data.dataframe().col(&name.to_cow()?);
         Ok(PyGroupedData::new(
             self.grouped_data.pivot(pcol, vals).to_pyerr()?,
         ))
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 5ed29850..1f164b94 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -2054,6 +2054,7 @@ impl DataFrame {
         let plan = LogicalPlan::WithColumnMetadata {
             input: Arc::clone(&self.plan),
             column_name: column_name.to_string(),
+            column: self.col(column_name).expression().clone(),
             metadata_json: metadata_json.to_string(),
         };
         DataFrame::new(self.session.clone(), plan)
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 559c81b5..185cca19 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -181,6 +181,8 @@ pub enum LogicalPlan {
     WithColumnMetadata {
         input: Arc<LogicalPlan>,
         column_name: String,
+        /// The column as a reference to the input's plan ID, as PySpark's 
`self[columnName]`.
+        column: Expression,
         metadata_json: String,
     },
     /// WithColumnsRenamed: `df.withColumnRenamed(old, new)` or 
`df.withColumnsRenamed(...)`.
@@ -1048,16 +1050,16 @@ impl LogicalPlan {
             LogicalPlan::WithColumnMetadata {
                 input,
                 column_name,
+                column,
                 metadata_json,
             } => {
                 let mut wc = proto::WithColumns::default();
                 wc.input = Some(Box::new(relation_of(input, plan_ids)));
-                let col_expr = 
crate::expression::Expression::column(column_name.clone());
-                let mut alias = proto::expression::Alias::default();
-                alias.expr = Some(Box::new(col_expr.to_proto()));
-                alias.name = vec![column_name.clone()];
-                alias.metadata = Some(metadata_json.clone());
-                wc.aliases.push(alias);
+                wc.aliases.push(proto::expression::Alias {
+                    expr: Some(Box::new(column.to_proto())),
+                    name: vec![column_name.clone()],
+                    metadata: Some(metadata_json.clone()),
+                });
                 relation.rel_type = 
Some(proto::relation::RelType::WithColumns(Box::new(wc)));
             }
 
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index e70e9fbe..5972870e 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1512,6 +1512,19 @@ def 
test_binary_values_of_collected_rows_follow_binary_as_bytes_as_pyspark(sessi
 
 
 
+def 
test_with_metadata_and_pivot_of_a_missing_column_are_dataframe_errors_as_pyspark(session):
+    # As PySpark's: withMetadata and pivot refer to the column through the 
DataFrame, so a
+    # missing column is CANNOT_RESOLVE_DATAFRAME_COLUMN, not UNRESOLVED_COLUMN.
+    df = session.createDataFrame([("a", 1)], "k string, v int")
+    with pytest.raises(AnalysisException) as error:
+        df.withMetadata("nope", {"a": "b"}).collect()
+    assert error.value.getCondition() == "CANNOT_RESOLVE_DATAFRAME_COLUMN"
+    with pytest.raises(AnalysisException) as error:
+        df.groupBy("k").pivot("nope").count().collect()
+    assert error.value.getCondition() == "CANNOT_RESOLVE_DATAFRAME_COLUMN"
+    assert df.withMetadata("v", {"a": "b"}).schema["v"].metadata == {"a": "b"}
+
+
 def 
test_internal_values_of_collected_rows_are_those_of_spark_as_pyspark(session):
     # As PySpark's: the toInternal of a collected row is the internal value of 
Spark that the
     # server computes, the days and the microseconds since the epoch, and its 
fromInternal is
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index a8d89486..2ce2d6d4 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6677,6 +6677,39 @@ def 
test_sample_sort_group_by_and_drop_take_their_arguments_as_pyspark(mock_conn
         assert error.value.getMessageParameters() == parameters
 
 
+def 
test_with_metadata_and_pivot_refer_to_their_column_through_the_dataframe_as_pyspark(
+        mock_connect):
+    # As PySpark's withMetadata (self[columnName]) and pivot 
(self._df[pivot_col]): the column
+    # is an UnresolvedAttribute with the plan ID of the DataFrame, which is 
that of its input.
+    spark, service = mock_connect
+    df = spark.range(2)
+
+    def root(dataframe):
+        dataframe.collect()
+        # ExecutePlanRequest.plan=3, Plan.root=1.
+        return 
parse_message(parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0])
+
+    def attribute_and_input_plan_ids(input_relation, expression):
+        # Relation.common=1, RelationCommon.plan_id=2; 
Expression.unresolved_attribute=2,
+        # UnresolvedAttribute.plan_id=2.
+        return (parse_message(parse_message(input_relation)[1][0])[2][0],
+                parse_message(parse_message(expression)[2][0])[2][0])
+
+    # Relation.with_columns=23: WithColumns.input=1, aliases=2; Alias.expr=1.
+    with_columns = parse_message(root(df.withMetadata("id", {"a": 
"b"}))[23][0])
+    alias = parse_message(with_columns[2][0])
+    input_plan_id, attribute_plan_id = attribute_and_input_plan_ids(
+        with_columns[1][0], alias[1][0])
+    assert attribute_plan_id == input_plan_id
+
+    # Relation.aggregate=9: Aggregate.input=1, pivot=5, Pivot.col=1.
+    aggregate = parse_message(root(df.groupBy("id").pivot("id").count())[9][0])
+    pivot = parse_message(aggregate[5][0])
+    input_plan_id, attribute_plan_id = attribute_and_input_plan_ids(
+        aggregate[1][0], pivot[1][0])
+    assert attribute_plan_id == input_plan_id
+
+
 def test_ordinals_and_names_of_columns_carry_the_origin_of_col_as_pyspark():
     # As PySpark's, `df[ordinal]`, and the columns of sort and groupBy of an 
ordinal or a
     # name, are of `F.col`, whose origin they carry. In a new interpreter, 
whose first column


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to