This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new e0fc43c6 [SPARK-61146][CONNECT][PYTHON] Create DataFrames of pandas 
frames whose struct fields repeat names in the Python client, as PySpark does
e0fc43c6 is described below

commit e0fc43c63da2d11091b32f210c08af84194d1f8e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:27:16 2026 +0900

    [SPARK-61146][CONNECT][PYTHON] Create DataFrames of pandas frames whose 
struct fields repeat names in the Python client, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61146](https://issues.apache.org/jira/browse/SPARK-61146): PySpark 
Connect's `createDataFrame` of a pandas DataFrame under a struct schema whose 
structs repeat field names, such as `struct<x: string, x: int>`:
    - converts the frame with the deduplicated names of 
`_deduplicate_field_names` (`x_0`, `x_1`)
    - sends that Arrow table with the schema as given, which the server applies
    
    The client converted the frame the same way, in `prepare_pandas_table`. But 
it then normalized the table again as an Arrow table of the caller 
(`prepare_table`), which may not repeat the field names of a struct. It now 
sends the converted table with the schema, as PySpark Connect does.
    
    The second normalization changed nothing else for this table. Its types are 
already those of the schema, and its timestamps are already of UTC. It also 
fetched the Arrow input settings a second time, so `createDataFrame` of such a 
frame now sends one `Config` request fewer (`integer_tuple_cases.py` counts 
them).
    
    ### Why are the changes needed?
    
    The client raised:
    
    ```
    UnsupportedOperationException: [DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT] 
Duplicated field names in Arrow Struct are not allowed, got ['x', 'x']
    ```
    
    Apache Spark's `test_createDataFrame_pandas_duplicate_field_names` of 
`test_parity_arrow.py` failed on this client.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A pandas frame under a schema whose structs repeat field names becomes 
a DataFrame, as with PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new offline test checks what is sent for such a frame: the Arrow table 
of the deduplicated names, of the values, and the schema as given. It fails 
without this change.
    - A new live test creates DataFrames of such a frame, with Arrow and 
without, and checks their schemas and rows. It passes with official PySpark 
4.3.0, and fails without this change.
    - The schemas and rows of the case of Apache Spark's test, and of 
timestamps, at the top level and in a struct under a client time zone unlike 
the session's, are those of official PySpark 4.3.0.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2205 passed, 8 skipped), and the 
live drop-in and Arrow suites (188 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #345 from HyukjinKwon/SPARK-61146-pandas-duplicate-names.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs    | 21 ++++++++++++++-------
 python/tests/integer_tuple_cases.py |  5 +++--
 python/tests/test_dropin_e2e.py     | 27 +++++++++++++++++++++++++++
 python/tests/test_dropin_offline.py | 24 ++++++++++++++++++++++++
 4 files changed, 68 insertions(+), 9 deletions(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 9e05146f..b19a8445 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -794,14 +794,21 @@ impl PySparkSession {
             safecheck,
             large,
         ))?;
-        let frame = crate::pyarrow_import::create_dataframe(
-            py,
-            &self.session,
-            &table,
-            Some(schema_py.bind(py)),
-        )?;
+        // As PySpark Connect, the table, cast to the Arrow types of the 
schema of deduplicated
+        // nested field names, is sent with the schema as given, which the 
server applies. It is
+        // not normalized again as an Arrow table of the caller, which may not 
repeat the names
+        // of the fields of a struct (DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT).
+        let spark_schema = crate::types::py_to_data_type(schema_py.bind(py))?;
+        let batches = crate::pyarrow_import::table_batches(&table)?;
+        let df = py
+            .detach(|| {
+                self.session
+                    .create_dataframe_from_arrow_with_options(batches, 
Some(spark_schema), &options)
+                    .map_err(Box::new)
+            })
+            .to_pyerr()?;
         // As the other pandas paths, the explicit names are applied with toDF.
-        Ok(Some(PyDataFrame::new(frame.dataframe.to_df(
+        Ok(Some(PyDataFrame::new(df.to_df(
             fields.iter().map(|field| field.name.as_str()).collect(),
         ))))
     }
diff --git a/python/tests/integer_tuple_cases.py 
b/python/tests/integer_tuple_cases.py
index ed49ae29..aa4fce94 100644
--- a/python/tests/integer_tuple_cases.py
+++ b/python/tests/integer_tuple_cases.py
@@ -145,8 +145,9 @@ class IntegerTupleTests(unittest.TestCase):
         frame = self.spark.createDataFrame(data, schema)
         self.assertEqual(len(self.captured), before)  # construction remains 
offline
         # Explicit-schema pandas input reads the Arrow conversion settings 
during
-        # construction, as PySpark does: the normalization and safe-cast 
configuration.
-        expected = 3 if isinstance(data, pd.DataFrame) else 0
+        # construction, as PySpark does: the Arrow input settings and the 
safe-cast
+        # configuration, once each.
+        expected = 2 if isinstance(data, pd.DataFrame) else 0
         self.assertEqual(self.config_calls - config_before, expected)
         config_before = self.config_calls
         frame.toArrow()
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 2de54daa..01e51120 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2637,6 +2637,33 @@ def 
test_struct_types_convert_to_the_ddl_of_spark_as_pyspark(session):
 
 
 
+def 
test_pandas_frames_of_duplicate_nested_field_names_are_created_as_pyspark(session):
+    # As PySpark's test_createDataFrame_pandas_duplicate_field_names: the 
structs of the schema
+    # repeat field names, with Arrow and without.
+    import pandas as pd
+    from pyspark.sql.types import StructType
+
+    schema = (
+        StructType()
+        .add("struct", StructType().add("x", StringType()).add("x", 
IntegerType()))
+        .add(
+            "struct",
+            StructType().add("a", IntegerType()).add("x", 
IntegerType()).add("x", StringType()),
+        )
+    )
+    data = [Row(Row("a", 1), Row(2, 3, "b")), Row(Row("x", None), Row(7, 8, 
None))]
+    frame = pd.DataFrame.from_records(data, columns=schema.names)
+    try:
+        for arrow in ("true", "false"):
+            session.conf.set("spark.sql.execution.arrow.pyspark.enabled", 
arrow)
+            df = session.createDataFrame(frame, schema)
+            assert df.schema == schema and df.collect() == data, arrow
+    finally:
+        session.conf.unset("spark.sql.execution.arrow.pyspark.enabled")
+
+
+
+
 def test_maps_keep_their_entries_in_order_with_keys_of_their_type(session):
     row = session.range(1).selectExpr(
         "map(2, 'b', 1, 'a') AS ints",
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 654a8d84..5da128c7 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7086,6 +7086,30 @@ def 
test_local_input_native_number_values_and_types(mock_connect, value):
         assert actual == value
 
 
+def 
test_pandas_frames_of_duplicate_nested_field_names_are_sent_as_pyspark(mock_connect):
+    # As PySpark Connect: a pandas frame under a schema whose structs repeat 
field names is sent
+    # as the Arrow table of the deduplicated names (x_0, x_1), with the schema 
as given, which
+    # the server applies. Before, the client raised 
DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT.
+    pd = pytest.importorskip("pandas")
+    pa = pytest.importorskip("pyarrow")
+
+    spark, service = mock_connect
+    schema = T.StructType().add(
+        "s", T.StructType().add("x", T.StringType()).add("x", T.IntegerType())
+    )
+    frame = pd.DataFrame.from_records([(Row("a", 1),), (Row("b", None),)], 
columns=schema.names)
+    spark.createDataFrame(frame, schema).toArrow()
+    relation = 
parse_message(parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0])
+    while 18 in relation:  # Relation.to_df -> ToDF.input.
+        relation = parse_message(parse_message(relation[18][0])[1][0])
+    # Relation.local_relation=11 -> LocalRelation.data=1, schema=2.
+    local = parse_message(relation[11][0])
+    table = pa.ipc.open_stream(local[1][0]).read_all()
+    assert table.schema.field("s").type == pa.struct([("x_0", pa.string()), 
("x_1", pa.int32())])
+    assert table.to_pylist() == [{"s": {"x_0": "a", "x_1": 1}}, {"s": {"x_0": 
"b", "x_1": None}}]
+    assert T.StructType.fromJson(json.loads(local[2][0])) == schema
+
+
 def test_pandas_input_overflow_raises_as_pyspark(mock_connect):
     # Without a schema, PySpark infers the Arrow type of an object column, so 
an integer
     # outside the int64 range raises OverflowError rather than becoming a 
double.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to