This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new c568153 [SPARK-60662][CONNECT][PYTHON] Share field names across
pandas input rows
c568153 is described below
commit c56815384a31694f123dfb218cc25262fc0166fc
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 16:02:53 2026 +0900
[SPARK-60662][CONNECT][PYTHON] Share field names across pandas input rows
### What changes were proposed in this pull request?
`createDataFrame(pandas_df)` now shares one immutable allocation of the
field names (`Arc<[String]>`, through `StructValue`/`Row`) across all the rows
it creates, instead of cloning the name list for every row. Names, schema
precedence, values and errors are unchanged. Master's rewritten local-row path
(`create_from_local`) already shares names this way; this does the same for the
pandas row path.
A new offline fixture, `python/tests/input_name_cases.py`, runs a mock
Connect gRPC server and checks the names and values in the Arrow IPC that
reaches it: explicit schemas, renamed columns, duplicates, Unicode, nulls,
empty input, caller mutation and errors.
`test_create_dataframe_input_names_offline` runs it in a subprocess, so its
session stays out of the rest of the offline suite.
### Why are the changes needed?
Cloning every field name for every input row wastes CPU and memory. An
earlier version of this change, measured on macOS arm64 with Python 3.12
against an older master, saved 7-8% CPU and 22-26 MiB of held memory on pandas
input of 10,000 x 32 and 100,000 x 3 values. It has not been re-measured on the
current master.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- The new fixture (4 tests), run by the new offline test.
- Against the current master, merged into this branch, and a local Spark
4.3.0 server: the lib and every integration test of `apache-spark-connect` and
`apache-spark-connect-core`, e2e included (771 passed), the build of all tests
with `wasm-udf`, the offline Python suites (721 passed, 9 skipped), the live
drop-in and Arrow suites (76 passed) and the live cases (10 passed). `cargo fmt
--all --check` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the merge with master and the review
follow-ups by Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #113 from dbtsai/codex/rsi-share-input-names.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Co-authored-by: DB Tsai <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 8 +-
python/tests/input_name_cases.py | 227 ++++++++++++++++++++++++++++++++++++
python/tests/test_dropin_offline.py | 15 +++
3 files changed, 246 insertions(+), 4 deletions(-)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 76895af..1f7af82 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -492,10 +492,10 @@ impl PySparkSession {
(None, Some(names))
}
};
- let row_names: Vec<String> = match (&given, &names) {
+ let row_names: std::sync::Arc<[String]> = match (&given, &names) {
(Some(fields), _) => fields.iter().map(|f|
f.name.clone()).collect(),
- (None, Some(names)) => names.clone(),
- (None, None) => Vec::new(),
+ (None, Some(names)) => names.clone().into(),
+ (None, None) => Vec::new().into(),
};
let mut rows: Vec<Row> = Vec::with_capacity(data.len());
for row in data.iter() {
@@ -511,7 +511,7 @@ impl PySparkSession {
values.len(),
));
}
- rows.push(Row::new(row_names.clone(), values));
+ rows.push(StructValue::new(row_names.clone(), values).into());
}
// A given struct wins; otherwise each field takes the type of its
first non-null
// value (empty input has a struct schema, checked above), but for
these, which are
diff --git a/python/tests/input_name_cases.py b/python/tests/input_name_cases.py
new file mode 100644
index 0000000..4bc5f43
--- /dev/null
+++ b/python/tests/input_name_cases.py
@@ -0,0 +1,227 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Check input names and complete IPC values without a Spark server or JVM."""
+
+from concurrent.futures import ThreadPoolExecutor
+import json
+import unittest
+
+try:
+ import grpc
+ import pandas as pd
+ import pyarrow as pa
+except ImportError:
+ grpc = pd = pa = None
+
+from pyspark.sql import Row, SparkSession, types as T
+
+
+def _varint(value):
+ out = bytearray()
+ while value > 127:
+ out.append((value & 127) | 128)
+ value >>= 7
+ out.append(value)
+ return bytes(out)
+
+
+def _bytes(number, value):
+ return _varint(number * 8 + 2) + _varint(len(value)) + value
+
+
+def _fields(data):
+ """Read the varint/bytes envelopes used by the fixture's protobuf
messages."""
+ def read(pos):
+ value = shift = 0
+ while True:
+ byte = data[pos]
+ pos += 1
+ value |= (byte & 127) << shift
+ if byte < 128:
+ return value, pos
+ shift += 7
+
+ pos = 0
+ result = {}
+ while pos < len(data):
+ tag, pos = read(pos)
+ wire = tag & 7
+ if wire == 2:
+ size, pos = read(pos)
+ result.setdefault(tag >> 3, []).append(data[pos:pos + size])
+ pos += size
+ elif wire == 0:
+ _, pos = read(pos)
+ else:
+ raise AssertionError(f"Unexpected protobuf wire type: {wire}")
+ return result
+
+
[email protected](grpc is None or pa is None, "grpcio, pandas and pyarrow are
required")
+class InputNameTests(unittest.TestCase):
+ def setUp(self):
+ self.captured = []
+
+ def execute(raw, context):
+ # ExecutePlanRequest.plan(3) -> Plan.root(1) ->
Relation.local_relation(11).
+ request = _fields(raw)
+ root = _fields(_fields(request[3][0])[1][0])
+ renames = []
+ while 18 in root: # Relation.to_df -> ToDF.input.
+ rename = _fields(root[18][0])
+ renames.append([name.decode() for name in rename.get(2, [])])
+ root = _fields(rename[1][0])
+ self.captured.append((_fields(root[11][0]), renames))
+ yield (_bytes(1, request[1][0]) + _bytes(15, b"input-name-fixture")
+ + _bytes(12, request[6][0]) + _bytes(13, b"complete")
+ + _bytes(14, b""))
+
+ def config(raw, context):
+ request = _fields(raw)
+ operation = _fields(request[3][0])
+ if 2 in operation: # ConfigRequest.Get.keys(1).
+ keys = _fields(operation[2][0])[1]
+ else:
+ self.assertEqual(set(operation), {3}) #
GetWithDefault.pairs(1).
+ keys = [_fields(pair)[1][0] for pair in
_fields(operation[3][0])[1]]
+ values = {
+ b"spark.sql.session.localRelationCacheThreshold": str(1 <<
60).encode(),
+ b"spark.sql.connect.serverStacktrace.enabled": b"false",
+ b"spark.sql.pyspark.jvmStacktrace.enabled": b"false",
+ }
+ pairs = b"".join(_bytes(2, _bytes(1, key) + _bytes(2, values[key]))
+ for key in keys)
+ return _bytes(1, request[1][0]) + _bytes(4, b"input-name-fixture")
+ pairs
+
+ def release(raw, context):
+ return _bytes(1, _fields(raw)[1][0])
+
+ pool = ThreadPoolExecutor(max_workers=2)
+ self.addCleanup(pool.shutdown)
+ server = grpc.server(pool)
+ server.add_generic_rpc_handlers([grpc.method_handlers_generic_handler(
+ "spark.connect.SparkConnectService", {
+ "ExecutePlan": grpc.unary_stream_rpc_method_handler(execute),
+ "Config": grpc.unary_unary_rpc_method_handler(config),
+ "ReleaseExecute": grpc.unary_unary_rpc_method_handler(release),
+ "ReleaseSession": grpc.unary_unary_rpc_method_handler(release),
+ }
+ )])
+ port = server.add_insecure_port("127.0.0.1:0")
+ self.assertNotEqual(port, 0)
+ server.start()
+ self.addCleanup(lambda: server.stop(0).wait())
+ self.spark =
SparkSession.builder.remote(f"sc://127.0.0.1:{port}").create()
+ self.addCleanup(self.spark.stop)
+
+ def capture(self, frame, expected_ipc_names=None):
+ before = len(self.captured)
+ frame.toArrow()
+ self.assertEqual(len(self.captured), before + 1)
+ local, renames = self.captured[-1]
+ schema = json.loads(local[2][0])
+ payload_names = [field["name"] for field in schema["fields"]]
+ if 1 in local:
+ table = pa.ipc.open_stream(local[1][0]).read_all()
+ table.validate(full=True)
+ self.assertEqual(table.schema.names, payload_names)
+ values = [[table.column(j)[i].as_py() for j in
range(table.num_columns)]
+ for i in range(table.num_rows)]
+ else:
+ values = []
+ if expected_ipc_names is not None:
+ self.assertEqual(payload_names, expected_ipc_names)
+ for names in renames:
+ self.assertEqual(len(names), len(payload_names))
+ # toDF changes observable column names without changing the local IPC
schema.
+ # Wrappers were captured from outside in, so the outermost names win.
+ return (renames[0] if renames else payload_names), values
+
+ def test_name_precedence_and_values(self):
+ cases = [
+ ([(7, 8), (9, 10)], None, ["_1", "_2"], ["_1", "_2"]),
+ ([Row(z=7, a=8), Row(z=9, a=10)], None, ["z", "a"], ["z", "a"]),
+ ([Row(z=7, a=8), Row(z=9, a=10)], ["x", "y"], ["x", "y"], ["z",
"a"]),
+ ([{"z": 8, "a": 7}, {"z": 10, "a": 9}], None, ["a", "z"], ["a",
"z"]),
+ ([{"z": 8, "a": 7}, {"z": 10, "a": 9}], ["x", "y"], ["x", "y"],
["a", "z"]),
+ (pd.DataFrame([[7, 8], [9, 10]], columns=["café", "雪"]),
+ None, ["café", "雪"], ["café", "雪"]),
+ (pd.DataFrame([[7, 8], [9, 10]], columns=["old", "names"]),
+ ["x", "y"], ["x", "y"], ["x", "y"]),
+ ]
+ for data, schema, names, ipc_names in cases:
+ with self.subTest(names=names, source=type(data).__name__):
+ self.assertEqual(self.capture(self.spark.createDataFrame(data,
schema), ipc_names),
+ (names, [[7, 8], [9, 10]]))
+
+ def test_unusual_names_nulls_and_empty_input(self):
+ names = ["dup", "dup", "", "café", "雪"]
+ data = [(1, 2, 3, 4, 5), (6, 7, 8, 9, 10)]
+ for schema in (names, T.StructType([
+ T.StructField(name, T.LongType(), True) for name in names])):
+ self.assertEqual(self.capture(self.spark.createDataFrame(data,
schema)),
+ (names, [list(row) for row in data]))
+ schema = T.StructType([T.StructField("number", T.LongType(), True),
+ T.StructField("text", T.StringType(), True)])
+ data = pd.DataFrame({"old": pd.Series([1, None], dtype=object),
+ "names": ["a", None]})
+ before = data.copy(deep=True)
+ self.assertEqual(self.capture(self.spark.createDataFrame(data,
schema)),
+ (["number", "text"], [[1, "a"], [None, None]]))
+ pd.testing.assert_frame_equal(data, before)
+ self.assertEqual(self.capture(self.spark.createDataFrame([], schema)),
+ (["number", "text"], []))
+
+ def test_each_call_owns_explicit_and_pandas_names(self):
+ names = ["first", "second"]
+ first = self.spark.createDataFrame([(1, 2), (3, 4)], names)
+ names[:] = ["new", "labels"]
+ second = self.spark.createDataFrame([(5, 6)], names)
+ self.assertEqual(self.capture(first), (["first", "second"], [[1, 2],
[3, 4]]))
+ self.assertEqual(self.capture(second), (["new", "labels"], [[5, 6]]))
+ self.assertEqual(self.capture(first), (["first", "second"], [[1, 2],
[3, 4]]))
+ data = pd.DataFrame([[7, 8]], columns=["old", "names"])
+ frame = self.spark.createDataFrame(data)
+ data.columns = ["changed", "later"]
+ self.assertEqual(self.capture(frame), (["old", "names"], [[7, 8]]))
+
+ def test_explicit_names_still_parse_overridden_inputs(self):
+ trace = []
+
+ class BrokenKey:
+ def __str__(self):
+ trace.append("key")
+ raise ValueError("key conversion failed")
+
+ with self.assertRaisesRegex(ValueError, "key conversion failed"):
+ self.spark.createDataFrame([{BrokenKey(): 1}], ["override"])
+ self.assertEqual(trace, ["key"])
+ self.assertEqual(
+ self.capture(self.spark.createDataFrame(pd.DataFrame([[1]],
columns=[7]), ["override"])),
+ (["override"], [[1]]),
+ )
+ from pyspark.errors import PySparkTypeError
+
+ with self.assertRaises(PySparkTypeError) as error:
+ self.spark.createDataFrame([(object(),)], ["x", "y"])
+ self.assertEqual(error.exception.getCondition(),
"CANNOT_INFER_TYPE_FOR_FIELD")
+ self.assertEqual(error.exception.getMessageParameters(),
{"field_name": "x"})
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 380fe6d..f7e4110 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -32,6 +32,21 @@ from pyspark.sql import Row, SparkSession, types as T
from day_time_interval_cases import DayTimeIntervalTests # noqa: F401
+def test_create_dataframe_input_names_offline():
+ # Keep the fixture's active session out of this process's offline tests.
+ import subprocess
+ import sys
+ from pathlib import Path
+
+ for dependency in ("grpc", "pandas", "pyarrow"):
+ pytest.importorskip(dependency)
+ result = subprocess.run(
+ [sys.executable, str(Path(__file__).with_name("input_name_cases.py"))],
+ capture_output=True, text=True,
+ )
+ assert result.returncode == 0, result.stdout + result.stderr
+
+
# ---------------------------------------------------------------------------
types
ATOMIC = [
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]