This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 345a1e7 [SPARK-61029][CONNECT][PYTHON] Parse the DDL schema strings
of createDataFrame and the map functions on the server as PySpark does
345a1e7 is described below
commit 345a1e719ffda85932b6bacbfaefd1dd7b617a22
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 08:35:53 2026 +0900
[SPARK-61029][CONNECT][PYTHON] Parse the DDL schema strings of
createDataFrame and the map functions on the server as PySpark does
### What changes were proposed in this pull request?
[SPARK-61029](https://issues.apache.org/jira/browse/SPARK-61029): the
Python client now parses the DDL string schemas of `createDataFrame` and of the
map functions on the server, as PySpark Connect does.
PySpark Connect parses a DDL string with `_parse_ddl`, the `DDLParse` of an
`AnalyzePlanRequest`. `spark.createDataFrame(data, schema)`, `mapInPandas`,
`mapInArrow`, and the `applyInPandas` and `applyInArrow` of grouped and
cogrouped data parse their schemas that way. So they take any DDL that Spark
does:
- nested types, such as `"st struct<a:int,b:double>"` and `"m
map<string,struct<x:int>>"`
- backtick-quoted names, such as ``"`a b` int"``
- collations, such as `"s STRING COLLATE UTF8_LCASE"`
- `NOT NULL` and `COMMENT` clauses
They also raise the errors of Spark's parser, such as `PARSE_SYNTAX_ERROR`
for `"a INT,"`. `transformWithStateInPandas` sends a DDL output schema as an
`UnparsedDataType`, for the server to parse.
The core adds `SparkSession::parse_ddl`, which asks the server to parse a
DDL string, and `spark_session()` accessors for `GroupedData` and
`CoGroupedData`.
### Why are the changes needed?
The client parsed these schemas itself, and its parser differs from Spark's:
- It raised `ValueError: invalid type string 'st struct<a:int,b:double>':
SparkError { kind: ValueError, error_class: "INVALID_DATATYPE_FORMAT", ... }`
for the nested, quoted, `NOT NULL` and `COMMENT` forms above.
- It took `"a INT,"` as `a INT`.
- For `"s STRING COLLATE UTF8_LCASE"`, it sent a type that the server
failed to parse (`PARSE_SYNTAX_ERROR`).
### Does this PR introduce _any_ user-facing change?
Yes. These functions take the DDL schemas of PySpark 4.3.0 and raise its
errors.
### How was this patch tested?
- A new live test creates DataFrames of these schemas and checks their
types and rows, the collation, the errors, and the `NOT NULL` and `COMMENT` of
a `mapInPandas` schema. It passes with official PySpark 4.3.0 and fails without
this change.
- A new e2e test checks `SparkSession::parse_ddl` against the server,
including its errors.
- A new offline test checks the DDL strings that `createDataFrame` and the
map functions send to a mock Connect server. The mock servers of the offline
and the Arrow tests now answer `DDLParse` requests.
- On a local server, 59 DDL schemas give the same types and errors as
official PySpark 4.3.0 for `createDataFrame`. So do 36 cases of the map and
apply functions.
- The lib tests of both crates (571 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (1810 passed, 8 skipped), and the
live drop-in and Arrow suites (109 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #235 from HyukjinKwon/SPARK-61029-ddl.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 22 ++-
crates/pyspark-rs/src/group.rs | 28 +++-
crates/pyspark-rs/src/pyarrow_import.rs | 2 +-
crates/pyspark-rs/src/session.rs | 10 +-
crates/pyspark-rs/src/types.rs | 17 +++
crates/spark-connect/src/group.rs | 10 ++
crates/spark-connect/src/session.rs | 24 ++++
crates/spark-connect/tests/e2e_session_catalog.rs | 30 +++++
python/tests/_mock_connect_server.py | 156 ++++++++++++++++++++++
python/tests/test_arrow_handoff.py | 48 ++++++-
python/tests/test_dropin_e2e.py | 31 +++++
python/tests/test_dropin_offline.py | 57 +++++++-
12 files changed, 418 insertions(+), 17 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 0c200cb..0888ba2 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2061,6 +2061,7 @@ impl PyDataFrame {
let cols = py.detach(|| self.dataframe.columns()).to_pyerr()?;
let udf = build_map_udf(
py,
+ &self.dataframe.spark_session(),
&func,
&schema,
eval_type::SQL_MAP_PANDAS_ITER_UDF,
@@ -2086,7 +2087,14 @@ impl PyDataFrame {
) -> PyResult<PyDataFrame> {
validate_map_function(py, &func, eval_type::SQL_MAP_ARROW_ITER_UDF)?;
let cols = py.detach(|| self.dataframe.columns()).to_pyerr()?;
- let udf = build_map_udf(py, &func, &schema,
eval_type::SQL_MAP_ARROW_ITER_UDF, &cols)?;
+ let udf = build_map_udf(
+ py,
+ &self.dataframe.spark_session(),
+ &func,
+ &schema,
+ eval_type::SQL_MAP_ARROW_ITER_UDF,
+ &cols,
+ )?;
let profile_id = profile_id_of(profile.as_ref())?;
Ok(PyDataFrame::new(
self.dataframe
@@ -3181,15 +3189,18 @@ fn bind_arguments<'py, const N: usize>(
}))
}
-/// Resolve a Python schema argument (a `DataType` wrapper or a DDL string)
into a Rust DataType.
+/// Resolve a Python schema argument (a `DataType` wrapper or a DDL string)
into a Rust
+/// DataType: a DDL string as the server of `session` parses it, as PySpark's
`_parse_ddl`.
pub(crate) fn resolve_datatype(
+ py: Python<'_>,
+ session: &spark_connect::session::SparkSession,
schema: &Bound<'_, PyAny>,
) -> PyResult<spark_connect::types::DataType> {
if let Ok(dt) = schema.extract::<PyRef<PyDataType>>() {
return Ok(dt.inner.clone());
}
- if let Ok(s) = schema.extract::<String>() {
- return spark_connect::types::DataType::from_ddl(&s).to_pyerr();
+ if schema.is_instance_of::<pyo3::types::PyString>() {
+ return crate::types::schema_data_type(py, session, schema);
}
Err(pyo3::exceptions::PyTypeError::new_err(
"schema must be a DataType or a DDL string",
@@ -3259,13 +3270,14 @@ fn udf_name(func: &Bound<'_, PyAny>) ->
PyResult<String> {
/// carry the resolved schema as the output type (mirrors the scalar UDF
command layout).
pub(crate) fn build_map_udf(
py: Python<'_>,
+ session: &spark_connect::session::SparkSession,
func: &Bound<'_, PyAny>,
schema: &Bound<'_, PyAny>,
eval_type_id: i32,
input_cols: &[String],
) -> PyResult<CommonInlineUserDefinedFunctionExpression> {
let name = udf_name(func)?;
- let return_type = resolve_datatype(schema)?;
+ let return_type = resolve_datatype(py, session, schema)?;
// The worker reads the pickled type as a `DataType`, so a DDL string is
pickled as the
// type it parses to, as the reference client pickles the type of
`_parse_ddl`.
let pickled_type = if schema.is_instance_of::<pyo3::types::PyString>() {
diff --git a/crates/pyspark-rs/src/group.rs b/crates/pyspark-rs/src/group.rs
index a7463f2..2666823 100644
--- a/crates/pyspark-rs/src/group.rs
+++ b/crates/pyspark-rs/src/group.rs
@@ -169,7 +169,8 @@ impl PyGroupedData {
)?;
validate_map_function(py, &func, eval_type_id)?;
let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
- let udf = build_map_udf(py, &func, &schema, eval_type_id, &cols)?;
+ let session = self.grouped_data.spark_session();
+ let udf = build_map_udf(py, &session, &func, &schema, eval_type_id,
&cols)?;
Ok(PyDataFrame::new(self.grouped_data.apply_in_pandas(udf)))
}
@@ -204,7 +205,8 @@ impl PyGroupedData {
)?;
validate_map_function(py, &func, eval_type_id)?;
let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
- let udf = build_map_udf(py, &func, &schema, eval_type_id, &cols)?;
+ let session = self.grouped_data.spark_session();
+ let udf = build_map_udf(py, &session, &func, &schema, eval_type_id,
&cols)?;
Ok(PyDataFrame::new(self.grouped_data.apply_in_arrow(udf)))
}
@@ -230,14 +232,16 @@ impl PyGroupedData {
) -> PyResult<PyDataFrame> {
validate_map_function(py, &func,
eval_type::SQL_GROUPED_MAP_PANDAS_UDF_WITH_STATE)?;
let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
+ let session = self.grouped_data.spark_session();
let udf = build_map_udf(
py,
+ &session,
&func,
&outputStructType,
eval_type::SQL_GROUPED_MAP_PANDAS_UDF_WITH_STATE,
&cols,
)?;
- let state_schema =
crate::dataframe::resolve_datatype(&stateStructType)?;
+ let state_schema = crate::dataframe::resolve_datatype(py, &session,
&stateStructType)?;
Ok(PyDataFrame::new(
self.grouped_data.apply_in_pandas_with_state(
udf,
@@ -273,8 +277,17 @@ impl PyGroupedData {
true,
)?;
let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
- let udf = build_map_udf(py, &func, &outputStructType, et, &cols)?;
- let out = crate::types::py_to_data_type(&outputStructType)?;
+ let session = self.grouped_data.spark_session();
+ let udf = build_map_udf(py, &session, &func, &outputStructType, et,
&cols)?;
+ // As PySpark's, which sends a DDL string as an `UnparsedDataType`,
for the server to
+ // parse.
+ let out = if
outputStructType.is_instance_of::<pyo3::types::PyString>() {
+ spark_connect::types::DataType::Unparsed {
+ data_type_string: outputStructType.extract()?,
+ }
+ } else {
+ crate::types::py_to_data_type(&outputStructType)?
+ };
let etc =
(!eventTimeColumnName.is_empty()).then_some(eventTimeColumnName);
Ok(PyDataFrame::new(
self.grouped_data.transform_with_state_in_pandas(
@@ -311,7 +324,8 @@ impl PyGroupedData {
false,
)?;
let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
- let udf = build_map_udf(py, &func, &outputStructType, et, &cols)?;
+ let session = self.grouped_data.spark_session();
+ let udf = build_map_udf(py, &session, &func, &outputStructType, et,
&cols)?;
let etc =
(!eventTimeColumnName.is_empty()).then_some(eventTimeColumnName);
Ok(PyDataFrame::new(self.grouped_data.transform_with_state(
udf,
@@ -378,6 +392,7 @@ impl PyCoGroupedData {
let cols = py.detach(|| self.inner.input_columns()).to_pyerr()?;
let udf = build_map_udf(
py,
+ &self.inner.spark_session(),
&func,
&schema,
eval_type::SQL_COGROUPED_MAP_PANDAS_UDF,
@@ -398,6 +413,7 @@ impl PyCoGroupedData {
let cols = py.detach(|| self.inner.input_columns()).to_pyerr()?;
let udf = build_map_udf(
py,
+ &self.inner.spark_session(),
&func,
&schema,
eval_type::SQL_COGROUPED_MAP_ARROW_UDF,
diff --git a/crates/pyspark-rs/src/pyarrow_import.rs
b/crates/pyspark-rs/src/pyarrow_import.rs
index eb39cfb..e2fea5f 100644
--- a/crates/pyspark-rs/src/pyarrow_import.rs
+++ b/crates/pyspark-rs/src/pyarrow_import.rs
@@ -36,7 +36,7 @@ pub(crate) fn create_dataframe(
let schema = match schema {
Some(s) if s.extract::<String>().is_ok() =>
Some(crate::types::data_type_to_py(
py,
- &crate::types::py_to_data_type(s)?,
+ &crate::types::schema_data_type(py, session, s)?,
)?),
Some(s) => Some(s.clone().unbind()),
None => None,
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 35b1084..43405af 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -712,7 +712,11 @@ enum Spec {
/// The schema argument of `createDataFrame`: None, a list or tuple of names,
or a struct or
/// atomic type, given as a type or a DDL string. Other types raise
`NOT_EXPECTED_TYPE`.
-fn schema_spec(py: Python<'_>, schema: Option<&Bound<'_, PyAny>>) ->
PyResult<Spec> {
+fn schema_spec(
+ py: Python<'_>,
+ session: &spark_connect::session::SparkSession,
+ schema: Option<&Bound<'_, PyAny>>,
+) -> PyResult<Spec> {
let Some(schema) = schema.filter(|s| !s.is_none()) else {
return Ok(Spec::None);
};
@@ -745,7 +749,7 @@ fn schema_spec(py: Python<'_>, schema: Option<&Bound<'_,
PyAny>>) -> PyResult<Sp
{
return Err(not_expected(&crate::infer::type_name(schema)?));
}
- Ok(match crate::types::py_to_data_type(schema)? {
+ Ok(match crate::types::schema_data_type(py, session, schema)? {
DataType::Struct { fields } => Spec::Struct(fields),
atomic if crate::infer::is_atomic(&atomic) => Spec::Atomic(atomic),
other => return Err(not_expected(&crate::infer::class_name(&other))),
@@ -917,7 +921,7 @@ impl PySparkSession {
if is_loaded_instance(py, data, "pyarrow", "Table")? {
return crate::pyarrow_import::create_dataframe(py, &self.session,
data, schema);
}
- let spec = schema_spec(py, schema)?;
+ let spec = schema_spec(py, &self.session, schema)?;
if is_loaded_instance(py, data, "pandas", "DataFrame")? {
// Guarded integer columns share the existing Arrow importer.
Schema
// validation above and unsupported pandas conversions keep
master's
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 7aa22f2..033f006 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -2934,6 +2934,23 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt:
&DataType) -> PyResult<Py<PyAn
Ok(o)
}
+/// The type of `schema`, a DataType or a DDL string, which the server parses,
as PySpark
+/// Connect's `_parse_ddl` does: so any DDL of Spark, with nested types,
quoted names,
+/// collations and the NOT NULL and COMMENT of fields, parses as Spark parses
it.
+pub(crate) fn schema_data_type(
+ py: Python<'_>,
+ session: &spark_connect::session::SparkSession,
+ schema: &Bound<'_, PyAny>,
+) -> PyResult<DataType> {
+ if schema.is_instance_of::<pyo3::types::PyString>() {
+ let ddl: String = schema.extract()?;
+ return crate::errors::ResultExt::to_pyerr(
+ py.detach(|| session.parse_ddl(&ddl).map_err(Box::new)),
+ );
+ }
+ py_to_data_type(schema)
+}
+
pub(crate) fn py_to_data_type(obj: &Bound<'_, PyAny>) -> PyResult<DataType> {
if let Ok(g) = obj.extract::<PyRef<PyGeometryType>>() {
return Ok(DataType::Geometry { srid: g.srid });
diff --git a/crates/spark-connect/src/group.rs
b/crates/spark-connect/src/group.rs
index 915a6e4..3c976f4 100644
--- a/crates/spark-connect/src/group.rs
+++ b/crates/spark-connect/src/group.rs
@@ -120,6 +120,11 @@ impl GroupedData {
self.dataframe.columns()
}
+ /// The session of the grouped DataFrame.
+ pub fn spark_session(&self) -> crate::session::SparkSession {
+ self.dataframe.spark_session()
+ }
+
/// Grouping-key expressions for this grouped data.
fn grouping_expressions(&self) -> Vec<Expression> {
self.group_cols
@@ -668,6 +673,11 @@ impl CoGroupedData {
Ok(cols)
}
+ /// The session of the cogrouped DataFrames.
+ pub fn spark_session(&self) -> crate::session::SparkSession {
+ self.left.spark_session()
+ }
+
/// Apply a pandas UDF to each cogroup (`cogroup(...).applyInPandas`).
///
/// `func` is built on the Python side (cloudpickled, eval type
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index d31f33a..010ccac 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -260,6 +260,30 @@ impl SparkSession {
Ok(DataFrame::new(self.clone(), plan))
}
+ /// Parse a DDL string on the server, mirroring PySpark Connect's
`_parse_ddl`: the
+ /// `DDLParse` of an `AnalyzePlanRequest`, of a schema or a type, as Spark
parses it.
+ pub fn parse_ddl(&self, ddl: &str) -> Result<DataType> {
+ let request = proto::AnalyzePlanRequest {
+ session_id: self.client().session_id().to_string(),
+ user_context: Some(proto::UserContext::default()),
+ analyze: Some(proto::analyze_plan_request::Analyze::DdlParse(
+ proto::analyze_plan_request::DdlParse {
+ ddl_string: ddl.to_string(),
+ },
+ )),
+ ..Default::default()
+ };
+ let response = block_on(self.client().analyze_plan(request))?;
+ match response.result {
+ Some(proto::analyze_plan_response::Result::DdlParse(r)) => {
+ DataType::from_proto(&r.parsed.unwrap_or_default())
+ }
+ _ => Err(SparkError::connect_msg(
+ "AnalyzePlan response did not contain a DDL parse result",
+ )),
+ }
+ }
+
/// Execute a SQL statement and return a DataFrame.
///
/// Mirrors `pyspark.sql.SparkSession.sql(sqlQuery)`: the statement is
sent at once, so a
diff --git a/crates/spark-connect/tests/e2e_session_catalog.rs
b/crates/spark-connect/tests/e2e_session_catalog.rs
index 1e1fae8..3b40581 100644
--- a/crates/spark-connect/tests/e2e_session_catalog.rs
+++ b/crates/spark-connect/tests/e2e_session_catalog.rs
@@ -292,3 +292,33 @@ fn sql_resolves_the_dataframes_that_the_query_names() {
assert_eq!(shown.collect().expect("collect").len(), 2);
let _ = spark.stop();
}
+
+/// `parse_ddl` parses a DDL string on the server, as PySpark Connect's
`_parse_ddl`: of a
+/// schema or a type, with the NOT NULL and COMMENT of fields, and the
server's errors.
+#[test]
+fn parse_ddl_parses_on_the_server() {
+ if !should_run() {
+ return;
+ }
+ use spark_connect::types::DataType;
+ let spark = session();
+ let parsed = spark
+ .parse_ddl("a INT NOT NULL COMMENT 'c', b STRUCT<x: DOUBLE>")
+ .expect("parse");
+ let DataType::Struct { fields } = parsed else {
+ panic!("expected a struct, got {parsed:?}");
+ };
+ assert_eq!(
+ fields.iter().map(|f| f.name.as_str()).collect::<Vec<_>>(),
+ vec!["a", "b"]
+ );
+ assert!(!fields[0].nullable);
+ assert_eq!(
+ fields[0].metadata.get("comment"),
+ Some(&serde_json::Value::String("c".to_string()))
+ );
+ assert_eq!(spark.parse_ddl("bigint").expect("parse"), DataType::Long);
+ let error = spark.parse_ddl("a INT,").expect_err("a syntax error");
+ assert_eq!(error.error_class, "PARSE_SYNTAX_ERROR");
+ let _ = spark.stop();
+}
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index 2a7400a..bfe773c 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -16,7 +16,9 @@
#
"""Small, separately hosted Spark Connect service for transport-level tests."""
+import json
import multiprocessing
+import re
from concurrent.futures import ThreadPoolExecutor
from contextlib import contextmanager
from threading import Event
@@ -77,6 +79,153 @@ def parse_message(data):
return fields
+# The JSON names of the atomic types of DDL names, and the DataType kinds
(types.proto) of the
+# JSON names.
+_ATOMIC_JSON = {
+ "void": "void", "null": "void", "binary": "binary", "boolean": "boolean",
+ "bool": "boolean", "byte": "byte", "tinyint": "byte", "short": "short",
+ "smallint": "short", "int": "integer", "integer": "integer", "long":
"long",
+ "bigint": "long", "float": "float", "real": "float", "double": "double",
+ "string": "string", "date": "date", "timestamp": "timestamp",
"timestamp_ltz": "timestamp",
+ "timestamp_ntz": "timestamp_ntz", "variant": "variant",
+}
+_PROTO_KINDS = {
+ "void": 1, "binary": 2, "boolean": 3, "byte": 4, "short": 5, "integer": 6,
"long": 7,
+ "float": 8, "double": 9, "string": 11, "date": 14, "timestamp": 15,
"timestamp_ntz": 16,
+ "variant": 25,
+}
+_DAY_TIME = ["day", "hour", "minute", "second"]
+_YEAR_MONTH = ["year", "month"]
+_DDL_FIELD = re.compile(
+
r"\s*(?P<name>`[^`]+`|\w+)(?:\s*:\s*|\s+)(?P<type>.+?)(?P<not_null>\s+not\s+null)?"
+ r"(?:\s+comment\s+'(?P<comment>[^']*)')?\s*", re.IGNORECASE | re.DOTALL)
+
+
+def _top_level_split(text):
+ parts, depth, start = [], 0, 0
+ for i, char in enumerate(text):
+ depth += {"<": 1, ">": -1, "(": 1, ")": -1}.get(char, 0)
+ if char == "," and depth == 0:
+ parts.append(text[start:i])
+ start = i + 1
+ return parts + [text[start:]]
+
+
+def ddl_json(text):
+ """The JSON value of the type of a DDL string, as Spark's DDL parser
parses those of the
+ tests: atomic types, decimal(p, s), time(p), intervals, arrays, maps and
structs, and
+ schemas of `name type` and `name: type` fields, of NOT NULL and COMMENT.
None for any
+ other DDL."""
+ text = text.strip()
+ lower = text.lower()
+ if lower in _ATOMIC_JSON:
+ return _ATOMIC_JSON[lower]
+ match = re.fullmatch(
+ r"(?:decimal|dec|numeric)\s*(?:\(\s*(\d+)\s*(?:,\s*(\d+)\s*)?\))?",
lower)
+ if match:
+ return f"decimal({int(match[1] or 10)},{int(match[2] or 0)})"
+ match = re.fullmatch(r"time\s*(?:\(\s*(\d+)\s*\))?", lower)
+ if match:
+ return f"time({int(match[1] or 6)})"
+ match = re.fullmatch(r"(geometry|geography)\s*\(\s*(\d+)\s*\)", lower)
+ if match:
+ crs = "OGC:CRS84" if match[2] == "4326" else f"SRID:{match[2]}"
+ return f"geometry({crs})" if match[1] == "geometry" else
f"geography({crs}, SPHERICAL)"
+ match = re.fullmatch(r"interval\s+(\w+)(?:\s+to\s+(\w+))?", lower)
+ if match:
+ start, end = match[1], match[2] or match[1]
+ for units in (_DAY_TIME, _YEAR_MONTH):
+ if start in units and end in units and units.index(start) <=
units.index(end):
+ return f"interval {start}" + ("" if start == end else f" to
{end}")
+ return None
+ if lower.startswith("array<") and lower.endswith(">"):
+ element = ddl_json(text[6:-1])
+ return element and {"type": "array", "elementType": element,
"containsNull": True}
+ if lower.startswith("map<") and lower.endswith(">"):
+ types = [ddl_json(part) for part in _top_level_split(text[4:-1])]
+ if len(types) != 2 or None in types:
+ return None
+ return {"type": "map", "keyType": types[0], "valueType": types[1],
+ "valueContainsNull": True}
+ if lower.startswith("struct<") and lower.endswith(">"):
+ text = text[7:-1]
+ if not text.strip():
+ return {"type": "struct", "fields": []}
+ fields = []
+ for part in _top_level_split(text):
+ match = _DDL_FIELD.fullmatch(part)
+ data_type = match and ddl_json(match["type"])
+ if not data_type:
+ return None
+ metadata = {} if match["comment"] is None else {"comment":
match["comment"]}
+ fields.append({"name": match["name"].strip("`"), "type": data_type,
+ "nullable": not match["not_null"], "metadata":
metadata})
+ return {"type": "struct", "fields": fields}
+
+
+def data_type_proto(value):
+ """The DataType (types.proto) of the JSON value of a type."""
+ if isinstance(value, dict):
+ # Array: element_type=1, contains_null=2; Map: key_type=1,
value_type=2,
+ # value_contains_null=3; Struct.fields=1, StructField: name=1,
data_type=2,
+ # nullable=3, metadata=4.
+ if value["type"] == "array":
+ return message_field(20, message_field(1,
data_type_proto(value["elementType"]))
+ + varint_field(2, int(value["containsNull"])))
+ if value["type"] == "map":
+ return message_field(22, message_field(1,
data_type_proto(value["keyType"]))
+ + message_field(2,
data_type_proto(value["valueType"]))
+ + varint_field(3,
int(value["valueContainsNull"])))
+ fields = b""
+ for field in value["fields"]:
+ encoded = message_field(1, field["name"]) + message_field(
+ 2, data_type_proto(field["type"]))
+ if field["nullable"]:
+ encoded += varint_field(3, 1)
+ if field["metadata"]:
+ encoded += message_field(4, json.dumps(field["metadata"]))
+ fields += message_field(1, encoded)
+ return message_field(21, fields)
+ if value in _PROTO_KINDS:
+ return message_field(_PROTO_KINDS[value], b"")
+ # Decimal: scale=1, precision=2; Time: precision=1; DayTimeInterval=19 and
+ # YearMonthInterval=18: start_field=1, end_field=2.
+ match = re.fullmatch(r"decimal\((\d+),(\d+)\)", value)
+ if match:
+ return message_field(10, varint_field(1, int(match[2])) +
varint_field(2, int(match[1])))
+ match = re.fullmatch(r"time\((\d+)\)", value)
+ if match:
+ return message_field(28, varint_field(1, int(match[1])))
+ # Geometry=26 and Geography=27: srid=1.
+ match = re.fullmatch(r"(geometry|geography)\((?:OGC:CRS84|SRID:(\d+))(?:,
\w+)?\)", value)
+ if match:
+ return message_field(26 if match[1] == "geometry" else 27,
+ varint_field(1, int(match[2] or 4326)))
+ match = re.fullmatch(r"interval (\w+)(?: to (\w+))?", value)
+ units = _DAY_TIME if match[1] in _DAY_TIME else _YEAR_MONTH
+ start, end = units.index(match[1]), units.index(match[2] or match[1])
+ return message_field(19 if units is _DAY_TIME else 18,
+ varint_field(1, start) + varint_field(2, end))
+
+
+def ddl_parse_response(request):
+ """The AnalyzePlanResponse of an AnalyzePlanRequest of a DDLParse, as the
server parses the
+ DDL strings of the tests, or None for another request:
AnalyzePlanRequest.ddl_parse=11 ->
+ DDLParse.ddl_string=1; AnalyzePlanResponse.ddl_parse=9 ->
DDLParse.parsed=1."""
+ message = parse_message(request)
+ if 11 not in message:
+ return None
+ parsed = ddl_data_type(parse_message(message[11][0])[1][0].decode())
+ return message_field(9, message_field(1, parsed or b""))
+
+
+def ddl_data_type(text):
+ """The DataType of a DDL string, as the server's DDLParse parses those of
the tests (see
+ `ddl_json`); None for any other DDL."""
+ value = ddl_json(text)
+ return None if value is None else data_type_proto(value)
+
+
def struct_schema(name="id"):
# types.proto: DataType.long=7, .struct=21; Struct.fields=1;
# StructField: name=1, data_type=2, nullable=3, metadata=4.
@@ -144,6 +293,13 @@ def serve(connection):
)
if self.failure == "rpc":
context.abort(grpc.StatusCode.INVALID_ARGUMENT, "schema
unavailable")
+ # base.proto: AnalyzePlanRequest.ddl_parse=11 ->
DDLParse.ddl_string=1;
+ # AnalyzePlanResponse.ddl_parse=9 -> DDLParse.parsed=1: the type
of the DDL, as the
+ # server parses it, or `data_type` for one that `ddl_data_type`
does not parse.
+ if 11 in parse_message(request):
+ ddl =
parse_message(parse_message(request)[11][0])[1][0].decode()
+ parsed = ddl_data_type(ddl) or self.data_type
+ return message_field(9, message_field(1, parsed))
if self.failure == "no-result":
return b""
if self.failure == "missing-schema":
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index a41a792..9424ac5 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -463,7 +463,8 @@ def _arrow_input_service(
state = {"cache": {}, "uploads": 0, "relations": [], "message_sizes": [],
"config": config, "config_requests": [], "input_tables": [],
"tables": [],
"upload_contexts": [], "status_contexts": [],
- "status_failures": [], "upload_failures": [], "upload_attempts":
0}
+ "status_failures": [], "upload_failures": [], "upload_attempts":
0,
+ "ddl_strings": []}
server_id = "00000000-0000-0000-0000-000000000001"
def attribution(request, context):
@@ -495,6 +496,48 @@ def _arrow_input_service(
if config.get(k) is not None]
).SerializeToString()
+ def top_level_split(text):
+ parts, depth, start = [], 0, 0
+ for i, char in enumerate(text):
+ depth += {"<": 1, ">": -1}.get(char, 0)
+ if char == "," and depth == 0:
+ parts.append(text[start:i])
+ start = i + 1
+ return parts + [text[start:]]
+
+ def ddl_type(text):
+ """The DataType of a DDL of these tests, as the server's `ddl_parse`
parses it: atomic
+ types, arrays and maps of them, and structs of `name type` fields."""
+ text, kinds = text.strip(), {
+ "long": "long", "bigint": "long", "int": "integer", "integer":
"integer",
+ "double": "double", "float": "float", "string": "string", "date":
"date",
+ "boolean": "boolean", "binary": "binary", "timestamp": "timestamp"}
+ lower = text.lower()
+ if lower.startswith("array<") and lower.endswith(">"):
+ return protocol("DataType", array=dict(
+ element_type=ddl_type(text[6:-1]), contains_null=True))
+ if lower.startswith("map<") and lower.endswith(">"):
+ key, value = top_level_split(text[4:-1])
+ return protocol("DataType", map=dict(
+ key_type=ddl_type(key), value_type=ddl_type(value),
value_contains_null=True))
+ if lower in kinds:
+ return protocol("DataType", **{kinds[lower]: {}})
+ fields = [field.strip().split(None, 1) for field in
top_level_split(text)]
+ return protocol("DataType", struct=dict(fields=[
+ dict(name=name, data_type=ddl_type(kind), nullable=True) for name,
kind in fields]))
+
+ def analyze(raw, context):
+ # As the server, which parses the DDL schemas that the client sends
(`_parse_ddl`);
+ # the tests ask for no other analysis.
+ request = protocol("AnalyzePlanRequest").FromString(raw)
+ if request.WhichOneof("analyze") != "ddl_parse":
+ context.abort(grpc.StatusCode.UNIMPLEMENTED, "Method not found!")
+ state["ddl_strings"].append(request.ddl_parse.ddl_string)
+ return protocol("AnalyzePlanResponse", session_id=request.session_id,
+ server_side_session_id=server_id,
+
ddl_parse=dict(parsed=ddl_type(request.ddl_parse.ddl_string))
+ ).SerializeToString()
+
def statuses(raw, context):
request = protocol("ArtifactStatusesRequest").FromString(raw)
state["status_contexts"].append(attribution(request, context))
@@ -621,6 +664,7 @@ def _arrow_input_service(
server.add_generic_rpc_handlers([grpc.method_handlers_generic_handler(
"spark.connect.SparkConnectService", {
"Config": grpc.unary_unary_rpc_method_handler(configure),
+ "AnalyzePlan": grpc.unary_unary_rpc_method_handler(analyze),
"ArtifactStatus":
grpc.unary_unary_rpc_method_handler(statuses),
"AddArtifacts": grpc.stream_unary_rpc_method_handler(upload),
"ExecutePlan": grpc.unary_stream_rpc_method_handler(execute),
@@ -865,6 +909,8 @@ def test_arrow_input_ddl_width_errors(protocol, schema,
expected_width):
assert error.value.getMessageParameters() == {
"expected_length": expected_width, "actual_length": "2",
}
+ # As PySpark Connect's, the server parses the DDL schema.
+ assert state["ddl_strings"] == [schema]
assert not state["relations"]
assert state["uploads"] == 0
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 15df8bf..48551dc 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1335,6 +1335,37 @@ def
test_window_order_by_a_column_sorts_nulls_first_as_pyspark(session):
assert [(r.o, r.rn) for r in rows] == expected
+def test_create_dataframe_takes_the_ddl_of_spark_as_pyspark(session):
+ # As PySpark Connect's, which parses a DDL schema on the server: nested
structs, arrays and
+ # maps of them, backtick-quoted names, and NOT NULL and COMMENT clauses.
+ schema = (
+ "st struct<a:int,b:double>, arr array<struct<c:int>>, m
map<string,struct<x:int>>, "
+ "`a b` int NOT NULL COMMENT 'c'"
+ )
+ df = session.createDataFrame([((1, 2.0), [(3,)], {"k": (4,)}, 5)], schema)
+ assert df.schema.simpleString() == (
+ "struct<st:struct<a:int,b:double>,arr:array<struct<c:int>>,"
+ "m:map<string,struct<x:int>>,a b:int>"
+ )
+ assert df.collect() == [
+ Row(st=Row(a=1, b=2.0), arr=[Row(c=3)], m={"k": Row(x=4)}, **{"a b":
5})
+ ]
+ # Collations, which the server names, and the errors of the server's
parser.
+ collated = session.createDataFrame([("x",)], "s STRING COLLATE UTF8_LCASE")
+ assert collated.schema.simpleString() == "struct<s:string collate
UTF8_LCASE>"
+ from pyspark.errors import ParseException
+
+ for ddl, condition in [("a INT,", "PARSE_SYNTAX_ERROR"), ("a BOGUS",
"UNSUPPORTED_DATATYPE")]:
+ with pytest.raises(ParseException) as error:
+ session.createDataFrame([], ddl)
+ assert error.value.getCondition() == condition
+ # As do the map functions, of their schemas.
+ mapped = session.range(2).mapInPandas(lambda it: it, "id long NOT NULL
COMMENT 'k'")
+ assert [(f.name, f.nullable, f.metadata) for f in mapped.schema.fields] ==
[
+ ("id", False, {"comment": "k"})]
+ assert mapped.collect() == [Row(id=0), Row(id=1)]
+
+
def test_functions_send_their_optional_arguments_as_pyspark(session):
# The values of official PySpark 4.3.0; each optional argument used to be
dropped.
df = session.sql(
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 0ece022..319d4c0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -20,6 +20,8 @@ from threading import Barrier
import pytest
from _mock_connect_server import (
+ ddl_json,
+ ddl_parse_response,
message_field,
mock_connect_session,
parse_message,
@@ -4892,7 +4894,11 @@ def test_timedelta_values_convert_as_pyspark():
def _local_create_dataframe(*args):
- # The local relation and its toDF are built without a server.
+ # The local relation and its toDF are built without a server; a DDL
schema, which PySpark
+ # asks the server to parse, is parsed as the server would.
+ args = list(args)
+ if len(args) > 1 and isinstance(args[1], str) and ddl_json(args[1]) is not
None:
+ args[1] = T._parse_datatype_json_string(json.dumps(ddl_json(args[1])))
return _offline_session().createDataFrame(*args)
@@ -5429,6 +5435,16 @@ def _container_input_fields(data):
return result
+def _ddl_parse(request, context):
+ # As the server, of the DDL schemas that the client asks it to parse
(`_parse_ddl`).
+ import grpc
+
+ response = ddl_parse_response(request)
+ if response is None:
+ context.abort(grpc.StatusCode.UNIMPLEMENTED, "Method not found!")
+ return response
+
+
def _container_input_capture_server(connection):
from concurrent.futures import ThreadPoolExecutor
import grpc
@@ -5452,6 +5468,7 @@ def _container_input_capture_server(connection):
server.add_generic_rpc_handlers((grpc.method_handlers_generic_handler(
"spark.connect.SparkConnectService", {
"ExecutePlan": grpc.unary_stream_rpc_method_handler(execute),
+ "AnalyzePlan": grpc.unary_unary_rpc_method_handler(_ddl_parse),
}),))
port = server.add_insecure_port("127.0.0.1:0")
server.start()
@@ -5744,6 +5761,7 @@ def _exact_date_capture_server(connection):
"spark.connect.SparkConnectService", {
"ExecutePlan":
grpc.unary_stream_rpc_method_handler(_exact_date_execute),
"Config":
grpc.unary_unary_rpc_method_handler(_exact_date_config),
+ "AnalyzePlan": grpc.unary_unary_rpc_method_handler(_ddl_parse),
}),))
port = server.add_insecure_port("127.0.0.1:0")
server.start()
@@ -6624,6 +6642,43 @@ def
test_completion_matches_python_connect(completion_analyzed, names):
assert df.schema.fieldNames() == names
+def test_ddl_schemas_are_parsed_on_the_server_as_pyspark(mock_connect):
+ # As PySpark Connect's `_parse_ddl`, which asks the server to parse a DDL
schema, of
+ # createDataFrame and of the map functions; a type is sent as it is.
+ spark, service = mock_connect
+
+ def ddl_requests():
+ # base.proto: AnalyzePlanRequest.ddl_parse=11, DDLParse.ddl_string=1.
+ return [parse_message(parse_message(request)[11][0])[1][0].decode()
+ for request in service.analyze_requests if 11 in
parse_message(request)]
+
+ def sent_local_schema():
+ # The LocalRelation of the type that the server parsed:
AnalyzePlanRequest.schema=4 ->
+ # plan=1 -> root=1 -> (Relation.to_df=18 -> input=1 ->)
local_relation=11 -> schema=2.
+ relation = parse_message(parse_message(parse_message(
+ parse_message(service.analyze_requests[-1])[4][0])[1][0])[1][0])
+ while 18 in relation:
+ relation = parse_message(parse_message(relation[18][0])[1][0])
+ return
T._parse_datatype_json_string(parse_message(relation[11][0])[2][0].decode())
+
+ spark.createDataFrame([], "id long").columns
+ assert sent_local_schema().simpleString() == "struct<id:bigint>"
+ nested = "id struct<a:int,b:double> NOT NULL"
+ spark.createDataFrame([], nested).columns
+ assert sent_local_schema() == T.StructType([T.StructField("id",
T.StructType([
+ T.StructField("a", T.IntegerType()), T.StructField("b",
T.DoubleType())]), False)])
+ assert ddl_requests() == ["id long", nested]
+ df = spark.range(1)
+ df.mapInPandas(lambda it: it, "id long NOT NULL")
+ df.mapInArrow(lambda it: it, "id long COMMENT 'c'")
+ df.groupBy("id").applyInPandas(lambda pdf: pdf, "`id` long")
+ df.groupBy("id").cogroup(df.groupBy("id")).applyInArrow(lambda l, r: l,
"id: long")
+ assert ddl_requests()[2:] == [
+ "id long NOT NULL", "id long COMMENT 'c'", "`id` long", "id: long"]
+ df.mapInPandas(lambda it: it, T.StructType([T.StructField("id",
T.LongType())]))
+ assert len(ddl_requests()) == 6
+
+
def test_completion_lists_are_independent(completion_analyzed):
spark, service = completion_analyzed
df = spark.range(10)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]