This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 345a1e7  [SPARK-61029][CONNECT][PYTHON] Parse the DDL schema strings 
of createDataFrame and the map functions on the server as PySpark does
345a1e7 is described below

commit 345a1e719ffda85932b6bacbfaefd1dd7b617a22
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 08:35:53 2026 +0900

    [SPARK-61029][CONNECT][PYTHON] Parse the DDL schema strings of 
createDataFrame and the map functions on the server as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61029](https://issues.apache.org/jira/browse/SPARK-61029): the 
Python client now parses the DDL string schemas of `createDataFrame` and of the 
map functions on the server, as PySpark Connect does.
    
    PySpark Connect parses a DDL string with `_parse_ddl`, the `DDLParse` of an 
`AnalyzePlanRequest`. `spark.createDataFrame(data, schema)`, `mapInPandas`, 
`mapInArrow`, and the `applyInPandas` and `applyInArrow` of grouped and 
cogrouped data parse their schemas that way. So they take any DDL that Spark 
does:
    
    - nested types, such as `"st struct<a:int,b:double>"` and `"m 
map<string,struct<x:int>>"`
    - backtick-quoted names, such as ``"`a b` int"``
    - collations, such as `"s STRING COLLATE UTF8_LCASE"`
    - `NOT NULL` and `COMMENT` clauses
    
    They also raise the errors of Spark's parser, such as `PARSE_SYNTAX_ERROR` 
for `"a INT,"`. `transformWithStateInPandas` sends a DDL output schema as an 
`UnparsedDataType`, for the server to parse.
    
    The core adds `SparkSession::parse_ddl`, which asks the server to parse a 
DDL string, and `spark_session()` accessors for `GroupedData` and 
`CoGroupedData`.
    
    ### Why are the changes needed?
    
    The client parsed these schemas itself, and its parser differs from Spark's:
    
    - It raised `ValueError: invalid type string 'st struct<a:int,b:double>': 
SparkError { kind: ValueError, error_class: "INVALID_DATATYPE_FORMAT", ... }` 
for the nested, quoted, `NOT NULL` and `COMMENT` forms above.
    - It took `"a INT,"` as `a INT`.
    - For `"s STRING COLLATE UTF8_LCASE"`, it sent a type that the server 
failed to parse (`PARSE_SYNTAX_ERROR`).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These functions take the DDL schemas of PySpark 4.3.0 and raise its 
errors.
    
    ### How was this patch tested?
    
    - A new live test creates DataFrames of these schemas and checks their 
types and rows, the collation, the errors, and the `NOT NULL` and `COMMENT` of 
a `mapInPandas` schema. It passes with official PySpark 4.3.0 and fails without 
this change.
    - A new e2e test checks `SparkSession::parse_ddl` against the server, 
including its errors.
    - A new offline test checks the DDL strings that `createDataFrame` and the 
map functions send to a mock Connect server. The mock servers of the offline 
and the Arrow tests now answer `DDLParse` requests.
    - On a local server, 59 DDL schemas give the same types and errors as 
official PySpark 4.3.0 for `createDataFrame`. So do 36 cases of the map and 
apply functions.
    - The lib tests of both crates (571 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (1810 passed, 8 skipped), and the 
live drop-in and Arrow suites (109 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #235 from HyukjinKwon/SPARK-61029-ddl.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs                |  22 ++-
 crates/pyspark-rs/src/group.rs                    |  28 +++-
 crates/pyspark-rs/src/pyarrow_import.rs           |   2 +-
 crates/pyspark-rs/src/session.rs                  |  10 +-
 crates/pyspark-rs/src/types.rs                    |  17 +++
 crates/spark-connect/src/group.rs                 |  10 ++
 crates/spark-connect/src/session.rs               |  24 ++++
 crates/spark-connect/tests/e2e_session_catalog.rs |  30 +++++
 python/tests/_mock_connect_server.py              | 156 ++++++++++++++++++++++
 python/tests/test_arrow_handoff.py                |  48 ++++++-
 python/tests/test_dropin_e2e.py                   |  31 +++++
 python/tests/test_dropin_offline.py               |  57 +++++++-
 12 files changed, 418 insertions(+), 17 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 0c200cb..0888ba2 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2061,6 +2061,7 @@ impl PyDataFrame {
         let cols = py.detach(|| self.dataframe.columns()).to_pyerr()?;
         let udf = build_map_udf(
             py,
+            &self.dataframe.spark_session(),
             &func,
             &schema,
             eval_type::SQL_MAP_PANDAS_ITER_UDF,
@@ -2086,7 +2087,14 @@ impl PyDataFrame {
     ) -> PyResult<PyDataFrame> {
         validate_map_function(py, &func, eval_type::SQL_MAP_ARROW_ITER_UDF)?;
         let cols = py.detach(|| self.dataframe.columns()).to_pyerr()?;
-        let udf = build_map_udf(py, &func, &schema, 
eval_type::SQL_MAP_ARROW_ITER_UDF, &cols)?;
+        let udf = build_map_udf(
+            py,
+            &self.dataframe.spark_session(),
+            &func,
+            &schema,
+            eval_type::SQL_MAP_ARROW_ITER_UDF,
+            &cols,
+        )?;
         let profile_id = profile_id_of(profile.as_ref())?;
         Ok(PyDataFrame::new(
             self.dataframe
@@ -3181,15 +3189,18 @@ fn bind_arguments<'py, const N: usize>(
     }))
 }
 
-/// Resolve a Python schema argument (a `DataType` wrapper or a DDL string) 
into a Rust DataType.
+/// Resolve a Python schema argument (a `DataType` wrapper or a DDL string) 
into a Rust
+/// DataType: a DDL string as the server of `session` parses it, as PySpark's 
`_parse_ddl`.
 pub(crate) fn resolve_datatype(
+    py: Python<'_>,
+    session: &spark_connect::session::SparkSession,
     schema: &Bound<'_, PyAny>,
 ) -> PyResult<spark_connect::types::DataType> {
     if let Ok(dt) = schema.extract::<PyRef<PyDataType>>() {
         return Ok(dt.inner.clone());
     }
-    if let Ok(s) = schema.extract::<String>() {
-        return spark_connect::types::DataType::from_ddl(&s).to_pyerr();
+    if schema.is_instance_of::<pyo3::types::PyString>() {
+        return crate::types::schema_data_type(py, session, schema);
     }
     Err(pyo3::exceptions::PyTypeError::new_err(
         "schema must be a DataType or a DDL string",
@@ -3259,13 +3270,14 @@ fn udf_name(func: &Bound<'_, PyAny>) -> 
PyResult<String> {
 /// carry the resolved schema as the output type (mirrors the scalar UDF 
command layout).
 pub(crate) fn build_map_udf(
     py: Python<'_>,
+    session: &spark_connect::session::SparkSession,
     func: &Bound<'_, PyAny>,
     schema: &Bound<'_, PyAny>,
     eval_type_id: i32,
     input_cols: &[String],
 ) -> PyResult<CommonInlineUserDefinedFunctionExpression> {
     let name = udf_name(func)?;
-    let return_type = resolve_datatype(schema)?;
+    let return_type = resolve_datatype(py, session, schema)?;
     // The worker reads the pickled type as a `DataType`, so a DDL string is 
pickled as the
     // type it parses to, as the reference client pickles the type of 
`_parse_ddl`.
     let pickled_type = if schema.is_instance_of::<pyo3::types::PyString>() {
diff --git a/crates/pyspark-rs/src/group.rs b/crates/pyspark-rs/src/group.rs
index a7463f2..2666823 100644
--- a/crates/pyspark-rs/src/group.rs
+++ b/crates/pyspark-rs/src/group.rs
@@ -169,7 +169,8 @@ impl PyGroupedData {
         )?;
         validate_map_function(py, &func, eval_type_id)?;
         let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
-        let udf = build_map_udf(py, &func, &schema, eval_type_id, &cols)?;
+        let session = self.grouped_data.spark_session();
+        let udf = build_map_udf(py, &session, &func, &schema, eval_type_id, 
&cols)?;
         Ok(PyDataFrame::new(self.grouped_data.apply_in_pandas(udf)))
     }
 
@@ -204,7 +205,8 @@ impl PyGroupedData {
         )?;
         validate_map_function(py, &func, eval_type_id)?;
         let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
-        let udf = build_map_udf(py, &func, &schema, eval_type_id, &cols)?;
+        let session = self.grouped_data.spark_session();
+        let udf = build_map_udf(py, &session, &func, &schema, eval_type_id, 
&cols)?;
         Ok(PyDataFrame::new(self.grouped_data.apply_in_arrow(udf)))
     }
 
@@ -230,14 +232,16 @@ impl PyGroupedData {
     ) -> PyResult<PyDataFrame> {
         validate_map_function(py, &func, 
eval_type::SQL_GROUPED_MAP_PANDAS_UDF_WITH_STATE)?;
         let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
+        let session = self.grouped_data.spark_session();
         let udf = build_map_udf(
             py,
+            &session,
             &func,
             &outputStructType,
             eval_type::SQL_GROUPED_MAP_PANDAS_UDF_WITH_STATE,
             &cols,
         )?;
-        let state_schema = 
crate::dataframe::resolve_datatype(&stateStructType)?;
+        let state_schema = crate::dataframe::resolve_datatype(py, &session, 
&stateStructType)?;
         Ok(PyDataFrame::new(
             self.grouped_data.apply_in_pandas_with_state(
                 udf,
@@ -273,8 +277,17 @@ impl PyGroupedData {
             true,
         )?;
         let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
-        let udf = build_map_udf(py, &func, &outputStructType, et, &cols)?;
-        let out = crate::types::py_to_data_type(&outputStructType)?;
+        let session = self.grouped_data.spark_session();
+        let udf = build_map_udf(py, &session, &func, &outputStructType, et, 
&cols)?;
+        // As PySpark's, which sends a DDL string as an `UnparsedDataType`, 
for the server to
+        // parse.
+        let out = if 
outputStructType.is_instance_of::<pyo3::types::PyString>() {
+            spark_connect::types::DataType::Unparsed {
+                data_type_string: outputStructType.extract()?,
+            }
+        } else {
+            crate::types::py_to_data_type(&outputStructType)?
+        };
         let etc = 
(!eventTimeColumnName.is_empty()).then_some(eventTimeColumnName);
         Ok(PyDataFrame::new(
             self.grouped_data.transform_with_state_in_pandas(
@@ -311,7 +324,8 @@ impl PyGroupedData {
             false,
         )?;
         let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
-        let udf = build_map_udf(py, &func, &outputStructType, et, &cols)?;
+        let session = self.grouped_data.spark_session();
+        let udf = build_map_udf(py, &session, &func, &outputStructType, et, 
&cols)?;
         let etc = 
(!eventTimeColumnName.is_empty()).then_some(eventTimeColumnName);
         Ok(PyDataFrame::new(self.grouped_data.transform_with_state(
             udf,
@@ -378,6 +392,7 @@ impl PyCoGroupedData {
         let cols = py.detach(|| self.inner.input_columns()).to_pyerr()?;
         let udf = build_map_udf(
             py,
+            &self.inner.spark_session(),
             &func,
             &schema,
             eval_type::SQL_COGROUPED_MAP_PANDAS_UDF,
@@ -398,6 +413,7 @@ impl PyCoGroupedData {
         let cols = py.detach(|| self.inner.input_columns()).to_pyerr()?;
         let udf = build_map_udf(
             py,
+            &self.inner.spark_session(),
             &func,
             &schema,
             eval_type::SQL_COGROUPED_MAP_ARROW_UDF,
diff --git a/crates/pyspark-rs/src/pyarrow_import.rs 
b/crates/pyspark-rs/src/pyarrow_import.rs
index eb39cfb..e2fea5f 100644
--- a/crates/pyspark-rs/src/pyarrow_import.rs
+++ b/crates/pyspark-rs/src/pyarrow_import.rs
@@ -36,7 +36,7 @@ pub(crate) fn create_dataframe(
     let schema = match schema {
         Some(s) if s.extract::<String>().is_ok() => 
Some(crate::types::data_type_to_py(
             py,
-            &crate::types::py_to_data_type(s)?,
+            &crate::types::schema_data_type(py, session, s)?,
         )?),
         Some(s) => Some(s.clone().unbind()),
         None => None,
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 35b1084..43405af 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -712,7 +712,11 @@ enum Spec {
 
 /// The schema argument of `createDataFrame`: None, a list or tuple of names, 
or a struct or
 /// atomic type, given as a type or a DDL string. Other types raise 
`NOT_EXPECTED_TYPE`.
-fn schema_spec(py: Python<'_>, schema: Option<&Bound<'_, PyAny>>) -> 
PyResult<Spec> {
+fn schema_spec(
+    py: Python<'_>,
+    session: &spark_connect::session::SparkSession,
+    schema: Option<&Bound<'_, PyAny>>,
+) -> PyResult<Spec> {
     let Some(schema) = schema.filter(|s| !s.is_none()) else {
         return Ok(Spec::None);
     };
@@ -745,7 +749,7 @@ fn schema_spec(py: Python<'_>, schema: Option<&Bound<'_, 
PyAny>>) -> PyResult<Sp
     {
         return Err(not_expected(&crate::infer::type_name(schema)?));
     }
-    Ok(match crate::types::py_to_data_type(schema)? {
+    Ok(match crate::types::schema_data_type(py, session, schema)? {
         DataType::Struct { fields } => Spec::Struct(fields),
         atomic if crate::infer::is_atomic(&atomic) => Spec::Atomic(atomic),
         other => return Err(not_expected(&crate::infer::class_name(&other))),
@@ -917,7 +921,7 @@ impl PySparkSession {
         if is_loaded_instance(py, data, "pyarrow", "Table")? {
             return crate::pyarrow_import::create_dataframe(py, &self.session, 
data, schema);
         }
-        let spec = schema_spec(py, schema)?;
+        let spec = schema_spec(py, &self.session, schema)?;
         if is_loaded_instance(py, data, "pandas", "DataFrame")? {
             // Guarded integer columns share the existing Arrow importer. 
Schema
             // validation above and unsupported pandas conversions keep 
master's
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 7aa22f2..033f006 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -2934,6 +2934,23 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt: 
&DataType) -> PyResult<Py<PyAn
     Ok(o)
 }
 
+/// The type of `schema`, a DataType or a DDL string, which the server parses, 
as PySpark
+/// Connect's `_parse_ddl` does: so any DDL of Spark, with nested types, 
quoted names,
+/// collations and the NOT NULL and COMMENT of fields, parses as Spark parses 
it.
+pub(crate) fn schema_data_type(
+    py: Python<'_>,
+    session: &spark_connect::session::SparkSession,
+    schema: &Bound<'_, PyAny>,
+) -> PyResult<DataType> {
+    if schema.is_instance_of::<pyo3::types::PyString>() {
+        let ddl: String = schema.extract()?;
+        return crate::errors::ResultExt::to_pyerr(
+            py.detach(|| session.parse_ddl(&ddl).map_err(Box::new)),
+        );
+    }
+    py_to_data_type(schema)
+}
+
 pub(crate) fn py_to_data_type(obj: &Bound<'_, PyAny>) -> PyResult<DataType> {
     if let Ok(g) = obj.extract::<PyRef<PyGeometryType>>() {
         return Ok(DataType::Geometry { srid: g.srid });
diff --git a/crates/spark-connect/src/group.rs 
b/crates/spark-connect/src/group.rs
index 915a6e4..3c976f4 100644
--- a/crates/spark-connect/src/group.rs
+++ b/crates/spark-connect/src/group.rs
@@ -120,6 +120,11 @@ impl GroupedData {
         self.dataframe.columns()
     }
 
+    /// The session of the grouped DataFrame.
+    pub fn spark_session(&self) -> crate::session::SparkSession {
+        self.dataframe.spark_session()
+    }
+
     /// Grouping-key expressions for this grouped data.
     fn grouping_expressions(&self) -> Vec<Expression> {
         self.group_cols
@@ -668,6 +673,11 @@ impl CoGroupedData {
         Ok(cols)
     }
 
+    /// The session of the cogrouped DataFrames.
+    pub fn spark_session(&self) -> crate::session::SparkSession {
+        self.left.spark_session()
+    }
+
     /// Apply a pandas UDF to each cogroup (`cogroup(...).applyInPandas`).
     ///
     /// `func` is built on the Python side (cloudpickled, eval type
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index d31f33a..010ccac 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -260,6 +260,30 @@ impl SparkSession {
         Ok(DataFrame::new(self.clone(), plan))
     }
 
+    /// Parse a DDL string on the server, mirroring PySpark Connect's 
`_parse_ddl`: the
+    /// `DDLParse` of an `AnalyzePlanRequest`, of a schema or a type, as Spark 
parses it.
+    pub fn parse_ddl(&self, ddl: &str) -> Result<DataType> {
+        let request = proto::AnalyzePlanRequest {
+            session_id: self.client().session_id().to_string(),
+            user_context: Some(proto::UserContext::default()),
+            analyze: Some(proto::analyze_plan_request::Analyze::DdlParse(
+                proto::analyze_plan_request::DdlParse {
+                    ddl_string: ddl.to_string(),
+                },
+            )),
+            ..Default::default()
+        };
+        let response = block_on(self.client().analyze_plan(request))?;
+        match response.result {
+            Some(proto::analyze_plan_response::Result::DdlParse(r)) => {
+                DataType::from_proto(&r.parsed.unwrap_or_default())
+            }
+            _ => Err(SparkError::connect_msg(
+                "AnalyzePlan response did not contain a DDL parse result",
+            )),
+        }
+    }
+
     /// Execute a SQL statement and return a DataFrame.
     ///
     /// Mirrors `pyspark.sql.SparkSession.sql(sqlQuery)`: the statement is 
sent at once, so a
diff --git a/crates/spark-connect/tests/e2e_session_catalog.rs 
b/crates/spark-connect/tests/e2e_session_catalog.rs
index 1e1fae8..3b40581 100644
--- a/crates/spark-connect/tests/e2e_session_catalog.rs
+++ b/crates/spark-connect/tests/e2e_session_catalog.rs
@@ -292,3 +292,33 @@ fn sql_resolves_the_dataframes_that_the_query_names() {
     assert_eq!(shown.collect().expect("collect").len(), 2);
     let _ = spark.stop();
 }
+
+/// `parse_ddl` parses a DDL string on the server, as PySpark Connect's 
`_parse_ddl`: of a
+/// schema or a type, with the NOT NULL and COMMENT of fields, and the 
server's errors.
+#[test]
+fn parse_ddl_parses_on_the_server() {
+    if !should_run() {
+        return;
+    }
+    use spark_connect::types::DataType;
+    let spark = session();
+    let parsed = spark
+        .parse_ddl("a INT NOT NULL COMMENT 'c', b STRUCT<x: DOUBLE>")
+        .expect("parse");
+    let DataType::Struct { fields } = parsed else {
+        panic!("expected a struct, got {parsed:?}");
+    };
+    assert_eq!(
+        fields.iter().map(|f| f.name.as_str()).collect::<Vec<_>>(),
+        vec!["a", "b"]
+    );
+    assert!(!fields[0].nullable);
+    assert_eq!(
+        fields[0].metadata.get("comment"),
+        Some(&serde_json::Value::String("c".to_string()))
+    );
+    assert_eq!(spark.parse_ddl("bigint").expect("parse"), DataType::Long);
+    let error = spark.parse_ddl("a INT,").expect_err("a syntax error");
+    assert_eq!(error.error_class, "PARSE_SYNTAX_ERROR");
+    let _ = spark.stop();
+}
diff --git a/python/tests/_mock_connect_server.py 
b/python/tests/_mock_connect_server.py
index 2a7400a..bfe773c 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -16,7 +16,9 @@
 #
 """Small, separately hosted Spark Connect service for transport-level tests."""
 
+import json
 import multiprocessing
+import re
 from concurrent.futures import ThreadPoolExecutor
 from contextlib import contextmanager
 from threading import Event
@@ -77,6 +79,153 @@ def parse_message(data):
     return fields
 
 
+# The JSON names of the atomic types of DDL names, and the DataType kinds 
(types.proto) of the
+# JSON names.
+_ATOMIC_JSON = {
+    "void": "void", "null": "void", "binary": "binary", "boolean": "boolean",
+    "bool": "boolean", "byte": "byte", "tinyint": "byte", "short": "short",
+    "smallint": "short", "int": "integer", "integer": "integer", "long": 
"long",
+    "bigint": "long", "float": "float", "real": "float", "double": "double",
+    "string": "string", "date": "date", "timestamp": "timestamp", 
"timestamp_ltz": "timestamp",
+    "timestamp_ntz": "timestamp_ntz", "variant": "variant",
+}
+_PROTO_KINDS = {
+    "void": 1, "binary": 2, "boolean": 3, "byte": 4, "short": 5, "integer": 6, 
"long": 7,
+    "float": 8, "double": 9, "string": 11, "date": 14, "timestamp": 15, 
"timestamp_ntz": 16,
+    "variant": 25,
+}
+_DAY_TIME = ["day", "hour", "minute", "second"]
+_YEAR_MONTH = ["year", "month"]
+_DDL_FIELD = re.compile(
+    
r"\s*(?P<name>`[^`]+`|\w+)(?:\s*:\s*|\s+)(?P<type>.+?)(?P<not_null>\s+not\s+null)?"
+    r"(?:\s+comment\s+'(?P<comment>[^']*)')?\s*", re.IGNORECASE | re.DOTALL)
+
+
+def _top_level_split(text):
+    parts, depth, start = [], 0, 0
+    for i, char in enumerate(text):
+        depth += {"<": 1, ">": -1, "(": 1, ")": -1}.get(char, 0)
+        if char == "," and depth == 0:
+            parts.append(text[start:i])
+            start = i + 1
+    return parts + [text[start:]]
+
+
+def ddl_json(text):
+    """The JSON value of the type of a DDL string, as Spark's DDL parser 
parses those of the
+    tests: atomic types, decimal(p, s), time(p), intervals, arrays, maps and 
structs, and
+    schemas of `name type` and `name: type` fields, of NOT NULL and COMMENT. 
None for any
+    other DDL."""
+    text = text.strip()
+    lower = text.lower()
+    if lower in _ATOMIC_JSON:
+        return _ATOMIC_JSON[lower]
+    match = re.fullmatch(
+        r"(?:decimal|dec|numeric)\s*(?:\(\s*(\d+)\s*(?:,\s*(\d+)\s*)?\))?", 
lower)
+    if match:
+        return f"decimal({int(match[1] or 10)},{int(match[2] or 0)})"
+    match = re.fullmatch(r"time\s*(?:\(\s*(\d+)\s*\))?", lower)
+    if match:
+        return f"time({int(match[1] or 6)})"
+    match = re.fullmatch(r"(geometry|geography)\s*\(\s*(\d+)\s*\)", lower)
+    if match:
+        crs = "OGC:CRS84" if match[2] == "4326" else f"SRID:{match[2]}"
+        return f"geometry({crs})" if match[1] == "geometry" else 
f"geography({crs}, SPHERICAL)"
+    match = re.fullmatch(r"interval\s+(\w+)(?:\s+to\s+(\w+))?", lower)
+    if match:
+        start, end = match[1], match[2] or match[1]
+        for units in (_DAY_TIME, _YEAR_MONTH):
+            if start in units and end in units and units.index(start) <= 
units.index(end):
+                return f"interval {start}" + ("" if start == end else f" to 
{end}")
+        return None
+    if lower.startswith("array<") and lower.endswith(">"):
+        element = ddl_json(text[6:-1])
+        return element and {"type": "array", "elementType": element, 
"containsNull": True}
+    if lower.startswith("map<") and lower.endswith(">"):
+        types = [ddl_json(part) for part in _top_level_split(text[4:-1])]
+        if len(types) != 2 or None in types:
+            return None
+        return {"type": "map", "keyType": types[0], "valueType": types[1],
+                "valueContainsNull": True}
+    if lower.startswith("struct<") and lower.endswith(">"):
+        text = text[7:-1]
+        if not text.strip():
+            return {"type": "struct", "fields": []}
+    fields = []
+    for part in _top_level_split(text):
+        match = _DDL_FIELD.fullmatch(part)
+        data_type = match and ddl_json(match["type"])
+        if not data_type:
+            return None
+        metadata = {} if match["comment"] is None else {"comment": 
match["comment"]}
+        fields.append({"name": match["name"].strip("`"), "type": data_type,
+                       "nullable": not match["not_null"], "metadata": 
metadata})
+    return {"type": "struct", "fields": fields}
+
+
+def data_type_proto(value):
+    """The DataType (types.proto) of the JSON value of a type."""
+    if isinstance(value, dict):
+        # Array: element_type=1, contains_null=2; Map: key_type=1, 
value_type=2,
+        # value_contains_null=3; Struct.fields=1, StructField: name=1, 
data_type=2,
+        # nullable=3, metadata=4.
+        if value["type"] == "array":
+            return message_field(20, message_field(1, 
data_type_proto(value["elementType"]))
+                                 + varint_field(2, int(value["containsNull"])))
+        if value["type"] == "map":
+            return message_field(22, message_field(1, 
data_type_proto(value["keyType"]))
+                                 + message_field(2, 
data_type_proto(value["valueType"]))
+                                 + varint_field(3, 
int(value["valueContainsNull"])))
+        fields = b""
+        for field in value["fields"]:
+            encoded = message_field(1, field["name"]) + message_field(
+                2, data_type_proto(field["type"]))
+            if field["nullable"]:
+                encoded += varint_field(3, 1)
+            if field["metadata"]:
+                encoded += message_field(4, json.dumps(field["metadata"]))
+            fields += message_field(1, encoded)
+        return message_field(21, fields)
+    if value in _PROTO_KINDS:
+        return message_field(_PROTO_KINDS[value], b"")
+    # Decimal: scale=1, precision=2; Time: precision=1; DayTimeInterval=19 and
+    # YearMonthInterval=18: start_field=1, end_field=2.
+    match = re.fullmatch(r"decimal\((\d+),(\d+)\)", value)
+    if match:
+        return message_field(10, varint_field(1, int(match[2])) + 
varint_field(2, int(match[1])))
+    match = re.fullmatch(r"time\((\d+)\)", value)
+    if match:
+        return message_field(28, varint_field(1, int(match[1])))
+    # Geometry=26 and Geography=27: srid=1.
+    match = re.fullmatch(r"(geometry|geography)\((?:OGC:CRS84|SRID:(\d+))(?:, 
\w+)?\)", value)
+    if match:
+        return message_field(26 if match[1] == "geometry" else 27,
+                             varint_field(1, int(match[2] or 4326)))
+    match = re.fullmatch(r"interval (\w+)(?: to (\w+))?", value)
+    units = _DAY_TIME if match[1] in _DAY_TIME else _YEAR_MONTH
+    start, end = units.index(match[1]), units.index(match[2] or match[1])
+    return message_field(19 if units is _DAY_TIME else 18,
+                         varint_field(1, start) + varint_field(2, end))
+
+
+def ddl_parse_response(request):
+    """The AnalyzePlanResponse of an AnalyzePlanRequest of a DDLParse, as the 
server parses the
+    DDL strings of the tests, or None for another request: 
AnalyzePlanRequest.ddl_parse=11 ->
+    DDLParse.ddl_string=1; AnalyzePlanResponse.ddl_parse=9 -> 
DDLParse.parsed=1."""
+    message = parse_message(request)
+    if 11 not in message:
+        return None
+    parsed = ddl_data_type(parse_message(message[11][0])[1][0].decode())
+    return message_field(9, message_field(1, parsed or b""))
+
+
+def ddl_data_type(text):
+    """The DataType of a DDL string, as the server's DDLParse parses those of 
the tests (see
+    `ddl_json`); None for any other DDL."""
+    value = ddl_json(text)
+    return None if value is None else data_type_proto(value)
+
+
 def struct_schema(name="id"):
     # types.proto: DataType.long=7, .struct=21; Struct.fields=1;
     # StructField: name=1, data_type=2, nullable=3, metadata=4.
@@ -144,6 +293,13 @@ def serve(connection):
                 )
             if self.failure == "rpc":
                 context.abort(grpc.StatusCode.INVALID_ARGUMENT, "schema 
unavailable")
+            # base.proto: AnalyzePlanRequest.ddl_parse=11 -> 
DDLParse.ddl_string=1;
+            # AnalyzePlanResponse.ddl_parse=9 -> DDLParse.parsed=1: the type 
of the DDL, as the
+            # server parses it, or `data_type` for one that `ddl_data_type` 
does not parse.
+            if 11 in parse_message(request):
+                ddl = 
parse_message(parse_message(request)[11][0])[1][0].decode()
+                parsed = ddl_data_type(ddl) or self.data_type
+                return message_field(9, message_field(1, parsed))
             if self.failure == "no-result":
                 return b""
             if self.failure == "missing-schema":
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index a41a792..9424ac5 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -463,7 +463,8 @@ def _arrow_input_service(
     state = {"cache": {}, "uploads": 0, "relations": [], "message_sizes": [],
              "config": config, "config_requests": [], "input_tables": [], 
"tables": [],
              "upload_contexts": [], "status_contexts": [],
-             "status_failures": [], "upload_failures": [], "upload_attempts": 
0}
+             "status_failures": [], "upload_failures": [], "upload_attempts": 
0,
+             "ddl_strings": []}
     server_id = "00000000-0000-0000-0000-000000000001"
 
     def attribution(request, context):
@@ -495,6 +496,48 @@ def _arrow_input_service(
                                if config.get(k) is not None]
                         ).SerializeToString()
 
+    def top_level_split(text):
+        parts, depth, start = [], 0, 0
+        for i, char in enumerate(text):
+            depth += {"<": 1, ">": -1}.get(char, 0)
+            if char == "," and depth == 0:
+                parts.append(text[start:i])
+                start = i + 1
+        return parts + [text[start:]]
+
+    def ddl_type(text):
+        """The DataType of a DDL of these tests, as the server's `ddl_parse` 
parses it: atomic
+        types, arrays and maps of them, and structs of `name type` fields."""
+        text, kinds = text.strip(), {
+            "long": "long", "bigint": "long", "int": "integer", "integer": 
"integer",
+            "double": "double", "float": "float", "string": "string", "date": 
"date",
+            "boolean": "boolean", "binary": "binary", "timestamp": "timestamp"}
+        lower = text.lower()
+        if lower.startswith("array<") and lower.endswith(">"):
+            return protocol("DataType", array=dict(
+                element_type=ddl_type(text[6:-1]), contains_null=True))
+        if lower.startswith("map<") and lower.endswith(">"):
+            key, value = top_level_split(text[4:-1])
+            return protocol("DataType", map=dict(
+                key_type=ddl_type(key), value_type=ddl_type(value), 
value_contains_null=True))
+        if lower in kinds:
+            return protocol("DataType", **{kinds[lower]: {}})
+        fields = [field.strip().split(None, 1) for field in 
top_level_split(text)]
+        return protocol("DataType", struct=dict(fields=[
+            dict(name=name, data_type=ddl_type(kind), nullable=True) for name, 
kind in fields]))
+
+    def analyze(raw, context):
+        # As the server, which parses the DDL schemas that the client sends 
(`_parse_ddl`);
+        # the tests ask for no other analysis.
+        request = protocol("AnalyzePlanRequest").FromString(raw)
+        if request.WhichOneof("analyze") != "ddl_parse":
+            context.abort(grpc.StatusCode.UNIMPLEMENTED, "Method not found!")
+        state["ddl_strings"].append(request.ddl_parse.ddl_string)
+        return protocol("AnalyzePlanResponse", session_id=request.session_id,
+                        server_side_session_id=server_id,
+                        
ddl_parse=dict(parsed=ddl_type(request.ddl_parse.ddl_string))
+                        ).SerializeToString()
+
     def statuses(raw, context):
         request = protocol("ArtifactStatusesRequest").FromString(raw)
         state["status_contexts"].append(attribution(request, context))
@@ -621,6 +664,7 @@ def _arrow_input_service(
         server.add_generic_rpc_handlers([grpc.method_handlers_generic_handler(
             "spark.connect.SparkConnectService", {
                 "Config": grpc.unary_unary_rpc_method_handler(configure),
+                "AnalyzePlan": grpc.unary_unary_rpc_method_handler(analyze),
                 "ArtifactStatus": 
grpc.unary_unary_rpc_method_handler(statuses),
                 "AddArtifacts": grpc.stream_unary_rpc_method_handler(upload),
                 "ExecutePlan": grpc.unary_stream_rpc_method_handler(execute),
@@ -865,6 +909,8 @@ def test_arrow_input_ddl_width_errors(protocol, schema, 
expected_width):
         assert error.value.getMessageParameters() == {
             "expected_length": expected_width, "actual_length": "2",
         }
+        # As PySpark Connect's, the server parses the DDL schema.
+        assert state["ddl_strings"] == [schema]
         assert not state["relations"]
         assert state["uploads"] == 0
 
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 15df8bf..48551dc 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1335,6 +1335,37 @@ def 
test_window_order_by_a_column_sorts_nulls_first_as_pyspark(session):
         assert [(r.o, r.rn) for r in rows] == expected
 
 
+def test_create_dataframe_takes_the_ddl_of_spark_as_pyspark(session):
+    # As PySpark Connect's, which parses a DDL schema on the server: nested 
structs, arrays and
+    # maps of them, backtick-quoted names, and NOT NULL and COMMENT clauses.
+    schema = (
+        "st struct<a:int,b:double>, arr array<struct<c:int>>, m 
map<string,struct<x:int>>, "
+        "`a b` int NOT NULL COMMENT 'c'"
+    )
+    df = session.createDataFrame([((1, 2.0), [(3,)], {"k": (4,)}, 5)], schema)
+    assert df.schema.simpleString() == (
+        "struct<st:struct<a:int,b:double>,arr:array<struct<c:int>>,"
+        "m:map<string,struct<x:int>>,a b:int>"
+    )
+    assert df.collect() == [
+        Row(st=Row(a=1, b=2.0), arr=[Row(c=3)], m={"k": Row(x=4)}, **{"a b": 
5})
+    ]
+    # Collations, which the server names, and the errors of the server's 
parser.
+    collated = session.createDataFrame([("x",)], "s STRING COLLATE UTF8_LCASE")
+    assert collated.schema.simpleString() == "struct<s:string collate 
UTF8_LCASE>"
+    from pyspark.errors import ParseException
+
+    for ddl, condition in [("a INT,", "PARSE_SYNTAX_ERROR"), ("a BOGUS", 
"UNSUPPORTED_DATATYPE")]:
+        with pytest.raises(ParseException) as error:
+            session.createDataFrame([], ddl)
+        assert error.value.getCondition() == condition
+    # As do the map functions, of their schemas.
+    mapped = session.range(2).mapInPandas(lambda it: it, "id long NOT NULL 
COMMENT 'k'")
+    assert [(f.name, f.nullable, f.metadata) for f in mapped.schema.fields] == 
[
+        ("id", False, {"comment": "k"})]
+    assert mapped.collect() == [Row(id=0), Row(id=1)]
+
+
 def test_functions_send_their_optional_arguments_as_pyspark(session):
     # The values of official PySpark 4.3.0; each optional argument used to be 
dropped.
     df = session.sql(
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 0ece022..319d4c0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -20,6 +20,8 @@ from threading import Barrier
 import pytest
 
 from _mock_connect_server import (
+    ddl_json,
+    ddl_parse_response,
     message_field,
     mock_connect_session,
     parse_message,
@@ -4892,7 +4894,11 @@ def test_timedelta_values_convert_as_pyspark():
 
 
 def _local_create_dataframe(*args):
-    # The local relation and its toDF are built without a server.
+    # The local relation and its toDF are built without a server; a DDL 
schema, which PySpark
+    # asks the server to parse, is parsed as the server would.
+    args = list(args)
+    if len(args) > 1 and isinstance(args[1], str) and ddl_json(args[1]) is not 
None:
+        args[1] = T._parse_datatype_json_string(json.dumps(ddl_json(args[1])))
     return _offline_session().createDataFrame(*args)
 
 
@@ -5429,6 +5435,16 @@ def _container_input_fields(data):
     return result
 
 
+def _ddl_parse(request, context):
+    # As the server, of the DDL schemas that the client asks it to parse 
(`_parse_ddl`).
+    import grpc
+
+    response = ddl_parse_response(request)
+    if response is None:
+        context.abort(grpc.StatusCode.UNIMPLEMENTED, "Method not found!")
+    return response
+
+
 def _container_input_capture_server(connection):
     from concurrent.futures import ThreadPoolExecutor
     import grpc
@@ -5452,6 +5468,7 @@ def _container_input_capture_server(connection):
         server.add_generic_rpc_handlers((grpc.method_handlers_generic_handler(
             "spark.connect.SparkConnectService", {
                 "ExecutePlan": grpc.unary_stream_rpc_method_handler(execute),
+                "AnalyzePlan": grpc.unary_unary_rpc_method_handler(_ddl_parse),
             }),))
         port = server.add_insecure_port("127.0.0.1:0")
         server.start()
@@ -5744,6 +5761,7 @@ def _exact_date_capture_server(connection):
             "spark.connect.SparkConnectService", {
                 "ExecutePlan": 
grpc.unary_stream_rpc_method_handler(_exact_date_execute),
                 "Config": 
grpc.unary_unary_rpc_method_handler(_exact_date_config),
+                "AnalyzePlan": grpc.unary_unary_rpc_method_handler(_ddl_parse),
             }),))
         port = server.add_insecure_port("127.0.0.1:0")
         server.start()
@@ -6624,6 +6642,43 @@ def 
test_completion_matches_python_connect(completion_analyzed, names):
     assert df.schema.fieldNames() == names
 
 
+def test_ddl_schemas_are_parsed_on_the_server_as_pyspark(mock_connect):
+    # As PySpark Connect's `_parse_ddl`, which asks the server to parse a DDL 
schema, of
+    # createDataFrame and of the map functions; a type is sent as it is.
+    spark, service = mock_connect
+
+    def ddl_requests():
+        # base.proto: AnalyzePlanRequest.ddl_parse=11, DDLParse.ddl_string=1.
+        return [parse_message(parse_message(request)[11][0])[1][0].decode()
+                for request in service.analyze_requests if 11 in 
parse_message(request)]
+
+    def sent_local_schema():
+        # The LocalRelation of the type that the server parsed: 
AnalyzePlanRequest.schema=4 ->
+        # plan=1 -> root=1 -> (Relation.to_df=18 -> input=1 ->) 
local_relation=11 -> schema=2.
+        relation = parse_message(parse_message(parse_message(
+            parse_message(service.analyze_requests[-1])[4][0])[1][0])[1][0])
+        while 18 in relation:
+            relation = parse_message(parse_message(relation[18][0])[1][0])
+        return 
T._parse_datatype_json_string(parse_message(relation[11][0])[2][0].decode())
+
+    spark.createDataFrame([], "id long").columns
+    assert sent_local_schema().simpleString() == "struct<id:bigint>"
+    nested = "id struct<a:int,b:double> NOT NULL"
+    spark.createDataFrame([], nested).columns
+    assert sent_local_schema() == T.StructType([T.StructField("id", 
T.StructType([
+        T.StructField("a", T.IntegerType()), T.StructField("b", 
T.DoubleType())]), False)])
+    assert ddl_requests() == ["id long", nested]
+    df = spark.range(1)
+    df.mapInPandas(lambda it: it, "id long NOT NULL")
+    df.mapInArrow(lambda it: it, "id long COMMENT 'c'")
+    df.groupBy("id").applyInPandas(lambda pdf: pdf, "`id` long")
+    df.groupBy("id").cogroup(df.groupBy("id")).applyInArrow(lambda l, r: l, 
"id: long")
+    assert ddl_requests()[2:] == [
+        "id long NOT NULL", "id long COMMENT 'c'", "`id` long", "id: long"]
+    df.mapInPandas(lambda it: it, T.StructType([T.StructField("id", 
T.LongType())]))
+    assert len(ddl_requests()) == 6
+
+
 def test_completion_lists_are_independent(completion_analyzed):
     spark, service = completion_analyzed
     df = spark.range(10)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to