This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 2b8650a0 [SPARK-61177][CONNECT] Quote the field names of 
DataType::to_ddl, and canonicalize the collation name of DataType::from_ddl, in 
the Rust client
2b8650a0 is described below

commit 2b8650a03da346436dffd9ef4abc0972d8164a2b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 20:30:13 2026 +0900

    [SPARK-61177][CONNECT] Quote the field names of DataType::to_ddl, and 
canonicalize the collation name of DataType::from_ddl, in the Rust client
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61177](https://issues.apache.org/jira/browse/SPARK-61177): 
`StructType.toDDL()` quotes each field name with `quoteIfNeeded`, which wraps 
the name in backticks, doubles the backticks in it, and leaves a name alone 
only when it matches `[a-zA-Z_][a-zA-Z0-9_]*`. The types are written as 
`DataType.sql`, so a nested struct is ``STRUCT<`my col`: INT NOT NULL COMMENT 
'c'>``. `DataType.fromDDL` canonicalizes the collation name, as the server does.
    
    The Rust client now does the same in `DataType::to_ddl` and in its 
`from_ddl` parser:
    
    - `to_ddl` quotes the top-level names and the names of nested struct 
fields. It writes the types in the `DataType.sql` form: `STRUCT<...>`, 
`ARRAY<...>`, `MAP<K, V>`, and `STRING COLLATE <NAME>` for a non-default 
collation.
    - `from_ddl` takes the collation name from the input, not from the 
lowercased copy, and canonicalizes it with `canonical_collation_name`. A 
`UTF8_*` or `UNICODE*` name is in upper case. An ICU locale name, 
`language[_Script][_COUNTRY][_SPECIFIERS]`, has a lower-case language, a 
title-case four-letter script and upper-case other parts. The server gives 
these names for the input in the first column:
    
      | written | canonical |
      |---|---|
      | `utf8_lcase` | `UTF8_LCASE` |
      | `Unicode_Ci_Ai` | `UNICODE_CI_AI` |
      | `unicode_rtrim` | `UNICODE_RTRIM` |
      | `utf8_lcase_rtrim` | `UTF8_LCASE_RTRIM` |
      | `sr_ci_ai`, `SR_CI_AI`, `sr_CI_AI` | `sr_CI_AI` |
      | `EN` | `en` |
      | `de_ci` | `de_CI` |
      | `zh_hant_hkg` | `zh_Hant_HKG` |
      | `SR_CYRL_SRB_CI_AI` | `sr_Cyrl_SRB_CI_AI` |
    
      The server rejects `en_us` with COLLATION_INVALID_NAME. `from_ddl` does 
not validate names, so it gives `en_US`.
    
    ### Why are the changes needed?
    
    With the old code, `to_ddl` wrote the names as they were. The server cannot 
parse the result for a name that needs quoting:
    
    ```
    createDataFrame([], "my col int")
      ParseException [PARSE_SYNTAX_ERROR] Syntax error at or near 'int': extra 
input 'int'.
    createDataFrame([], "a`b int")
      ParseException [PARSE_SYNTAX_ERROR] Syntax error at or near '`'.
    createDataFrame([], "s struct<my col:int>")
      ParseException [PARSE_SYNTAX_ERROR] Syntax error at or near '<'.
    ```
    
    The strings are what the old `to_ddl` wrote for a struct with a field `my 
col` of type `INT`, a field ``a`b`` of type `INT`, and a struct field `s` of 
type `struct<my col:int>`.
    
    `from_ddl("a STRING COLLATE UTF8_LCASE")` gave the collation `utf8_lcase`. 
Spark gives `UTF8_LCASE`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `DataType::to_ddl` now returns the Spark DDL form, for example `` `my 
col` INT,`a``b` INT `` in place of ``my col int,a`b int``. A nested struct, 
array or map is written with the SQL type names. `DataType::from_ddl` 
canonicalizes the collation name, for example `UTF8_LCASE` for `STRING COLLATE 
utf8_lcase` and `sr_CI_AI` for `STRING COLLATE SR_CI_AI`, as the server does.
    
    ### How was this patch tested?
    
    - `struct_to_ddl_quotes_names_and_writes_sql_types` in 
`crates/spark-connect/tests/types_behavior.rs` covers quoted and unquoted names 
(spaces, backticks, dots, leading digits, all digits, leading underscore, 
unicode), `NOT NULL`, comments with a single quote, nested struct fields, 
arrays and maps of structs, atomic types, and collations. The expected strings 
are the output of `StructType.toDDL()` on the server of PySpark 4.3.0. This 
test fails without the change.
    - `from_ddl_canonicalizes_the_collation_name` in 
`crates/spark-connect/tests/types_behavior.rs` checks the twelve names of the 
table above, with the canonical name the server gives for each (`en_us` as 
`en_US`), and a `to_ddl` round trip. It fails without the change (`utf8_lcase` 
before the change).
    - `struct_to_ddl_tree_and_nullable` was updated to the new output `a INT,b 
STRING NOT NULL`.
    - Against the server of PySpark 4.3.0, `createDataFrame([], "c STRING 
COLLATE <name>")` gives the canonical names of the table above, which the new 
test checks.
    - Against the server of PySpark 4.3.0, the DDL strings written by `to_ddl` 
parse back (`createDataFrame([], ddl).schema`) to the expected types, for the 
names, the nested structs, the arrays and maps, and the collations in the cases 
above. The CHAR and VARCHAR case fails in `createDataFrame` with an internal 
check on the local relation. This is not a parse error.
    - The lib tests of both crates (608 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2227 passed, 8 skipped), and the 
live drop-in and Arrow suites (210 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #374 from HyukjinKwon/SPARK-61177-ddl-quoting.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect/src/types.rs            | 120 ++++++++++++--
 crates/spark-connect/tests/types_behavior.rs | 225 ++++++++++++++++++++++++++-
 2 files changed, 329 insertions(+), 16 deletions(-)

diff --git a/crates/spark-connect/src/types.rs 
b/crates/spark-connect/src/types.rs
index 0a402636..c849174e 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -920,20 +920,19 @@ impl DataType {
     }
 
     /// DDL string for a StructType, mirroring `StructType.toDDL()`:
-    /// comma-separated `name type[ NOT NULL][ COMMENT '...']` per field.
+    /// comma-separated `name type[ NOT NULL][ COMMENT '...']` per field. The 
names are quoted
+    /// and the types written as `DataType.sql`, so the server parses the DDL 
back.
     pub fn to_ddl(&self) -> Result<String> {
         match self {
             DataType::Struct { fields } => Ok(fields
                 .iter()
                 .map(|f| {
-                    let mut s = format!("{} {}", f.name, 
f.data_type.simple_string());
-                    if !f.nullable {
-                        s.push_str(" NOT NULL");
-                    }
-                    if let Some(comment) = 
f.metadata.get("comment").and_then(|c| c.as_str()) {
-                        s.push_str(&format!(" COMMENT '{}'", 
comment.replace('\'', "\\'")));
-                    }
-                    s
+                    format!(
+                        "{} {}{}",
+                        quote_if_needed(&f.name),
+                        f.data_type.sql_type(),
+                        field_ddl_suffix(f)
+                    )
                 })
                 .collect::<Vec<_>>()
                 .join(",")),
@@ -944,6 +943,39 @@ impl DataType {
         }
     }
 
+    /// The SQL form of a type, mirroring `DataType.sql`: `INT`, `STRING 
COLLATE UTF8_LCASE`,
+    /// `ARRAY<...>`, `MAP<K, V>` and `STRUCT<name: TYPE, ...>`, the form 
`toDDL` writes.
+    fn sql_type(&self) -> String {
+        match self {
+            DataType::Struct { fields } => {
+                let fields: Vec<String> = fields
+                    .iter()
+                    .map(|f| {
+                        format!(
+                            "{}: {}{}",
+                            quote_if_needed(&f.name),
+                            f.data_type.sql_type(),
+                            field_ddl_suffix(f)
+                        )
+                    })
+                    .collect();
+                format!("STRUCT<{}>", fields.join(", "))
+            }
+            DataType::Array { element_type, .. } => format!("ARRAY<{}>", 
element_type.sql_type()),
+            DataType::Map {
+                key_type,
+                value_type,
+                ..
+            } => format!("MAP<{}, {}>", key_type.sql_type(), 
value_type.sql_type()),
+            DataType::String { collation }
+                if !(collation.is_empty() || collation == "UTF8_BINARY") =>
+            {
+                format!("STRING COLLATE {}", collation.to_uppercase())
+            }
+            _ => self.simple_string().to_uppercase(),
+        }
+    }
+
     /// Tree-string for a StructType, mirroring `StructType.treeString()`.
     pub fn tree_string(&self) -> Result<String> {
         self.tree_string_with_depth(i32::MAX)
@@ -1176,6 +1208,32 @@ fn interval_string(start_field: i32, end_field: i32, 
fields: &[(&str, i32)]) ->
     }
 }
 
+/// Quotes a field name with backticks, as Spark's 
`QuotingUtils.quoteIfNeeded` does: a name
+/// that is not `[a-zA-Z_][a-zA-Z0-9_]*` is quoted, and each backtick in it is 
doubled.
+fn quote_if_needed(name: &str) -> String {
+    let mut chars = name.chars();
+    let plain = matches!(chars.next(), Some(c) if c == '_' || 
c.is_ascii_alphabetic())
+        && chars.all(|c| c == '_' || c.is_ascii_alphanumeric());
+    if plain {
+        name.to_string()
+    } else {
+        format!("`{}`", name.replace('`', "``"))
+    }
+}
+
+/// The ` NOT NULL` and ` COMMENT '...'` suffix of a field, as 
`StructField.toDDL` writes it. The
+/// single quotes of the comment are escaped as Spark's 
`escapeSingleQuotedString` does.
+fn field_ddl_suffix(field: &StructField) -> String {
+    let mut s = String::new();
+    if !field.nullable {
+        s.push_str(" NOT NULL");
+    }
+    if let Some(comment) = field.metadata.get("comment").and_then(|c| 
c.as_str()) {
+        s.push_str(&format!(" COMMENT '{}'", comment.replace('\'', "\\'")));
+    }
+    s
+}
+
 /// The key of the metadata of a struct field of the collations of its 
strings, as Spark's.
 const COLLATIONS_METADATA_KEY: &str = "__COLLATIONS";
 
@@ -1907,6 +1965,35 @@ fn split_top_level_comma(s: &str) -> Vec<&str> {
     result
 }
 
+/// The canonical form of a collation name, as Spark's server gives it. A 
`UTF8_*` or `UNICODE*`
+/// name is in upper case. An ICU locale name, 
`language[_Script][_COUNTRY][_SPECIFIERS]`, has a
+/// lower-case language, a title-case script (a four-letter part) and 
upper-case other parts, such
+/// as `sr_Cyrl_SRB_CI_AI`. Collation names are case-insensitive, so any case 
is accepted.
+fn canonical_collation_name(name: &str) -> String {
+    let provider = name.split('_').next().unwrap_or_default();
+    if provider.eq_ignore_ascii_case("UTF8") || 
provider.eq_ignore_ascii_case("UNICODE") {
+        return name.to_uppercase();
+    }
+    name.split('_')
+        .enumerate()
+        .map(|(i, part)| {
+            if i == 0 {
+                part.to_lowercase()
+            } else if part.len() == 4 && part.chars().all(|c| 
c.is_ascii_alphabetic()) {
+                let mut chars = part.chars();
+                let first = chars.next().map(|c| c.to_ascii_uppercase());
+                first
+                    .into_iter()
+                    .chain(chars.map(|c| c.to_ascii_lowercase()))
+                    .collect()
+            } else {
+                part.to_uppercase()
+            }
+        })
+        .collect::<Vec<_>>()
+        .join("_")
+}
+
 /// Parse a single type (primitive or complex)
 fn parse_single_type(input: &str) -> Result<DataType> {
     let trimmed = input.trim().to_lowercase();
@@ -1959,12 +2046,15 @@ fn parse_single_type(input: &str) -> Result<DataType> {
             } else if trimmed.starts_with("interval") {
                 parse_interval_string(&trimmed)
             } else if trimmed.starts_with("string collate") {
-                let collation = trimmed
-                    .trim_start_matches("string")
-                    .trim_start_matches("collate")
-                    .trim()
-                    .to_string();
-                Ok(DataType::String { collation })
+                // The name is taken from the input, not from `trimmed`, which 
is lowercased.
+                let collation = input
+                    .split_whitespace()
+                    .skip(2)
+                    .collect::<Vec<_>>()
+                    .join(" ");
+                Ok(DataType::String {
+                    collation: canonical_collation_name(&collation),
+                })
             } else if trimmed.starts_with("geometry(") {
                 parse_geometry(&trimmed)
             } else if trimmed.starts_with("geography(") {
diff --git a/crates/spark-connect/tests/types_behavior.rs 
b/crates/spark-connect/tests/types_behavior.rs
index 56ee941f..a705ce05 100644
--- a/crates/spark-connect/tests/types_behavior.rs
+++ b/crates/spark-connect/tests/types_behavior.rs
@@ -277,7 +277,7 @@ fn struct_to_ddl_tree_and_nullable() {
             },
         ],
     };
-    assert_eq!(st.to_ddl().unwrap(), "a int,b string NOT NULL");
+    assert_eq!(st.to_ddl().unwrap(), "a INT,b STRING NOT NULL");
     let tree = st.tree_string().unwrap();
     assert!(tree.starts_with("root\n"));
     assert!(tree.contains("a: integer (nullable = true)"));
@@ -293,6 +293,229 @@ fn struct_to_ddl_tree_and_nullable() {
     assert!(DataType::Integer.tree_string().is_err());
 }
 
+/// The expected strings are the output of `StructType.toDDL()` of PySpark 
4.3.0 on a server.
+#[test]
+fn struct_to_ddl_quotes_names_and_writes_sql_types() {
+    use spark_connect::types::StructField;
+    use std::collections::BTreeMap;
+
+    fn field(name: &str, data_type: DataType, nullable: bool) -> StructField {
+        StructField {
+            name: name.to_string(),
+            data_type,
+            nullable,
+            metadata: BTreeMap::new(),
+        }
+    }
+    fn commented(name: &str, data_type: DataType, nullable: bool, comment: 
&str) -> StructField {
+        let mut f = field(name, data_type, nullable);
+        f.metadata
+            .insert("comment".to_string(), serde_json::json!(comment));
+        f
+    }
+    fn ddl(fields: Vec<StructField>) -> String {
+        DataType::Struct { fields }.to_ddl().unwrap()
+    }
+    let int = || DataType::Integer;
+    let string = || DataType::String {
+        collation: "UTF8_BINARY".to_string(),
+    };
+
+    // Names are quoted unless they are `[a-zA-Z_][a-zA-Z0-9_]*`; a backtick 
is doubled.
+    assert_eq!(
+        ddl(vec![
+            field("my col", int(), true),
+            field("a`b", int(), true),
+            field("a.b", int(), true),
+            field("1abc", int(), true),
+            field("123", int(), true),
+            field("_x", int(), true),
+            field("ünï", int(), true),
+        ]),
+        "`my col` INT,`a``b` INT,`a.b` INT,`1abc` INT,`123` INT,_x INT,`ünï` 
INT"
+    );
+    assert_eq!(
+        ddl(vec![commented("x", int(), false, "it's")]),
+        "x INT NOT NULL COMMENT 'it\\'s'"
+    );
+    // Nested struct fields are written as `name: TYPE`, with the same quoting 
and suffixes.
+    assert_eq!(
+        ddl(vec![field(
+            "s",
+            DataType::Struct {
+                fields: vec![commented("my col", int(), false, "c")],
+            },
+            true,
+        )]),
+        "s STRUCT<`my col`: INT NOT NULL COMMENT 'c'>"
+    );
+    assert_eq!(
+        ddl(vec![field(
+            "arr",
+            DataType::Array {
+                element_type: Box::new(DataType::Struct {
+                    fields: vec![field("a b", int(), true)],
+                }),
+                contains_null: true,
+            },
+            true,
+        )]),
+        "arr ARRAY<STRUCT<`a b`: INT>>"
+    );
+    assert_eq!(
+        ddl(vec![field(
+            "m",
+            DataType::Map {
+                key_type: Box::new(string()),
+                value_type: Box::new(DataType::Struct {
+                    fields: vec![field("k`v", DataType::Long, false)],
+                }),
+                value_contains_null: true,
+            },
+            true,
+        )]),
+        "m MAP<STRING, STRUCT<`k``v`: BIGINT NOT NULL>>"
+    );
+    assert_eq!(
+        ddl(vec![field(
+            "a",
+            DataType::Array {
+                element_type: Box::new(DataType::Struct {
+                    fields: vec![commented("q", string(), true, "x'y")],
+                }),
+                contains_null: true,
+            },
+            true,
+        )]),
+        "a ARRAY<STRUCT<q: STRING COMMENT 'x\\'y'>>"
+    );
+    // Atomic types are written as `DataType.sql`.
+    assert_eq!(
+        ddl(vec![
+            field(
+                "d",
+                DataType::Decimal {
+                    precision: 10,
+                    scale: 2
+                },
+                true
+            ),
+            field("ntz", DataType::TimestampNtz, true),
+            field(
+                "ym",
+                DataType::YearMonthInterval {
+                    start_field: 0,
+                    end_field: 1
+                },
+                true
+            ),
+            field(
+                "dt",
+                DataType::DayTimeInterval {
+                    start_field: 0,
+                    end_field: 3
+                },
+                true
+            ),
+            field("ch", DataType::Char { length: 3 }, true),
+            field("vc", DataType::Varchar { length: 4 }, true),
+            field("tm", DataType::Time { precision: 3 }, true),
+            field("bin", DataType::Binary, true),
+            field("b", DataType::Boolean, true),
+            field("v", DataType::Variant, true),
+        ]),
+        "d DECIMAL(10,2),ntz TIMESTAMP_NTZ,ym INTERVAL YEAR TO MONTH,dt 
INTERVAL DAY TO SECOND,\
+         ch CHAR(3),vc VARCHAR(4),tm TIME(3),bin BINARY,b BOOLEAN,v VARIANT"
+    );
+    // A non-default collation is written as `STRING COLLATE <name>`.
+    assert_eq!(
+        ddl(vec![field(
+            "c",
+            DataType::String {
+                collation: "UTF8_LCASE".to_string(),
+            },
+            true,
+        )]),
+        "c STRING COLLATE UTF8_LCASE"
+    );
+    assert_eq!(
+        ddl(vec![field(
+            "s",
+            DataType::Struct {
+                fields: vec![field(
+                    "c d",
+                    DataType::String {
+                        collation: "UNICODE_CI".to_string(),
+                    },
+                    true,
+                )],
+            },
+            true,
+        )]),
+        "s STRUCT<`c d`: STRING COLLATE UNICODE_CI>"
+    );
+}
+
+#[test]
+fn from_ddl_canonicalizes_the_collation_name() {
+    // The canonical names of the server of PySpark 4.3.0, from 
`createDataFrame([], "c STRING
+    // COLLATE <name>")`. `en_us` is rejected by the server 
(COLLATION_INVALID_NAME), so `from_ddl`
+    // only canonicalizes its case, `en_US`.
+    let cases = [
+        ("utf8_lcase", "UTF8_LCASE"),
+        ("Unicode_Ci_Ai", "UNICODE_CI_AI"),
+        ("unicode_rtrim", "UNICODE_RTRIM"),
+        ("utf8_lcase_rtrim", "UTF8_LCASE_RTRIM"),
+        ("sr_ci_ai", "sr_CI_AI"),
+        ("SR_CI_AI", "sr_CI_AI"),
+        ("sr_CI_AI", "sr_CI_AI"),
+        ("EN", "en"),
+        ("de_ci", "de_CI"),
+        ("zh_hant_hkg", "zh_Hant_HKG"),
+        ("SR_CYRL_SRB_CI_AI", "sr_Cyrl_SRB_CI_AI"),
+        ("en_us", "en_US"),
+    ];
+    for (written, canonical) in cases {
+        assert_eq!(
+            DataType::from_ddl(&format!("STRING COLLATE {written}")).unwrap(),
+            DataType::String {
+                collation: canonical.to_string(),
+            },
+            "STRING COLLATE {written}"
+        );
+    }
+    match DataType::from_ddl("a STRING COLLATE UTF8_LCASE").unwrap() {
+        DataType::Struct { fields } => assert_eq!(
+            fields[0].data_type,
+            DataType::String {
+                collation: "UTF8_LCASE".to_string(),
+            }
+        ),
+        other => panic!("expected struct, got {other:?}"),
+    }
+    assert_eq!(
+        DataType::from_ddl("array<string collate UNICODE_CI>").unwrap(),
+        DataType::Array {
+            element_type: Box::new(DataType::String {
+                collation: "UNICODE_CI".to_string(),
+            }),
+            contains_null: true,
+        }
+    );
+    // The DDL written by `to_ddl` parses back to the same collation.
+    let st = DataType::Struct {
+        fields: vec![spark_connect::types::StructField {
+            name: "a".to_string(),
+            data_type: DataType::String {
+                collation: "UTF8_LCASE".to_string(),
+            },
+            nullable: true,
+            metadata: Default::default(),
+        }],
+    };
+    assert_eq!(DataType::from_ddl(&st.to_ddl().unwrap()).unwrap(), st);
+}
+
 #[test]
 fn tree_string_with_depth_truncates_nested_structs() {
     use spark_connect::types::StructField;


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to