This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 2a7e0df5 [SPARK-61206][CONNECT] Escape the backslashes of the comments 
of DataType::to_ddl, as Spark's DDL does
2a7e0df5 is described below

commit 2a7e0df5a0d91b776a32888268cafe32747a8121
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:05:45 2026 +0900

    [SPARK-61206][CONNECT] Escape the backslashes of the comments of 
DataType::to_ddl, as Spark's DDL does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61206](https://issues.apache.org/jira/browse/SPARK-61206): Escape 
the backslashes of the comments in `DataType::to_ddl`, as Spark's DDL does.
    
    `DataType::to_ddl` (and the nested `sql_type` of its struct fields) writes 
a field comment as `COMMENT '...'`, with only its single quotes escaped. The 
backslashes are now escaped too, first and then the quotes, as Spark's 
`escapeSingleQuotedString` does:
    
    - `a\b` is written as `COMMENT 'a\\b'`,
    - `it's` as `COMMENT 'it\'s'`, and `ends\` as `COMMENT 'ends\\'`.
    
    The official `StructType.toDDL()` writes the same text for these comments, 
and the server reads each one back as it was. The newline and the other 
characters are written as before.
    
    ### Why are the changes needed?
    
    With the old escaping, the server's parser read the backslash of a comment 
as an escape sequence. A comment `a\b` was written as `COMMENT 'a\b'`, which 
the server reads as `a` followed by a backspace character (U+0008). A trailing 
backslash, or a backslash before a quote, broke the literal. The data changed 
silently, or the DDL failed to parse, for any comment with a backslash.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `DataType::to_ddl` (Rust API only) writes the backslashes of a comment 
escaped. The DDL of a comment without a backslash does not change.
    
    ### How was this patch tested?
    
    - New tests at the end of `crates/spark-connect/tests/types_roundtrip.rs`:
      - `to_ddl_escapes_the_comments_as_spark_does` checks the DDL of comments 
with a backslash, a quote, an escaped quote, a trailing backslash and a 
newline, with and without NOT NULL.
      - `to_ddl_comments_round_trip_through_the_server_parser` parses the DDL 
with the server (`session.parse_ddl`) and checks that each comment comes back 
as it was. It runs when `SPARK_REMOTE` is set.
      - Both tests fail on the code before this change (the backslash comments) 
and pass with it.
    - The official PySpark 4.3.0 `StructType.toDDL()` gives the same DDL text 
for each of these comments, and the same round trip, against the same server.
    - With `SPARK_REMOTE` set to a local Spark 4.3.0 server, `cargo test -p 
apache-spark-connect --release --test types_roundtrip` (12 passed), the server 
round trip included.
    - The lib tests of both crates (618 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2456 passed, 8 skipped); 
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs 
the e2e and live suites.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #392 from 
HyukjinKwon/SPARK-61206-r4-escape-backslashes-in-to-ddl-comments.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect/src/types.rs             |  6 ++-
 crates/spark-connect/tests/types_roundtrip.rs | 72 +++++++++++++++++++++++++++
 2 files changed, 76 insertions(+), 2 deletions(-)

diff --git a/crates/spark-connect/src/types.rs 
b/crates/spark-connect/src/types.rs
index c849174e..31b2e03e 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -1222,14 +1222,16 @@ fn quote_if_needed(name: &str) -> String {
 }
 
 /// The ` NOT NULL` and ` COMMENT '...'` suffix of a field, as 
`StructField.toDDL` writes it. The
-/// single quotes of the comment are escaped as Spark's 
`escapeSingleQuotedString` does.
+/// backslashes and single quotes of the comment are escaped as Spark's 
`escapeSingleQuotedString`
+/// does: the backslashes first, so that the escape of a quote is not escaped 
again.
 fn field_ddl_suffix(field: &StructField) -> String {
     let mut s = String::new();
     if !field.nullable {
         s.push_str(" NOT NULL");
     }
     if let Some(comment) = field.metadata.get("comment").and_then(|c| 
c.as_str()) {
-        s.push_str(&format!(" COMMENT '{}'", comment.replace('\'', "\\'")));
+        let escaped = comment.replace('\\', "\\\\").replace('\'', "\\'");
+        s.push_str(&format!(" COMMENT '{escaped}'"));
     }
     s
 }
diff --git a/crates/spark-connect/tests/types_roundtrip.rs 
b/crates/spark-connect/tests/types_roundtrip.rs
index d28a2eff..80b7407d 100644
--- a/crates/spark-connect/tests/types_roundtrip.rs
+++ b/crates/spark-connect/tests/types_roundtrip.rs
@@ -456,3 +456,75 @@ fn spatial_json_names_the_crs_as_pyspark() {
         DataType::Geography { srid: -1 }
     );
 }
+
+/// The COMMENT of `toDDL` escapes the backslashes and the single quotes of a 
comment as Spark's
+/// `escapeSingleQuotedString` does, so that the parser reads back the comment 
as it is.
+#[test]
+fn to_ddl_escapes_the_comments_as_spark_does() {
+    fn with_comment(comment: &str, nullable: bool) -> DataType {
+        DataType::Struct {
+            fields: vec![StructField {
+                name: "c".to_string(),
+                data_type: DataType::Integer,
+                nullable,
+                metadata: BTreeMap::from([(
+                    "comment".to_string(),
+                    serde_json::Value::String(comment.to_string()),
+                )]),
+            }],
+        }
+    }
+    for (comment, ddl) in [
+        ("a\\b", r"c INT COMMENT 'a\\b'"),
+        ("it's", r"c INT COMMENT 'it\'s'"),
+        (r"it\'s", r"c INT COMMENT 'it\\\'s'"),
+        ("ends\\", r"c INT COMMENT 'ends\\'"),
+        ("line\nbreak", "c INT COMMENT 'line\nbreak'"),
+    ] {
+        assert_eq!(
+            with_comment(comment, true).to_ddl().unwrap(),
+            ddl,
+            "{comment:?}"
+        );
+    }
+    // A NOT NULL field keeps its comment's escapes, after the NOT NULL.
+    assert_eq!(
+        with_comment("a\\b", false).to_ddl().unwrap(),
+        r"c INT NOT NULL COMMENT 'a\\b'"
+    );
+}
+
+/// The comments that `toDDL` writes come back from the server's DDL parser as 
they were, with the
+/// backslashes and quotes of the comment. Runs only with `SPARK_REMOTE` set.
+#[test]
+fn to_ddl_comments_round_trip_through_the_server_parser() {
+    let Ok(url) = std::env::var("SPARK_REMOTE") else {
+        return;
+    };
+    let session = spark_connect::session::SparkSession::builder()
+        .remote(&url)
+        .get_or_create()
+        .expect("session");
+    for comment in ["a\\b", "it's", "it\\'s", "ends\\", "line\nbreak"] {
+        let schema = DataType::Struct {
+            fields: vec![StructField {
+                name: "c".to_string(),
+                data_type: DataType::Integer,
+                nullable: true,
+                metadata: BTreeMap::from([(
+                    "comment".to_string(),
+                    serde_json::Value::String(comment.to_string()),
+                )]),
+            }],
+        };
+        let parsed = session.parse_ddl(&schema.to_ddl().unwrap()).unwrap();
+        let DataType::Struct { fields } = parsed else {
+            panic!("expected a struct");
+        };
+        assert_eq!(
+            fields[0].metadata.get("comment"),
+            Some(&serde_json::Value::String(comment.to_string())),
+            "{comment:?}"
+        );
+    }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to