This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 2a7e0df5 [SPARK-61206][CONNECT] Escape the backslashes of the comments
of DataType::to_ddl, as Spark's DDL does
2a7e0df5 is described below
commit 2a7e0df5a0d91b776a32888268cafe32747a8121
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:05:45 2026 +0900
[SPARK-61206][CONNECT] Escape the backslashes of the comments of
DataType::to_ddl, as Spark's DDL does
### What changes were proposed in this pull request?
[SPARK-61206](https://issues.apache.org/jira/browse/SPARK-61206): Escape
the backslashes of the comments in `DataType::to_ddl`, as Spark's DDL does.
`DataType::to_ddl` (and the nested `sql_type` of its struct fields) writes
a field comment as `COMMENT '...'`, with only its single quotes escaped. The
backslashes are now escaped too, first and then the quotes, as Spark's
`escapeSingleQuotedString` does:
- `a\b` is written as `COMMENT 'a\\b'`,
- `it's` as `COMMENT 'it\'s'`, and `ends\` as `COMMENT 'ends\\'`.
The official `StructType.toDDL()` writes the same text for these comments,
and the server reads each one back as it was. The newline and the other
characters are written as before.
### Why are the changes needed?
With the old escaping, the server's parser read the backslash of a comment
as an escape sequence. A comment `a\b` was written as `COMMENT 'a\b'`, which
the server reads as `a` followed by a backspace character (U+0008). A trailing
backslash, or a backslash before a quote, broke the literal. The data changed
silently, or the DDL failed to parse, for any comment with a backslash.
### Does this PR introduce _any_ user-facing change?
Yes. `DataType::to_ddl` (Rust API only) writes the backslashes of a comment
escaped. The DDL of a comment without a backslash does not change.
### How was this patch tested?
- New tests at the end of `crates/spark-connect/tests/types_roundtrip.rs`:
- `to_ddl_escapes_the_comments_as_spark_does` checks the DDL of comments
with a backslash, a quote, an escaped quote, a trailing backslash and a
newline, with and without NOT NULL.
- `to_ddl_comments_round_trip_through_the_server_parser` parses the DDL
with the server (`session.parse_ddl`) and checks that each comment comes back
as it was. It runs when `SPARK_REMOTE` is set.
- Both tests fail on the code before this change (the backslash comments)
and pass with it.
- The official PySpark 4.3.0 `StructType.toDDL()` gives the same DDL text
for each of these comments, and the same round trip, against the same server.
- With `SPARK_REMOTE` set to a local Spark 4.3.0 server, `cargo test -p
apache-spark-connect --release --test types_roundtrip` (12 passed), the server
round trip included.
- The lib tests of both crates (618 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2456 passed, 8 skipped);
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs
the e2e and live suites.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #392 from
HyukjinKwon/SPARK-61206-r4-escape-backslashes-in-to-ddl-comments.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect/src/types.rs | 6 ++-
crates/spark-connect/tests/types_roundtrip.rs | 72 +++++++++++++++++++++++++++
2 files changed, 76 insertions(+), 2 deletions(-)
diff --git a/crates/spark-connect/src/types.rs
b/crates/spark-connect/src/types.rs
index c849174e..31b2e03e 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -1222,14 +1222,16 @@ fn quote_if_needed(name: &str) -> String {
}
/// The ` NOT NULL` and ` COMMENT '...'` suffix of a field, as
`StructField.toDDL` writes it. The
-/// single quotes of the comment are escaped as Spark's
`escapeSingleQuotedString` does.
+/// backslashes and single quotes of the comment are escaped as Spark's
`escapeSingleQuotedString`
+/// does: the backslashes first, so that the escape of a quote is not escaped
again.
fn field_ddl_suffix(field: &StructField) -> String {
let mut s = String::new();
if !field.nullable {
s.push_str(" NOT NULL");
}
if let Some(comment) = field.metadata.get("comment").and_then(|c|
c.as_str()) {
- s.push_str(&format!(" COMMENT '{}'", comment.replace('\'', "\\'")));
+ let escaped = comment.replace('\\', "\\\\").replace('\'', "\\'");
+ s.push_str(&format!(" COMMENT '{escaped}'"));
}
s
}
diff --git a/crates/spark-connect/tests/types_roundtrip.rs
b/crates/spark-connect/tests/types_roundtrip.rs
index d28a2eff..80b7407d 100644
--- a/crates/spark-connect/tests/types_roundtrip.rs
+++ b/crates/spark-connect/tests/types_roundtrip.rs
@@ -456,3 +456,75 @@ fn spatial_json_names_the_crs_as_pyspark() {
DataType::Geography { srid: -1 }
);
}
+
+/// The COMMENT of `toDDL` escapes the backslashes and the single quotes of a
comment as Spark's
+/// `escapeSingleQuotedString` does, so that the parser reads back the comment
as it is.
+#[test]
+fn to_ddl_escapes_the_comments_as_spark_does() {
+ fn with_comment(comment: &str, nullable: bool) -> DataType {
+ DataType::Struct {
+ fields: vec![StructField {
+ name: "c".to_string(),
+ data_type: DataType::Integer,
+ nullable,
+ metadata: BTreeMap::from([(
+ "comment".to_string(),
+ serde_json::Value::String(comment.to_string()),
+ )]),
+ }],
+ }
+ }
+ for (comment, ddl) in [
+ ("a\\b", r"c INT COMMENT 'a\\b'"),
+ ("it's", r"c INT COMMENT 'it\'s'"),
+ (r"it\'s", r"c INT COMMENT 'it\\\'s'"),
+ ("ends\\", r"c INT COMMENT 'ends\\'"),
+ ("line\nbreak", "c INT COMMENT 'line\nbreak'"),
+ ] {
+ assert_eq!(
+ with_comment(comment, true).to_ddl().unwrap(),
+ ddl,
+ "{comment:?}"
+ );
+ }
+ // A NOT NULL field keeps its comment's escapes, after the NOT NULL.
+ assert_eq!(
+ with_comment("a\\b", false).to_ddl().unwrap(),
+ r"c INT NOT NULL COMMENT 'a\\b'"
+ );
+}
+
+/// The comments that `toDDL` writes come back from the server's DDL parser as
they were, with the
+/// backslashes and quotes of the comment. Runs only with `SPARK_REMOTE` set.
+#[test]
+fn to_ddl_comments_round_trip_through_the_server_parser() {
+ let Ok(url) = std::env::var("SPARK_REMOTE") else {
+ return;
+ };
+ let session = spark_connect::session::SparkSession::builder()
+ .remote(&url)
+ .get_or_create()
+ .expect("session");
+ for comment in ["a\\b", "it's", "it\\'s", "ends\\", "line\nbreak"] {
+ let schema = DataType::Struct {
+ fields: vec![StructField {
+ name: "c".to_string(),
+ data_type: DataType::Integer,
+ nullable: true,
+ metadata: BTreeMap::from([(
+ "comment".to_string(),
+ serde_json::Value::String(comment.to_string()),
+ )]),
+ }],
+ };
+ let parsed = session.parse_ddl(&schema.to_ddl().unwrap()).unwrap();
+ let DataType::Struct { fields } = parsed else {
+ panic!("expected a struct");
+ };
+ assert_eq!(
+ fields[0].metadata.get("comment"),
+ Some(&serde_json::Value::String(comment.to_string())),
+ "{comment:?}"
+ );
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]