This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new a59f8c5c6861 [SPARK-52832][3.5][SQL] Fix JDBC dialect identifier
quoting
a59f8c5c6861 is described below
commit a59f8c5c68610beaa59fa68e69231014343d9b1c
Author: Holden Karau <[email protected]>
AuthorDate: Mon Aug 31 16:14:05 2026 -0700
[SPARK-52832][3.5][SQL] Fix JDBC dialect identifier quoting
### What changes were proposed in this pull request?
Backport of #51533 to `branch-3.5`, plus its comment-only followup #51620.
The original change is SPARK-52832, `2bc575785465de53376b075b35a23a318d0fcd93`;
the followup is `47b08a0e9588955aa389a5f21d0f994765a04fec`.
- `JdbcDialect.quoteIdentifier` escapes a double quote in the identifier by
doubling it, per the ANSI standard.
- `MySQLDialect.quoteIdentifier` escapes a backtick by doubling it, per the
MySQL manual.
The original authors are carried through as authors of this commit. Any
mistake in adapting it to `branch-3.5` is mine, not theirs.
### Backport notes
The cherry-pick does not apply cleanly. Three deviations from a verbatim
pick:
- `DatabricksDialect` does not exist on `branch-3.5` (it arrives in 4.0.0
with SPARK-45139), so its `quoteIdentifier` hunk is dropped.
- The `databricksDialect` assertions come out of the `JDBCSuite` test for
the same reason. They could not simply be left in place: `branch-3.5` registers
no Databricks dialect, so `jdbc:databricks://host/db` resolves to `NoopDialect`
and emits ANSI double quotes, which fails the upstream backtick expectation.
- The `MySQLDialect` comment is the post-followup text, folding in #51620,
which deleted a trailing sentence that did not read sensibly without the
screenshot from the MySQL docs. This makes `branch-3.5` match `master` rather
than `branch-4.0`, which took the fix but not the followup.
The two main-code hunks are otherwise byte-identical to upstream.
### Why are the changes needed?
An identifier that contains the dialect's own quote character terminates
the quoted identifier early, so the remainder of the name is emitted as SQL
text instead of as part of the identifier, and the generated statement is wrong.
`quoteIdentifier` is what every `branch-3.5` dialect uses to quote column,
table, schema and index names in generated SQL. They all inherit the base
implementation except `MySQLDialect`, fixed here too, and `AggregatedDialect`,
which delegates to `dialects.head`. So these two hunks cover the branch.
### Does this PR introduce _any_ user-facing change?
Yes, the same changes upstream made:
- Identifiers containing a double quote, or a backtick under
`MySQLDialect`, are now quoted correctly rather than producing a malformed
statement.
- `JDBCRelation.verifyAndGetNormalizedPartitionColumn` matches the
`partitionColumn` option against `dialect.quoteIdentifier(f.name)`, so a caller
who worked around the old behaviour by passing a naively quoted name for a
column that itself contains a quote (field `a"b`, option `"a"b"`) now has to
pass `"a""b"`. No test covers this on any branch.
### How was this patch tested?
`JDBCSuite`, "quote column names by jdbc dialect", which the original
commit extended to cover a double quote and a backtick in the identifier --
here for the MySQL, Postgres, Derby and Oracle dialects. It fails on
`branch-3.5` without the main-code change:
Expected List("`abc`", "`key`", "`double_quote"`", "`back```"),
but got List("`abc`", "`key`", "`double_quote"`", "`back``")
Green on this branch with the change, JDK 17, Scala 2.12:
JDBCSuite 105 tests
JDBCWriteSuite 32 tests, 1 ignored
JDBCV2Suite 70 tests
The docker integration suites were not run.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
Lead-authored-by: Uros Stankovic <uros.stankovicdatabricks.com>
(cherry picked from commit 2bc575785465de53376b075b35a23a318d0fcd93)
Closes #58393 from holdenk/SPARK-52832-jdbc-quote-identifier-branch-3.5.
Lead-authored-by: Holden Karau <[email protected]>
Co-authored-by: Uros Stankovic <[email protected]>
Signed-off-by: Holden Karau <[email protected]>
---
.../org/apache/spark/sql/jdbc/JdbcDialects.scala | 4 +++-
.../org/apache/spark/sql/jdbc/MySQLDialect.scala | 8 ++++++-
.../org/apache/spark/sql/jdbc/JDBCSuite.scala | 25 ++++++++++++----------
3 files changed, 24 insertions(+), 13 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
index bea31a6ecd88..ca5d9a8d922b 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala
@@ -170,7 +170,9 @@ abstract class JdbcDialect extends Serializable with
Logging {
* name is a reserved keyword, or in case it contains characters that
require quotes (e.g. space).
*/
def quoteIdentifier(colName: String): String = {
- s""""$colName""""
+ // By ANSI standard, quotes are escaped with another quotes.
+ val escapedColName = colName.replace("\"", "\"\"")
+ s""""$escapedColName""""
}
/**
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala
b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala
index 5cdba5f1b91a..a40ed80130cb 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala
@@ -132,7 +132,13 @@ private case object MySQLDialect extends JdbcDialect with
SQLConfHelper {
}
override def quoteIdentifier(colName: String): String = {
- s"`$colName`"
+ // Per MySQL documentation:
https://dev.mysql.com/doc/refman/8.4/en/identifiers.html
+ //
+ // Identifier quote characters can be included within an identifier if you
quote the
+ // identifier. If the character to be included within the identifier is
the same as
+ // that used to quote the identifier itself, then you need to double the
character.
+ val escapedColName = colName.replace("`", "``")
+ s"`$escapedColName`"
}
override def schemasExists(conn: Connection, options: JDBCOptions, schema:
String): Boolean = {
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
index f74aff8efb41..3ec7d4d99430 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
@@ -834,17 +834,20 @@ class JDBCSuite extends QueryTest with SharedSparkSession
{
}
test("quote column names by jdbc dialect") {
- val MySQL = JdbcDialects.get("jdbc:mysql://127.0.0.1/db")
- val Postgres = JdbcDialects.get("jdbc:postgresql://127.0.0.1/db")
- val Derby = JdbcDialects.get("jdbc:derby:db")
-
- val columns = Seq("abc", "key")
- val MySQLColumns = columns.map(MySQL.quoteIdentifier(_))
- val PostgresColumns = columns.map(Postgres.quoteIdentifier(_))
- val DerbyColumns = columns.map(Derby.quoteIdentifier(_))
- assert(MySQLColumns === Seq("`abc`", "`key`"))
- assert(PostgresColumns === Seq(""""abc"""", """"key""""))
- assert(DerbyColumns === Seq(""""abc"""", """"key""""))
+ val mySQLDialect = JdbcDialects.get("jdbc:mysql://127.0.0.1/db")
+ val postgresDialect = JdbcDialects.get("jdbc:postgresql://127.0.0.1/db")
+ val derbyDialect = JdbcDialects.get("jdbc:derby:db")
+ val oracleDialect =
JdbcDialects.get("jdbc:oracle:thin:@//localhost:1521/orcl")
+
+ val columns = Seq("abc", "key", "double_quote\"", "back`")
+ val mySQLColumns = columns.map(mySQLDialect.quoteIdentifier)
+ val postgresColumns = columns.map(postgresDialect.quoteIdentifier)
+ val derbyColumns = columns.map(derbyDialect.quoteIdentifier)
+ val oracleColumns = columns.map(oracleDialect.quoteIdentifier)
+ assertResult(Seq("`abc`", "`key`", "`double_quote\"`",
"`back```"))(mySQLColumns)
+ assertResult(Seq("\"abc\"", "\"key\"", "\"double_quote\"\"\"",
"\"back`\""))(postgresColumns)
+ assertResult(Seq("\"abc\"", "\"key\"", "\"double_quote\"\"\"",
"\"back`\""))(derbyColumns)
+ assertResult(Seq("\"abc\"", "\"key\"", "\"double_quote\"\"\"",
"\"back`\""))(oracleColumns)
}
test("compile filters") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
