This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 245179e93641 [SPARK-57580][SQL] Redact credentials embedded in JDBC
URLs
245179e93641 is described below
commit 245179e936419c099a7382f3b615ca8a94111d76
Author: Wenchen Fan <[email protected]>
AuthorDate: Sun Jun 21 13:40:06 2026 -0700
[SPARK-57580][SQL] Redact credentials embedded in JDBC URLs
### What changes were proposed in this pull request?
`JDBCOptions.getRedactUrl()` is used to surface the JDBC URL in logs and in
the
`FAILED_JDBC.*` error messages (e.g. `FAILED_JDBC.CONNECTION`). Today it is
implemented as:
```scala
def getRedactUrl(): String =
Utils.redact(SQLConf.get.stringRedactionPattern, url)
```
`SQLConf.get.stringRedactionPattern` (`spark.sql.redaction.string.regex`)
is **unset by
default**, so `Utils.redact(None, url)` returns the URL unchanged. JDBC
URLs routinely embed
credentials — either as userinfo in the authority
(`jdbc:mysql://user:passwordhost/db`), as
connection properties (`...?password=secret`, `...;token=abc`), or in
driver-specific inline
forms (Oracle Thin's `jdbc:oracle:thin:user/passwordhost`) — so by default
those secrets are
printed verbatim in error messages and logs.
This PR makes `getRedactUrl()` redact credentials unconditionally,
independent of the optional
`spark.sql.redaction.string.regex`. A JDBC URL has the form
`jdbc:<subprotocol>:<subname>`, where
`<subprotocol>` is a registered driver name (`mysql`, `oracle`,
`postgresql`, …) and `<subname>`
is entirely driver-specific. Credentials can appear anywhere in `<subname>`
and in arbitrary,
open-ended syntaxes, so rather than enumerate every driver's syntax (and
inevitably miss one and
leak), `JDBCOptions.redactUrl(url, regex)` takes an **allowlist** approach:
it keeps only the
`jdbc:<subprotocol>:` prefix — which is credential-free by construction —
and redacts everything
after it.
This is deliberately stronger than trying to preserve the
host/port/database: the set of
credential-bearing positions and property names is driver-specific and
unbounded, so any attempt
to match-and-strip them is a never-ending game of catch-up. Keeping only
the subprotocol is safe
regardless of the driver's URL grammar, while still preserving the most
useful triage signal
(which database engine the failure came from). The user-configured `regex`
is still applied on
top, preserving existing behavior.
### Why are the changes needed?
`FAILED_JDBC.*` errors and connection logs can currently leak JDBC
credentials to anyone who can
read query error messages or driver logs, because the default redaction is
a no-op. Redacting at
the single `getRedactUrl()` chokepoint fixes every `FAILED_JDBC.*` call
site at once.
### Does this PR introduce _any_ user-facing change?
Yes. `FAILED_JDBC.*` error messages (and any log line built from
`getRedactUrl()`) now show only
the `jdbc:<subprotocol>:` prefix followed by `*********(redacted)`, instead
of the full URL. For
example `jdbc:mysql://user:secrethost:3306/db?password=p` becomes
`jdbc:mysql:*********(redacted)`.
### How was this patch tested?
Unit test `redactUrl keeps only the jdbc:<subprotocol>: prefix` in
`JdbcUtilsSuite`, covering:
the `//user:pwdhost` authority form, Oracle Thin's `user/pwdhost` inline
form (with and without
a trailing `//host` and query string), `?`- and `;`-delimited connection
properties, credential-
free URLs (still reduced to the prefix), a malformed URL with no second
colon (redacted
wholesale), the user-configured `regex` being applied on top of the kept
prefix, and null/empty
inputs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
Closes #56630 from cloud-fan/SPARK-55706-followup-redact-jdbc-url.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 6588f6d14692515216687b132eed0a9ae1b552dd)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../execution/datasources/jdbc/JDBCOptions.scala | 35 +++++++++++++++++-
.../datasources/jdbc/JdbcUtilsSuite.scala | 41 ++++++++++++++++++++++
.../org/apache/spark/sql/jdbc/JDBCSuite.scala | 6 ++--
3 files changed, 79 insertions(+), 3 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
index 3c85b6e65dee..7188c2b8b2e8 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
@@ -20,6 +20,8 @@ package org.apache.spark.sql.execution.datasources.jdbc
import java.sql.{Connection, DriverManager}
import java.util.{Locale, Properties}
+import scala.util.matching.Regex
+
import org.apache.commons.io.FilenameUtils
import org.apache.spark.SparkFiles
@@ -268,7 +270,7 @@ class JDBCOptions(
case _ => false
}
- def getRedactUrl(): String =
Utils.redact(SQLConf.get.stringRedactionPattern, url)
+ def getRedactUrl(): String = JDBCOptions.redactUrl(url,
SQLConf.get.stringRedactionPattern)
}
class JdbcOptionsInWrite(
@@ -302,6 +304,37 @@ object JDBCOptions {
name
}
+ /**
+ * Redacts a JDBC URL so it is safe to surface in logs and error messages.
+ *
+ * A JDBC URL has the form `jdbc:<subprotocol>:<subname>`, where
`<subprotocol>` is a registered
+ * driver name (`mysql`, `oracle`, `postgresql`, ...) and `<subname>` is
entirely driver-specific.
+ * Credentials can appear anywhere in `<subname>` and in arbitrary syntaxes
-- userinfo in a
+ * `//user:pwd@host` authority, Oracle Thin's `user/pwd@host`, `?`/`;`
connection properties, etc.
+ * Rather than enumerate every driver's syntax (and inevitably miss one and
leak), we keep only
+ * the `jdbc:<subprotocol>:` prefix -- which is credential-free by
construction -- and redact
+ * everything after it. The driver type stays visible for debugging; nothing
else does.
+ *
+ * This redaction is unconditional, unlike the optional, user-configured
+ * `spark.sql.redaction.string.regex` (which is unset by default and would
leave the URL in the
+ * clear). The configured `regex` is still applied on top, preserving
existing behavior.
+ */
+ def redactUrl(url: String, regex: Option[Regex]): String = {
+ if (url == null || url.isEmpty) {
+ url
+ } else {
+ // The second colon terminates the subprotocol: "jdbc" ':'
"<subprotocol>" ':' "<subname>".
+ val subprotocolEnd = url.indexOf(':', url.indexOf(':') + 1)
+ val redacted = if (subprotocolEnd < 0) {
+ // No subname delimiter -- the URL is malformed, so don't trust any of
it.
+ Utils.REDACTION_REPLACEMENT_TEXT
+ } else {
+ url.substring(0, subprotocolEnd + 1) + Utils.REDACTION_REPLACEMENT_TEXT
+ }
+ Utils.redact(regex, redacted)
+ }
+ }
+
val JDBC_URL = newOption("url")
val JDBC_TABLE_NAME = newOption("dbtable")
val JDBC_QUERY_STRING = newOption("query")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtilsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtilsSuite.scala
index d3723881bfa2..0f03f06b7425 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtilsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtilsSuite.scala
@@ -21,6 +21,7 @@ import org.apache.spark.SparkFunSuite
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.parser.ParseException
import org.apache.spark.sql.types._
+import org.apache.spark.util.Utils
class JdbcUtilsSuite extends SparkFunSuite {
@@ -69,4 +70,44 @@ class JdbcUtilsSuite extends SparkFunSuite {
condition = "PARSE_SYNTAX_ERROR",
parameters = Map("error" -> "'.'", "hint" -> ""))
}
+
+ test("redactUrl keeps only the jdbc:<subprotocol>: prefix") {
+ val redaction = Utils.REDACTION_REPLACEMENT_TEXT
+
+ // Only the "jdbc:<subprotocol>:" prefix is kept; everything after it
(host, port, database,
+ // userinfo and connection properties) is redacted, regardless of the
driver-specific syntax.
+ // This covers credentials embedded as a "//user:pwd@host" authority, ...
+ assert(JDBCOptions.redactUrl("jdbc:mysql://user:secret@host:3306/db",
None) ===
+ s"jdbc:mysql:$redaction")
+
assert(JDBCOptions.redactUrl("jdbc:mysql://user:p@ss@host:3306/db?password=other",
None) ===
+ s"jdbc:mysql:$redaction")
+ // ... as Oracle Thin's "user/pwd@host" form (no "//" authority), ...
+ assert(JDBCOptions.redactUrl("jdbc:oracle:thin:scott/tiger@host:1521/svc",
None) ===
+ s"jdbc:oracle:$redaction")
+
assert(JDBCOptions.redactUrl("jdbc:oracle:thin:scott/tiger@//host:1521/svc?x=1",
None) ===
+ s"jdbc:oracle:$redaction")
+ // ... and as "?"- or ";"-delimited connection properties.
+ assert(JDBCOptions.redactUrl(
+ "jdbc:postgresql://host/db?user=alice&password=secret", None) ===
+ s"jdbc:postgresql:$redaction")
+ assert(JDBCOptions.redactUrl(
+ "jdbc:sqlserver://localhost:1433;databaseName=testdb;password=secret",
None) ===
+ s"jdbc:sqlserver:$redaction")
+
+ // Even URLs that carry no credentials are reduced to the prefix --
nothing past the
+ // subprotocol is assumed safe.
+ assert(JDBCOptions.redactUrl("jdbc:mysql://localhost/db", None) ===
s"jdbc:mysql:$redaction")
+ assert(JDBCOptions.redactUrl("jdbc:h2:mem:testdb", None) ===
s"jdbc:h2:$redaction")
+
+ // A URL with no subname delimiter (no second colon) is redacted wholesale.
+ assert(JDBCOptions.redactUrl("jdbc:weird-url", None) === redaction)
+
+ // The user-configured regex is still applied on top of the kept prefix.
+ assert(JDBCOptions.redactUrl("jdbc:mysql://host/db", Some("mysql".r)) ===
+ s"jdbc:$redaction:$redaction")
+
+ // Null and empty inputs are passed through.
+ assert(JDBCOptions.redactUrl(null, None) === null)
+ assert(JDBCOptions.redactUrl("", None) === "")
+ }
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
index 889b2faa3e5e..29e3e0e6fb68 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
@@ -2143,7 +2143,8 @@ class JDBCSuite extends SharedSparkSession {
spark.read.format("jdbc").options(opts).load()
},
condition = "FAILED_JDBC.CONNECTION",
- parameters = Map("url" -> url)
+ // getRedactUrl() keeps only the "jdbc:<subprotocol>:" prefix and
redacts the rest.
+ parameters = Map("url" ->
s"jdbc:mysql:${Utils.REDACTION_REPLACEMENT_TEXT}")
)
}
@@ -2496,7 +2497,8 @@ class JDBCSuite extends SharedSparkSession {
}
},
condition = "FAILED_JDBC.CONNECTION",
- parameters = Map("url" -> url)
+ // getRedactUrl() keeps only the "jdbc:<subprotocol>:" prefix and
redacts the rest.
+ parameters = Map("url" ->
s"$connectionUrl:${Utils.REDACTION_REPLACEMENT_TEXT}")
)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
