This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new f21fdf4d4172 [SPARK-59118][4.X][SQL] Warn when doAs is enabled but not
enforced
f21fdf4d4172 is described below
commit f21fdf4d41727b170f5e1af7f5385f427fdbfa52
Author: Holden Karau <[email protected]>
AuthorDate: Tue Oct 27 14:04:15 2026 +0000
[SPARK-59118][4.X][SQL] Warn when doAs is enabled but not enforced
### What changes were proposed in this pull request?
This is a softer follow on of https://github.com/apache/spark/pull/58414
for 4.X branch which warns when doAs is enabled but not enforced. We should log
rather than fail a previously running job.
### Why are the changes needed?
People might expect doAs to run as a different user, which is not supported.
### Does this PR introduce _any_ user-facing change?
The Spark Thrift Server logs a startup warning when
`hive.server2.enable.doAs` is enabled with an authenticating auth type, unless
the new static conf `spark.sql.hive.thriftServer.allowIneffectiveDoAs` is set
to `true`.
### How was this patch tested?
New unit test.
### Was this patch authored or co-authored using generative AI tooling?
Yes, claude
Closes #58438 from holdenk/SPARK-59118-doAs-warn-4.x-r2.
Lead-authored-by: Holden Karau <[email protected]>
Co-authored-by: Holden Karau <[email protected]>
Co-authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
docs/sql-migration-guide.md | 1 +
.../apache/spark/sql/internal/StaticSQLConf.scala | 17 ++++
.../sql/hive/thriftserver/HiveThriftServer2.scala | 47 ++++++++++
.../thriftserver/HiveThriftServer2DoAsSuite.scala | 104 +++++++++++++++++++++
.../SparkExecuteStatementOperationSuite.scala | 24 +++++
5 files changed, 193 insertions(+)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index d228ba14050b..647151f9ba57 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -57,6 +57,7 @@ license: |
- Since Spark 4.4, with whole-stage codegen enabled, a filter condition,
including the `FILTER` clause of an aggregate function, evaluates the remaining
top-level conjuncts of the condition on a row after one of them evaluates to
`NULL`, as the SQL standard requires, and does so in the order they appear in
the condition, as Spark already did with whole-stage codegen disabled, which
also evaluates a subexpression shared by several conjuncts before the
condition. Only a `FALSE` conjunct st [...]
- Since Spark 4.4, when a format is specified, `to_timestamp`,
`try_to_timestamp`, and `to_timestamp_ltz` cast a numeric first argument to a
string and parse it with the format, for example `to_timestamp(20201231,
'yyyyMMdd')` returns `2020-12-31 00:00:00`. This matches the existing behavior
when `spark.sql.timestampType` is `TIMESTAMP_NTZ`. Previously, with the default
`TIMESTAMP_LTZ` timestamp type, such calls failed with `INTERNAL_ERROR`. Note
that the string form of the number is par [...]
+- Since Spark 4.4, the Spark Thrift Server logs a warning at startup when
`hive.server2.enable.doAs` is `true` and `hive.server2.authentication` verifies
user identities (any recognized type other than `NONE`/`NOSASL`), because
executor-side data access still runs as the server's own service identity
rather than the connecting user (SPARK-5159). Set
`spark.sql.hive.thriftServer.allowIneffectiveDoAs` to `true` to acknowledge the
limitation and silence the warning. Spark 5.0 is expected to [...]
## Upgrading from Spark SQL 4.2 to 4.3
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
index d00036b131d7..24b5647ca187 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
@@ -147,6 +147,23 @@ object StaticSQLConf {
.booleanConf
.createWithDefault(true)
+ val HIVE_THRIFT_SERVER_ALLOW_INEFFECTIVE_DOAS =
+ buildStaticConf("spark.sql.hive.thriftServer.allowIneffectiveDoAs")
+ .doc("With hive.server2.enable.doAs=true the Spark Thrift Server
impersonates the " +
+ "connecting user on the driver, but executor-side data access still
runs as the " +
+ "server's own service identity (SPARK-5159), so storage-level
permissions are " +
+ "checked against the privileged service principal instead of the
impersonated " +
+ "user. Because that can silently grant users access to data they could
not read " +
+ "with their own credentials, the server warns at startup when " +
+ "hive.server2.enable.doAs=true and hive.server2.authentication
verifies user " +
+ "identities (any recognized type other than NONE/NOSASL), unless this
option is " +
+ "set to true to acknowledge the limitation. Spark 5.0 is expected to
refuse to " +
+ "start on that configuration unless this option is set.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
val SPARK_SESSION_EXTENSIONS = buildStaticConf("spark.sql.extensions")
.doc("A comma-separated list of classes that implement " +
"Function1[SparkSessionExtensions, Unit] used to configure Spark Session
extensions. The " +
diff --git
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala
index 3825c6ed750f..ded58feda956 100644
---
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala
+++
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala
@@ -23,12 +23,15 @@ import java.util.concurrent.atomic.AtomicBoolean
import org.apache.hadoop.hive.common.ServerUtils
import org.apache.hadoop.hive.conf.HiveConf
import org.apache.hadoop.hive.conf.HiveConf.ConfVars
+import org.apache.hive.service.auth.HiveAuthFactory.AuthTypes
import org.apache.hive.service.cli.thrift.{ThriftBinaryCLIService,
ThriftHttpCLIService}
import org.apache.hive.service.server.HiveServer2
import org.apache.spark.SparkContext
import org.apache.spark.annotation.{DeveloperApi, Since}
+import org.apache.spark.internal.LogEntry
import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys.{CONFIG, CONFIG2}
import org.apache.spark.internal.config.UI.UI_ENABLED
import org.apache.spark.sql.{SparkSession, SQLContext}
import org.apache.spark.sql.hive.HiveUtils
@@ -105,6 +108,48 @@ object HiveThriftServer2 extends Logging {
}
}
+ // Sessions are impersonated on the driver (metastore calls, driver-side
file system access),
+ // but executor-side data access runs as the service identity (SPARK-5159),
so storage ACLs
+ // are checked against the wrong principal. NONE/NOSASL are exempt: they
establish no verified
+ // identity (the client picks any username it likes, and doAs then
impersonates that
+ // unverified name for metastore calls), so there is no security boundary to
warn about --
+ // and Hive's out-of-the-box config (auth NONE, doAs true) must stay quiet.
Unrecognized auth
+ // types are left alone too: HiveAuthFactory rejects them with its own
"Unsupported
+ // authentication type" error, which points at the actual problem.
+ private[thriftserver] def ineffectiveDoAsWarning(
+ hiveConf: HiveConf,
+ allowIneffectiveDoAs: Boolean): Option[LogEntry] = {
+ val authType = hiveConf.getVar(ConfVars.HIVE_SERVER2_AUTHENTICATION)
+ val verifyingAuthTypes =
+ AuthTypes.values().filterNot(Set(AuthTypes.NONE,
AuthTypes.NOSASL)).map(_.getAuthName)
+ // getVar returns the default (NONE) when unset and "" when set empty,
never null.
+ val authVerifiesUser =
verifyingAuthTypes.exists(_.equalsIgnoreCase(authType))
+ if (authVerifiesUser &&
hiveConf.getBoolVar(ConfVars.HIVE_SERVER2_ENABLE_DOAS) &&
+ !allowIneffectiveDoAs) {
+ Some(log"${MDC(CONFIG, ConfVars.HIVE_SERVER2_ENABLE_DOAS.varname)} is
set to true, " +
+ log"but the Spark Thrift Server impersonates the connecting user only
on the driver, " +
+ log"for Hive metastore calls and driver-side file system access:
executor-side data " +
+ log"access still runs as the server's own service identity, so storage
permissions " +
+ log"are checked against the service principal rather than the
connecting user " +
+ log"(SPARK-5159). That can expose data the connecting user is not
authorized to " +
+ log"read. Setting it to false silences this, but note that it stops
impersonating " +
+ log"metastore calls too, so it is not a no-op (and unsetting it does
not help -- " +
+ log"Hive's own default is true). To acknowledge the limitation and
silence this " +
+ log"instead, set " +
+ log"${MDC(CONFIG2,
StaticSQLConf.HIVE_THRIFT_SERVER_ALLOW_INEFFECTIVE_DOAS.key)} to " +
+ log"true; it is a static conf, so it must be set before the
SparkSession is created. " +
+ log"Spark 5.0 is expected to refuse to start on this configuration
unless it is set.")
+ } else {
+ None
+ }
+ }
+
+ private[thriftserver] def warnIfIneffectiveDoAs(
+ hiveConf: HiveConf,
+ allowIneffectiveDoAs: Boolean): Unit = {
+ ineffectiveDoAsWarning(hiveConf,
allowIneffectiveDoAs).foreach(logWarning(_))
+ }
+
def main(args: Array[String]): Unit = {
// If the arguments contains "-h" or "--help", print out the usage and
exit.
if (args.contains("-h") || args.contains("--help")) {
@@ -154,6 +199,8 @@ private[hive] class HiveThriftServer2(sparkSession:
SparkSession)
private val started = new AtomicBoolean(false)
override def init(hiveConf: HiveConf): Unit = {
+ HiveThriftServer2.warnIfIneffectiveDoAs(
+ hiveConf,
sparkSession.conf.get(StaticSQLConf.HIVE_THRIFT_SERVER_ALLOW_INEFFECTIVE_DOAS))
val sparkSqlCliService = new SparkSQLCLIService(this, sparkSession)
setSuperField(this, "cliService", sparkSqlCliService)
addService(sparkSqlCliService)
diff --git
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2DoAsSuite.scala
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2DoAsSuite.scala
new file mode 100644
index 000000000000..5fd89731f4a6
--- /dev/null
+++
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2DoAsSuite.scala
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.thriftserver
+
+import org.apache.hadoop.hive.conf.HiveConf
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars
+import org.apache.hive.service.auth.HiveAuthFactory.AuthTypes
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.internal.StaticSQLConf
+
+/**
+ * Tests for the SPARK-59118 startup warning: `hive.server2.enable.doAs` is
accepted but
+ * impersonation does not reach executor-side data access. Spark 5.0 is
expected to refuse
+ * to start instead, on the same configuration and with the same opt-out conf.
+ */
+class HiveThriftServer2DoAsSuite extends SparkFunSuite {
+
+ private def hiveConf(authType: String, doAs: Boolean): HiveConf = {
+ val conf = new HiveConf()
+ conf.setVar(ConfVars.HIVE_SERVER2_AUTHENTICATION, authType)
+ conf.setBoolVar(ConfVars.HIVE_SERVER2_ENABLE_DOAS, doAs)
+ conf
+ }
+
+ /**
+ * The warning text (if any) that checking `conf` would log. These tests
assert on the
+ * returned entry rather than captured log output, which is hostage to
JVM-global logging
+ * state when the whole module shares one forked JVM (SERIAL_SBT_TESTS=1 in
CI).
+ */
+ private def doAsWarnings(conf: HiveConf, allowIneffectiveDoAs: Boolean =
false): Seq[String] = {
+ HiveThriftServer2.ineffectiveDoAsWarning(conf,
allowIneffectiveDoAs).map(_.message).toSeq
+ }
+
+ // The auth types that establish a user identity worth impersonating.
+ private val verifyingAuthTypes =
+ AuthTypes.values().filterNot(Set(AuthTypes.NONE,
AuthTypes.NOSASL)).map(_.getAuthName)
+
+ test("SPARK-5159 / SPARK-59118 warn when doAs is enabled but not enforced") {
+ val warnings = doAsWarnings(hiveConf("KERBEROS", doAs = true))
+ assert(warnings.length == 1)
+ assert(warnings.head.contains("SPARK-5159"))
+ // Auth type matching is case-insensitive.
+ assert(doAsWarnings(hiveConf("kerberos", doAs = true)).length == 1)
+ }
+
+ test("SPARK-59118 allowIneffectiveDoAs=true silences an otherwise-warned
config") {
+ assert(doAsWarnings(hiveConf("KERBEROS", doAs = true),
allowIneffectiveDoAs = true).isEmpty)
+ }
+
+ test("SPARK-59118 the warning names the conf that silences it") {
+ val warning = doAsWarnings(hiveConf("KERBEROS", doAs = true)).head
+
assert(warning.contains(StaticSQLConf.HIVE_THRIFT_SERVER_ALLOW_INEFFECTIVE_DOAS.key))
+ }
+
+ test("SPARK-59118 warn for every auth type that verifies the user") {
+ // Guards against a new AuthTypes value silently landing on the quiet side.
+ assert(verifyingAuthTypes.nonEmpty)
+ verifyingAuthTypes.foreach { authType =>
+ assert(doAsWarnings(hiveConf(authType, doAs = true)).length == 1, s"no
warning: $authType")
+ }
+ }
+
+ test("SPARK-59118 does not warn when auth type establishes no user
identity") {
+ Seq("NONE", "NOSASL", "none", "noSasl").foreach { authType =>
+ assert(doAsWarnings(hiveConf(authType, doAs = true)).isEmpty, s"warned:
$authType")
+ }
+ }
+
+ test("SPARK-59118 does not warn when doAs is disabled") {
+ verifyingAuthTypes.foreach { authType =>
+ assert(doAsWarnings(hiveConf(authType, doAs = false)).isEmpty, s"warned:
$authType")
+ }
+ }
+
+ test("SPARK-59118 does not warn on an untouched Hive config") {
+ // Hive defaults doAs to true and auth to NONE, so a stock config must
stay quiet.
+ assert(doAsWarnings(new HiveConf()).isEmpty)
+ }
+
+ test("SPARK-59118 an unrecognized or empty auth type is left to Hive's own
error") {
+ // HiveAuthFactory rejects these with "Unsupported authentication type",
where the
+ // warning's advice would not help. getVar returns "" (never null) for an
explicitly
+ // empty value, so this must not NPE.
+ Seq("NOT_AN_AUTH_TYPE", "").foreach { authType =>
+ assert(doAsWarnings(hiveConf(authType, doAs = true)).isEmpty, s"warned:
$authType")
+ }
+ }
+}
diff --git
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperationSuite.scala
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperationSuite.scala
index ddd7f19ef238..6e62395cedc4 100644
---
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperationSuite.scala
+++
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperationSuite.scala
@@ -23,9 +23,11 @@ import java.util.concurrent.Semaphore
import scala.concurrent.duration._
import org.apache.hadoop.hive.conf.HiveConf
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars
import org.apache.hive.service.cli.OperationState
import org.apache.hive.service.cli.session.{HiveSession, HiveSessionImpl}
import org.apache.hive.service.rpc.thrift.{TProtocolVersion, TTypeId}
+import org.apache.logging.log4j.Level
import org.mockito.Mockito.{doReturn, mock, spy, when, RETURNS_DEEP_STUBS}
import org.mockito.invocation.InvocationOnMock
@@ -135,6 +137,28 @@ class SparkExecuteStatementOperationSuite extends
SharedSparkSession {
}
}
+ test("SPARK-59118 HiveThriftServer2.init warns when doAs is not enforced") {
+ // The warning is only worth anything if init() actually calls it.
+ val hiveConf = new HiveConf()
+ hiveConf.setVar(ConfVars.HIVE_SERVER2_AUTHENTICATION, "KERBEROS")
+ hiveConf.setBoolVar(ConfVars.HIVE_SERVER2_ENABLE_DOAS, true)
+ // init() boots Hive's service graph; operation logging attaches an
appender to the root
+ // logger that nothing ever removes, so keep it off rather than leak it
into this JVM.
+ hiveConf.setBoolVar(ConfVars.HIVE_SERVER2_LOGGING_OPERATION_ENABLED, false)
+ // Attach to the root logger, not the HiveThriftServer2 logger by name:
withLogAppender on
+ // a named logger leaves behind a non-additive LoggerConfig (log4j2 copies
root's additivity)
+ // that swallows that logger's output for every later suite in this JVM.
+ val appender = new LogAppender("doAs impersonation warning")
+ withLogAppender(appender, level = Some(Level.WARN)) {
+ new HiveThriftServer2(spark).init(hiveConf)
+ }
+ val warnings = appender.loggingEvents.map(_.getMessage.getFormattedMessage)
+ assert(warnings.exists { warning =>
+ warning.contains(ConfVars.HIVE_SERVER2_ENABLE_DOAS.varname) &&
+ warning.contains("SPARK-5159")
+ })
+ }
+
private class MySparkExecuteStatementOperation(
session: SparkSession,
hiveSession: HiveSession,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
