LuciferYang commented on code in PR #12367: URL: https://github.com/apache/gluten/pull/12367#discussion_r3499917851
########## backends-velox/src/test/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApiFsConfSuite.scala: ########## @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.gluten.backendsapi.velox + +import org.apache.gluten.execution.{GlutenPartition, WholeStageTransformContext} +import org.apache.gluten.substrait.plan.PlanBuilder + +import org.apache.spark.sql.test.SharedSparkSession + +/** + * Tests that [[VeloxIteratorApi.genPartitions]] captures fs.azure.*, fs.s3a.*, and fs.gs.* keys + * from the driver-side Hadoop configuration and embeds them in [[GlutenPartition.fsConf]], so they + * are available on executors where Spark's SQLConf propagation does not reach "fs.*" keys. + * + * Keys must be set on `sparkContext.hadoopConfiguration` (the mutable base configuration) because + * `sessionState.newHadoopConf()` creates a fresh copy each time - mutations to its return value are + * discarded before the next call. + */ +class VeloxIteratorApiFsConfSuite extends SharedSparkSession { + + private val api = new VeloxIteratorApi + + /** + * Build a minimal WholeStageTransformContext backed by an empty Substrait plan. genPartitions + * only calls wsCtx.root.toProtobuf.toByteArray, so a plan with no relations is sufficient for the + * purpose of this test. + */ + private def emptyWsCtx: WholeStageTransformContext = + WholeStageTransformContext(PlanBuilder.empty()) + + /** + * Set Hadoop conf keys on the underlying mutable configuration and restore their previous values + * (or unset them) after the block. `sessionState.newHadoopConf()` copies from + * `sparkContext.hadoopConfiguration`, so this is the correct mutation point. + */ + private def withHadoopConf(pairs: (String, String)*)(body: => Unit): Unit = { + // scalastyle:off hadoopconfiguration + val hadoopConf = spark.sparkContext.hadoopConfiguration + // scalastyle:on hadoopconfiguration + val prev: Seq[(String, Option[String])] = pairs.map { + case (k, _) => k -> Option(hadoopConf.get(k)) + } + pairs.foreach { case (k, v) => hadoopConf.set(k, v) } + try body + finally prev.foreach { + case (k, Some(old)) => hadoopConf.set(k, old) + case (k, None) => hadoopConf.unset(k) + } + } + + test("genPartitions embeds fs.azure.* keys from Hadoop conf into GlutenPartition.fsConf") { + withHadoopConf( + "fs.azure.account.auth.type.myaccount.dfs.core.windows.net" -> "OAuth", + "fs.azure.account.oauth.provider.type" -> "ClientCredentials" + ) { + val partitions = api.genPartitions(emptyWsCtx, Seq(Seq.empty), Seq.empty) + assert(partitions.size == 1) + val fsConf = partitions.head.asInstanceOf[GlutenPartition].fsConf + assert( + fsConf.contains("fs.azure.account.auth.type.myaccount.dfs.core.windows.net"), + s"Expected fs.azure key not found; got: ${fsConf.keys.mkString(", ")}") + assert(fsConf("fs.azure.account.auth.type.myaccount.dfs.core.windows.net") == "OAuth") + assert( + fsConf.contains("fs.azure.account.oauth.provider.type"), + s"Expected fs.azure key not found; got: ${fsConf.keys.mkString(", ")}") + assert(fsConf("fs.azure.account.oauth.provider.type") == "ClientCredentials") + } + } + + test("genPartitions embeds fs.s3a.* keys from Hadoop conf into GlutenPartition.fsConf") { + withHadoopConf( + "fs.s3a.access.key" -> "AKIAIOSFODNN7EXAMPLE", Review Comment: `AKIAIOSFODNN7EXAMPLE` is AWS's documentation example and matches the AKIA/ASIA pattern used by GitHub secret scanning, gitleaks, and most fork‑level scanners. Even as a test fixture, this routinely fails CI in downstream forks and surfaces as false‑positive credential alerts. Use clearly synthetic placeholders, e.g. `"dummy-access-key"` / `"dummy-secret-value"`. ########## backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala: ########## @@ -125,12 +125,40 @@ class VeloxIteratorApi extends IteratorApi with Logging { // Only serialize plan once, save lots time when plan is complex. val planByteArray = wsCtx.root.toProtobuf.toByteArray + // Capture fs.azure.* / fs.s3a.* / fs.gs.* keys from the driver-side + // Hadoop configuration NOW, while we are still on the driver, and embed + // them in every GlutenPartition. These keys are set by the user via + // spark.conf.set("fs.azure.account.auth.type", ...) or Review Comment: Two things： 1. The comment block at lines 128–144 still advertises `spark.conf.set(...)`, `DataFrameReader.option(...)`, and `sessionState.newHadoopConf()` as user surfaces — none of those reach the new code path, which only reads `SparkContext.getOrCreate().hadoopConfiguration`. The actual recipes that work are submit‑time `--conf spark.hadoop.fs.*=…` (which Spark copies into `SparkContext.hadoopConfiguration` at boot) or runtime `sparkContext.hadoopConfiguration.set("fs.*", ...)`. `spark.conf.set("spark.hadoop.fs.*", ...)` set at runtime writes to SQLConf only and is invisible to the new code. 2. The paragraph starting "Capture fs.azure.* …" is duplicated at line 137 — looks like a merge artefact. Please collapse into one paragraph and rewrite the recipe. ########## backends-velox/src/test/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApiFsConfSuite.scala: ########## @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.gluten.backendsapi.velox + +import org.apache.gluten.execution.{GlutenPartition, WholeStageTransformContext} +import org.apache.gluten.substrait.plan.PlanBuilder + +import org.apache.spark.sql.test.SharedSparkSession + +/** + * Tests that [[VeloxIteratorApi.genPartitions]] captures fs.azure.*, fs.s3a.*, and fs.gs.* keys + * from the driver-side Hadoop configuration and embeds them in [[GlutenPartition.fsConf]], so they + * are available on executors where Spark's SQLConf propagation does not reach "fs.*" keys. + * + * Keys must be set on `sparkContext.hadoopConfiguration` (the mutable base configuration) because + * `sessionState.newHadoopConf()` creates a fresh copy each time - mutations to its return value are + * discarded before the next call. + */ +class VeloxIteratorApiFsConfSuite extends SharedSparkSession { + + private val api = new VeloxIteratorApi + + /** + * Build a minimal WholeStageTransformContext backed by an empty Substrait plan. genPartitions + * only calls wsCtx.root.toProtobuf.toByteArray, so a plan with no relations is sufficient for the + * purpose of this test. + */ + private def emptyWsCtx: WholeStageTransformContext = + WholeStageTransformContext(PlanBuilder.empty()) Review Comment: `emptyWsCtx` only works because `genPartitions` happens to touch only `wsCtx.root.toProtobuf.toByteArray`, but `WholeStageTransformContext` has other fields — a future shape change silently breaks the helper. Either refactor `genPartitions` to take `Array[Byte]` directly (cleanest), or add a one‑line comment recording the implicit dependency. Non‑blocking. ########## gluten-arrow/src/main/scala/org/apache/gluten/runtime/Runtimes.scala: ########## @@ -18,10 +18,48 @@ package org.apache.gluten.runtime import org.apache.spark.task.TaskResources +import java.security.MessageDigest import java.util object Runtimes { + /** + * Produce a stable, value-free cache key for a (backendName, name, extraConf) triple. + * + * Two problems with the old `s"$backendName:$name:$extraConf"` key: + * + * 1. **Credential leakage** – `Map.toString` embeds secret values (e.g. `fs.s3a.secret.key`, + * `fs.azure.account.oauth2.client.secret`) in a plain heap string that can appear in logs, + * thread dumps, and heap snapshots. + * 2. **Nondeterminism** – Scala `Map.toString` does not guarantee insertion order, so two maps + * with identical entries can produce different strings, causing spurious duplicate + * `VeloxRuntime` registrations within a task. + * + * Fix: sort keys, hash them with SHA-256, and use only the hex digest as the key. Values are + * intentionally excluded from the digest – distinct configs (different credentials for the same + * key set) that need separate runtimes are already separated at the task level through + * `GlutenPartition.fsConf`. Within a single task the key set is stable, so the digest is stable. + */ + private def stableKey( + backendName: String, + name: String, + extraConf: util.Map[String, String]): String = { + val digest = MessageDigest.getInstance("SHA-256") + digest.update(backendName.getBytes("UTF-8")) + digest.update(0.toByte) // field separator + digest.update(name.getBytes("UTF-8")) + digest.update(0.toByte) + // Sort keys for determinism; hash only keys, not values, to avoid leaking secrets. + val sortedKeys = new java.util.ArrayList(extraConf.keySet) + java.util.Collections.sort(sortedKeys) + sortedKeys.forEach { + k => + digest.update(k.getBytes("UTF-8")) + digest.update(0.toByte) + } + digest.digest().map("%02x".format(_)).mkString Review Comment: **CWE‑694**：`stableKey` hashes only the keys. Two `Runtimes.contextInstance` calls inside the same task that share a keyset but differ in values (e.g. `IcebergWrite#write` with different `icebergProperties`) will collide on the same `TaskResources` registration, and the second caller silently gets the runtime built from the first. Please hash sorted `(key, value)` pairs — values are still hidden from logs because only the digest escapes, and the existing key‑sort already makes the digest deterministic. ########## backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala: ########## @@ -125,12 +125,40 @@ class VeloxIteratorApi extends IteratorApi with Logging { // Only serialize plan once, save lots time when plan is complex. val planByteArray = wsCtx.root.toProtobuf.toByteArray + // Capture fs.azure.* / fs.s3a.* / fs.gs.* keys from the driver-side + // Hadoop configuration NOW, while we are still on the driver, and embed + // them in every GlutenPartition. These keys are set by the user via + // spark.conf.set("fs.azure.account.auth.type", ...) or + // sparkContext.hadoopConfiguration.set(...) + // Spark's withSQLConfPropagated only forwards keys starting with "spark" + // as task-local-properties, so "fs.*" keys never reach the executor's + // SQLConf. Serialising them inside the partition is the only safe way + // to make them available to the native runtime on the executor. + // Capture fs.azure.* / fs.s3a.* / fs.gs.* keys while on the driver. + // SparkPlan.sqlContext is available on the driver -- using the first leaf + // gives us access to sessionState.newHadoopConf() which includes all keys + // set via spark.conf.set(), sparkContext.hadoopConfiguration, and + // DataFrameReader.option(). These are NOT propagated to executors by + // Spark's withSQLConfPropagated (it only forwards keys starting with + // "spark"), so embedding them in the serialised GlutenPartition is the + // only reliable transport mechanism. + val fsPrefixes = Seq("fs.azure.", "fs.s3a.", "fs.gs.") + // scalastyle:off hadoopconfiguration + val baseHadoopConf = org.apache.spark.SparkContext.getOrCreate().hadoopConfiguration Review Comment: `org.apache.spark.SparkContext.getOrCreate()` is spelled out with full FQN; `SparkContext` is already imported and used by short name elsewhere in this file. Move the short name in and drop the inline FQN. On the other hand, The simplified capture reads `SparkContext.getOrCreate().hadoopConfiguration` directly, which only contains keys present at SparkContext construction (from `SparkConf` `spark.hadoop.*`) plus anything mutated via `sparkContext.hadoopConfiguration.set(...)`. It does **not** see per‑session overrides made via `spark.conf.set("spark.hadoop.fs.*", ...)` at runtime, nor session‑scoped `DataFrameReader.option(...)` keys — both of which live on `SparkSession.sessionState.newHadoopConf()`. If the original use case in issue #10113 includes runtime session‑level credential injection, the current code path won't satisfy it. Either document the supported recipes explicitly (boot‑time `--conf spark.hadoop.fs.*` or runtime `sparkContext.hadoopConfiguration.set`) or read from `leaves.head.sqlContext.sessionState.newHadoopConf()` when leaves are available, falling back to `SparkContext.hadoopConfiguration` only when they aren't. ########## backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala: ########## @@ -125,12 +125,40 @@ class VeloxIteratorApi extends IteratorApi with Logging { // Only serialize plan once, save lots time when plan is complex. val planByteArray = wsCtx.root.toProtobuf.toByteArray + // Capture fs.azure.* / fs.s3a.* / fs.gs.* keys from the driver-side + // Hadoop configuration NOW, while we are still on the driver, and embed + // them in every GlutenPartition. These keys are set by the user via + // spark.conf.set("fs.azure.account.auth.type", ...) or + // sparkContext.hadoopConfiguration.set(...) + // Spark's withSQLConfPropagated only forwards keys starting with "spark" + // as task-local-properties, so "fs.*" keys never reach the executor's + // SQLConf. Serialising them inside the partition is the only safe way + // to make them available to the native runtime on the executor. + // Capture fs.azure.* / fs.s3a.* / fs.gs.* keys while on the driver. + // SparkPlan.sqlContext is available on the driver -- using the first leaf + // gives us access to sessionState.newHadoopConf() which includes all keys + // set via spark.conf.set(), sparkContext.hadoopConfiguration, and + // DataFrameReader.option(). These are NOT propagated to executors by + // Spark's withSQLConfPropagated (it only forwards keys starting with + // "spark"), so embedding them in the serialised GlutenPartition is the + // only reliable transport mechanism. + val fsPrefixes = Seq("fs.azure.", "fs.s3a.", "fs.gs.") + // scalastyle:off hadoopconfiguration + val baseHadoopConf = org.apache.spark.SparkContext.getOrCreate().hadoopConfiguration + // scalastyle:on hadoopconfiguration + val fsConf: Map[String, String] = fsPrefixes.flatMap { + prefix => + baseHadoopConf.getPropsWithPrefix(prefix).asScala + .map { case (suffix, v) => (prefix + suffix) -> v } + }.toMap + splitInfos.zipWithIndex.map { case (splitInfos, index) => GlutenPartition( index, planByteArray, - splitInfos.toArray + splitInfos.toArray, + fsConf = fsConf Review Comment: Assigning `fsConf = fsConf` puts an independent serialised copy of the map on every `GlutenPartition` — reference sharing is driver‑side only, the wire cost scales with partition count. ########## gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala: ########## @@ -37,8 +37,13 @@ case class GlutenPartition( index: Int, plan: Array[Byte], splitInfos: Array[SplitInfo] = Array.empty[SplitInfo], - files: Array[String] = - Array.empty[String] // touched files, for implementing UDF input_file_name + files: Array[String] = Array.empty[String], // touched files, for UDF input_file_name + // fs.azure.* / fs.s3a.* / fs.gs.* keys captured on the driver from + // sessionState.newHadoopConf() and serialised here so they survive the + // RDD partition boundary. Spark's withSQLConfPropagated only propagates + // keys that start with "spark", so these keys are otherwise invisible on + // the executor side (the executor's SQLConf never sees them). + fsConf: Map[String, String] = Map.empty Review Comment: **CWE‑532 / CWE‑200**：Case‑class auto‑`toString` prints every field, so any log line or exception that stringifies a partition (or the wrapping `FirstZippedPartitionsPartition`) will dump `fs.s3a.secret.key=…` verbatim. Please override `toString` to print fsConf keys only — or redact values on the driver side at capture time using `SECRET_REDACTION_PATTERN` (`spark.redaction.regex`). Note `SQLConf.get.stringRedactionPattern` is a different config (default `None`) and won't help here, and `SQLConf.get` itself isn't safe to call off‑task. Please also add a regression test that asserts `toString` does not contain a seeded secret value. I notice Copilot already flagged this security‑related risk earlier. However, the latest commit still hasn’t addressed the issue. ########## backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala: ########## @@ -199,7 +227,16 @@ class VeloxIteratorApi extends IteratorApi with Logging { iter => new ColumnarBatchInIterator(BackendsApiManager.getBackendName, iter.asJava) } - val extraConf = Map(GlutenConfig.COLUMNAR_CUDF_ENABLED.key -> enableCudf.toString).asJava + // Merge the fs.* keys captured on the driver (stored in GlutenPartition.fsConf) + // into the extraConf passed to NativePlanEvaluator / VeloxRuntime. + // Runtimes.contextInstance() will call GlutenConfig.getNativeSessionConf() which + // merges extraConf on top of SQLConf.get.getAllConfs. Because the executor-side + // SQLConf never receives "fs.*" keys (Spark only propagates "spark.*" keys via + // task local properties), this is the only path these credentials can take to + // reach the native session config and ultimately the Velox ABFS connector. + val partitionFsConf = inputPartition.asInstanceOf[GlutenPartition].fsConf + val extraConf = (partitionFsConf + Review Comment: **CWE‑532:** This line is where `fsConf` reaches `extraConf` and ultimately flows into the native session config printed by `printConfig` (cpp side). `printConfig` redacts on key match against `spark.redaction.regex`, default `(?i)secret|password|token|access[.]?key`. That default misses `fs.azure.account.key.*` (the Azure storage account key — the *primary* credential), `fs.s3a.encryption.key` and the deprecated alias `fs.s3a.server-side-encryption.key`, `fs.gs.auth.service.account.private.key`, and `fs.gs.auth.service.account.json.keyfile`. With `spark.gluten.sql.debug=true` these reach executor stdout in cleartext. The fix is on the C++ side: please hard‑redact the credential‑bearing prefixes inside `printConfig` *in addition to* applying the operator regex — minimum list: `fs.azure.account.key.`, `fs.azure.sas.`, `fs.azure.account.oauth2.`, `fs.s3a.access.key`, `fs.s3a.secret.key`, `fs.s3a.session.token`, `fs.s3a.encryption.key`, `fs.s3a.server-side-encryption.key`, `fs.gs.auth.service.account.private.key`, `fs.gs.auth.service.account.json.keyfile`. Consider also extending Gluten's default `spark.redaction.regex` to cover them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
