zhouyuan commented on code in PR #12367:
URL: https://github.com/apache/gluten/pull/12367#discussion_r3492625807
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala:
##########
@@ -125,12 +125,41 @@ class VeloxIteratorApi extends IteratorApi with Logging {
// Only serialize plan once, save lots time when plan is complex.
val planByteArray = wsCtx.root.toProtobuf.toByteArray
+ // Capture fs.azure.* / fs.s3a.* / fs.gs.* keys from the driver-side
+ // Hadoop configuration NOW, while we are still on the driver, and embed
+ // them in every GlutenPartition. These keys are set by the user via
+ // spark.conf.set("fs.azure.account.auth.type", ...) or
+ // sparkContext.hadoopConfiguration.set(...)
+ // Spark's withSQLConfPropagated only forwards keys starting with "spark"
+ // as task-local-properties, so "fs.*" keys never reach the executor's
+ // SQLConf. Serialising them inside the partition is the only safe way
+ // to make them available to the native runtime on the executor.
+ // Capture fs.azure.* / fs.s3a.* / fs.gs.* keys while on the driver.
+ // SparkPlan.sqlContext is available on the driver -- using the first leaf
+ // gives us access to sessionState.newHadoopConf() which includes all keys
+ // set via spark.conf.set(), sparkContext.hadoopConfiguration, and
+ // DataFrameReader.option(). These are NOT propagated to executors by
+ // Spark's withSQLConfPropagated (it only forwards keys starting with
+ // "spark"), so embedding them in the serialised GlutenPartition is the
+ // only reliable transport mechanism.
+ val fsPrefixes = Seq("fs.azure.", "fs.s3a.", "fs.gs.")
+ val hadoopConf = leaves.headOption
+ .map(_ =>
org.apache.spark.sql.SparkSession.active.sessionState.newHadoopConf())
+
.getOrElse(org.apache.spark.SparkContext.getOrCreate().hadoopConfiguration)
+ val fsConf = {
+ hadoopConf.iterator().asScala
Review Comment:
Good point. Hadoop Configuration has getPropsWithPrefix in the Hadoop
versions Gluten supports, so I’ll replace the full iterator() scan with prefix
lookups for fs.azure., fs.s3a., and fs.gs.. This keeps the same propagation
behavior but avoids iterating the entire configuration set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
