jackylee-ch commented on code in PR #12549:
URL: https://github.com/apache/gluten/pull/12549#discussion_r3804833868
##########
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala:
##########
@@ -641,23 +644,37 @@ object VeloxConfig extends ConfigRegistry {
val CACHE_PREFETCH_MINPCT =
buildStaticConf("spark.gluten.sql.columnar.backend.velox.cachePrefetchMinPct")
+ .passToNative()
.doc("Set prefetch cache min pct for velox file scan")
.intConf
.createWithDefault(0)
+ // The three confs below are read by native `createHiveConnectorConfig`,
which runs both at
+ // backend initialization (for the reused HiveConnector) and per write on
each native runtime
+ // (`VeloxParquetDataSourceS3::initSink`, `IcebergWriter`), hence they are
modifiable confs.
val AWS_SDK_LOG_LEVEL =
buildConf("spark.gluten.velox.awsSdkLogLevel")
.internal()
+ .passToNative()
.doc("Log granularity of AWS C++ SDK in velox.")
.stringConf
.createWithDefault("FATAL")
- val AWS_S3_RETRY_MODE =
- buildConf("spark.gluten.velox.fs.s3a.retry.mode")
+ val S3_USE_PROXY_FROM_ENV =
+ buildConf("spark.gluten.velox.s3UseProxyFromEnv")
.internal()
- .doc("Retry mode for AWS s3 connection error: legacy, standard and
adaptive.")
+ .passToNative()
+ .doc("Whether to use proxy from environment variables for S3 C++
client.")
+ .booleanConf
+ .createWithDefault(false)
+
+ val S3_PAYLOAD_SIGNING_POLICY =
+ buildConf("spark.gluten.velox.s3PayloadSigningPolicy")
+ .internal()
+ .passToNative()
Review Comment:
Yeap, this is the main change for this PR. Developer must set `passToNative`
if he want pass the value to native engine, if not, it will use the native
default values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
