Copilot commented on code in PR #12405:
URL: https://github.com/apache/gluten/pull/12405#discussion_r3558035501
##########
cpp/velox/compute/VeloxRuntime.cc:
##########
@@ -284,23 +283,26 @@ void VeloxRuntime::initializeExecutors() {
void VeloxRuntime::registerConnectors() {
auto* backend = VeloxBackend::get();
+
+ auto merged = mergeWithSessionOverrides(backend->getStaticConnectorConfig(),
veloxCfg_->rawConfigs());
+
Review Comment:
registerConnectors() is now invoked from WholeStageResultIterator’s
constructor, which means it can run multiple times on the same VeloxRuntime
(e.g. cpp/velox/benchmarks/GenericBenchmark.cc calls createResultIterator() in
a loop). As implemented, repeated calls will attempt to re-register connectors
with the same scoped IDs and will fail. Either make registration idempotent or
unregister previously-registered connectors before re-registering.
##########
cpp/velox/utils/ConfigExtractor.cc:
##########
@@ -294,6 +294,30 @@ std::string getConfigValue(
return got->second;
}
+std::shared_ptr<facebook::velox::config::ConfigBase> mergeWithSessionOverrides(
+ const std::shared_ptr<facebook::velox::config::ConfigBase>& baseConf,
+ const std::unordered_map<std::string, std::string>& sessionConf) {
+ auto merged = baseConf->rawConfigs(); // copy — we must not mutate the
shared base
+
+ // ── Step 1: forward all per-account and generic fs.azure.* / fs.s3a.* /
fs.gs.*
+ // keys from the session config into the merged map. Session value always
wins.
+ static const std::vector<std::string_view> kForwardPrefixes = {
+ "fs.azure.",
+ "fs.s3a.",
+ "fs.gs.",
+ };
+ for (const auto& [k, v] : sessionConf) {
+ for (const auto& prefix : kForwardPrefixes) {
+ if (k.starts_with(prefix)) {
+ merged[k] = v;
+ break;
+ }
+ }
+ }
Review Comment:
mergeWithSessionOverrides only forwards keys that already start with
"fs.azure." / "fs.s3a." / "fs.gs.". However, Gluten passes Hadoop filesystem
session configs to native as "spark.hadoop.fs.*" (see getAbfsHiveConfig
stripping "spark.hadoop." into the connector config). As a result,
session-level overrides like "spark.hadoop.fs.azure.*" won’t be applied to the
merged connector config, so ABFS (and similar) session overrides still won’t
take effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
