sunchao commented on code in PR #5503:
URL: https://github.com/apache/datafusion-comet/pull/5503#discussion_r3970448022
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -469,9 +469,12 @@ fn create_hdfs_object_store(
})
}
-type ObjectStoreCache = RwLock<HashMap<(String, u64), Arc<dyn ObjectStore>>>;
+type ObjectStoreCacheKey = (String, u64, bool);
Review Comment:
Added documentation directly above `ObjectStoreCacheKey` in deaf061bb. The
fields are the scheme/host/port, configuration hash, and whether the Hadoop
backend is selected (`true` for Hadoop, including custom schemes routed through
it; `false` for native). Native `s3a` is normalized to `s3`, while
Hadoop-selected schemes retain their spelling.
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -581,28 +584,262 @@ pub(crate) fn prepare_object_store_with_configs(
(store, path)
};
- let object_store_url = ObjectStoreUrl::parse(url_key.clone())?;
- runtime_env.register_object_store(&url, object_store);
+ // A RuntimeEnv can plan multiple scans with different backends or
credentials
+ // for the same bucket. Use the same identity as the cache, even for the
first
+ // registration, so neither later registration nor planning order changes
the
+ // store used by an existing scan. Native s3/s3a share the normalized s3
scheme;
+ // a Hadoop-selected scheme retains its physical spelling.
+ //
+ // Native LocalFileSystem ignores these Hadoop options and keeps file://
for
+ // compatibility. An explicitly Hadoop-routed file scheme is still
isolated.
+ let object_store_url = if scheme == "file" && !is_hdfs_scheme {
+ ObjectStoreUrl::parse(url_key)?
+ } else {
+ let backend = if is_hdfs_scheme { "hdfs" } else { "native" };
+ ObjectStoreUrl::parse(format!(
+ "{scheme}+comet-{config_hash:016x}-{backend}://{}",
Review Comment:
`+comet-` is a reserved marker for our internal registration URLs.
DataFusion keys stores only by scheme and authority, so we put the
configuration hash and backend identity in the scheme while preserving the
physical authority. This prevents a later scan of the same bucket with
different credentials or backend selection from replacing an earlier scan's
store. Encryption lookup recognizes and strips the complete suffix to recover
the physical URI. Added a comment explaining this in deaf061bb.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
