blackmwk commented on code in PR #2651:
URL: https://github.com/apache/iceberg-rust/pull/2651#discussion_r4640922892
##########
crates/storage/opendal/src/resolving.rs:
##########
@@ -272,12 +279,30 @@ impl OpenDalResolvingStorage {
return Ok(storage.clone());
}
- let storage = build_storage_for_scheme(
- scheme,
- &self.props,
- #[cfg(feature = "opendal-s3")]
- &self.customized_credential_load,
- )?;
+ // A prefix such as `s3://bucket/` or `s3` names its scheme, and never
+ // matches the paths of another.
+ let storage_credentials = self
+ .storage_credentials
+ .iter()
+ .filter(|credential| {
+ let prefix = credential.prefix();
+ let name = prefix.split_once(':').map_or(prefix, |(name, _)|
name);
Review Comment:
Why we need to do this split?
##########
crates/storage/opendal/src/resolving.rs:
##########
@@ -325,10 +350,21 @@ impl Storage for OpenDalResolvingStorage {
let mut grouped: HashMap<&'static str, Vec<String>> = HashMap::new();
while let Some(path) = paths.next().await {
let scheme = extract_scheme(&path)?;
- grouped.entry(scheme).or_default().push(path);
+ let batch = grouped.entry(scheme).or_default();
+ batch.push(path);
+ if batch.len() >= DELETE_BATCH_SIZE {
Review Comment:
We should make `DELETE_BATCH_SIZE` a config rather a const, create an issue
to track this.
##########
crates/storage/opendal/src/resolving.rs:
##########
@@ -235,9 +239,12 @@ impl StorageFactory for OpenDalResolvingStorageFactory {
pub struct OpenDalResolvingStorage {
/// Configuration properties shared across all backends.
props: HashMap<String, String>,
+ /// Credentials that apply to locations under their prefixes.
+ #[serde(default, skip_serializing_if = "Vec::is_empty")]
+ storage_credentials: Vec<StorageCredential>,
Review Comment:
We should replace this with StorageConfig?
##########
crates/storage/opendal/src/resolving.rs:
##########
@@ -272,12 +279,30 @@ impl OpenDalResolvingStorage {
return Ok(storage.clone());
}
- let storage = build_storage_for_scheme(
- scheme,
- &self.props,
- #[cfg(feature = "opendal-s3")]
- &self.customized_credential_load,
- )?;
+ // A prefix such as `s3://bucket/` or `s3` names its scheme, and never
+ // matches the paths of another.
+ let storage_credentials = self
+ .storage_credentials
+ .iter()
+ .filter(|credential| {
+ let prefix = credential.prefix();
+ let name = prefix.split_once(':').map_or(prefix, |(name, _)|
name);
+ parse_scheme(&name.to_ascii_lowercase()).map_or(true, |s| s ==
scheme)
Review Comment:
`map_or(true, …)` admits a credential with an unknown scheme into every
backend. I verified that a `future://bucket/` credential with an invalid
`opendal.io-timeout-ms` makes `exists("memory:/file")` fail because the
resolver parses that credential’s config even though its prefix can never match
the path. Could we skip credentials with an unsupported URI scheme here and add
that case to `test_resolve_ignores_credentials_for_other_schemes`?
##########
crates/storage/opendal/src/resolving.rs:
##########
@@ -272,12 +279,30 @@ impl OpenDalResolvingStorage {
return Ok(storage.clone());
}
- let storage = build_storage_for_scheme(
- scheme,
- &self.props,
- #[cfg(feature = "opendal-s3")]
- &self.customized_credential_load,
- )?;
+ // A prefix such as `s3://bucket/` or `s3` names its scheme, and never
+ // matches the paths of another.
+ let storage_credentials = self
+ .storage_credentials
+ .iter()
+ .filter(|credential| {
+ let prefix = credential.prefix();
+ let name = prefix.split_once(':').map_or(prefix, |(name, _)|
name);
Review Comment:
WHy we can't just extract_schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]