jordepic commented on code in PR #2441:
URL: https://github.com/apache/iceberg-rust/pull/2441#discussion_r3282253921
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -218,6 +234,20 @@ pub enum OpenDalStorage {
/// Azure DLS configuration.
config: Arc<AzdlsConfig>,
},
+ /// HDFS storage variant (via OpenDAL `services-hdfs-native`).
+ ///
+ /// Accepts paths of the form `hdfs://<authority>/<rel>` (or
+ /// `hdfs:///<rel>` for authority-less paths, which defer to
+ /// `fs.defaultFS` from the `hdfs-native`-loaded Hadoop config). The
+ /// authority - or `None` when absent - keys a per-name-node `Operator`
+ /// cache, so distinct name nodes coexist in a single storage instance.
+ #[cfg(feature = "opendal-hdfs-native")]
+ Hdfs {
+ /// Operator cache. `Some("hdfs://<authority>")` for paths with an
+ /// authority; `None` for authority-less paths (`fs.defaultFS`).
+ #[serde(skip, default)]
+ operators: Arc<RwLock<HashMap<Option<String>, Operator>>>,
Review Comment:
Good catch on the Java cache key. hdfs-native doesn't model per-call
credentials the way Java's UGI does — Kerberos goes through libgssapi_krb5
reading process env (KRB5CCNAME / KRB5_CONFIG), and the
HdfsNativeConfig builder doesn't accept a UGI or HDFS user. Effectively one
principal per process, so (scheme, authority) is sufficient as the cache key.
Java needs UGI in the key because FileSystem.get() is invoked across UGIs in
the same JVM (Spark executors, multi-tenant services using
UserGroupInformation.doAs()). hdfs-native has no equivalent today — if it ever
grows per-call credentials we'd add UGI as a third dimension, but with the
current API there's nothing to vary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
