mixermt opened a new pull request, #3111: URL: https://github.com/apache/iceberg-rust/pull/3111
## Which issue does this PR close? - Closes #2440. - Part of #1130. This revives #1131's successor #2441 by @jordepic (closed by the stale bot after a first review round), rebased onto the current `Storage`-trait layout and updated for opendal 0.58, where `services-hdfs-native` changed behavior in ways that required design changes (details below). ## What changes are included in this PR? Adds an `opendal-hdfs-native` cargo feature to `iceberg-storage-opendal`, with `OpenDalStorageFactory::Hdfs` / `OpenDalStorage::Hdfs` variants and `hdfs://` routing in `OpenDalResolvingStorage`. The backend uses OpenDAL's `services-hdfs-native` (pure-Rust HDFS RPC via `hdfs-native` — no JNI/libhdfs). The feature is experimental and not part of `opendal-all`, matching `opendal-oss`/`opendal-azdls`. NameNode resolution (differs from #2441, forced by opendal 0.58 where `name_node` is mandatory and its comma-split list is the HA mechanism): 1. `hdfs.name-node` property when set — a single endpoint or a comma-separated list for HA failover (new `HdfsConfig` + constants in `iceberg::io`, following the existing per-backend config convention). 2. Otherwise the path authority (`hdfs://host:port/path`). 3. Authority-less paths without the property are rejected with a pointed error (opendal 0.58 can no longer defer to `fs.defaultFS`). `hadoop.`-prefixed properties pass through to the HDFS client config, overriding `$HADOOP_CONF_DIR` values (mirroring the `hadoop.` catalog-property convention of the Java integrations); `hdfs-native` still loads `core-site.xml`/`hdfs-site.xml` from `$HADOOP_CONF_DIR`/`$HADOOP_HOME` for everything else, and Kerberos works via `libgssapi_krb5` (runtime dlopen). Operators are cached per effective NameNode since each holds live RPC connections. Relative paths are returned opendal-style without a leading `/` — `opendal::Deleter::delete` (used by `delete_stream`) rejects leading slashes, which an integration test caught. Test infrastructure: single-node HDFS docker fixture (`apache/hadoop:3.5.0`, host networking — required because `hdfs-native` dials DataNodes by their registered IP, unroutable on a bridge). The DataNode healthcheck gates on NameNode registration so `--wait` means writable. Tests are `#[ignore]`d (host networking is Linux-only); CI opts in via `cargo nextest --run-ignored=only -E 'test(file_io_hdfs)'`. ## Are these changes tested? - 24 unit tests across config parsing, path parsing (authority/port/authority-less/wrong-scheme), NameNode precedence, operator caching, relativize, and scheme resolution; 3 config tests in `iceberg::io`. - 12 integration tests against the docker fixture covering exists/read/write/overwrite/metadata/range reads/streaming writes/delete/delete_prefix/delete_stream (+empty), resolving storage over `hdfs://`, and the HA flow (logical authority in the path + `hdfs.name-node` property). All 12 verified green from a cold cluster locally; the suite runs in CI on Linux. - `make check`, full-workspace `--all-features` lib tests, and the existing s3/gcs/resolving integration suites pass; `public-api.txt` regenerated for both crates. ## AI Disclosure Developed with AI assistance (Claude Code): drafting code/tests/fixtures starting from #2441, and cross-checking the design against the vendored opendal 0.58.1 / hdfs-native 0.14.5 sources. I reviewed the implementation and ran all verification locally. Areas worth reviewer attention: the NameNode-resolution semantics above (opendal's synthetic-nameservice behavior constrains what `hdfs://<nameservice>` paths can do without the property), and the Windows `--all-features` build of `hdfs-native`, which I could only verify via CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
