mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r3893167714
##########
.github/workflows/ci.yml:
##########
@@ -232,6 +232,16 @@ jobs:
cargo test --no-fail-fast ${{ matrix.test-suite.args }}
fi
+ # HDFS tests are `#[ignore]`d (docker fixture needs Linux host
+ # networking); opt them in here by test-name filter.
+ - name: Run HDFS integration tests
Review Comment:
Done in 2b0ca95 — removed the step and the `#[ignore]`s; the suite now runs
under the default `cargo nextest run` (the fixture is already started by `make
docker-up`, and the Tests job is Linux-only).
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+use std::collections::HashMap;
+
+use serde::{Deserialize, Serialize};
+use typed_builder::TypedBuilder;
+
+use super::StorageConfig;
+use crate::Result;
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
+
+/// HDFS storage configuration.
+///
+/// This struct contains all the configuration options for connecting to HDFS.
+/// Use the builder pattern via `HdfsConfig::builder()` to construct instances.
+#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize,
TypedBuilder)]
+pub struct HdfsConfig {
Review Comment:
Done in 2b0ca95. One note: with `pub(crate)` the struct has no non-test
consumer inside the core crate yet (`iceberg-storage-opendal` shares only the
key constants), so it carries `#[allow(dead_code)]` for now — happy to drop the
struct entirely instead if you prefer.
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+use std::collections::HashMap;
+
+use serde::{Deserialize, Serialize};
+use typed_builder::TypedBuilder;
+
+use super::StorageConfig;
+use crate::Result;
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
+
+/// HDFS storage configuration.
+///
+/// This struct contains all the configuration options for connecting to HDFS.
+/// Use the builder pattern via `HdfsConfig::builder()` to construct instances.
+#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize,
TypedBuilder)]
+pub struct HdfsConfig {
Review Comment:
Done in 2b0ca95 — migrated to `#[derive(Properties)]` with `key`/`prefix`
attributes and dropped the hand-written `TryFrom` + `TypedBuilder`/serde
derives.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
