Xuanwo commented on code in PR #3600:
URL:
https://github.com/apache/incubator-opendal/pull/3600#discussion_r1395953991
##########
core/src/services/hdfs/backend.rs:
##########
@@ -219,20 +219,37 @@ impl Accessor for HdfsBackend {
async fn write(&self, path: &str, op: OpWrite) -> Result<(RpWrite,
Self::Writer)> {
let p = build_rooted_abs_path(&self.root, path);
- let parent = PathBuf::from(&p)
- .parent()
- .ok_or_else(|| {
- Error::new(
- ErrorKind::Unexpected,
- "path should have parent but not, it must be malformed",
- )
- .with_context("input", &p)
- })?
- .to_path_buf();
-
- self.client
- .create_dir(&parent.to_string_lossy())
- .map_err(new_std_io_error)?;
+ if let Err(err) = self.client.metadata(&p) {
+ // Early return if other error happened.
+ if err.kind() != io::ErrorKind::NotFound {
+ return Err(new_std_io_error(err));
+ }
+
+ let parent = PathBuf::from(&p)
Review Comment:
We can use `oio::get_parent()` instead.
##########
.github/workflows/service_test_hdfs.yml:
##########
Review Comment:
default fs (ChecksumFileSystem) doesn't support append, let's add a new
config for hdfs called `enable_append` to allow users to pick if their
deployment support append or not.
##########
.github/workflows/service_test_hdfs.yml:
##########
@@ -96,6 +98,8 @@ jobs:
-e WEBHDFS_CONF_dfs_webhdfs_enabled=true \
-e CORE_CONF_hadoop_http_staticuser_user=root \
-e HDFS_CONF_dfs_permissions_enabled=false \
+ -e HDFS_CONF_dfs_support_append=true \
+ -e HDFS_CONF_dfs_replication=1 \
Review Comment:
You are right. We need to configure `dfs_replication` correct.
We can add a `hdfs-site.xml` file in `fixtures/hdfs` and copy into
`/home/runner/hadoop-3.2.4/etc/hadoop` in CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
