Xuanwo commented on code in PR #3763:
URL:
https://github.com/apache/incubator-opendal/pull/3763#discussion_r1431285027
##########
core/src/services/hdfs/backend.rs:
##########
@@ -38,6 +39,7 @@ pub struct HdfsBuilder {
name_node: Option<String>,
kerberos_ticket_cache_path: Option<String>,
user: Option<String>,
+ atomic_write_dir: Option<PathBuf>,
Review Comment:
We don't need to use `PathBuf` here.
##########
core/src/services/hdfs/backend.rs:
##########
@@ -262,12 +317,36 @@ impl Accessor for HdfsBackend {
open_options.write(true);
}
- let f = open_options
- .async_open(&p)
- .await
- .map_err(new_std_io_error)?;
+ if let Some(tmp_path) = &tmp_path {
+ let mut t = open_options
+ .async_open(tmp_path.to_str().unwrap())
+ .await
+ .map_err(new_std_io_error)?;
- Ok((RpWrite::new(), HdfsWriter::new(f)))
+ let tmp_path = tmp_path.clone();
+ let target_path = target_path.clone();
+
+ t.flush().await.map_err(new_std_io_error)?;
Review Comment:
Please impelment those logic in `HdfsWriter` instead. We should rename file
during `poll_close`.
##########
core/src/services/hdfs/backend.rs:
##########
@@ -95,6 +97,16 @@ impl HdfsBuilder {
self.enable_append = enable_append;
self
}
+
+ /// Set temp dir for atomic write.
+ pub fn atomic_write_dir(&mut self, dir: &str) -> &mut Self {
Review Comment:
We can add a new test workflow in
https://github.com/apache/incubator-opendal/blob/main/.github/workflows/service_test_hdfs.yml
with `OPENDAL_HDFS_ATOMIC_WRITE_DIR=path/to/tmp` to test this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
