bokket commented on issue #3724:
URL:
https://github.com/apache/incubator-opendal/issues/3724#issuecomment-1852714859
I have written this logic with reference to `fs`
```rust
impl HdfsBackend {
async fn atomic_write(&self,path: &str, tmp: &str) -> Result<PathBuf> {
// /root/
// /root/path
let p = build_rooted_abs_path(&self.root, path);
self.client.metadata(&p).map_err(new_std_io_error)?;
let t = build_rooted_abs_path(&self.root, tmp);
let result = self.client.metadata(&t);
match result {
Err(err) => {
// Early return if other error happened.
if err.kind() != io::ErrorKind::NotFound {
return Err(new_std_io_error(err));
}
let parent = PathBuf::from(&t)
.parent()
.ok_or_else(|| {
Error::new(
ErrorKind::Unexpected,
"path should have parent but not, it must be
malformed",
)
.with_context("input", &t)
})?
.to_path_buf();
self.client
.create_dir(&parent.to_string_lossy())
.map_err(new_std_io_error)?;
let mut f = self
.client
.open_file()
.create(true)
.write(true)
.async_open(&t)
.await
.map_err(new_std_io_error)?;
/// Build a future for `poll_close`.
/// Is Close the writer and make sure all data has been
flushed？
/// Should us flush()?
f.close().await.map_err(new_std_io_error)?;
}
Ok(metadata) => {
if metadata.is_file() {
self.client
.remove_file(&t)
.map_err(new_std_io_error)?;
} else {
return Err(Error::new(ErrorKind::IsADirectory, "path
should be a file")
.with_context("input", &t));
}
}
}
self.client
.rename_file(&t, &p)
.map_err(new_std_io_error)?;
Ok(PathBuf::from(&p))
}
}
```
But I think this is ineffective. At least we need to change the structure of
`HdfsWriter` like `fs::FsWriter`
```rust
pub struct FsWriter<F> {
target_path: PathBuf,
tmp_path: Option<PathBuf>,
f: Option<F>,
fut: Option<BoxFuture<'static, Result<()>>>,
}
fn poll_close(&mut self, cx: &mut Context<'_>) -> Poll<Result<()>> {
loop {
........
}
let mut f = self.f.take().expect("FsWriter must be initialized");
let tmp_path = self.tmp_path.clone();
let target_path = self.target_path.clone();
self.fut = Some(Box::pin(async move {
f.flush().await.map_err(new_std_io_error)?;
f.sync_all().await.map_err(new_std_io_error)?;
if let Some(tmp_path) = &tmp_path {
tokio::fs::rename(tmp_path, &target_path)
.await
.map_err(new_std_io_error)?;
}
Ok(())
}));
}
}
```
I can clearly see that `atomic_dir` is rename after being sync. I think
`HdfsBackend` does not support such capabilities.Becasue `HdfsWriter` internal
implementation is different.Data don't write into `tmp_dir`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
