corwinjoy commented on code in PR #7019:
URL: https://github.com/apache/arrow-rs/pull/7019#discussion_r1945693385
##########
object_store/src/local.rs:
##########
@@ -678,6 +622,93 @@ impl ObjectStore for LocalFileSystem {
}
}
+impl LocalFileSystem {
+ fn list_with_maybe_offset(
+ &self,
+ prefix: Option<&Path>,
+ maybe_offset: Option<&Path>,
+ ) -> BoxStream<'static, Result<ObjectMeta>> {
+ let config = Arc::clone(&self.config);
+
+ let root_path = match prefix {
+ Some(prefix) => match config.prefix_to_filesystem(prefix) {
+ Ok(path) => path,
+ Err(e) => return
futures::future::ready(Err(e)).into_stream().boxed(),
+ },
+ None => config.root.to_file_path().unwrap(),
+ };
+
+ let walkdir = WalkDir::new(root_path)
Review Comment:
@alamb I agree with all this and thanks for the helpful comments! I think
your concern about not having a deterministic order is a good one and agree
this is the existing behavior. I also think we should change this, but propose
we do that in a follow-up PR since it does change the behavior and could
(potentially) break an assumption somewhere. I think we can just use
https://docs.rs/walkdir/latest/walkdir/struct.WalkDir.html#method.sort_by_file_name
which should be fine and I believe matches what the offset filter expects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
