alamb commented on PR #2207:
URL: https://github.com/apache/arrow-rs/pull/2207#issuecomment-1201316230
I ran a little experiment (code below) with some various symlinks and the
TLDR is I don't see any difference in behavior with this PR comared to master
```shell
$ ls -l /tmp/object_store/
total 16
-rw-r--r-- 1 alamb wheel 4 Aug 1 10:47 file1.txt
-rw-r--r-- 1 alamb wheel 7 Aug 1 10:47 file2.txt
lrwxr-xr-x 1 alamb wheel 27 Aug 1 10:48 file_ln.txt ->
/tmp/object_store/file1.txt
lrwxr-xr-x 1 alamb wheel 21 Aug 1 10:50 file_ln_outside.txt ->
/tmp/outsize_root.txt
```
Using the `master` branch, my test program shows
```shell
Using object store root: /tmp/object_store
file2.txt, size: 7, 7
file1.txt, size: 4, 4
```
Using this PR's branch my test prgram shows:
```
Using object store root: /tmp/object_store
file1.txt, size: 4, 4
file2.txt, size: 7, 7
```
Test code:
```rust
//! Basic 'ls' client
use std::sync::Arc;
use futures::stream::{FuturesOrdered, StreamExt};
use object_store::{local::LocalFileSystem, path::Path, ObjectStore};
#[tokio::main]
async fn main() {
// create an ObjectStore
let object_store: Arc<dyn ObjectStore> = get_local_store();
// list all objects in the store
let path: Path = "/".try_into().unwrap();
let list_stream = object_store
.list(Some(&path))
.await
.expect("Error listing files");
// List all files in the store
list_stream
.map(|meta| async {
let meta = meta.expect("Error listing");
// fetch the bytes from object store
let stream = object_store
.get(&meta.location)
.await
.unwrap()
.into_stream();
// Get the size size
let measured_size = stream
.map(|bytes| {
let bytes = bytes.unwrap();
bytes.len()
})
.collect::<Vec<usize>>()
.await
.into_iter()
.sum::<usize>();
(meta, measured_size)
})
.collect::<FuturesOrdered<_>>()
.await
.collect::<Vec<_>>()
.await
.into_iter()
.for_each(|(meta, measured_size)| {
println!("{}, size: {}, {}", meta.location, meta.size,
measured_size);
});
}
fn get_local_store() -> Arc<dyn ObjectStore> {
let root = "/tmp/object_store";
println!("Using object store root: {}", root);
let local_fs =
LocalFileSystem::new_with_prefix(root)
.expect("Error creating local file system");
Arc::new(local_fs)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
