0z5a opened a new issue, #905:
URL: https://github.com/apache/arrow-rs-object-store/issues/905

   **Describe the bug**
   
   The default `ObjectStore::get_ranges` implementation (`coalesce_ranges`) 
panics with `range start must not be greater than end` when one of the 
requested ranges starts after the end of the object and is merged with an 
earlier, in-bounds range. If the range starts exactly at the end of the object, 
it returns an empty `Bytes` instead of an error.
   
   Requesting the same range on its own with `get_range` returns an error. So 
does `get_ranges` on `InMemory` and `LocalFileSystem`, which have their own 
implementations. `AmazonS3`, `GoogleCloudStorage`, `MicrosoftAzure` and 
`HttpStore` use the default implementation, so they take this path. For 
example, a reader working from a truncated file or bad metadata can ask for 
such ranges, and the caller then gets a panic instead of an error it can handle.
   
   **To Reproduce**
   
   With `object_store` from main and `tokio` (features `macros`, 
`rt-multi-thread`):
   
   ```rust
   use object_store::memory::InMemory;
   use object_store::path::Path;
   use object_store::{OBJECT_STORE_COALESCE_DEFAULT, ObjectStoreExt, 
coalesce_ranges};
   
   #[tokio::main]
   async fn main() {
       let store = InMemory::new();
       let path = Path::from("obj");
       store.put(&path, "arbitrary data".into()).await.unwrap(); // 14 bytes
   
       // A range starting past the end of the object is an error when 
requested on its own
       let r = store.get_range(&path, 200..300).await;
       println!("get_range(200..300) -> {:?}", r.map_err(|e| e.to_string()));
   
       // Body of the default `ObjectStore::get_ranges`
       for ranges in [vec![0..5, 14..20], vec![0..5, 200..300]] {
           let r = coalesce_ranges(
               &ranges,
               |r| store.get_range(&path, r),
               OBJECT_STORE_COALESCE_DEFAULT,
           )
           .await;
           println!("get_ranges({ranges:?}) -> {:?}", r.map_err(|e| 
e.to_string()));
       }
   }
   ```
   
   Output:
   
   ```
   get_range(200..300) -> Err("Generic InMemory error: Invalid range: Wanted 
range starting at 200, but object was only 14 bytes long")
   get_ranges([0..5, 14..20]) -> Ok([b"arbit", b""])
   
   thread 'main' panicked at .../bytes-1.12.1/src/bytes.rs:374:28:
   range start must not be greater than end: 200 <= 14
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   ```
   
   The loop body is the same as the default `get_ranges`. I also checked a real 
store: `HttpStore` against a small server that follows RFC 9110. That server 
answers `bytes=0-299` on the 14-byte object with `206` and `Content-Range: 
bytes 0-13/14`, and answers `bytes=200-299` with `416`. Against it, 
`get_range(200..300)` returns the 416 error, `get_ranges(&[0..5, 200..300])` 
panics with the same message, and `get_ranges(&[0..5, 14..20])` returns 
`[b"arbit", b""]`. Calling `InMemory::get_ranges` or 
`LocalFileSystem::get_ranges` with the same ranges returns an error. In-bounds 
ranges such as `[0..5, 7..9]` work on all of them.
   
   The cause is in `coalesce_ranges` in `src/util.rs`:
   - `0..5` and `200..300` are merged into a single fetch of `0..300`.
   - A bounded range that runs past the end of the object is truncated. This is 
the documented `GetRange::Bounded` behaviour, and the HTTP client accepts the 
`bytes 0-13/14` response for it. So the merged fetch returns 14 bytes.
   - Each requested range is then sliced out of that buffer. The end of the 
slice is clamped to the buffer length, but the start is not checked. 
`slice(200..14)` panics, and `slice(14..14)` returns an empty buffer.
   
   **Expected behavior**
   
   `get_ranges` should return an error for a range that starts at or after the 
end of the object. That matches `get_range` for the same range, and 
`InMemory`/`LocalFileSystem::get_ranges`. RFC 9110 section 14.1.2 treats such a 
range (first position at or past the representation length) as unsatisfiable. 
The `GetRange::Bounded` docs say "If the given range is zero-length or starts 
after the end of the object, an error will be returned". `put_get_delete_list` 
in `src/integration.rs` already expects `get_range(200..300)` to be a non-fatal 
error.
   
   Proposed fix: in `coalesce_ranges`, when the bytes returned for a merged 
fetch end before a requested range starts, fetch that range on its own and 
return what the store returns, which is normally its out-of-range error. The 
public signature stays the same, and the extra request only happens in this 
error case. Ranges that start inside the object but run past its end are still 
truncated as today.
   
   **Additional context**
   
   - object_store main at c216ede (version 0.14.2)
   - macOS 26.6 arm64, rustc 1.100.0
   - The end clamp was added in apache/arrow-rs#5222. That change made bounded 
ranges past the end get truncated instead of rejected, but the start of each 
slice was never checked.
   - I have a fix with a regression test: a unit test for `coalesce_ranges`, 
plus a `get_ranges` assertion in `put_get_delete_list` so the S3/GCS/Azure/HTTP 
integration runs cover it. I'm happy to open a PR.
   
   This issue was drafted with Claude Code (claude-opus-5-5); the results above 
are from the runs described.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to