0z5a opened a new issue, #905:
URL: https://github.com/apache/arrow-rs-object-store/issues/905
**Describe the bug**
The default `ObjectStore::get_ranges` implementation (`coalesce_ranges`)
panics with `range start must not be greater than end` when one of the
requested ranges starts after the end of the object and is merged with an
earlier, in-bounds range. If the range starts exactly at the end of the object,
it returns an empty `Bytes` instead of an error.
Requesting the same range on its own with `get_range` returns an error. So
does `get_ranges` on `InMemory` and `LocalFileSystem`, which have their own
implementations. `AmazonS3`, `GoogleCloudStorage`, `MicrosoftAzure` and
`HttpStore` use the default implementation, so they take this path. For
example, a reader working from a truncated file or bad metadata can ask for
such ranges, and the caller then gets a panic instead of an error it can handle.
**To Reproduce**
With `object_store` from main and `tokio` (features `macros`,
`rt-multi-thread`):
```rust
use object_store::memory::InMemory;
use object_store::path::Path;
use object_store::{OBJECT_STORE_COALESCE_DEFAULT, ObjectStoreExt,
coalesce_ranges};
#[tokio::main]
async fn main() {
let store = InMemory::new();
let path = Path::from("obj");
store.put(&path, "arbitrary data".into()).await.unwrap(); // 14 bytes
// A range starting past the end of the object is an error when
requested on its own
let r = store.get_range(&path, 200..300).await;
println!("get_range(200..300) -> {:?}", r.map_err(|e| e.to_string()));
// Body of the default `ObjectStore::get_ranges`
for ranges in [vec![0..5, 14..20], vec![0..5, 200..300]] {
let r = coalesce_ranges(
&ranges,
|r| store.get_range(&path, r),
OBJECT_STORE_COALESCE_DEFAULT,
)
.await;
println!("get_ranges({ranges:?}) -> {:?}", r.map_err(|e|
e.to_string()));
}
}
```
Output:
```
get_range(200..300) -> Err("Generic InMemory error: Invalid range: Wanted
range starting at 200, but object was only 14 bytes long")
get_ranges([0..5, 14..20]) -> Ok([b"arbit", b""])
thread 'main' panicked at .../bytes-1.12.1/src/bytes.rs:374:28:
range start must not be greater than end: 200 <= 14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
The loop body is the same as the default `get_ranges`. I also checked a real
store: `HttpStore` against a small server that follows RFC 9110. That server
answers `bytes=0-299` on the 14-byte object with `206` and `Content-Range:
bytes 0-13/14`, and answers `bytes=200-299` with `416`. Against it,
`get_range(200..300)` returns the 416 error, `get_ranges(&[0..5, 200..300])`
panics with the same message, and `get_ranges(&[0..5, 14..20])` returns
`[b"arbit", b""]`. Calling `InMemory::get_ranges` or
`LocalFileSystem::get_ranges` with the same ranges returns an error. In-bounds
ranges such as `[0..5, 7..9]` work on all of them.
The cause is in `coalesce_ranges` in `src/util.rs`:
- `0..5` and `200..300` are merged into a single fetch of `0..300`.
- A bounded range that runs past the end of the object is truncated. This is
the documented `GetRange::Bounded` behaviour, and the HTTP client accepts the
`bytes 0-13/14` response for it. So the merged fetch returns 14 bytes.
- Each requested range is then sliced out of that buffer. The end of the
slice is clamped to the buffer length, but the start is not checked.
`slice(200..14)` panics, and `slice(14..14)` returns an empty buffer.
**Expected behavior**
`get_ranges` should return an error for a range that starts at or after the
end of the object. That matches `get_range` for the same range, and
`InMemory`/`LocalFileSystem::get_ranges`. RFC 9110 section 14.1.2 treats such a
range (first position at or past the representation length) as unsatisfiable.
The `GetRange::Bounded` docs say "If the given range is zero-length or starts
after the end of the object, an error will be returned". `put_get_delete_list`
in `src/integration.rs` already expects `get_range(200..300)` to be a non-fatal
error.
Proposed fix: in `coalesce_ranges`, when the bytes returned for a merged
fetch end before a requested range starts, fetch that range on its own and
return what the store returns, which is normally its out-of-range error. The
public signature stays the same, and the extra request only happens in this
error case. Ranges that start inside the object but run past its end are still
truncated as today.
**Additional context**
- object_store main at c216ede (version 0.14.2)
- macOS 26.6 arm64, rustc 1.100.0
- The end clamp was added in apache/arrow-rs#5222. That change made bounded
ranges past the end get truncated instead of rejected, but the start of each
slice was never checked.
- I have a fix with a regression test: a unit test for `coalesce_ranges`,
plus a `get_ranges` assertion in `put_get_delete_list` so the S3/GCS/Azure/HTTP
integration runs cover it. I'm happy to open a PR.
This issue was drafted with Claude Code (claude-opus-5-5); the results above
are from the runs described.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]