ionnich opened a new issue, #902:
URL: https://github.com/apache/arrow-rs-object-store/issues/902
### Describe the bug
`GetResult::bytes` (`src/lib.rs`) passes the expected length to
`collect_bytes` (`src/util.rs`) only as a capacity hint. If the body stream
ends cleanly before `range.end`, `bytes()` returns `Ok` with fewer bytes than
the range it validated.
This happens, for example, when a `206` response says `Content-Range: bytes
0-999/1000` but sends `Content-Length: 500` and 500 bytes. hyper sees a
complete 500-byte body, `get_range_meta` accepts the `Content-Range`, and
`get_range(0..1000)` returns 500 bytes. Callers such as Parquet and Avro
readers then get a short buffer without any error. It is still true on main /
0.14.2.
### To Reproduce
Run against `object_store = { version = "=0.14.2", features = ["http"] }`:
```rust
//! A server answers a ranged GET with `206`, `Content-Range: bytes
0-999/1000` and a clean 500-byte body
//! (`Content-Length: 500`). object_store's `get_range(0..1000)` returns Ok
with 500 bytes instead of an error.
use std::io::{BufRead, BufReader, Write};
use std::net::TcpListener;
use object_store::http::HttpBuilder;
use object_store::path::Path;
use object_store::{ClientOptions, ObjectStoreExt};
#[tokio::main]
async fn main() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
std::thread::spawn(move || {
for stream in listener.incoming() {
let mut stream = stream.unwrap();
let mut reader = BufReader::new(stream.try_clone().unwrap());
let mut line = String::new();
while reader.read_line(&mut line).unwrap() > 2 {
line.clear();
}
let body = vec![b'x'; 500];
write!(
stream,
"HTTP/1.1 206 Partial Content\r\nContent-Range: bytes
0-999/1000\r\nContent-Length: 500\r\nETag: \"e\"\r\nLast-Modified: Tue, 10 Nov
2026 00:00:00 GMT\r\nConnection: close\r\n\r\n"
)
.unwrap();
stream.write_all(&body).unwrap();
}
});
let store = HttpBuilder::new()
.with_url(format!("http://{addr}";))
.with_client_options(ClientOptions::new().with_allow_http(true))
.build()
.unwrap();
match store.get_range(&Path::from("obj"), 0..1000).await {
Ok(bytes) => println!("get_range(0..1000) = Ok, {} bytes (expected
1000)", bytes.len()),
Err(e) => println!("get_range(0..1000) = Err: {e}"),
}
}
```
Output: `get_range(0..1000) = Ok, 500 bytes (expected 1000)`
### Expected behavior
`bytes()` returns an error when the collected length differs from `range.end
- range.start`. A body that ends early could also be treated like a body error
in `retry_stream`, so it is resumed rather than accepted.
### Additional context
Found while investigating transient read failures of Iceberg manifests in
Sail (lakehq/sail, object_store 0.13.2). There, a short manifest body surfaces
as `Avro read value error: Read into buffer failed: failed to fill whole
buffer`. Related: #690, which fixed the unchecked `Content-Range` on resumed
requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
