Sruhvx-jpg opened a new issue, #3166:
URL: https://github.com/apache/iceberg-rust/issues/3166
### Problem / Context
In #3165 (reviving #2257 for #2259), the foundational
`ObjectStoreStorage::S3` backend is established. To ensure production
robustness and performance in the write pipeline, several edge cases and
optimizations should be addressed as follow-up work:
### Proposed Improvements
1. **Zero-Copy Writes in `ObjectStoreWriter`**:
- `FileWrite::write` receives owned `bytes::Bytes`.
- `object_store::WriteMultipart` provides a `put(Bytes)` method that
buffers without slice copying, avoiding redundant `extend_from_slice`
allocations.
2. **S3 URL Boundary Validation**:
- Empty bucket strings (e.g. `s3:///path/to/file` or `s3://`) currently
pass initial URL host parsing as `""`. These should explicitly return
`ErrorKind::DataInvalid`.
- Ensure paths without trailing slashes (`s3://bucket`) extract `bucket`
and empty relative path cleanly.
3. **Stream Handling & State Machine Guards**:
- Validate `delete_stream` on empty streams (`futures::stream::empty()`)
and ensure immediate error propagation on invalid URLs.
- Add unit test coverage for `ObjectStoreWriter` state guards (e.g.,
verifying `close()` and `write()` error handling after writer has been closed).
4. **Backpressure & Memory Throttling**:
- For high-throughput streaming, evaluate
`WriteMultipart::wait_for_capacity` to throttle producers and prevent unbounded
5MB chunk allocations in memory ahead of S3 network uploads.
### Related
- #2258 (Umbrella: Implement Storage using object_store)
- #2259 (Implement ObjectStoreStorage::S3)
- #3165 (feat(storage): Implement ObjectStoreStorage::S3)
## AI Disclosure
Drafted with AI assistance; reviewed and verified by contributor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
