Phoenix500526 opened a new issue, #10445:
URL: https://github.com/apache/arrow-rs/issues/10445
### Is your feature request related to a problem or challenge?
`StreamEncoder::encode` currently returns `Vec<Buffer>`. This keeps the API
simple and sans-IO friendly, but it requires the encoder to collect all output
buffers for a batch before the caller can forward them to its destination.
### Describe the solution you'd like
An alternative API could push encoded pieces directly into a sink-like
abstraction, for example:
```rust
pub fn encode_to<S: IpcMessageSink>(
&mut self,
batch: &RecordBatch,
sink: &mut S,
) -> Result<(), ArrowError>
```
### Describe alternatives you've considered
or make `StreamEncoder` generic over a sink.
This could reduce intermediate buffering and copying for callers that can
consume IPC pieces immediately, such as async writers. The tradeoff is that the
encoder starts to look more like a writer, and exposing a sink abstraction may
make the public API more complex.
Things to evaluate:
- Whether this should be additive, such as `encode_to`, instead of replacing
`encode`
- Whether the sink abstraction should be public
- Whether this preserves the intended sans-IO API boundary
- How it affects copy behavior and memory usage
- Benchmark impact compared with `encode -> Vec<Buffer>`
### Additional context
Follow-up from #10277 and review discussion:
https://github.com/apache/arrow-rs/pull/10277#discussion_r3651419972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
