tustvold commented on code in PR #3967:
URL: https://github.com/apache/arrow-rs/pull/3967#discussion_r1150479860
##########
parquet/src/arrow/async_writer/mod.rs:
##########
@@ -88,22 +85,25 @@ pub struct AsyncArrowWriter<W> {
impl<W: AsyncWrite + Unpin + Send> AsyncArrowWriter<W> {
/// Try to create a new Async Arrow Writer.
///
- /// `buffer_flush_threshold` will be used to trigger flush of the inner
buffer.
+ /// `buffer_size` determines the size of the intermediate buffer
+ ///
+ /// Flush will automatically be called by [`Self::write`] if
+ /// the buffer is at least half full
pub fn try_new(
writer: W,
arrow_schema: SchemaRef,
- buffer_flush_threshold: usize,
+ buffer_size: usize,
props: Option<WriterProperties>,
) -> Result<Self> {
- let shared_buffer = SharedBuffer::default();
+ let shared_buffer = SharedBuffer::new(buffer_size);
Review Comment:
> fake feature to let the writer do flush only when all encoded bytes are
read
Yeah, at that point you might as well just use the sync writer 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
