alamb commented on PR #9093: URL: https://github.com/apache/arrow-rs/pull/9093#issuecomment-3730681145
> from experiment: Pre-allocation overhead may offset the savings from avoiding incremental growth @lyang24 -- I am not sure that the `num_rows` you are passing in is actually the total number of rows which would be output. Looking at the internals of the reader, it almost looks like the record_reader doesn't currently get told how many records it may get -- however, the higher level APIs certainly know the max size (it is the `batch_size`) Maybe we could pass in the batch size to the reader... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
