Niqnil commented on issue #10468: URL: https://github.com/apache/arrow-rs/issues/10468#issuecomment-5174105571
Thanks for the tip @alamb. It prompted a recheck of our position. We have moved off the record API. Our decoder now reads through `ColumnReaderImpl::read_records` into reused batch buffers (variant B in the table above) so we no longer have a stake in this issue. The arrow reader is not a good fit for us now because: 1. Enabling `arrow` on `parquet 59.1.0` vs `parquet` with `default-features = false, features = ["zstd"]` would increase our library's `cargo tree` from 35 to 47 crates. 2. Only ~76 ns of variant B's ~472 ns/row is decompression and page decode. The remainder is our own per-row conversions. Going through `RecordBatch` would add a columnar-to-row walk. That is specific to our narrow flat schema and row-at-a-time output. A correction to my previous report: item (1) is not a cost at the 4-column schema I measured. `size_of::<(String, Field)>()` is 72, which puts it in `RawVec`'s `MIN_NON_ZERO_CAP == 4` bracket, so `Vec::new()` allocates capacity 4 on its very first push — already identical to `with_capacity(4)`, same single allocation, same capacity. Tracing capacity through one `push` per column: 1–4 columns saves nothing at all, 5–8 saves one allocation (plus one 288-byte memcpy), and 20 columns goes from 4 allocations to 1. So item (1) is really a wide-schema fix. As others might still be affected, I leave it to you to keep this issue open or to close it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
