comphead opened a new issue, #3572: URL: https://github.com/apache/iceberg-rust/issues/3572
`RecordBatchTransformer::generate_batch_transform` converts the whole snapshot schema to Arrow and maps every top-level field on the first batch of each file, even when the result is a pass-through. The cost grows with the table's width, not with the number of projected columns. Wide tables stored as many small files pay it most. #3540 avoids this for the row filter transformer by giving it a schema that holds only the filter fields. @mbutrovich measured building that transformer and processing one 8192-row batch: 135 µs with the full schema of a 1001-column table, 2 µs with the filter fields only. The output transformer in `reader/pipeline.rs` should do the same, either at the call site or inside `generate_batch_transform`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
