XiangpengHao commented on issue #5855: URL: https://github.com/apache/arrow-rs/issues/5855#issuecomment-2154852784
A low-hanging fruit would be selectively decoding row groups. Currently, we build all row group metadata, which takes 30% of the decoding time (as will shown in the #5770 blog post). https://github.com/apache/arrow-rs/blob/087f34b70e97ee85e1a54b3c45c5ed814f500b0a/parquet/src/file/footer.rs#L99-L101 If we can filter out the row group early based on the column stats, we can easily skip those row groups. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
