zhuqi-lucas commented on issue #11384: URL: https://github.com/apache/arrow-rs/issues/11384#issuecomment-6098310611
Thanks @alamb — it does not change that, and I think the two are complementary rather than overlapping. The page-index fetch planning is `ColumnFetch` in `parquet/src/arrow/in_memory_row_group.rs`. The selection prunes data pages, but `dictionary_range` never looks at it — the dictionary range is always chained into `ranges()`, even when no data page is selected. So @Abhisheklearn12's statement is literally what the code does. #11168 does not touch that file. It changes `column/reader.rs` and `serialized_reader.rs` — what happens to the bytes *after* they are fetched. On the async path the chunk ranges are pulled up front, so it saves the dictionary's decompression, not the read. (On the sync path, where `get_bytes` is per page, a chunk skipped end to end does avoid the read as well.) So for `WHERE id BETWEEN 1000 AND 1100` the 2.1 MB dictionary page is still fetched. This issue removes it at write time, which nothing on the read side can do. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
