etseidl commented on PR #11157: URL: https://github.com/apache/arrow-rs/pull/11157#issuecomment-6009734153
Thanks @alamb! I opened #11395 to track unifying the policy and mask, and left TODO's in the code so I remember all of the places that need updating. > Longer term I think we should come up with an example of how someone would incrementally load parts of the page index "on demand" -- maybe we should pull it into its own "PageIndexParser" or something (and similarly for BloomFilter ...) 🤔 I've got an addition to the `ParquetMetaDataPushDecoder` pretty much ready to go that will only parse the page indexes...hopefully that will be more ergonomic for folks. When coupled with the cloneable `ParquetMetaData`, I think we'll have all the pieces needed for downstream users to create a caching page index provider. I have started thinking about the bloom filters. Discovered there are 3 implmentations of read-the-bloom-filter-for-rowgroup-and-column, one for `Read`, one for `ChunkReader`, and one for `AsyncFileReader`. I think that's ripe for pushing into the push decoder. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
