etseidl commented on PR #11159: URL: https://github.com/apache/arrow-rs/pull/11159#issuecomment-5768417631
> > Reads of the data, or just reads of the page index? > > Presumably both would be valuable, but I'm not sure how much we do full e2e data read benchmarks here vs. datafusion. Even if we just had the page index ones (which we could easily merge as a precursor PR) then it would strengthen the story here. I asked only because I saw [#11159 (comment)](https://github.com/apache/arrow-rs/pull/11159#issuecomment-5767106264) was basically flat but I assume it's not measuring the thing we are tryign to improve here anwya. Ahh, thanks. Answering in reverse order, yeah, that bench (added in #11157) is just demonstrating that selectively reading the page index is much faster than reading the whole thing. The run you referenced is just showing that using the Grid didn't negatively impact reading the thing. We don't have a whole lot of end-to-end benchmarks in this crate, and I'm pretty sure the ones we do have don't do any pruning. I agree it would be nice to have something to demonstrate the value here. With this particular PR, I think the value comes more from saving memory without adversely affecting performance. To that end, we probably should add some kind of a bench that does range based reads with column projection to see if the metadata handling enters into the equation in a meaningful way. I'm sure at huge scales it will. Maybe I'll have Codex/Claude whip up a test for me 😅 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
