etseidl commented on PR #11159:
URL: https://github.com/apache/arrow-rs/pull/11159#issuecomment-5780045620
@adriangb I had Codex create a simple point lookup test for me so I could do
some benchmarking at varying table widths. The code generates a table with a
specified number of rows and columns, then does a series of random point
lookups filtering on a "key" column that is unique and sorted. The queries
return a varying number of columns (the "projected" column below). It's naive,
so each query opens the file, parses the footer metadata, and then optionally
parses the page index (both a full scan a la what's done currently, and a read
that's targeted to just the predicate and projected columns). Pruning is either
not done at all, done at the row group level using the chunk metadata, or done
at the page level using the index. (Note this is against a file on NVMe, fully
in buffer cache, so it's really just measuring the decoding work. Files in
object store would change the calculus immensely :P).
At 100 columns, things look pretty ok, but even at this size reading the
full page index slows things down relative to just using the column stats. The
page index pruning clearly helps here.
```
projected pruning metadata size footer/query index
load index eval data/query total/query
1 none 456.30 KiB 687.412µs
0ns 0ns 13.695408ms 14.451238ms
1 row group 456.30 KiB 599.734µs
0ns 0ns 1.454493ms 2.117207ms
1 page index (full) 1012.61 KiB 588.467µs
1.200928ms 920ns 601.318µs 2.456797ms
1 page index (selective) 457.08 KiB 595.609µs
91.832µs 842ns 608.451µs 1.359397ms
25 none 456.30 KiB 661.783µs
0ns 0ns 40.989044ms 41.72175ms
25 row group 456.30 KiB 676.718µs
0ns 0ns 8.570402ms 9.316573ms
25 page index (full) 1012.61 KiB 633.917µs
1.372732ms 1.118µs 6.442088ms 8.519533ms
25 page index (selective) 461.50 KiB 677.392µs
115.79µs 1.003µs 5.989914ms 6.851483ms
```
At 1000 columns you begin to see metadata times dominating the query time.
The selective page index is still beating row group pruning, but reading the
entire index is a big L.
```
projected pruning metadata size footer/query index
load index eval data/query total/query
1 none 4.45 MiB 6.487408ms
0ns 0ns 15.08447ms 21.897408ms
1 row group 4.45 MiB 6.50841ms
0ns 0ns 1.841498ms 8.66429ms
1 page index (full) 9.88 MiB 5.747513ms
16.112672ms 1.421µs 1.200488ms 23.402837ms
1 page index (selective) 4.45 MiB 5.611439ms
275.741µs 1.095µs 730.992µs 6.9187ms
25 none 4.45 MiB 6.461029ms
0ns 0ns 45.832282ms 52.61971ms
25 row group 4.45 MiB 7.006866ms
0ns 0ns 10.332351ms 17.681206ms
25 page index (full) 9.88 MiB 6.549746ms
17.171582ms 1.476µs 7.764038ms 31.846389ms
25 page index (selective) 4.45 MiB 6.285695ms
348.461µs 1.229µs 6.565722ms 13.510559ms
```
This continues at the 10000 column size. For a single projected column, even
the filtered page index barely loses to simply using column stats, but by 25
columns it squeaks ahead. Again, reading the full page index by this point is
worse than not pruning at all!
```
projected pruning metadata size footer/query index
load index eval data/query total/query
1 none 44.46 MiB 61.37101ms
0ns 0ns 16.463509ms 80.517332ms
1 row group 44.46 MiB 57.46164ms
0ns 0ns 2.734904ms 62.666304ms
1 page index (full) 98.77 MiB 55.465729ms
158.227488ms 1.914µs 1.78146ms 218.15268ms
1 page index (selective) 44.46 MiB 59.214822ms
1.969541ms 1.702µs 1.79331ms 65.421946ms
25 none 44.46 MiB 67.72911ms
0ns 0ns 48.087823ms 118.719795ms
25 row group 44.46 MiB 67.420542ms
0ns 0ns 12.070592ms 82.314726ms
25 page index (full) 98.77 MiB 63.277779ms
175.549233ms 2.056µs 9.84537ms 251.711592ms
25 page index (selective) 44.46 MiB 63.131708ms
2.451171ms 1.934µs 8.800725ms 77.007979ms
```
So I think the advantage to pruning the page index is pretty apparent here.
The need for being able to selectively populate the file metadata is also
pretty clear. In the short term we could try the skip index route, but I'm
cautiously optimistic we might get a redo of the footer metadata done right.
A side effect of this study was seeing how difficult it was to get decent
query performance out of the existing APIs. Codex originally went through the
ArrowReaderMetadata API, but that at first resulted in re-decoding the footer
metadata, and later cloning it. I switched to using the ParquetMetaDataReader
where I could re-use the footer metadata directly when reading the page index
and that helped immensely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
