viirya commented on PR #58050: URL: https://github.com/apache/spark/pull/58050#issuecomment-5332103359
Thanks @qlong -- and you're right that the lift only shows up when the field is shredded, the predicate is a literal, the data is sorted on that field, and a file has many row groups, which is why the join-key / one-row-group-per-file workloads didn't move. I added a benchmark (`VariantShreddedPredicatePushdownBenchmark`, 60e877a) that sets up exactly that layout -- a shredded `bigint` field sorted ascending, small block size so one file holds many row groups -- and measures a literal predicate on/off. Results (JDK 21, Apple M4 Max, 20M rows): ``` Can skip all row groups: 774ms -> 36ms 21.6x Can skip some row groups: 841ms -> 46ms 18.3x Can skip no row groups: 1003ms -> 1044ms 1.0x ``` So when the layout cooperates the skip is a large win, and the "skip no row groups" case shows negligible overhead (~4%, within noise) when nothing can be skipped. It's reproducible via `build/sbt "sql/Test/runMain org.apache.spark.sql.execution.benchmark.VariantShreddedPredicatePushdownBenchmark"`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
