andygrove commented on issue #3457: URL: https://github.com/apache/datafusion-comet/issues/3457#issuecomment-5034646880
Closing this in favour of #4988, which carries the investigation forward with measurements. Most of what this issue asked for already shipped in #4722: `COMET_RESPECT_PARQUET_FILTER_PUSHDOWN` is gone, and format-level pruning (row group statistics, page index, bloom filters) now runs by default whenever Spark's `spark.sql.parquet.filterPushdown` is enabled. The chart in the description predates that change and the DataFusion 54 upgrade, so it no longer describes current behaviour. What remains is narrower than the original title: row-level `RowFilter` evaluation plus late materialization, behind `spark.comet.parquet.rowFilterPushdown.enabled` (default `false`). I benchmarked that flag on TPC-H SF100 Q1 and Q6. Q1 is roughly neutral, Q6 regresses 47-81% depending on cache state, and scan metrics show why: bytes scanned is byte-for-byte identical with the flag on, while decode time is 2.5x higher, because Q6's predicate is uncorrelated with physical row order so late materialization can never eliminate a page. Details, per-iteration scan metrics, and proposed work items are in #4988. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
