alamb commented on issue #13983:
URL: https://github.com/apache/datafusion/issues/13983#issuecomment-2612198384
And in this case enabling predicate pushdown results in a 2x speedup
```sql
set datafusion.execution.parquet.pushdown_filters = false;
SELECT * FROM hits_partitioned WHERE "URL" LIKE '%google%' ORDER BY
to_timestamp_seconds("EventTime") LIMIT 10;
```
Elapsed 4.108 seconds.
Elapsed 5.430 seconds.
Elapsed 4.659 seconds.
```sql
set datafusion.execution.parquet.pushdown_filters = true;
SELECT * FROM hits_partitioned WHERE "URL" LIKE '%google%' ORDER BY
to_timestamp_seconds("EventTime") LIMIT 10;
```
Elapsed 2.415 seconds.
Elapsed 2.070 seconds.
Elapsed 2.279 seconds.
Here is the flamegraph for no pushdown:
It would be cool to test with @XiangpengHao 's change to the parquet decoder
here:
- https://github.com/apache/arrow-rs/pull/6921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
