ndemir opened a new pull request, #7360:
URL: https://github.com/apache/arrow-rs/pull/7360
# Which issue does this PR close?
This is NOT closing an issue yet.
I am opening this issue to illustrate how the #7348 can be solved.
# Rationale for this change
I run tests
arrow::arrow_reader::tests::test_predicate_pushdown_vs_row_filter and I can see
the performance increase clearly.
TEST.1
=== PERFORMANCE COMPARISON ===
Filter type | Time | Row count
-------------------|-----------|----------
Row Filter | 21.144189ms | 590
Predicate Pushdown| 15.067947ms | 590
Improvement | 1.40x |
TEST.2
=== PERFORMANCE COMPARISON ===
Filter type | Time | Row count
-------------------|-----------|----------
Row Filter | 26.991091ms | 80
Predicate Pushdown| 7.604246ms | 80
Improvement | 3.55x |
# What changes are included in this PR?
In this PR, we have a) filtering out the row groups that we do not need b)
then creating the predicates and adding them to RowFilter
# Are there any user-facing changes?
Yes, we will have, once the full implementation is completed.
This PR is just to show a possible solution for #7348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
