DhamoPS opened a new pull request, #3482:
URL: https://github.com/apache/arrow-datafusion/pull/3482
* Added the new optimizer rule reduce_cross_join which would convert
cross joins to inner joins if the filter has the join predicates for the
corresponding tables.
* Join Predicates which are part of OR expressions of FILTER PREDICATES,
were not pulled properly. Due to that, CROSS JOINs are chosen for those joins
instead of innerjoin and causes query to run longer. Actually TPCH Q19 was
taking more than 2000 sec in my laptop. With Fix, TPCH Q19 is taking ~3 sec.
# Which issue does this PR close?
-- reduce_cross_join.rs is the new optimizer rule added to pull the
Joined predicates from Filters and convert cross_join into inner_join
-- tests/sql/subqueries.rs::added testcases for the above fix
Closes #78.
# Rationale for this change
In `[PR#3334`](https://github.com/apache/arrow-datafusion/pull/3334), I have
tried to fix this issue in planner.rs. But to handle DataFrameAPI queries, we
have decided to move this rule to optimizer.
# What changes are included in this PR?
* new optimizer rule reduce_cross_join
* unit test cases for the optimizer rule
* test cases by running SQL stmts
Logs of TPCH Q19:
arrow-datafusion/target/release/tpch benchmark datafusion --iterations 3
--path ./data --format tbl --query 19 --batch-size 4096`
Running benchmarks with the following options: DataFusionBenchmarkOpt {
query: 19, debug: false, iterations: 3, partitions: 2, batch_size: 4096, path:
"./data", file_format: "tbl", mem_table: false, output_path: None }
Query 19 iteration 0 took 2640.2 ms and returned 1 rows
Query 19 iteration 1 took 2099.2 ms and returned 1 rows
Query 19 iteration 2 took 2199.9 ms and returned 1 rows
Query 19 avg time: 2313.08 ms
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
