Omega359 commented on PR #17276: URL: https://github.com/apache/datafusion/pull/17276#issuecomment-3211985591
Here is an instrumented output from running the new logical_plan_optimize benchmark. Compared to the results I see with my actual code the type_coercion in execute_and_check is much faster (I suspect it's because the benchmark isn't using a ListingTable with many files behind it) however the pushdown filter rules takes more time. Adjusting the column count up in the benchmark increases the time significantly. ``` Benchmarking logical_plan_optimize: Warming up for 3.0000 s[2025-08-21T20:17:34Z INFO datafusion_optimizer::analyzer] Analyzer::execute_and_check rule resolve_grouping_function took 464 ms [2025-08-21T20:17:37Z INFO datafusion_optimizer::analyzer] Analyzer::execute_and_check rule type_coercion took 2953 ms [2025-08-21T20:17:37Z INFO datafusion_optimizer::analyzer] Analyzer::execute_and_check rule processing took 3417 ms [2025-08-21T20:17:37Z INFO datafusion_optimizer::analyzer] Analyzer::execute_and_check took 3619 ms [2025-08-21T20:17:38Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_nested_union took > 50ms: 180 ms [2025-08-21T20:17:40Z INFO datafusion_optimizer::optimizer] Optimization for rule simplify_expressions took > 50ms: 2038 ms [2025-08-21T20:17:40Z INFO datafusion_optimizer::optimizer] Optimization for rule replace_distinct_aggregate took > 50ms: 158 ms [2025-08-21T20:17:40Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_join took > 50ms: 159 ms [2025-08-21T20:17:41Z INFO datafusion_optimizer::optimizer] Optimization for rule decorrelate_predicate_subquery took > 50ms: 293 ms [2025-08-21T20:17:41Z INFO datafusion_optimizer::optimizer] Optimization for rule scalar_subquery_to_join took > 50ms: 202 ms [2025-08-21T20:17:41Z INFO datafusion_optimizer::optimizer] Optimization for rule decorrelate_lateral_join took > 50ms: 158 ms [2025-08-21T20:17:41Z INFO datafusion_optimizer::optimizer] Optimization for rule extract_equijoin_predicate took > 50ms: 159 ms [2025-08-21T20:17:41Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_duplicated_expr took > 50ms: 159 ms [2025-08-21T20:17:42Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_filter took > 50ms: 159 ms [2025-08-21T20:17:42Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_limit took > 50ms: 159 ms [2025-08-21T20:17:42Z INFO datafusion_optimizer::optimizer] Optimization for rule propagate_empty_relation took > 50ms: 160 ms [2025-08-21T20:17:42Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_one_union took > 50ms: 162 ms [2025-08-21T20:17:42Z INFO datafusion_optimizer::optimizer] Optimization for rule filter_null_join_keys took > 50ms: 163 ms [2025-08-21T20:17:42Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_outer_join took > 50ms: 160 ms [2025-08-21T20:17:43Z INFO datafusion_optimizer::optimizer] Optimization for rule push_down_limit took > 50ms: 160 ms [2025-08-21T20:19:24Z INFO datafusion_optimizer::optimizer] Optimization for rule push_down_filter took > 50ms: 101005 ms [2025-08-21T20:19:24Z INFO datafusion_optimizer::optimizer] Optimization for rule single_distinct_aggregation_to_group_by took > 50ms: 290 ms [2025-08-21T20:19:24Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_group_by_constant took > 50ms: 277 ms [2025-08-21T20:19:25Z INFO datafusion_optimizer::optimizer] Optimization for rule common_sub_expression_eliminate took > 50ms: 946 ms [2025-08-21T20:19:30Z INFO datafusion_optimizer::optimizer] Optimization for rule optimize_projections took > 50ms: 4573 ms [2025-08-21T20:19:30Z INFO datafusion_optimizer::optimizer] Processing time for pass 0 was 112145ms [2025-08-21T20:19:30Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_nested_union took > 50ms: 267 ms [2025-08-21T20:19:32Z INFO datafusion_optimizer::optimizer] Optimization for rule simplify_expressions took > 50ms: 1898 ms [2025-08-21T20:19:32Z INFO datafusion_optimizer::optimizer] Optimization for rule replace_distinct_aggregate took > 50ms: 145 ms [2025-08-21T20:19:32Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_join took > 50ms: 145 ms [2025-08-21T20:19:33Z INFO datafusion_optimizer::optimizer] Optimization for rule decorrelate_predicate_subquery took > 50ms: 242 ms [2025-08-21T20:19:33Z INFO datafusion_optimizer::optimizer] Optimization for rule scalar_subquery_to_join took > 50ms: 170 ms [2025-08-21T20:19:33Z INFO datafusion_optimizer::optimizer] Optimization for rule decorrelate_lateral_join took > 50ms: 144 ms [2025-08-21T20:19:33Z INFO datafusion_optimizer::optimizer] Optimization for rule extract_equijoin_predicate took > 50ms: 144 ms [2025-08-21T20:19:33Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_duplicated_expr took > 50ms: 144 ms [2025-08-21T20:19:33Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_filter took > 50ms: 144 ms [2025-08-21T20:19:34Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_limit took > 50ms: 144 ms [2025-08-21T20:19:34Z INFO datafusion_optimizer::optimizer] Optimization for rule propagate_empty_relation took > 50ms: 291 ms [2025-08-21T20:19:34Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_one_union took > 50ms: 101 ms [2025-08-21T20:19:34Z INFO datafusion_optimizer::optimizer] Optimization for rule filter_null_join_keys took > 50ms: 94 ms [2025-08-21T20:19:34Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_outer_join took > 50ms: 94 ms [2025-08-21T20:19:34Z INFO datafusion_optimizer::optimizer] Optimization for rule push_down_limit took > 50ms: 94 ms [2025-08-21T20:19:35Z INFO datafusion_optimizer::optimizer] Optimization for rule push_down_filter took > 50ms: 291 ms [2025-08-21T20:19:35Z INFO datafusion_optimizer::optimizer] Optimization for rule single_distinct_aggregation_to_group_by took > 50ms: 99 ms [2025-08-21T20:19:35Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_group_by_constant took > 50ms: 98 ms [2025-08-21T20:19:35Z INFO datafusion_optimizer::optimizer] Optimization for rule common_sub_expression_eliminate took > 50ms: 337 ms [2025-08-21T20:19:37Z INFO datafusion_optimizer::optimizer] Optimization for rule optimize_projections took > 50ms: 1628 ms [2025-08-21T20:19:37Z INFO datafusion_optimizer::optimizer] Processing time for pass 1 was 6904ms [2025-08-21T20:19:37Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_nested_union took > 50ms: 107 ms [2025-08-21T20:19:38Z INFO datafusion_optimizer::optimizer] Optimization for rule simplify_expressions took > 50ms: 711 ms [2025-08-21T20:19:38Z INFO datafusion_optimizer::optimizer] Optimization for rule replace_distinct_aggregate took > 50ms: 84 ms [2025-08-21T20:19:38Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_join took > 50ms: 81 ms [2025-08-21T20:19:38Z INFO datafusion_optimizer::optimizer] Optimization for rule decorrelate_predicate_subquery took > 50ms: 124 ms [2025-08-21T20:19:38Z INFO datafusion_optimizer::optimizer] Optimization for rule scalar_subquery_to_join took > 50ms: 92 ms [2025-08-21T20:19:38Z INFO datafusion_optimizer::optimizer] Optimization for rule decorrelate_lateral_join took > 50ms: 85 ms [2025-08-21T20:19:38Z INFO datafusion_optimizer::optimizer] Optimization for rule extract_equijoin_predicate took > 50ms: 83 ms [2025-08-21T20:19:38Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_duplicated_expr took > 50ms: 82 ms [2025-08-21T20:19:38Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_filter took > 50ms: 82 ms [2025-08-21T20:19:39Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_limit took > 50ms: 82 ms [2025-08-21T20:19:39Z INFO datafusion_optimizer::optimizer] Optimization for rule propagate_empty_relation took > 50ms: 82 ms [2025-08-21T20:19:39Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_one_union took > 50ms: 82 ms [2025-08-21T20:19:39Z INFO datafusion_optimizer::optimizer] Optimization for rule filter_null_join_keys took > 50ms: 82 ms [2025-08-21T20:19:39Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_outer_join took > 50ms: 82 ms [2025-08-21T20:19:39Z INFO datafusion_optimizer::optimizer] Optimization for rule push_down_limit took > 50ms: 82 ms [2025-08-21T20:19:39Z INFO datafusion_optimizer::optimizer] Optimization for rule push_down_filter took > 50ms: 285 ms [2025-08-21T20:19:39Z INFO datafusion_optimizer::optimizer] Optimization for rule single_distinct_aggregation_to_group_by took > 50ms: 80 ms [2025-08-21T20:19:39Z INFO datafusion_optimizer::optimizer] Optimization for rule eliminate_group_by_constant took > 50ms: 79 ms [2025-08-21T20:19:40Z INFO datafusion_optimizer::optimizer] Optimization for rule common_sub_expression_eliminate took > 50ms: 283 ms [2025-08-21T20:19:41Z INFO datafusion_optimizer::optimizer] Optimization for rule optimize_projections took > 50ms: 1261 ms [2025-08-21T20:19:41Z INFO datafusion_optimizer::optimizer] Processing time for pass 2 was 4183ms [2025-08-21T20:19:41Z INFO datafusion_optimizer::optimizer] Processing time for 3 passes was 123504ms [2025-08-21T20:19:41Z INFO datafusion_optimizer::optimizer] Optimizer took 123599 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
