[
https://issues.apache.org/jira/browse/SPARK-33798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252185#comment-17252185
]
Yuming Wang commented on SPARK-33798:
-------------------------------------
{noformat}
=== Metrics of Analyzer/Optimizer Rules ===
Total number of runs: 8160599
Total time: 384.166067212 seconds
Rule
Effective Time / Total Time Effective
Runs / Total Runs
org.apache.spark.sql.catalyst.optimizer.Optimizer$OptimizeSubqueries
39693600575 / 46725545597 1125 / 27003
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations
31810498644 / 32443489761 11315 /
77746
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveSubquery
20255122691 / 21209087230 1260 / 77578
org.apache.spark.sql.catalyst.optimizer.ColumnPruning
3662408905 / 14865979018 6250 / 64063
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions
3839545719 / 8615434071 535 / 77559
org.apache.spark.sql.execution.datasources.FindDataSourceTable
5926386338 / 6266041323 3133 / 77413
org.apache.spark.sql.catalyst.optimizer.PushDownPredicates
1004935400 / 5318937518 3059 / 64077
org.apache.spark.sql.catalyst.optimizer.PruneFilters
20196806 / 5197169487 165 / 50348
org.apache.spark.sql.catalyst.optimizer.RemoveNoopOperators
382393781 / 4495401710 2631 / 63636
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences
2223564118 / 4457762727 18153 /
77660
org.apache.spark.sql.catalyst.analysis.Analyzer$AddMetadataColumns
0 / 4449402746 0 / 77746
org.apache.spark.sql.catalyst.optimizer.CollapseProject
1632101108 / 4327933620 7632 / 50775
org.apache.spark.sql.catalyst.optimizer.RemoveDispensableExpressions
1410614 / 3723070497 14 / 37060
org.apache.spark.sql.catalyst.optimizer.SimplifyCasts
858952677 / 3679387506 4398 / 37060
org.apache.spark.sql.catalyst.optimizer.ConstantFolding
882780177 / 3584991204 3906 / 37060
org.apache.spark.sql.catalyst.optimizer.CollapseRepartition
0 / 3525788470 0 / 37487
org.apache.spark.sql.catalyst.optimizer.CombineUnions
14386207 / 3484693787 48 / 51202
org.apache.spark.sql.catalyst.optimizer.UnwrapCastInBinaryComparison
46878113 / 3445978351 125 / 37060
org.apache.spark.sql.catalyst.optimizer.LikeSimplification
1794827 / 3415501538 7 / 37060
org.apache.spark.sql.catalyst.optimizer.OptimizeJsonExprs
0 / 3313864979 0 / 37060
org.apache.spark.sql.catalyst.optimizer.BooleanSimplification
20310663 / 3306249541 86 / 37060
org.apache.spark.sql.catalyst.optimizer.NullPropagation
60290890 / 3192991337 429 / 37487
org.apache.spark.sql.catalyst.optimizer.EliminateOuterJoin
390401546 / 3189932899 310 / 37487
org.apache.spark.sql.catalyst.optimizer.CollapseWindow
0 / 3098148291 0 / 37487
org.apache.spark.sql.catalyst.optimizer.EliminateResolvedHint
0 / 3084228124 0 / 13715
org.apache.spark.sql.catalyst.optimizer.ReorderJoin
24282784 / 3080214384 37 / 37487
org.apache.spark.sql.catalyst.optimizer.SimplifyBinaryComparison
6555738 / 3076719247 28 / 37060
org.apache.spark.sql.catalyst.optimizer.ReorderAssociativeOperator
0 / 3038611489 0 / 37060
org.apache.spark.sql.catalyst.optimizer.OptimizeIn
1377857 / 3014960087 8 / 37487
org.apache.spark.sql.catalyst.optimizer.SimplifyConditionals
14895081 / 2965371144 187 / 37060
org.apache.spark.sql.catalyst.optimizer.PushFoldableIntoBranches
5542395 / 2958097009 15 / 37060
{noformat}
> Simplify EqualTo(CaseWhen/If, Literal) always false
> ---------------------------------------------------
>
> Key: SPARK-33798
> URL: https://issues.apache.org/jira/browse/SPARK-33798
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Yuming Wang
> Assignee: Yuming Wang
> Priority: Major
> Fix For: 3.2.0
>
>
> Simplify CaseWhen/If with EqualTo if all values are Literal and always false,
> this is a real case from production:
> {code:sql}
> create table t1 using parquet as select * from range(100);
> create table t2 using parquet as select * from range(200);
> create temp view v1 as
>
> select 'a' as event_type, * from t1
>
> union all
>
> select CASE WHEN id = 1 THEN 'b' WHEN id = 3 THEN 'c' end as event_type, *
> from t2
> explain select * from v1 where event_type = 'a';
> {code}
> Before this PR:
> {noformat}
> == Physical Plan ==
> Union
> :- *(1) Project [a AS event_type#30533, id#30535L]
> : +- *(1) ColumnarToRow
> : +- FileScan parquet default.t1[id#30535L] Batched: true, DataFilters:
> [], Format: Parquet
> +- *(2) Project [CASE WHEN (id#30536L = 1) THEN b WHEN (id#30536L = 3) THEN c
> END AS event_type#30534, id#30536L]
> +- *(2) Filter (CASE WHEN (id#30536L = 1) THEN b WHEN (id#30536L = 3) THEN
> c END = a)
> +- *(2) ColumnarToRow
> +- FileScan parquet default.t2[id#30536L] Batched: true,
> DataFilters: [(CASE WHEN (id#30536L = 1) THEN b WHEN (id#30536L = 3) THEN c
> END = a)], Format: Parquet
> {noformat}
> After this PR:
> {noformat}
> == Physical Plan ==
> *(1) Project [a AS event_type#8, id#4L]
> +- *(1) ColumnarToRow
> +- FileScan parquet default.t1[id#4L] Batched: true, DataFilters: [],
> Format: Parquet
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
