wangyum commented on issue #27518: [WIP][SPARK-30768][SQL] Constraints should be inferred from inequality attributes URL: https://github.com/apache/spark/pull/27518#issuecomment-586929232 Metrics of Analyzer/Optimizer Rules for `TPCDSQuerySuite` Before this PR: ```scala 02:32:26.475 WARN org.apache.spark.sql.TPCDSQuerySuite: === Metrics of Analyzer/Optimizer Rules === Total number of runs: 224786 Total time: 45.803692546 seconds Rule Effective Time / Total Time Effective Runs / Total Runs org.apache.spark.sql.catalyst.optimizer.Optimizer$OptimizeSubqueries 6898974687 / 8840587642 47 / 772 org.apache.spark.sql.catalyst.optimizer.ColumnPruning 671684614 / 2740648387 327 / 2364 org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveSubquery 1637851386 / 1780942834 51 / 2159 org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions 624455301 / 1744459696 49 / 2159 org.apache.spark.sql.catalyst.analysis.DecimalPrecision 1136163267 / 1355664883 361 / 2159 org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences 958843179 / 1132184862 813 / 2159 org.apache.spark.sql.catalyst.optimizer.PruneFilters 21041535 / 875480663 5 / 1978 org.apache.spark.sql.catalyst.optimizer.BooleanSimplification 7949392 / 757989983 4 / 1592 org.apache.spark.sql.catalyst.analysis.TypeCoercion$ImplicitTypeCasts 301831588 / 657645402 78 / 2159 org.apache.spark.sql.catalyst.optimizer.PushDownPredicates 374351767 / 641828516 758 / 1979 org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveMissingReferences 14541719 / 632536654 10 / 2159 org.apache.spark.sql.catalyst.optimizer.NullPropagation 39937489 / 587603524 42 / 1592 org.apache.spark.sql.catalyst.optimizer.ReorderJoin 244728546 / 572349474 177 / 1592 org.apache.spark.sql.catalyst.optimizer.ConstantFolding 158687531 / 564225840 194 / 1592 org.apache.spark.sql.catalyst.optimizer.ReorderAssociativeOperator 0 / 547165121 0 / 1592 org.apache.spark.sql.catalyst.optimizer.SimplifyConditionals 0 / 532921745 0 / 1592 org.apache.spark.sql.catalyst.optimizer.SimplifyBinaryComparison 0 / 527530671 0 / 1592 org.apache.spark.sql.catalyst.optimizer.RemoveNoopOperators 48706059 / 526075410 116 / 2364 org.apache.spark.sql.catalyst.optimizer.SimplifyExtractValueOps 0 / 511807420 0 / 1592 org.apache.spark.sql.catalyst.optimizer.SimplifyCasts 45722796 / 510124455 83 / 1592 org.apache.spark.sql.catalyst.optimizer.SimplifyCaseConversionExpressions 0 / 493219669 0 / 1592 org.apache.spark.sql.catalyst.optimizer.OptimizeIn 12300150 / 491448204 27 / 1592 org.apache.spark.sql.catalyst.analysis.UpdateAttributeNullability 20208595 / 484413473 12 / 674 org.apache.spark.sql.catalyst.optimizer.RemoveDispensableExpressions 0 / 480755936 0 / 1592 org.apache.spark.sql.catalyst.optimizer.LikeSimplification 799298 / 463609379 1 / 1592 org.apache.spark.sql.catalyst.optimizer.CollapseProject 92682031 / 457556748 215 / 1978 org.apache.spark.sql.catalyst.optimizer.ReplaceNullWithFalseInPredicate 0 / 456658852 0 / 1592 org.apache.spark.sql.catalyst.analysis.TypeCoercion$FunctionArgumentConversion 227217097 / 442233854 56 / 2159 org.apache.spark.sql.catalyst.optimizer.InferFiltersFromConstraints 406969136 / 435459925 278 / 386 ``` After this PR: ```scala 02:28:49.937 WARN org.apache.spark.sql.TPCDSQuerySuite: === Metrics of Analyzer/Optimizer Rules === Total number of runs: 224786 Total time: 47.011460872 seconds Rule Effective Time / Total Time Effective Runs / Total Runs org.apache.spark.sql.catalyst.optimizer.Optimizer$OptimizeSubqueries 7073196527 / 8950854926 47 / 772 org.apache.spark.sql.catalyst.optimizer.ColumnPruning 724531405 / 2931435267 327 / 2364 org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveSubquery 1789988207 / 1942196179 51 / 2159 org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions 717122415 / 1838219499 49 / 2159 org.apache.spark.sql.catalyst.analysis.DecimalPrecision 1276704718 / 1524939842 361 / 2159 org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences 1003010234 / 1192920997 813 / 2159 org.apache.spark.sql.catalyst.optimizer.PruneFilters 21471500 / 952660373 5 / 1978 org.apache.spark.sql.catalyst.optimizer.BooleanSimplification 9761473 / 773205469 4 / 1592 org.apache.spark.sql.catalyst.analysis.TypeCoercion$ImplicitTypeCasts 329430374 / 734067405 78 / 2159 org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveMissingReferences 12112700 / 709719203 10 / 2159 org.apache.spark.sql.catalyst.optimizer.PushDownPredicates 399755753 / 663934266 758 / 1979 org.apache.spark.sql.catalyst.optimizer.ReorderJoin 266383835 / 593070778 177 / 1592 org.apache.spark.sql.catalyst.optimizer.NullPropagation 35781183 / 578417823 42 / 1592 org.apache.spark.sql.catalyst.optimizer.RemoveNoopOperators 48387593 / 540807035 116 / 2364 org.apache.spark.sql.catalyst.optimizer.SimplifyBinaryComparison 0 / 535708634 0 / 1592 org.apache.spark.sql.catalyst.optimizer.ConstantFolding 114115625 / 533552246 194 / 1592 org.apache.spark.sql.catalyst.optimizer.OptimizeIn 12632788 / 529487507 27 / 1592 org.apache.spark.sql.catalyst.optimizer.SimplifyCaseConversionExpressions 0 / 513745956 0 / 1592 org.apache.spark.sql.catalyst.optimizer.SimplifyConditionals 0 / 508586160 0 / 1592 org.apache.spark.sql.catalyst.optimizer.InferFiltersFromConstraints 459543325 / 490828096 278 / 386 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
