wangyum commented on a change in pull request #22038: [SPARK-25056][SQL] Unify
the InConversion and BinaryComparison behavior
URL: https://github.com/apache/spark/pull/22038#discussion_r373838773
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
##########
@@ -495,9 +495,12 @@ object TypeCoercion {
i
}
- case i @ In(a, b) if b.exists(_.dataType != a.dataType) =>
- findWiderCommonType(i.children.map(_.dataType)) match {
- case Some(finalDataType) => i.withNewChildren(i.children.map(Cast(_,
finalDataType)))
+ case i @ In(value, list) if list.exists(_.dataType != value.dataType) =>
+ findWiderCommonType(list.map(_.dataType)) match {
Review comment:
@cloud-fan @maropu It seems the logic is `PromoteStrings` ->
`DecimalPrecision` -> `ImplicitTypeCasts`:
```scala
scala> spark.conf.set("spark.sql.optimizer.planChangeLog.level", "WARN")
scala> spark.sql("select bigint(1) > double(2), 1 > '1', bigint(1) > cast(1
as decimal(10, 0)) ").explain
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1: Batch Hints has no
effect.
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1: Batch Simple Sanity
Check has no effect.
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1: Batch Substitution
has no effect.
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions ===
!'Project [unresolvedalias(('bigint(1) > 'double(2)), None),
unresolvedalias((1 > 1), None), unresolvedalias(('bigint(1) > cast(1 as
decimal(10,0))), None)] 'Project [unresolvedalias((cast(1 as bigint) > cast(2
as double)), None), unresolvedalias((1 > 1), None), unresolvedalias((cast(1 as
bigint) > cast(1 as decimal(10,0))), None)]
+- OneRowRelation
+- OneRowRelation
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule org.apache.spark.sql.catalyst.analysis.ResolveTimeZone ===
'Project [unresolvedalias((cast(1 as bigint) > cast(2 as double)), None),
unresolvedalias((1 > 1), None), unresolvedalias((cast(1 as bigint) > cast(1 as
decimal(10,0))), None)] 'Project [unresolvedalias((cast(1 as bigint) > cast(2
as double)), None), unresolvedalias((1 > 1), None), unresolvedalias((cast(1 as
bigint) > cast(1 as decimal(10,0))), None)]
+- OneRowRelation
+- OneRowRelation
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule
org.apache.spark.sql.catalyst.analysis.TypeCoercion$PromoteStrings ===
!'Project [unresolvedalias((cast(1 as bigint) > cast(2 as double)), None),
unresolvedalias((1 > 1), None), unresolvedalias((cast(1 as bigint) > cast(1 as
decimal(10,0))), None)] 'Project [unresolvedalias((cast(1 as bigint) > cast(2
as double)), None), unresolvedalias((1 > cast(1 as int)), None),
unresolvedalias((cast(1 as bigint) > cast(1 as decimal(10,0))), None)]
+- OneRowRelation
+- OneRowRelation
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule org.apache.spark.sql.catalyst.analysis.DecimalPrecision ===
!'Project [unresolvedalias((cast(1 as bigint) > cast(2 as double)), None),
unresolvedalias((1 > cast(1 as int)), None), unresolvedalias((cast(1 as bigint)
> cast(1 as decimal(10,0))), None)] 'Project [unresolvedalias((cast(1 as
bigint) > cast(2 as double)), None), unresolvedalias((1 > cast(1 as int)),
None), unresolvedalias((cast(cast(1 as bigint) as decimal(20,0)) > cast(1 as
decimal(10,0))), None)]
+- OneRowRelation
+- OneRowRelation
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule
org.apache.spark.sql.catalyst.analysis.TypeCoercion$ImplicitTypeCasts ===
!'Project [unresolvedalias((cast(1 as bigint) > cast(2 as double)), None),
unresolvedalias((1 > cast(1 as int)), None), unresolvedalias((cast(cast(1 as
bigint) as decimal(20,0)) > cast(1 as decimal(10,0))), None)] 'Project
[unresolvedalias((cast(cast(1 as bigint) as double) > cast(2 as double)),
None), unresolvedalias((1 > cast(1 as int)), None),
unresolvedalias((cast(cast(1 as bigint) as decimal(20,0)) > cast(1 as
decimal(10,0))), None)]
+- OneRowRelationscala>
spark.conf.set("spark.sql.optimizer.planChangeLog.level", "WARN")
scala> spark.sql("select bigint(1) > double(2), 1 > '1', bigint(1) > cast(1
as decimal(10, 0)) ").explain
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1: Batch Hints has no
effect.
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1: Batch Simple Sanity
Check has no effect.
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1: Batch Substitution
has no effect.
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions ===
!'Project [unresolvedalias(('bigint(1) > 'double(2)), None),
unresolvedalias((1 > 1), None), unresolvedalias(('bigint(1) > cast(1 as
decimal(10,0))), None)] 'Project [unresolvedalias((cast(1 as bigint) > cast(2
as double)), None), unresolvedalias((1 > 1), None), unresolvedalias((cast(1 as
bigint) > cast(1 as decimal(10,0))), None)]
+- OneRowRelation
+- OneRowRelation
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule org.apache.spark.sql.catalyst.analysis.ResolveTimeZone ===
'Project [unresolvedalias((cast(1 as bigint) > cast(2 as double)), None),
unresolvedalias((1 > 1), None), unresolvedalias((cast(1 as bigint) > cast(1 as
decimal(10,0))), None)] 'Project [unresolvedalias((cast(1 as bigint) > cast(2
as double)), None), unresolvedalias((1 > 1), None), unresolvedalias((cast(1 as
bigint) > cast(1 as decimal(10,0))), None)]
+- OneRowRelation
+- OneRowRelation
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule
org.apache.spark.sql.catalyst.analysis.TypeCoercion$PromoteStrings ===
!'Project [unresolvedalias((cast(1 as bigint) > cast(2 as double)), None),
unresolvedalias((1 > 1), None), unresolvedalias((cast(1 as bigint) > cast(1 as
decimal(10,0))), None)] 'Project [unresolvedalias((cast(1 as bigint) > cast(2
as double)), None), unresolvedalias((1 > cast(1 as int)), None),
unresolvedalias((cast(1 as bigint) > cast(1 as decimal(10,0))), None)]
+- OneRowRelation
+- OneRowRelation
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule org.apache.spark.sql.catalyst.analysis.DecimalPrecision ===
!'Project [unresolvedalias((cast(1 as bigint) > cast(2 as double)), None),
unresolvedalias((1 > cast(1 as int)), None), unresolvedalias((cast(1 as bigint)
> cast(1 as decimal(10,0))), None)] 'Project [unresolvedalias((cast(1 as
bigint) > cast(2 as double)), None), unresolvedalias((1 > cast(1 as int)),
None), unresolvedalias((cast(cast(1 as bigint) as decimal(20,0)) > cast(1 as
decimal(10,0))), None)]
+- OneRowRelation
+- OneRowRelation
20/02/02 19:31:10 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule
org.apache.spark.sql.catalyst.analysis.TypeCoercion$ImplicitTypeCasts ===
!'Project [unresolvedalias((cast(1 as bigint) > cast(2 as double)), None),
unresolvedalias((1 > cast(1 as int)), None), unresolvedalias((cast(cast(1 as
bigint) as decimal(20,0)) > cast(1 as decimal(10,0))), None)] 'Project
[unresolvedalias((cast(cast(1 as bigint) as double) > cast(2 as double)),
None), unresolvedalias((1 > cast(1 as int)), None),
unresolvedalias((cast(cast(1 as bigint) as decimal(20,0)) > cast(1 as
decimal(10,0))), None)]
+- OneRowRelation
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
