huangtengfei created SPARK-38434: ------------------------------------ Summary: Correct semantic of CheckAnalysis.getDataTypesAreCompatibleFn method Key: SPARK-38434 URL: https://issues.apache.org/jira/browse/SPARK-38434 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.2.1 Reporter: huangtengfei
Currently, in `CheckAnalysis` method [getDataTypesAreCompatibleFn |https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L606] implemented as: {code:java} private def getDataTypesAreCompatibleFn(plan: LogicalPlan): (DataType, DataType) => Boolean = { val isUnion = plan.isInstanceOf[Union] if (isUnion) { (dt1: DataType, dt2: DataType) => !DataType.equalsStructurally(dt1, dt2, true) } else { // SPARK-18058: we shall not care about the nullability of columns (dt1: DataType, dt2: DataType) => TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty } } {code} Return false when data types are compatible, otherwise return true, which is pretty confusing. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org