Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/21320#discussion_r190479026
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
---
@@ -162,7 +162,9 @@ case class FilterExec(condition: Expression, child:
SparkPlan)
val generatedIsNotNullChecks = new Array[Boolean](notNullPreds.length)
val generated = otherPreds.map { c =>
val nullChecks = c.references.map { r =>
- val idx = notNullPreds.indexWhere { n =>
n.asInstanceOf[IsNotNull].child.semanticEquals(r)}
+ val idx = notNullPreds.indexWhere { n =>
+ n.asInstanceOf[IsNotNull].child.references.contains(r)
--- End diff --
Yes. IIRC a community member identified a bug and @viirya contributed a fix
and unit test. See
https://github.com/VideoAmp/spark-public/commit/8b5661cefc3aa12cc79d2af162b0d84ab2afa39f.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
