Jordan Halterman created SPARK-17897: ----------------------------------------
Summary: not isnotnull is converted to the always false condition isnotnull && not isnotnull Key: SPARK-17897 URL: https://issues.apache.org/jira/browse/SPARK-17897 Project: Spark Issue Type: Bug Components: Optimizer Affects Versions: 2.0.1, 2.0.0 Reporter: Jordan Halterman When a logical plan is built containing the following somewhat nonsensical filter: {{Filter (NOT isnotnull($f0#212))}} During optimization the filter is converted into a condition that will always fail: {{Filter (isnotnull($f0#212) && NOT isnotnull($f0#212))}} This appears to be caused by the following check for {{NullIntolerant}}: https://github.com/apache/spark/commit/df68beb85de59bb6d35b2a8a3b85dbc447798bf5#diff-203ac90583cebe29a92c1d812c07f102R63 Which recurses through the expression and extracts nested {{IsNotNull}} calls, converting them to {{IsNotNull}} calls on the attribute at the root level: https://github.com/apache/spark/commit/df68beb85de59bb6d35b2a8a3b85dbc447798bf5#diff-203ac90583cebe29a92c1d812c07f102R49 This results in the nonsensical condition above. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org