Jordan Halterman created SPARK-17897:

             Summary: not isnotnull is converted to the always false condition 
isnotnull && not isnotnull
                 Key: SPARK-17897
             Project: Spark
          Issue Type: Bug
          Components: Optimizer
    Affects Versions: 2.0.1, 2.0.0
            Reporter: Jordan Halterman

When a logical plan is built containing the following somewhat nonsensical 
{{Filter (NOT isnotnull($f0#212))}}

During optimization the filter is converted into a condition that will always 
{{Filter (isnotnull($f0#212) && NOT isnotnull($f0#212))}}

This appears to be caused by the following check for {{NullIntolerant}}:

Which recurses through the expression and extracts nested {{IsNotNull}} calls, 
converting them to {{IsNotNull}} calls on the attribute at the root level:

This results in the nonsensical condition above.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to