[
https://issues.apache.org/jira/browse/SPARK-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583872#comment-15583872
]
Apache Spark commented on SPARK-17981:
--------------------------------------
User 'gatorsmile' has created a pull request for this issue:
https://github.com/apache/spark/pull/15523
> Incorrectly Set Nullability to False in FilterExec
> --------------------------------------------------
>
> Key: SPARK-17981
> URL: https://issues.apache.org/jira/browse/SPARK-17981
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.1, 2.1.0
> Reporter: Xiao Li
> Assignee: Xiao Li
> Priority: Critical
>
> When FilterExec contains isNotNull, which could be inferred and pushed down
> or users specified, we convert the nullability of the involved columns if the
> top-layer expression is null-intolerant. However, this is not true, if the
> top-layer expression is not a leaf expression, it could still tolerate the
> null when it has null-tolerant child expression.
> For example, cast(coalesce(a#5, a#15) as double). Although cast is a
> null-intolerant expression, but obviously coalesce is a null-tolerant.
> When the nullability is wrong, we could generate incorrect results in
> different cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
