[jira] [Commented] (SPARK-22181) ReplaceExceptWithNotFilter if one or both of the datasets are fully derived out of Filters from a same parent

Sun, 01 Oct 2017 13:27:18 -0700 

    [ 
https://issues.apache.org/jira/browse/SPARK-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187526#comment-16187526
 ] 

Sathiya Kumar commented on SPARK-22181:
---------------------------------------

I will create the PR if you agree..

> ReplaceExceptWithNotFilter if one or both of the datasets are fully derived 
> out of Filters from a same parent
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-22181
>                 URL: https://issues.apache.org/jira/browse/SPARK-22181
>             Project: Spark
>          Issue Type: New Feature
>          Components: Optimizer, SQL
>    Affects Versions: 2.1.1, 2.2.0
>            Reporter: Sathiya Kumar
>            Priority: Minor
>
> While applying Except operator between two datasets, if one or both of the 
> datasets are purely transformed using filter operations, then instead of 
> rewriting the Except operator using expensive join operation, we can rewrite 
> it using filter operation by flipping the filter condition of the right node.
> ![Case-1](https://github.com/sathiyapk/Blog-Posts/blob/master/images/spark-optimizer/ReplaceExceptWithNotFilter-case1.png)
> ![Case-2](https://github.com/sathiyapk/Blog-Posts/blob/master/images/spark-optimizer/ReplaceExceptWithNotFilter-case2.png)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to