GitHub user viirya opened a pull request:

    https://github.com/apache/spark/pull/21745

    [SPARK-24781][SQL] Using a reference from Dataset in Filter/Sort might not 
work

    ## What changes were proposed in this pull request?
    
    When we use a reference from Dataset in filter or sort, which was not used 
in the prior select, an AnalysisException occurs, e.g.,
    
    ```scala
    val df = Seq(("test1", 0), ("test2", 1)).toDF("name", "id")
    df.select(df("name")).filter(df("id") === 0).show()
    ```
    
    ```scala
    org.apache.spark.sql.AnalysisException: Resolved attribute(s) id#6 missing 
from name#5 in operator !Filter (id#6 = 0).;;
    !Filter (id#6 = 0)
       +- AnalysisBarrier
          +- Project [name#5]
             +- Project [_1#2 AS name#5, _2#3 AS id#6]
                +- LocalRelation [_1#2, _2#3]
    ```
    
    This change adds a condition `missingInput.isEmpty` to `resolved` of 
`LogicalPlan`. Previously a logical plan is resolved if all expressions are 
resolved and its children are resolved. However, as we possibly add a resolved 
reference like `df("name")` into a query plan, it is possible that all 
expressions in a query plan are resolved but have missing inputs.
    
    ## How was this patch tested?
    
    Added tests.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/viirya/spark-1 SPARK-24781

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21745.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21745
    
----
commit 97837a46b790ceb1f0df38cc7a3094b1cb4eb556
Author: Liang-Chi Hsieh <viirya@...>
Date:   2018-07-11T07:44:43Z

    Resolved references from Dataset should be checked if it is missed from 
plan.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to