GitHub user rdblue opened a pull request:

    https://github.com/apache/spark/pull/20488

    [SPARK-23321][SQL]: Validate datasource v2 writes

    ## What changes were proposed in this pull request?
    
    DataSourceV2 does not currently apply any validation rules when writing. 
Other write paths attempt to validate that a data frame can be written to a 
target table or path and these changes add the same logic to v2.
    
    This updates the logical plan to use InsertIntoTable and applies the insert 
preprocess rules to writes. It also adds a conversion rule from InserIntoTable 
to DataSourceV2Write because InsertIntoTable cannot be used in logical plans 
after analysis.
    
    InsertIntoTable is not necessarily the right logical plan. It assumes that 
the table exists and can report its schema.
    
    ## How was this patch tested?
    
    Added a test that fails analysis in the preprocess rule.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rdblue/spark 
SPARK-23321-validate-datasource-v2-writes

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20488
    
----
commit 62c569672083c0fa633da1d6edaba40d0bb05819
Author: Ryan Blue <blue@...>
Date:   2018-01-17T21:58:12Z

    SPARK-22386: DataSourceV2: Use immutable logical plans.

commit f0bd45d3c931941b8092cdac738cb29954e0acdd
Author: Ryan Blue <blue@...>
Date:   2018-01-24T19:34:42Z

    SPARK-23203: Fix scala style check.

commit 2fdeb4556cd22a092630b341a22a16a59e377183
Author: Ryan Blue <blue@...>
Date:   2018-01-24T19:54:10Z

    SPARK-23203: Fix Kafka tests, use StreamingDataSourceV2Relation.
    
    This also removes unused imports.

commit ab945a19efe666c41deae9c044002f3455220c1d
Author: Ryan Blue <blue@...>
Date:   2018-02-02T20:30:33Z

    SPARK-23204: DataFrameReader: Remove v2 table identifier parsing.

commit 3580daf15497a1d49112a0eddd556f74b9b3e280
Author: Ryan Blue <blue@...>
Date:   2018-02-02T19:04:23Z

    SPARK-23321: Apply preprocess insert rules to DataSourceV2.
    
    This updates the DataSourceV2 write path to use DataSourceV2Relation and
    InsertIntoTable to apply the insert preprocess rules.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to