GitHub user rdblue reopened a pull request:

    https://github.com/apache/spark/pull/20490

    [SPARK-23323][SQL]: Support commit coordinator for DataSourceV2 writes

    ## What changes were proposed in this pull request?
    
    DataSourceV2 batch writes should use the output commit coordinator if it is 
required by the data source. This adds a new method, 
`DataWriterFactory#useCommitCoordinator`, that determines whether the 
coordinator will be used. If the write factory returns true, 
`WriteToDataSourceV2` will use the coordinator for batch writes.
    
    ## How was this patch tested?
    
    This relies on existing write tests, which now use the commit coordinator.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rdblue/spark 
SPARK-23323-add-commit-coordinator

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20490
    
----
commit ebe9d56094e53d1a8f7083eae781aa490d96d80b
Author: Ryan Blue <blue@...>
Date:   2018-02-02T22:21:48Z

    SPARK-23323: DataSourceV2: support commit coordinator.

commit 14b4a95b9c0ce0024e304d3cd48880a260df0f81
Author: Ryan Blue <blue@...>
Date:   2018-02-06T19:30:51Z

    Update documentation in DataSourceWriter for commit coordination.

commit 2ac1fa23781b04172b9bf33380656a5e9c885db7
Author: Ryan Blue <blue@...>
Date:   2018-02-06T20:04:35Z

    Fix javadoc for DataWriterFactory.

commit c982d3a5d0a895ad33a696a7b0fbd9453724fdb4
Author: Ryan Blue <blue@...>
Date:   2018-02-06T22:28:55Z

    Remove link to OutputCommitCoordinator because Javadoc can't find it.

commit 9353074ae18da971ebb0fadc2a986933442b46f1
Author: Ryan Blue <blue@...>
Date:   2018-02-07T16:59:21Z

    Remove unused import.

commit a2a0ec8b440152be0f643fd89dcce2c0f51612c1
Author: Ryan Blue <blue@...>
Date:   2018-02-08T17:32:34Z

    Move useCommitCoordinator to DataSourceWriter.
    
    This should be configured by the writer, not the factory that creates
    DataWriters.

commit e9964ca2fc831819662056210db594f613bce5d0
Author: Ryan Blue <blue@...>
Date:   2018-02-08T20:13:31Z

    Avoid catching writer in Java serialization.

commit ec968563605f961d3d874913de51265683a8c132
Author: Ryan Blue <blue@...>
Date:   2018-02-09T19:20:25Z

    Only one => at most one.

commit 538bc864f8ebb8d1b7e63c26806f209f2c3b0fc4
Author: Ryan Blue <blue@...>
Date:   2018-02-12T18:32:13Z

    Fix docs and style nit.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to