GitHub user rdblue opened a pull request:

    [SPARK-23323][SQL]: Add support for commit coordinator for DataSourceV2 

    ## What changes were proposed in this pull request?
    DataSourceV2 batch writes should use the output commit coordinator if it is 
required by the data source. This adds a new method, 
`DataWriterFactory#useCommitCoordinator`, that determines whether the 
coordinator will be used. If the write factory returns true, 
`WriteToDataSourceV2` will use the coordinator for batch writes.
    This relies on the commits in #20387. Once that is committed, this will be 
rebased. Only the last commit is part of this PR.
    ## How was this patch tested?
    This relies on existing write tests, which now use the commit coordinator.

You can merge this pull request into a Git repository by running:

    $ git pull 

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20490
commit 62c569672083c0fa633da1d6edaba40d0bb05819
Author: Ryan Blue <blue@...>
Date:   2018-01-17T21:58:12Z

    SPARK-22386: DataSourceV2: Use immutable logical plans.

commit f0bd45d3c931941b8092cdac738cb29954e0acdd
Author: Ryan Blue <blue@...>
Date:   2018-01-24T19:34:42Z

    SPARK-23203: Fix scala style check.

commit 2fdeb4556cd22a092630b341a22a16a59e377183
Author: Ryan Blue <blue@...>
Date:   2018-01-24T19:54:10Z

    SPARK-23203: Fix Kafka tests, use StreamingDataSourceV2Relation.
    This also removes unused imports.

commit ab945a19efe666c41deae9c044002f3455220c1d
Author: Ryan Blue <blue@...>
Date:   2018-02-02T20:30:33Z

    SPARK-23204: DataFrameReader: Remove v2 table identifier parsing.

commit f1d9872a2699cdbd5c87b02e702dc8103335131d
Author: Ryan Blue <blue@...>
Date:   2018-02-02T21:48:29Z

    SPARK-23203: Remove import changes from DataSourceV2Utils.

commit 288af6a2729c769e0d4075a8f9190958ab5a211c
Author: Ryan Blue <blue@...>
Date:   2018-02-02T22:21:48Z

    SPARK-23323: DataSourceV2: support commit coordinator.



To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to