GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/20490
[SPARK-23323][SQL]: Add support for commit coordinator for DataSourceV2
writes
## What changes were proposed in this pull request?
DataSourceV2 batch writes should use the output commit coordinator if it is
required by the data source. This adds a new method,
`DataWriterFactory#useCommitCoordinator`, that determines whether the
coordinator will be used. If the write factory returns true,
`WriteToDataSourceV2` will use the coordinator for batch writes.
This relies on the commits in #20387. Once that is committed, this will be
rebased. Only the last commit is part of this PR.
## How was this patch tested?
This relies on existing write tests, which now use the commit coordinator.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/spark
SPARK-23323-add-commit-coordinator
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20490.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20490
----
commit 62c569672083c0fa633da1d6edaba40d0bb05819
Author: Ryan Blue <blue@...>
Date: 2018-01-17T21:58:12Z
SPARK-22386: DataSourceV2: Use immutable logical plans.
commit f0bd45d3c931941b8092cdac738cb29954e0acdd
Author: Ryan Blue <blue@...>
Date: 2018-01-24T19:34:42Z
SPARK-23203: Fix scala style check.
commit 2fdeb4556cd22a092630b341a22a16a59e377183
Author: Ryan Blue <blue@...>
Date: 2018-01-24T19:54:10Z
SPARK-23203: Fix Kafka tests, use StreamingDataSourceV2Relation.
This also removes unused imports.
commit ab945a19efe666c41deae9c044002f3455220c1d
Author: Ryan Blue <blue@...>
Date: 2018-02-02T20:30:33Z
SPARK-23204: DataFrameReader: Remove v2 table identifier parsing.
commit f1d9872a2699cdbd5c87b02e702dc8103335131d
Author: Ryan Blue <blue@...>
Date: 2018-02-02T21:48:29Z
SPARK-23203: Remove import changes from DataSourceV2Utils.
commit 288af6a2729c769e0d4075a8f9190958ab5a211c
Author: Ryan Blue <blue@...>
Date: 2018-02-02T22:21:48Z
SPARK-23323: DataSourceV2: support commit coordinator.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
