GitHub user rdblue opened a pull request: https://github.com/apache/spark/pull/20490
[SPARK-23323][SQL]: Add support for commit coordinator for DataSourceV2 writes ## What changes were proposed in this pull request? DataSourceV2 batch writes should use the output commit coordinator if it is required by the data source. This adds a new method, `DataWriterFactory#useCommitCoordinator`, that determines whether the coordinator will be used. If the write factory returns true, `WriteToDataSourceV2` will use the coordinator for batch writes. This relies on the commits in #20387. Once that is committed, this will be rebased. Only the last commit is part of this PR. ## How was this patch tested? This relies on existing write tests, which now use the commit coordinator. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rdblue/spark SPARK-23323-add-commit-coordinator Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20490.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20490 ---- commit 62c569672083c0fa633da1d6edaba40d0bb05819 Author: Ryan Blue <blue@...> Date: 2018-01-17T21:58:12Z SPARK-22386: DataSourceV2: Use immutable logical plans. commit f0bd45d3c931941b8092cdac738cb29954e0acdd Author: Ryan Blue <blue@...> Date: 2018-01-24T19:34:42Z SPARK-23203: Fix scala style check. commit 2fdeb4556cd22a092630b341a22a16a59e377183 Author: Ryan Blue <blue@...> Date: 2018-01-24T19:54:10Z SPARK-23203: Fix Kafka tests, use StreamingDataSourceV2Relation. This also removes unused imports. commit ab945a19efe666c41deae9c044002f3455220c1d Author: Ryan Blue <blue@...> Date: 2018-02-02T20:30:33Z SPARK-23204: DataFrameReader: Remove v2 table identifier parsing. commit f1d9872a2699cdbd5c87b02e702dc8103335131d Author: Ryan Blue <blue@...> Date: 2018-02-02T21:48:29Z SPARK-23203: Remove import changes from DataSourceV2Utils. commit 288af6a2729c769e0d4075a8f9190958ab5a211c Author: Ryan Blue <blue@...> Date: 2018-02-02T22:21:48Z SPARK-23323: DataSourceV2: support commit coordinator. ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org