GitHub user lw-lin reopened a pull request: https://github.com/apache/spark/pull/13518
[WIP][SPARK-15472][SQL] Add support for writing in `csv`, `json`, `text` formats in Structured Streaming ## What changes were proposed in this pull request? This patch adds support for writing in `csv`, `json`, `text` formats in Structured Streaming: **1. at a high level, this patch forms the following hierarchy**(`text` as an example): ``` <OutputWriter> â TextOutputWriterBase â â BatchTextOutputWriter StreamingTextOutputWriter ``` ``` <OutputWriterFactory> â â BatchTextOutputWriterFactory StreamingOutputWriterFactory â StreamingTextOutputWriterFactory ``` The `StreamingTextOutputWriter` and other 'streaming' output writers would write data **without** using an `OutputCommitter`. This was the same approach taken by [SPARK-14716](https://github.com/apache/spark/pull/12409). **2. to support compression, this patch attaches an extension to the path assigned by `FileStreamSink`**, which is slightly different from [SPARK-14716](https://github.com/apache/spark/pull/12409). For example, if we are writing out using the `gzip` compression and `FileStreamSink` assigns path `${uuid}` to a text writer, then in the end the file written out will be `${uuid}.txt.gz` -- so that when we read the file back, we'll correctly interpret it as `gzip` compressed. ## How was this patch tested? `FileStreamSinkSuite` is expanded much more to cover the added `csv`, `json`, `text` formats: ```scala test(" csv - unpartitioned data - codecs: none/gzip") test("json - unpartitioned data - codecs: none/gzip") test("text - unpartitioned data - codecs: none/gzip") test(" csv - partitioned data - codecs: none/gzip") test("json - partitioned data - codecs: none/gzip") test("text - partitioned data - codecs: none/gzip") test(" csv - unpartitioned writing and batch reading - codecs: none/gzip") test("json - unpartitioned writing and batch reading - codecs: none/gzip") test("text - unpartitioned writing and batch reading - codecs: none/gzip") test(" csv - partitioned writing and batch reading - codecs: none/gzip") test("json - partitioned writing and batch reading - codecs: none/gzip") test("text - partitioned writing and batch reading - codecs: none/gzip") ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/lw-lin/spark add-csv-json-text-for-ss Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13518.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13518 ---- commit 97034f9aeb092b10e1606e60a8e6b4878ebd54cf Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-05T09:03:04Z Add csv, json, text commit 2035b597b44aa519d8da3b155036446f88b3050e Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-05T09:03:15Z Fix parquet extension commit 4737361489fd680405b291ec498ab91374685ffe Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-05T11:52:14Z Fix style commit 90d02c4a10c14af83bbed985e36ef99a1edaa48b Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T08:02:32Z Fix tests commit daec480bd16ed52137d32f332debe3806953f4d2 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T09:03:08Z Revert "Fix tests" This reverts commit 90d02c4a10c14af83bbed985e36ef99a1edaa48b. commit 43b68d426e9b64061095eca7a1db0e762843adef Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T09:09:10Z Fix tests commit 56dbb9b4f0f7e2bf76935e0d1d2fc6c6cdf141ff Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T12:34:07Z Investigate test commit 91e51aed5caf663d5068057e5cf28f21eb768310 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T12:43:39Z Investigate test commit eb2090ce9fa04efbd23370f7d3e6cb98fd0b4c74 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-06T13:00:26Z Update run-tests commit 1e9af1cef892706de8b07728c192dd8ca5e5851e Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-07T04:10:34Z Investigate test commit f76b1d9fe5e4970ad66dfb6d4b2fed939b68728c Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-07T04:49:12Z Fix tests commit 7fca579d9fec2589f13403b0eb0f2d5f5e6bd52a Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-07T04:50:24Z Fix tests commit 2ead307d01d8f908951fbc059b8e49bbc77947b1 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-07T05:02:55Z Fix tests commit b64afc64d3121479eb5c3f8c8b5663b6e05349b7 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-07T05:35:34Z Fix tests commit 2c52c2c6ac961d0b71c2dd3802cd53d51ba1926a Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-07T05:58:58Z Fix style commit aa61f1dab4fa367296fa83948c173985a289a629 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-07T06:39:22Z Fix tests commit c5ac63e139c37e9e29c071ef17bfd0c45b40676b Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-07T06:53:44Z Fix tests commit cc842733d7ad2644ed8ddb7ebd9dcc4def301130 Author: Liwei Lin <lwl...@gmail.com> Date: 2016-06-07T07:12:46Z Investigate tests ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org