GitHub user lw-lin reopened a pull request:
https://github.com/apache/spark/pull/13518
[WIP][SPARK-15472][SQL] Add support for writing in `csv`, `json`, `text`
formats in Structured Streaming
## What changes were proposed in this pull request?
This patch adds support for writing in `csv`, `json`, `text` formats in
Structured Streaming:
**1. at a high level, this patch forms the following hierarchy**(`text` as
an example):
```
<OutputWriter>
â
TextOutputWriterBase
â â
BatchTextOutputWriter StreamingTextOutputWriter
```
```
<OutputWriterFactory>
â â
BatchTextOutputWriterFactory StreamingOutputWriterFactory
â
StreamingTextOutputWriterFactory
```
The `StreamingTextOutputWriter` and other 'streaming' output writers would
write data **without** using an `OutputCommitter`. This was the same approach
taken by [SPARK-14716](https://github.com/apache/spark/pull/12409).
**2. to support compression, this patch attaches an extension to the path
assigned by `FileStreamSink`**, which is slightly different from
[SPARK-14716](https://github.com/apache/spark/pull/12409). For example, if we
are writing out using the `gzip` compression and `FileStreamSink` assigns path
`${uuid}` to a text writer, then in the end the file written out will be
`${uuid}.txt.gz` -- so that when we read the file back, we'll correctly
interpret it as `gzip` compressed.
## How was this patch tested?
`FileStreamSinkSuite` is expanded much more to cover the added `csv`,
`json`, `text` formats:
```scala
test(" csv - unpartitioned data - codecs: none/gzip")
test("json - unpartitioned data - codecs: none/gzip")
test("text - unpartitioned data - codecs: none/gzip")
test(" csv - partitioned data - codecs: none/gzip")
test("json - partitioned data - codecs: none/gzip")
test("text - partitioned data - codecs: none/gzip")
test(" csv - unpartitioned writing and batch reading - codecs: none/gzip")
test("json - unpartitioned writing and batch reading - codecs: none/gzip")
test("text - unpartitioned writing and batch reading - codecs: none/gzip")
test(" csv - partitioned writing and batch reading - codecs: none/gzip")
test("json - partitioned writing and batch reading - codecs: none/gzip")
test("text - partitioned writing and batch reading - codecs: none/gzip")
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lw-lin/spark add-csv-json-text-for-ss
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13518.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13518
----
commit 97034f9aeb092b10e1606e60a8e6b4878ebd54cf
Author: Liwei Lin <[email protected]>
Date: 2016-06-05T09:03:04Z
Add csv, json, text
commit 2035b597b44aa519d8da3b155036446f88b3050e
Author: Liwei Lin <[email protected]>
Date: 2016-06-05T09:03:15Z
Fix parquet extension
commit 4737361489fd680405b291ec498ab91374685ffe
Author: Liwei Lin <[email protected]>
Date: 2016-06-05T11:52:14Z
Fix style
commit 90d02c4a10c14af83bbed985e36ef99a1edaa48b
Author: Liwei Lin <[email protected]>
Date: 2016-06-06T08:02:32Z
Fix tests
commit daec480bd16ed52137d32f332debe3806953f4d2
Author: Liwei Lin <[email protected]>
Date: 2016-06-06T09:03:08Z
Revert "Fix tests"
This reverts commit 90d02c4a10c14af83bbed985e36ef99a1edaa48b.
commit 43b68d426e9b64061095eca7a1db0e762843adef
Author: Liwei Lin <[email protected]>
Date: 2016-06-06T09:09:10Z
Fix tests
commit 56dbb9b4f0f7e2bf76935e0d1d2fc6c6cdf141ff
Author: Liwei Lin <[email protected]>
Date: 2016-06-06T12:34:07Z
Investigate test
commit 91e51aed5caf663d5068057e5cf28f21eb768310
Author: Liwei Lin <[email protected]>
Date: 2016-06-06T12:43:39Z
Investigate test
commit eb2090ce9fa04efbd23370f7d3e6cb98fd0b4c74
Author: Liwei Lin <[email protected]>
Date: 2016-06-06T13:00:26Z
Update run-tests
commit 1e9af1cef892706de8b07728c192dd8ca5e5851e
Author: Liwei Lin <[email protected]>
Date: 2016-06-07T04:10:34Z
Investigate test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
