GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/13342
[SPARK-15593][SQL]Add DataFrameWriter.foreach to allow the user consuming
data in ContinuousQuery
## What changes were proposed in this pull request?
* Add DataFrameWriter.foreach to allow the user consuming data in
ContinuousQuery
* ForeachWriter is the interface for the user to consume partitions of
data
* Add a type parameter T to DataFrameWriter
Usage
```Scala
val ds = spark.read....stream().as[String]
ds.....write
.queryName(...)
.option("checkpointLocation", ...)
.foreach(new ForeachWriter[Int] {
override def open(version: Long): Unit = {
// prepare some resources for a partition
}
override def process(value: Int): Unit = {
// process data
}
override def close(): Unit = {
// release resources for a partition
}
})
```
## How was this patch tested?
New unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark foreach
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13342.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13342
----
commit cda6ca329ec334083f5f2d0ac9b4e6fe76f0587d
Author: Shixiong Zhu <[email protected]>
Date: 2016-05-27T00:16:31Z
Add DataFrameWriter.foreach to allow the user consuming data in
ContinuousQuery
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
