GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/14691
[SPARK-16407][STREAMING] Allow users to supply custom streamsink provider
## What changes were proposed in this pull request?
This change allows the user to supply a specific instance of a stream sink
provider rather than using reflection to construct it from a class name. The
useful of this is illustrated within the testcase where we provide a custom
ForeachSink which behaves more like the old `foreachRDD` in the `DStream` API
than the current `foreach` implementation. This can serve as part of a useful
building block for creating ML systems on top of Structured Streaming (see
https://github.com/holdenk/spark-structured-streaming-ml for a use case).
## How was this patch tested?
Additional unit tests using the new API.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark
SPARK-16407-allow-users-to-supply-custom-streamsink-provider
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14691.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14691
----
commit 5ff14f61e0b06d1945be794e80e6be850b763d60
Author: Holden Karau <[email protected]>
Date: 2016-08-11T18:20:29Z
Start work to allow users to specify a sinkProvider directily rather than
class name
commit 46d59096818b598feeb1ebad783cba1df2782d8d
Author: Holden Karau <[email protected]>
Date: 2016-08-15T17:43:21Z
Make it easier for users to specify custom sinks and add a test (WIP)
commit 5b7eb2e020ccdf45f610f744c0dc24de3d524725
Author: Holden Karau <[email protected]>
Date: 2016-08-17T19:30:34Z
Fix the query stream writer tests a bit
commit 70e663c256e723ecd6bf69094f687861405561c9
Author: Holden Karau <[email protected]>
Date: 2016-08-17T19:56:37Z
Improve docstring slightly
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
