[
https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491819#comment-15491819
]
Michael Armbrust commented on SPARK-16407:
------------------------------------------
I think it is likely that we will want to make it easy for users to instantiate
Sources / Sinks programmatically. I have two concerns with merging this
particular PR at this time:
- The source/sink APIs are in a lot of flux. The fact that they are not
labeled experimental in 2.0.0 was an unfortunate oversight that we'll correct
in 2.0.1. For example [~freiss] pointed out that it was impossible to know
when to flush state from the source. Another example is we'll probably have to
make changes to allow for rate limiting. Until we have a better idea of what
the final API will look like, I would not want to expose it in the end-user
APIs like DataStreamWriter.
- When we added the {{foreach}} API, it was a conscious choice to avoid
exposing dataframes / batches to the user. The idea is we could adapt code to
a non-microbatch model eventually if we wanted. This API would break that.
Its not out of the question, but thats a larger shift.
> Allow users to supply custom StreamSinkProviders
> ------------------------------------------------
>
> Key: SPARK-16407
> URL: https://issues.apache.org/jira/browse/SPARK-16407
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: holdenk
>
> The current DataStreamWriter allows users to specify a class name as format,
> however it could be easier for people to directly pass in a specific provider
> instance - e.g. for user equivalent of ForeachSink or other sink with
> non-string parameters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
