[
https://issues.apache.org/jira/browse/SPARK-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15528403#comment-15528403
]
Frederick Reiss commented on SPARK-17631:
-----------------------------------------
I can think of two use cases where one might want to have the option to send
the output of a Structured Streaming application out over HTTP.
The first use case involves using HTTP as a lightweight protocol for
transferring bulk data out of a Structured Streaming query to a remote system.
In this case, I would think the best approach would be to use a
{{ForeachSink}}. That way, the HTTP connections would originate from the
executors instead of having all the data pulled into the driver. I'm not sure
that there's really a need to add a built-in sink type for this use case, as
the existing {{ForeachSink}} provides most of what one would need.
The second use case involves tunneling data out of the Spark cluster to an
existing legacy system that speaks HTTP or HTTPS. For example, one might want a
Structured Streaming application to send ML model updates to an application
server via a secured connection. In this use case, I would think that it would
be appropriate for the data to be pulled to the driver as in the PR. However,
you would want a lot more configurability in any facility built into Spark;
otherwise, there isn't much benefit over what a user could throw together in a
few minutes over the (as yet unopened, but hopefully soon available) Sink API.
> Structured Streaming - Do we need to output results through http API?
> ---------------------------------------------------------------------
>
> Key: SPARK-17631
> URL: https://issues.apache.org/jira/browse/SPARK-17631
> Project: Spark
> Issue Type: New Feature
> Components: SQL, Streaming
> Affects Versions: 2.0.0
> Reporter: zhangxinyu
> Priority: Minor
>
> Streaming query results can be sinked to http server through http post request
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
