[jira] [Commented] (SPARK-17631) Structured Streaming - Add Http Stream Sink
[
https://issues.apache.org/jira/browse/SPARK-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519970#comment-15519970
]
zhangxinyu commented on SPARK-17631:
h4. A short design for this feature
h5. Goal
Build an http sink for structured streaming. Streaming query results can be
written out to http servers.
h5. Usage
# The streaming query results should have a single string column.
# We should configure ".format("http").option("url", yourHttpUrl)" in our
programs to create http sinks.
e.g.
val query = counts.writeStream
.outputMode("complete")
.format("http")
.option("url", "yourHttpUrl")
.start()
h5. Design
# Add a class "HttpSink" that extends trait "Sink", and override function
"addBatch".
Override "addBatch": echo Row in dataFrame will be written out through
sending an http post request.
# Add a class "HttpStreamSink" that extends both trait "StreamSinkProvider" and
trait "DataSourceRegister". It overrides two functions:
- shortname: return an string "http"
- createSink: return an HttpSink instance
h5. Other features to debate
# should we support https too?
# do we need to set any headers (i.e. maybe the batch id?)
> Structured Streaming - Add Http Stream Sink
> ---
>
> Key: SPARK-17631
> URL: https://issues.apache.org/jira/browse/SPARK-17631
> Project: Spark
> Issue Type: New Feature
> Components: SQL, Streaming
>Affects Versions: 2.0.0
>Reporter: zhangxinyu
>Priority: Minor
>
> Streaming query results can be sinked to http server through http post request
> github: https://github.com/apache/spark/pull/15194
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17631) Structured Streaming - Add Http Stream Sink
[ https://issues.apache.org/jira/browse/SPARK-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512781#comment-15512781 ] Apache Spark commented on SPARK-17631: -- User 'zhangxinyu1' has created a pull request for this issue: https://github.com/apache/spark/pull/15197 > Structured Streaming - Add Http Stream Sink > --- > > Key: SPARK-17631 > URL: https://issues.apache.org/jira/browse/SPARK-17631 > Project: Spark > Issue Type: New Feature > Components: SQL, Streaming >Affects Versions: 2.0.0 >Reporter: zhangxinyu >Priority: Minor > Fix For: 2.0.0 > > > Streaming query results can be sinked to http server through http post request > github: https://github.com/apache/spark/pull/15194 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17631) Structured Streaming - Add Http Stream Sink
[ https://issues.apache.org/jira/browse/SPARK-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512716#comment-15512716 ] Apache Spark commented on SPARK-17631: -- User 'zhangxinyu1' has created a pull request for this issue: https://github.com/apache/spark/pull/15194 > Structured Streaming - Add Http Stream Sink > --- > > Key: SPARK-17631 > URL: https://issues.apache.org/jira/browse/SPARK-17631 > Project: Spark > Issue Type: New Feature > Components: SQL, Streaming >Affects Versions: 2.0.0 >Reporter: zhangxinyu >Priority: Minor > Fix For: 2.0.0 > > > Streaming query results can be sinked to http server through http post request > github: https://github.com/apache/spark/pull/15194 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
