[GitHub] spark issue #15197: [SPARK-17631] [SQL] Add HttpStreamSink for structured st...
Github user zhangxinyu1 commented on the issue: https://github.com/apache/spark/pull/15197 @marmbrus Thanks for all your sugguestion! I have done the following things: - Write a short desing on [JIRA](https://issues.apache.org/jira/browse/SPARK-17631) - I replace `HttpDataFormat` with the input which only has one string column. I like this idea very much! thanks again. - Complete `HttpStreamSinkSuite` to test. It's my first time to write this, could you please help me test it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15197: [SPARK-17631] [SQL] Add HttpStreamSink for structured st...
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15197 Thanks for working on this, it does seem like it could be useful. I'm not sure if this should go into Spark or into a separate package. It really depends on how many people want this feature. Regardless, a few high level comments on this PR: - Check out the [contributing to Spark guide](https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark). Patches need to have tests and follow the style guide. - I would not define a new `HttpDataFormat` interface. Instead I would mandate that the input is a single string column (similar to what we do for `df.write.text`). Users can use all of the existing DataFrame/Dataset operations to convert their data into a string. - It would be good to write up a short design on JIRA and debate there. A few things that I can think of off the top of my head: - should we support https too? - do we need to set any headers (i.e. maybe the batch id?) - We'd also need to add docs for this feature. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15197: [SPARK-17631] [SQL] Add HttpStreamSink for structured st...
Github user zhangxinyu1 commented on the issue: https://github.com/apache/spark/pull/15197 @marmbrus I propose this feature mainly because we require output streaming query results via http api. One of use cases is real-time alarm. It's essential to send alarm messages in real time via Http Api, when we use structured streaming to analyze logs and find some Exceptions in logs. What do you think about it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15197: [SPARK-17631] [SQL] Add HttpStreamSink for structured st...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15197 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
