There has been efforts to come up with a unified syntax for streaming (see [1] [2]), but I guess there will be differences based on the streaming features supported by a system.
Agree it needs a detailed design and it can be as close to the Spark batch SQL syntax as possible. Also I am not sure if its possible or makes sense to express all the operations via pure sql. e.g. the query start/stop, triggers, watermark etc might be better expressed via APIs. [1] https://docs.google.com/document/d/1wrla8mF_mmq-NW9sdJHYVgMyZsgCmHumJJ5f5WUzTiM/edit#heading=h.vfrf26d6b3ne [2] https://calcite.apache.org/docs/stream.html On Fri, 21 Dec 2018 at 18:13, Wenchen Fan <cloud0...@gmail.com> wrote: > It will be great to add pure-SQL support to structured streaming. I think > it goes without saying that how important SQL support is, but we should > make a completed design first. > > Looking at the Kafka streaming syntax > <https://www.confluent.io/blog/ksql-streaming-sql-for-apache-kafka/>, it > has CREATE STREAM, it has WINDOW TUMBLING. Shall we check other streaming > systems with SQL support, and justify places where we are going to differ? > > We should also take into account the full lifecycle: > 1. how to restart a streaming query from checkpoint? > 2. how to stop a streaming query? > 3. how to check status/progress of a streaming query? > 4. ... > > Basically, we should check what functions the DataStreamReader/Writer API > support, and see if we can support it with SQL as well. > > > Thanks for your proposal! > Wenchen > > On Mon, Oct 22, 2018 at 11:15 AM JackyLee <qcsd2...@163.com> wrote: > >> The code of SQLStreaming has been pushed: >> >> https://github.com/apache/spark/pull/22575 >> >> >> >> -- >> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
