For the 2.0 release, look for "Unsupported Operations" here: http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html Also, there are bigger gaps - like no Kafka support, no way to plug user-defined sources or sinks etc
Ofir Manor Co-Founder & CTO | Equalum Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io On Wed, Jul 27, 2016 at 11:24 AM, Chang Chen <baibaic...@gmail.com> wrote: > > I don't understand what kind of low level control that DStream can do > while Structure Streaming can not > > Thanks > Chang > > On Wednesday, July 27, 2016, Matei Zaharia <matei.zaha...@gmail.com> > wrote: > >> Yup, they will definitely coexist. Structured Streaming is currently >> alpha and will probably be complete in the next few releases, but Spark >> Streaming will continue to exist, because it gives the user more low-level >> control. It's similar to DataFrames vs RDDs (RDDs are the lower-level API >> for when you want control, while DataFrames do more optimizations >> automatically by restricting the computation model). >> >> Matei >> >> On Jul 27, 2016, at 12:03 AM, Ofir Manor <ofir.ma...@equalum.io> wrote: >> >> Structured Streaming in 2.0 is declared as alpha - plenty of bits still >> missing: >> >> http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html >> I assume that it will be declared stable / GA in a future 2.x release, >> and then it will co-exist with DStream for quite a while before someone >> will suggest to start a deprecation process that will eventually lead to >> its removal... >> As a user, I guess we will need to apply judgement about when to switch >> to Structured Streaming - each of us have a different risk/value tradeoff, >> based on our specific situation... >> >> Ofir Manor >> >> Co-Founder & CTO | Equalum >> >> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io >> >> On Wed, Jul 27, 2016 at 8:02 AM, Chang Chen <baibaic...@gmail.com> wrote: >> >>> Hi guys >>> >>> Structure Stream is coming with spark 2.0, but I noticed that DStream >>> is still here >>> >>> What's the future of the DStream, will it be deprecated and removed >>> eventually? Or co-existed with Structure Stream forever? >>> >>> Thanks >>> Chang >>> >>> >> >>
