1. Not all types of joins are supported. Here is the list. - Right outer joins - stream-batch not allowed, batch-stream allowed - Left outer joins - batch-stream not allowed, stream-batch allowed (reverse of Right outer join) - Stream-stream joins are not allowed
In the cases of outer joins, the not-allowed-cases are fundamentally hard because to do them correctly, every time there is new data in the stream, all the past data in the stream needs to be processed. Since we cannot stored ever-increasing amount of data in memory, this is not feasible. 2. For the update mode, the timeline is Spark 2.1. TD On Mon, Jun 6, 2016 at 6:54 AM, raaggarw <raagg...@adobe.com> wrote: > Thanks > So, > > 1) For joins (stream-batch) - are all types of joins supported - i mean > inner, leftouter etc or specific ones? > Also what is the timeline for complete support - I mean stream-stream > joins? > > 2) So now outputMode is exposed via DataFrameWriter but will work in > specific cases as you mentioned? We were looking for delta & append output > modes for aggregation/groupBy. What is the timeline for that? > > Thanks > Ravi > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Timeline-for-supporting-basic-operations-like-groupBy-joins-etc-on-Streaming-DataFrames-tp27091p27093.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
