I was reading StructuredStreamingProgrammingAbstractionSemanticsandAPIs-ApacheJIRA.pdf attached to SPARK-8360
On page 12, there was mentioning of .format(“kafka”) but I searched the codebase and didn't find any occurrence. FYI On Fri, May 6, 2016 at 1:06 PM, Michael Malak < michaelma...@yahoo.com.invalid> wrote: > At first glance, it looks like the only streaming data sources available > out of the box from the github master branch are > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala > and > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala > . > Out of the Jira epic for Structured Streaming > https://issues.apache.org/jira/browse/SPARK-8360 it would seem the > still-open https://issues.apache.org/jira/browse/SPARK-10815 "API design: > data sources and sinks" is relevant here. > > In short, it would seem the code is not there yet to create a Kafka-fed > Dataframe/Dataset that can be queried with Structured Streaming; or if it > is, it's not obvious how to write such code. > > > ------------------------------ > *From:* Anthony May <anthony...@gmail.com> > *To:* Deepak Sharma <deepakmc...@gmail.com>; Sunita Arvind < > sunitarv...@gmail.com> > *Cc:* "user@spark.apache.org" <user@spark.apache.org> > *Sent:* Friday, May 6, 2016 11:50 AM > *Subject:* Re: Adhoc queries on Spark 2.0 with Structured Streaming > > Yeah, there isn't even a RC yet and no documentation but you can work off > the code base and test suites: > https://github.com/apache/spark > And this might help: > > https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/streaming/DataFrameReaderWriterSuite.scala > > On Fri, 6 May 2016 at 11:07 Deepak Sharma <deepakmc...@gmail.com> wrote: > > Spark 2.0 is yet to come out for public release. > I am waiting to get hands on it as well. > Please do let me know if i can download source and build spark2.0 from > github. > > Thanks > Deepak > > On Fri, May 6, 2016 at 9:51 PM, Sunita Arvind <sunitarv...@gmail.com> > wrote: > > Hi All, > > We are evaluating a few real time streaming query engines and spark is my > personal choice. The addition of adhoc queries is what is getting me > further excited about it, however the talks I have heard so far only > mention about it but do not provide details. I need to build a prototype to > ensure it works for our use cases. > > Can someone point me to relevant material for this. > > regards > Sunita > > > > > -- > Thanks > Deepak > www.bigdatabig.com > www.keosha.net > > > >
