Here is an example of how you can do. Lets say "myDStream" contains the
data that you may want to asynchornously query, say using, Spark SQL.
val sqlContext = new SqlContext(streamingContext.sparkContext)
myDStream.foreachRDD { rdd => // rdd is a RDD of case class
sqlContext.registerRDDAsTable(rdd, "mytable")
}
This will make sure that the table "mytable" always refers to the latest
RDD generated by the DStream.
Then from a diffrent thread you can asynchronously query
sqlContext.sql("select * from mytable")
Hope this helps.
TD
On Fri, Feb 13, 2015 at 3:59 AM, Sean Owen <so...@cloudera.com> wrote:
> Sure it's possible, but you would use Streaming to update some shared
> state, and create another service that accessed that shared state too.
>
> On Fri, Feb 13, 2015 at 11:57 AM, Tamas Jambor <jambo...@gmail.com> wrote:
> > Thanks for the reply, I am trying to setup a streaming as a service
> > approach, using the framework that is used for spark-jobserver. for that
> I
> > would need to handle asynchronous operations that are initiated from
> > outside of the stream. Do you think it is not possible?
> >
> > On Fri Feb 13 2015 at 10:14:18 Sean Owen <so...@cloudera.com> wrote:
> >>
> >> You call awaitTermination() in the main thread, and indeed it blocks
> >> there forever. From there Spark Streaming takes over, and is invoking
> >> the operations you set up. Your operations have access to the data of
> >> course. That's the model; you don't make external threads that reach
> >> in to Spark Streaming's objects, but can easily create operations that
> >> take whatever actions you want and invoke them in Streaming.
> >>
> >> On Fri, Feb 13, 2015 at 10:04 AM, jamborta <jambo...@gmail.com> wrote:
> >> > Hi all,
> >> >
> >> > I am trying to come up with a workflow where I can query streams
> >> > asynchronously. The problem I have is a ssc.awaitTermination() line
> >> > blocks
> >> > the whole thread, so it is not straightforward to me whether it is
> >> > possible
> >> > to get hold of objects from streams once they are started. any
> >> > suggestion on
> >> > what is the best way to implement this?
> >> >
> >> > thanks,
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> http://apache-spark-user-list.1001560.n3.nabble.com/Interact-with-streams-in-a-non-blocking-way-tp21640.html
> >> > Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> >> > For additional commands, e-mail: user-h...@spark.apache.org
> >> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
