I meant are you using RDD generated by DStreams, in Spark jobs out
side the DStreams computation?
Something like this:
var globalRDD = null
dstream.foreachRDD(rdd =>
// have a global pointer based on the rdds generate by dstream
if (runningFirstTime) globalRDD = rdd
)
ssc.start()
.....
// much much time later try to use the RDD in Spark jobs independent
of the streaming computation
globalRDD.count()
On Fri, Aug 1, 2014 at 3:52 PM, Kanwaldeep <kanwal...@gmail.com> wrote:
> All the operations being done are using the dstream. I do read an RDD in
> memory which is collected and converted into a map and used for lookups as
> part of DStream operations. This RDD is loaded only once and converted into
> map that is then used on streamed data.
>
> Do you mean non streaming jobs on RDD using raw kafka data?
>
> Log File attached:
> streaming.gz
> <http://apache-spark-user-list.1001560.n3.nabble.com/file/n11229/streaming.gz>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Could-not-compute-split-block-not-found-tp11186p11229.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
