Spark streaming archive results

[Udo Fholl]

Hi,

I want to aggregate on a small window and send downstream every 30 secs.
But I would also like to store in our archive the outcome every 20min.

My current approach (simplified version) is:

  val stream = //
  val statedStream = stream.mapWithState(stateSpec)
  val archiveStream = statedStream


  statedStream.foreachRDD(rdd => rdd.foreachPartition(d =>
sendDownstream(d)))

  archive.window(Minutes(20).foreachRDD(rdd => rdd.foreachPartition(d =>
archive(d)))


Is there a more suitable way to deal with this situations?

Thanks.

Udo