Is there a way to do this that preserves exactly once semantics for the write to Kafka?
On Tue, Sep 2, 2014 at 12:30 PM, Tim Smith <secs...@gmail.com> wrote: > I'd be interested in finding the answer too. Right now, I do: > > val kafkaOutMsgs = kafkInMessages.map(x=>myFunc(x._2,someParam)) > kafkaOutMsgs.foreachRDD((rdd,time) => { rdd.foreach(rec => { > writer.output(rec) }) } ) //where writer.ouput is a method that takes a > string and writer is an instance of a producer class. > > > > > > On Tue, Sep 2, 2014 at 10:12 AM, Massimiliano Tomassi < > max.toma...@gmail.com> wrote: > >> Hello all, >> after having applied several transformations to a DStream I'd like to >> publish all the elements in all the resulting RDDs to Kafka. What the best >> way to do that would be? Just using DStream.foreach and then RDD.foreach ? >> Is there any other built in utility for this use case? >> >> Thanks a lot, >> Max >> >> -- >> ------------------------------------------------ >> Massimiliano Tomassi >> ------------------------------------------------ >> e-mail: max.toma...@gmail.com >> ------------------------------------------------ >> > >