Re: [Spark Streaming+Kafka][How-to]

2017-03-21 Thread OUASSAIDI, Sami
t;assign", """{t0: {"1": }, t1:{"1": x}}""") > .load() > .coalesce(1) > .writeStream > .foreach(... code to write to cassandra ...) > > On Fri, Mar 17, 2017 at 7:35 AM, OUASSAIDI, Sami > wrote: > >> @Co

Re: [Spark Streaming+Kafka][How-to]

2017-03-17 Thread OUASSAIDI, Sami
@Cody : Duly noted. @Michael Ambrust : A repartition is out of the question for our project as it would be a fairly expensive operation. We tried looking into targeting a specific executor so as to avoid this extra cost and directly have well partitioned data after consuming the kafka topics. Also

[Spark Streaming+Kafka][How-to]

2017-03-16 Thread OUASSAIDI, Sami
Hi all, So I need to specify how an executor should consume data from a kafka topic. Let's say I have 2 topics : t0 and t1 with two partitions each, and two executors e0 and e1 (both can be on the same node so assign strategy does not work since in the case of a multi executor node it works based