Re: Pausing spark kafka streaming (direct) or exclude/include some partitions on the fly per batch

2016-09-03 Thread Cody Koeninger
wrap a >> consumer to do what you need. >> >> On Fri, Sep 2, 2016 at 12:28 PM, sagarcasual . >> wrote: >> > Hello, this is for >> > Pausing spark kafka streaming (direct) or exclude/include some >> partitions on >> > the fly per batch

Re: Pausing spark kafka streaming (direct) or exclude/include some partitions on the fly per batch

2016-09-02 Thread sagarcasual .
rite some code. The 0.10 integration makes the > underlying kafka consumer pluggable, so you may be able to wrap a > consumer to do what you need. > > On Fri, Sep 2, 2016 at 12:28 PM, sagarcasual . > wrote: > > Hello, this is for > > Pausing spark kafka streaming (direct)

Re: Pausing spark kafka streaming (direct) or exclude/include some partitions on the fly per batch

2016-09-02 Thread Cody Koeninger
le to wrap a consumer to do what you need. On Fri, Sep 2, 2016 at 12:28 PM, sagarcasual . wrote: > Hello, this is for > Pausing spark kafka streaming (direct) or exclude/include some partitions on > the fly per batch > = > I

Pausing spark kafka streaming (direct) or exclude/include some partitions on the fly per batch

2016-09-02 Thread sagarcasual .
Hello, this is for Pausing spark kafka streaming (direct) or exclude/include some partitions on the fly per batch = I have following code that creates a direct stream using Kafka connector for Spark. final JavaInputDStream msgRecords