If you do not need one to one semantics and does not want strict ordering guarantee , you can very well use the Receiver based approach, and this consumer from Spark-Packages ( https://github.com/dibbhatt/kafka-spark-consumer) can give much better alternatives in terms of performance and reliability for Receiver based approach.
Regards, Dibyendu On Thu, Oct 29, 2015 at 11:57 AM, varun sharma <varunsharman...@gmail.com> wrote: > Right now, there is one to one correspondence between kafka partitions and > spark partitions. > I dont have a requirement of one to one semantics. > I need more tasks to be generated in the job so that it can be > parallelised and batch can be completed fast. In the previous Receiver > based approach number of tasks created were independent of kafka > partitions, I need something like that only. > Any config available if I dont need one to one semantics? > Is there any way I can repartition without incurring any additional cost. > > Thanks > *VARUN SHARMA* > >
