As I understand the matter: Option 1) has benefits when you think that your network bandwidth may be a bottle neck, because Spark opens several network connections on possibly several different physical machines.
Option 2) - as you already pointed out - has the benefit that you occupy less worker cores with receiver tasks. Regards, Jeff 2015-02-26 9:38 GMT+01:00 [email protected] <[email protected]>: > Sure, Thanks Tathagata! > > ------------------------------ > [email protected] > > > *From:* Tathagata Das <[email protected]> > *Date:* 2015-02-26 14:47 > *To:* [email protected] > *CC:* Akhil Das <[email protected]>; user <[email protected]> > *Subject:* Re: Re: Many Receiver vs. Many threads per Receiver > Spark Streaming has a new Kafka direct stream, to be release as > experimental feature with 1.3. That uses a low level consumer. Not sure if > it satisfies your purpose. > If you want more control, its best to create your own Receiver with the > low level Kafka API. > > TD > > On Tue, Feb 24, 2015 at 12:09 AM, [email protected] <[email protected]> wrote: > >> Thanks Akhil. >> Not sure whether thelowlevel consumer. >> <https://github.com/dibbhatt/kafka-spark-consumer>will be officially >> supported by Spark Streaming. So far, I don't see it mentioned/documented >> in the spark streaming programming guide. >> >> ------------------------------ >> [email protected] >> >> >> *From:* Akhil Das <[email protected]> >> *Date:* 2015-02-24 16:21 >> *To:* [email protected] >> *CC:* user <[email protected]> >> *Subject:* Re: Many Receiver vs. Many threads per Receiver >> I believe when you go with 1, it will distribute the consumer across your >> cluster (possibly on 6 machines), but still it i don't see a away to tell >> from which partition it will consume etc. If you are looking to have a >> consumer where you can specify the partition details and all, then you are >> better off with the lowlevel consumer. >> <https://github.com/dibbhatt/kafka-spark-consumer> >> >> >> >> Thanks >> Best Regards >> >> On Tue, Feb 24, 2015 at 9:36 AM, [email protected] <[email protected]> wrote: >> >>> Hi, >>> I am experimenting Spark Streaming and Kafka Integration, To read >>> messages from Kafka in parallel, basically there are two ways >>> 1. Create many Receivers like (1 to 6).map(_ => >>> KakfaUtils.createStream). >>> 2. Specifiy many threads when calling KakfaUtils.createStream like val >>> topicMap("myTopic"=>6), this will create one receiver with 6 reading >>> threads. >>> >>> My question is which option is better, sounds option 2 is better is to >>> me because it saves a lot of cores(one Receiver one core), but I >>> learned from somewhere else that choice 1 is better, so I would ask and see >>> how you guys elaborate on this. Thank >>> >>> ------------------------------ >>> [email protected] >>> >> >> >
