Thanks David!. We are not using confluent at the moment .Since the work that needs to be done by each consumer is the same (read > write to hdfs) I am guessing my consumer code will just look the same and will need just one consumer group.
Thanks, Nishanth On Tue, Sep 12, 2017 at 8:53 AM, David Garcia <dav...@spiceworks.com> wrote: > Consumers can be split up based on partitions. So, you can tell a > consumer group to listen to several topics and it will divvy up the work. > Your use case sounds very canonical. I would take a look at Kafka connect > (if you’re using the confluent stack). > > -Daivd > > http://docs.confluent.io/current/connect/connect-hdfs/docs/index.html > > On 9/11/17, 4:48 PM, "Nishanth S" <nishanth.2...@gmail.com> wrote: > > All, > I am very new to kafka . We have a case where we need to ingest > multiple > avro record types . These avro record types vary vastly in volume and > size > and I am thinking of sending each of these message types to a > different > topic and creating partitions based on volume and through put needed. > What > the kafka consumer has to do is read the record of from partitions > and > write to different hdfs locations based on record type . I am > guessing > we should at least start with one consumer per topic . Is this > understanding correct or is there a better way to look at it? > > >
