Thanks Mattijs. I understand from your mail that there is no issue using HL consumer except for the loss of flexibility.
Regards, Kashyap On Mon, Mar 24, 2014 at 3:20 AM, Mattijs Ugen <[email protected]> wrote: > Is there any downside to using Kafka high level consumer as spout? >> > The main downside of the high level consumer is that you won't be able to > control exactly when it will request a broker for more data and that it > will always commit the latest offset you read from the stream it provides. > In a somewhat continuous stream of messages, the first part won't matter > much, you can tweak all of the client properties listed on the kafka site. > The latter part becomes somewhat complicated when you need to be able to > replay messages that fail within your topology, assuming you don't want > your client to commit anything that hasn't been ack()'d to the spout. > > There's up and downsides to using either the low or high level client, > feel free to examine the differences between https://github.com/ > wurstmeister/storm-kafka-0.8-plus (low level client) and > https://github.com/HolmesNL/kafka-spout/ (high level client). > > > I plan to spawn threads to read from various partitions of the topic in >> Kafka. >> > I reckon that won't be necessary from storm; storm will manage the > parallelism for you in terms of multiple spout instances among your > clusters. As long as you put spouts in the same consumer group, they'll > together consume all partitions (even a single client will switch > partitions now and then to ensure all are read). > > Kind regards, > > Mattijs > >
