As I read your email and the link within it, I believe that your question is:
* Does there exist or is anyone working on a KafkaSpout which stores its offsets in a highly durable Kafka Topic, instead of within ZooKeeper? Is that a correct interpretation of your question? Thank you for your time! +++++++++++++++++++++ Jeff Maass <[email protected]> linkedin.com/in/jeffmaass stackoverflow.com/users/373418/maassql +++++++++++++++++++++ On Fri, May 15, 2015 at 1:34 PM, Cristian Makoto Sandiga < [email protected]> wrote: > Hi, > > I want to replace my RedisSpout and i was developing a POC, with > KafkaSpout, but i'm worried for consumer offset manager with zookeeper and > HighLoad write, maybe loose offset, failover, etc. > > I was thinking about a zookeeper cluster for any data loose, but i think > is still better save in kafka server offset at least is a kafka > recommendation > https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka > . > > That has any documentation about offset.storage in KafkaSpout i read all > mailing list but i didn't find nothing talking about it, has anybody > working in a possible feature? or maybe somebody has a good recommendation > for me?. > > I really will appreciate your help. > > Application with normal traffic: 100 000 msgs/seg High 500 000 msgs/seg > > replication factor-> 2 > > 1 topic -> 10 partitions > 1 topic -> 10 partitions > 1 topic -> 5 partition > > storm-kafka -> 0.9.4, > kafka_2.10 -> 0.8.2.1 >
