+1 Topic Deletion with 0.8.1.1 is extremely problematic, and coupled with the fact that rebalance/broker membership changes pay a cost per partition today, whereby excessive partitions extend downtime in the case of a failure; this means fewer topics (e.g. hundreds or thousands) is a best practice in the published version of kafka.
There are also secondary impacts on topic count — e.g. useful operational tools such as: http://quantifind.com/KafkaOffsetMonitor/ start to become problematic in terms of UX with a massive number of topics. Once topic deletion is a supported feature, the use-case outlined might be more tenable. Best Regards, -Jonathan On Sep 5, 2014, at 4:20 AM, Sharninder <sharnin...@gmail.com> wrote: > I'm not really sure about your exact use-case but I don't think having a > topic per user is very efficient. Deleting topics in kafka, at the moment, > isn't really straightforward. You should rethink your date pipeline a bit. > > Also, just because kafka has the ability to store messages for a certain > time, don't think of it as a data store. Kafka is a streaming system, think > of it as a fast queue that gives you the ability to move your pointer back. > > -- > Sharninder > > > > On Fri, Sep 5, 2014 at 4:27 PM, Aris Alexis <aris.alexis....@gmail.com> > wrote: > >> Thanks for the reply. If I use it only for activity streams like twitter: >> >> I would want a topic for each #tag and a topic for each user and maybe >> foreach city. Would that be too many topics or it doesn't matter since most >> of them will be deleted in a specified interval. >> >> >> >> Best Regards, >> Aris Giachnis >> >> >> On Fri, Sep 5, 2014 at 6:57 AM, Sharninder <sharnin...@gmail.com> wrote: >> >>> Since you want all chats and mail history persisted all the time, I >>> personally wouldn't recommend kafka for your requirement. Kafka is more >>> suitable as a streaming system where events expire after a certain time. >>> Look at something more general purpose like hbase for persisting data >>> indefinitely. >>> >>> So, for example all activity streams can go into kafka from where >> consumers >>> will pick up messages to parse and put them to hbase or other clients. >>> >>> -- >>> Sharninder >>> >>> >>> >>> >>> >>> On Fri, Sep 5, 2014 at 12:05 AM, Aris Alexis <snowboard...@gmail.com> >>> wrote: >>> >>>> Hello, >>>> >>>> I am building a big web application that I want to be massively >> scalable >>> (I >>>> am using cassandra and titan as a general db). >>>> >>>> I want to implement the following: >>>> >>>> real time web chat that is persisted so that user a in the future can >>>> recall his chat with user b,c,d much like facebook. >>>> mail like messages in the web application (not sure about this as it is >>>> somewhat covered by the first one) >>>> user activity streams >>>> users subscribing to topics for example florida/musicevents >>>> >>>> Could i use kafka for this? can you recommend another technology maybe? >>>> >>> >>
