Hi Konrad,
Kafka does handle a throughput well over our requirements. The problem with Kafka is that it doesn’t support this amount of topics/partitions. So we’ll need to implement some kind of adapter that will split the events by entity(session) ID. Additionally to subscribing to a stream of events we also use the events for recovery. In this case Kafka is even less convenient, since, because of the lack of the required topic granularity, in order to recover an entity, all requests from the topic/partition should be replayed, to only filter the relevant ones. For recovery something like cassandra would be more convenient. Actually for anything other than Pub/Sub something like Cassandra would be convenient. Another option is using some combination of Kafka/Cassandra. This will require some adapting, handling 2 systems instead of one, possible lack of sync, writing more code, etc. I do have a feeling that one data store that can meet those kind of requirements would be very useful. My guess so far is that this kind of data store doesn’t exist. Just wondering if you know of anything like this that might be available in the near future. Thanks, Leonid On Tuesday, August 18, 2015 at 11:40:59 AM UTC+3, Konrad Malawski wrote: > > Hi Leonid, > If you need pub-sub at that scale I'd recommend using Kafka as a Journal. > With the new query side we anticipate the Kafka ReadJournal being one of > the most interesting applications actually. > (We drafted an implementation and it's not that hard and also very > powerful to expose Kafka subscribers as Akka Streams - see reactive-kafka > <https://github.com/softwaremill/reactive-kafka>). > > -- > Cheers, > Konrad Malawski > Akka @ Typesafe > > On 17 August 2015 at 10:06:15, [email protected] <javascript:> ( > [email protected] <javascript:>) wrote: > > One of the possible use cases of event sourcing we’d like to implement is > subscribing to domain events changes. > > The requirements are quite high: > Over 100000 live topics/entities, preferably up to 1,000,000 > Up to 50000 updates per second > Low latency - less than 10msec, average > > Topic scale (at low latency) seems to be the most challenging on this list. > > With the upcoming akka persistence query module, which provides a stream > instead of the polling it seems that the bottleneck moves to data store > implementation. > Persistent query would really benefit from a data store able to scale the > number of subscriptions while keeping the latency low. > > Can you recommend a data store that can provide the above (or close to > above) requirements? > > Thanks, > Leonid > > This message may contain confidential and/or privileged information. > If you are not the addressee or authorized to receive this on behalf of > the addressee you must not use, copy, disclose or take action based on this > message or any information herein. > If you have received this message in error, please advise the sender > immediately by reply email and delete this message. Thank you. > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
