Can someone (Martin?) please post some rough performance and scalability numbers per backing storage type? I see these DDD/ES/CQRS discussions lead to consumer-developer limitations based on performance and scalability, but I have not seen any actual numbers. So please post numbers in events per-second as I would prefer not trying to hunt down such numbers in old posts.
I keep saying this, but it seems without much success, but akka-persistence, even at its slowest, probably still performs 5x-10x better than most relational stores, and at its best perhaps 500x better. I often poll my IDDD Workhop students for realistic transactions per-second numbers and most of the time it is at or below 100 tps. Some of the higher ones are around 1000 tps. A few students identify with 10,000 tps. As far as I know, akka-persistence can regularly perform in the 30,000 tps. With some store tops out at, what, 500,000 tps? The point I am trying to make is, even if you put a singleton sequence in front of your slowest possible store, let's assume that it could cost 30%. That would still leave performance at 20,000 tps on your slowest store, which is 20x faster than many, many enterprise applications. (There are faster ways of producing incremental sequences than using a singleton atomic long.) I vote that you need to have a single sequence across all events in an event store. This is going to cover probably 99% of all actor persistence needs and it is going to make using akka-persistence way easier. A suggestion: rather than looking so carefully at akka-persistence for performance and scalability increases, I think a lot could be gained by looking at false sharing analysis and padding solutions. Vaughn On Thursday, August 7, 2014 12:01:26 PM UTC-6, ahjohannessen wrote: > > On Wednesday, August 6, 2014 9:08:14 AM UTC+1, Martin Krasser wrote: > > Kafka maintains an offset for each partition separately and a partition >> is bound to a single node (disregarding replication). For example, if a >> Kafka topic is configured to have 2 partitions, each partition starts >> with offset=0, and, if you consume from that topic you only obtain a >> partially ordered stream because Kafka doesn't define any ordering >> across partitions (see Kafka docs for details). This situation is >> comparable to other distributed datastores. For example, Cassandra only >> maintains an ordering for entries with the same partition key (i.e. for >> entries that reside on the same node). >> >> In general, if you want to maintain an ordering of entries, you either >> have to use >> >> - a single writer in the whole cluster (which is the case for persistent >> actors) or >> - keep entries (that are generated by multiple producers) on a single >> node so that the server is able to maintain a local counter (which is >> what Kafka does with offsets for each partition separately) >> >> Both limits scalability (as already mentioned by Patrik) for both write >> throughput and data volume. It may well be that some applications are >> fine with these limitations and benefit from a total ordering of entries >> per "tag" but this should not be the default in akka-persistence. IMO, >> it would make sense if akka-persistence allows applications to configure >> an optional ordering per "tag" so that users can decide to sacrifice >> scalability if total ordering is needed for a given tag (and it is up to >> journal implementations how to implement that ordering). >> > > Interesting idea, especially since not all of us think in terms of > scalability. > I agree with your opinion wrt optional ordering per tag. It would be nice > to > find a middle-ground that made sense wrt scalability *and* the need of > being able to use offset per group of persistent actors of same type. > > >> As already mentioned in a previous post, causal ordering could be a >> later extension to akka-persistence that goes beyond the limits of a >> single writer or co-located storage *and* allows for better scalability. >> I wish I had more time for hacking on a prototype that tracks causalities >> :) >> > > That seems to be a great solution, but something tells me that this is > not going to happen soon. > > So, my conclusion is that I will go with using a single persistent actor > as a journal, > thus getting a "tag" + seqNr, and use persistent views to what I otherwise > would > use a normal persistent actor for, e.g. as in this sketch: > - https://gist.github.com/ahjohannessen/70381de6da3bde1c743e > and use snapshotting to reduce recovery time of persistent views. > > Patrik, on a second thought, perhaps tags per event would not be that > silly, at > least it gives more flexibility. However, I suppose it all depends on what > you guys > want to use a "tag" for. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
