Hi Jaen, I hope you’re liking what you have so far with sharding and persistence! :-)
Before I dive into your questions, I have one myself (both because I’m curious, and because it’s an issue that should be brought explicitly up when doing these things): So when you have these two versions of the nodes running - joined in the same cluster - how do you deal with semantic differences of these? I’m curious how you guys approach this problem. Do you assume it’ll be fine or use version tagging of some kind to notice “I may be talking to an old node” and phase them out slowly? Back to your persistence questions though: 1) We do not currently have a nice utility that would help to migrate between journals. Please note that akka-persistence it still experimental, and we’ll be changing things in there for 2.4 (read this for more details: http://letitcrash.com/post/96687159512/akka-persistence-on-the-query-side-the-conclusion (roadmap is on the typesafe blog)). Having that said, I think it would be rather simple to implement once we have the new read-side of persistence, so I filed an issue to track this idea: https://github.com/akka/akka/issues/16332 Currently the way you’d go about writing the migration is Persistence(system).journalFor("") ! ReplayMessages and for each of those send this over to another ActorSystem PersistentActor that would persist into the second journal. We are aware of one limitation in the current API that makes this a bit harder: Ideally you’d like to “read all processors”, but there is not message for this, and you have to know which persistenceIds you want to replicate. There is a pending issue open for this: https://github.com/akka/akka/issues/13892 which would make it possible to get all processorIds, and then run the migration. It would have to be implemented by existing journal plugins though. So there isn’t a super easy migration path for this scenario yet - please remember it’s an “experimental” module until 2.4 early 2015. 2) The second question can be either very scary or not scary at all… If you mean to start the 2nd part of the cluster on a different journal and then hope once you kill the old one the sharding manager will be able to replicate over to the new one and resume “as if nothing has happened” – sadly no, this is not possible. The reason is that the shard coordinator stores shard locations etc in the journal. If you move it to a system that uses a different journal, it won’t have the data available there for it - so it will either crash or start anew - possibly forgetting about existing shard actors etc… So I’d highly suggest shutting down the cluster when switching journals here. Fun fact is that we have many “big and smart” (no names though) clients who advertise for this strategy even if they could run 2 versions of things in the cluster - it’s just a risky (see my question on semantic differences of the same message). Short answer: no, you can not live migrate a running cluster between journals when using cluster sharding. The journal is required for sharding to keep it’s information about the cluster’s state. So all in all, I think if you’re diving into it right now - before 2.4, try to set up the journal you’ll end up using in the long term before you’re stuck with one. The cassandra one is pretty popular and well maintained by Martin Krasser for example. I hope this helps! On Mon, Nov 17, 2014 at 11:04 AM, Jean Helou <[email protected]> wrote: > Hi, > > Some context > > We are working on a public accessible web application (openoox.com). At > the moment the application run on play 2.3.6 and akka 2.3.6 > > We have implemented sharding in our application to distribute the load of > some user actions throughout our cluster. At the moment cluster sharding is > the only thing we use which depends on akka persistence. > Our application uses mongodb as its main datastore, and when implementing > the sharding prototype we used > https://github.com/ironfish/akka-persistence-mongo to get a shared > journal. > > Using mongodb as a journal store doesn't seem to be the best option but we > don't really have the time to setup an alternate store at the moment. We > will probably do so as we start moving business actors to an akka > persistence implementation. > > One last point to note is we use a rolling deployment configuration where > "new" and "old" servers live and communicate together for about 30 minutes > before the old servers are taken down.Which means that akka nodes with the > new and old configuration live in the same cluster for a while. > > The questions > > - Is there anything to help us migrate the existing Cluster sharding > events to the new store ? It isn't clear to us how we will be able to do > the migration ... our understanding of the documentation is that an actor > system can only have one backing store for akka persistence. > This means a migration plan would probably require creating another actor > system with the new backing store, implementing a custom persistent actor > in both actor systems using the same persistence id as our shard > coordinators and forward the recovered events from the actor backed by the > old store to the actor backed with the new store. I think we should also > delay shard startup until the migration is over, would that be enough not > to loose events ? > > - In the event we are unable to migrate the events, what would the > consequences be for the sharded cluster ? More specifically would the nodes > backed by the new store sill be able to process their shards or should we > plan downtime for the website while we do the migration ? > > Does anyone have experience with these kind of problems ? Should we avoid > migration entirely and delay the deployment of sharding until we have time > to setup a more appropriate store for akka persistence ? > > We would welcome insight before we take time to experiment ... > > Thanks > Jean > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
