Re: [akka-user] Best practices using Akka Persistence with long-running projects?
Hi Martin, Have you also look at Apache Avro? I haven't used in it conjunction with Akka (Persistence) yet, but have used it in other projects with success. It does have a strong notion of schema evolution. Checkout http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html for an in-depth treatment. Cheers, Sander On Friday, July 25, 2014 6:42:11 PM UTC+2, Martin Simons wrote: Am Donnerstag, 24. Juli 2014 11:51:57 UTC+2 schrieb Konrad Malawski: I know I’ve just grown your to-read list by quite a bit, but I hope this helps! :-) Glad you did ;-) I at least skimmed all the articles you posted and decided that Protobuf is probably the most common thing to use. I went on to hack together a small example project, trying to keep the amount of boilerplate code per class as low as possible as it will be required for each and every event and state class that might be added in the future. No matter how one does it, it will always look more or less messy when translating Protobuf Java objects to Scala case classes, I'm afraid. The result of my humble attempts can be found here: https://github.com/lunikon/akka-persistence-serialization The code is still a mess, obviously, but do you see a general problem with this approach (the approach being that one defines an adapter for every case class and registers it with a static registry which is then used by the Akka Serializer implementation)? -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] Best practices using Akka Persistence with long-running projects?
Hello Martin, yeah, that looks like what I had in mind. And yeah, translating to/from proto is always a bit of verboseness somewhere, here's how we try to minimise the verboseness (serializer impl): https://github.com/akka/akka/blob/release-2.3/akka-persistence/src/main/scala/akka/persistence/serialization/MessageSerializer.scala#L51 Sadly, the serializers that are slick and nice to work with, don't usually have versioning/evolution built in from what I see available... Kryo is fast and using it is great, but no versioning / anything built in = no-go for this use case (IMO at least). On Fri, Jul 25, 2014 at 6:42 PM, Martin Simons mar...@lunikon.net wrote: Am Donnerstag, 24. Juli 2014 11:51:57 UTC+2 schrieb Konrad Malawski: I know I’ve just grown your to-read list by quite a bit, but I hope this helps! :-) Glad you did ;-) I at least skimmed all the articles you posted and decided that Protobuf is probably the most common thing to use. I went on to hack together a small example project, trying to keep the amount of boilerplate code per class as low as possible as it will be required for each and every event and state class that might be added in the future. No matter how one does it, it will always look more or less messy when translating Protobuf Java objects to Scala case classes, I'm afraid. The result of my humble attempts can be found here: https://github.com/lunikon/akka-persistence-serialization The code is still a mess, obviously, but do you see a general problem with this approach (the approach being that one defines an adapter for every case class and registers it with a static registry which is then used by the Akka Serializer implementation)? -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, Konrad 'ktoso' Malawski hAkker @ Typesafe http://typesafe.com -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] Best practices using Akka Persistence with long-running projects?
I had a reply prepped yesterday but lost the draft (argh offline editing!). Back to the topic though: First of all, thank you very much for the elaborate reply. I guess I'll get hacking away at some prototypes soon and so some performance testing close to our use-case. Looking forward to it, too :-) Great to hear, let us know about your findings please :-) So what you suggest is that the serializer first deserializes a stored event forgivingly, without throwing an exception for for example dropping removed field and initializing new fields with 0/null/default values, and then I compare the version and apply additional conversion logic? Or do you mean something more complex like storing a dehydrated version of the event and re-hydrating it every time an event is loaded from storage, promoting it to the latest version in this process? This very much depends on what serialiser you use, because they inherently support some kind of schema evolution. I’ve been mostly using protocol buffers in my career. There, what you do is: mostly use optional fields so these can be null, and you have to be prepped for this anyway. And then you can reserve some space for extensions, using that you’re able to apply this trick: http://www.indelible.org/ink/protobuf-polymorphism/ Because of how proto works, renames are safe for example - because it’s not using the name in the binary format, but the ID. etc etc… This should give you a feel how “bound” your evolution strategies will become to the serialisation format. Other basic tips are nicely summarised here: http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html Again though, dig deeper into the details of each serialisation format before you decide to go for it (we don’t really have a “use that one” opinion on this). Other serialisation formats, such as Simple Binary Encoding (designed to be low latency envs - check benchmarks) support this via keeping version numbers in the message header: https://github.com/real-logic/simple-binary-encoding/wiki/Message-Versioning In essence - version number = some number of extension fields, you cannot delete fields in this one. One other format I’d like to point to here is Cap n’ Proto (same author as original protobufs AFAIK), and their description on this (as it’s listing it up quite nicely what is safe and what is not): http://kentonv.github.io/capnproto/language.html#evolving_your_protocol Summing up… These strategies allow you to change your events and stay compatible with the same type - to still be able to deserialise. If you however at some point make some huge change, and need to get events B but you have events A serialised, or you need to add additional data you can add another layer that would promote A events to B events, and then emit them to the actor. This gets more complicated, but the point is - you’re not doomed even if you have to do bigger changes (try to avoid this ofc ;-)). I know I’ve just grown your to-read list by quite a bit, but I hope this helps! :-) -- Konrad 'ktoso' Malawski hAkker @ typesafe http://akka.io -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] Best practices using Akka Persistence with long-running projects?
First of all, thank you very much for the elaborate reply. I guess I'll get hacking away at some prototypes soon and so some performance testing close to our use-case. Looking forward to it, too :-) One question remains: Am Montag, 21. Juli 2014 13:28:27 UTC+2 schrieb Konrad Malawski: The other method I like *more *personally is promoting events. So an event comes in it's in version 2. You're running your app on version 5 events. You have promotions ready in your system and can promote an v2 event to v3, then to v4, then to v5, and after promoting it to your current runtime's version, you can give it to your actors. What's the wins? You don't keep the old implementations around and you and you can hook in the promotions into the custom serialiser in akka. Read in old events, promote them, return them to be used. So what you suggest is that the serializer first deserializes a stored event forgivingly, without throwing an exception for for example dropping removed field and initializing new fields with 0/null/default values, and then I compare the version and apply additional conversion logic? Or do you mean something more complex like storing a dehydrated version of the event and re-hydrating it every time an event is loaded from storage, promoting it to the latest version in this process? -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
