Hi guys, As the subject suggests, I'm really interested in using the Raft consensus protocol over XMPP. I'll give you some background on the project I'm working on, some info on Raft and will then try to explain why I think XMPP is a great choice for transporting the raft data. It's the first time I've considered working on an XEP, so any constructive or critical feedback will truly be welcome! Also, thanks in advance for taking the time to read through all this.
The project I'm working on provides a data store to clients over XMPP. The data is mostly static and at most would be updated a couple of times an hour, but generally perhaps once per day or so. Clients interact with the data store by sending <iq/> stanza requests and getting the relevant responses. So far so good, but I also need it to be highly available. To do that, I have added the JIDs of the various data stores to DNS (just a TXT record at a known location). With this the client can then send a request to a JID of its choice and request some data. However, this doesn't solve the high availability problem. I'd like to ensure that all nodes are in sync and are consistent. This lead to a lot of reading around until I came across the Raft consensus protocol (https://raftconsensus.github.io/). Raft was developed at Stanford University as an alternative to the PAXOS protocol which is hard to understand and harder to implement. Raft describes a complete protocol that is simple, easy to understand and relatively complete so that it is straight forward to implement. If you haven't come across it before, please do check out that link as it's interesting in its own right - they have a nice animated simulator (and a walk through (http://thesecretlivesofdata.com/raft/) as well as a number of presentations. Raft works on the basis of a replicated log. The protocol describes how to make sure that each node receives the correct log and that it is committed and executed in order. It specifies a small number of simple messages that should be exchanged between nodes in order for exchange of state and leadership election. However, Raft does not describe what goes into the log, what the backend store is or more interestingly the transport layer. What I would like to do is exchange Raft messages between nodes using XMPP. My applications are already XMPP aware and all of their work is based on their JID. If Raft could be used over XMPP, this would allow me to use a node's JID as the node's Raft ID. This is both convenient and beneficial in that I can re-use the existing transport layer. It also makes it really easy for other people to deploy Raft using applications around the Internet as XMPP takes care of delivering the messages - no need to mess with IP addresses, ports and firewall rules - you can just re-use your existing XMPP connection. I am already working on a prototype to let me do this using custom <message/> stanzas. It would be easy enough to do this as 'chat' and place the payload in <body/>, but because the data fits structured XML so nicely, it just seemed plain wrong to overload 'chat'. So, as I'm probably going to have to do this work anyway, I wanted to get in touch with the community and see whether or not it thinks this would be a suitable case for an XEP. To be clear, I'm not suggesting we implement Raft itself in XMPP, but merely define a mechanism for transporting Raft messages within a cluster. I'm very happy to do the leg work and I'll certainly take on board all feedback that I get. If the overall vibe is positive, I'll start putting together a proto-XEP for submission to the XEP Editor. Thanks again for taking the time to read through this long email and I look forward to working with you guys in the future! Kind Regards, Peter Membrey
