Hi Drizzlelites, Sorry about that, the presentation was larger than I thought. I have put it up on http://www.kindahl.net/pub/DynamicReplicationTopologies.pdf instead.
Best wishes, Mats Kindahl Mats Kindahl wrote: > > Adi Alurkar wrote: >> Greetings, >> >> It was great to meet everybody at Developer day. I have been catching up >> with all the docs on the protocol and discussing them with our CTO and in >> the last brainstorm with him we realized that although the proposed protocol >> offers great enhancements in terms of exposing shard id / QUERY / QUERY_RO >> in the protocol itself, the following idea would make it even better. >> >> To simplify and explain the problem consider the following architecture: >> (also note that this would apply to sharded DBD scenario as well) >> >> Master Slave/Replica >> M1 ---> R1 >> ---> R2 >> ---> R3 >> ... >> ---> R<n> >> >> * R1, R2 use a persistent storage engine (primarily) to serve as backups to >> M1 >> * R3...RN use memory based/memcache storage engines and used to handle most >> of the "READ ONLY" traffic. >> * Data is being updated at the M1. >> * Data is being replicated to each of the replicas R1..RN. >> * The replicas can be applying the changes at master at different rates and >> are thus at different consistency state's >> * The application assumes eventual consistency. This implies the following >> all request for data made by a thread should only be serviced with data that >> is at a consistency state higher than or equal to of the previous state. >> Using timestamp as a crude gauge to measure consistency if thread TH1 >> requests data at 10:00 all subsequent data request from TH1 should only see >> data that is consistent as of 10:00 or later. >> * In reality timestamp is not the correct way to track consistency, which is >> where we rely on SCN within oracle to achieve consistent reads. The simplest >> definition of SCN is a increasing counter which determines the consistency >> state of the database. >> * C<n> indicates the consistency state.(Needless to say this needs to be >> serialized and determined at the end of a transaction) >> >> Consider the following use case in the above architecture: >> >> * all request where QUERY_RO can be proxied to all servers but all QUERY >> i.e. writes/updates need to be sent to and can be served only by M1. >> * new thread TH1 updates (either, INSERT/UPDATE/DATE/DDL) the data at the >> master, it received a C1 as part of the confirmation of the update and is >> set in the protocol. >> * all subsequent QUERY_RO requests via TH1 include C1 as part of the request >> (set in protocol) thus can be served only by servers that have consistency >> state >=C1. Each satisfied request resets the consistency state in the >> protocol. >> * lets assume TH2 performs multiple updates resulting in the M1 to be at >> C50. R1 is at C45, R2 is at C38, R3 is at R24, etc >> * TH1 request data (QUERY_RO) and is at C24. The request can be routed to >> any of the candidate servers which have consistency state >= C24. >> * a new TH3 requests data (QUERY_RO) with empty consistent state (or 0) and >> thus can be served by the first available server. > > This only works if there is a single source of data (i.e., as you note below, > a > single master). > >> As indicated above adding the concept of SCN and exposing it via the >> protocol would make this a really awesome feature. Some other enhancements >> that would make the SCN concept even more rock solid include: >> * Oracle's SCN is tied to the Oracle instance and is not portable from one >> machine to another, except incase of physical standby. This makes it very >> difficult to implement partial replication(ability to replicate only certain >> objects/schemas), support for multi-master multi-slave replicated >> environment. Not to mention moving a slave from one server to another >> impossible unless the new master is a physical standby of the original >> server. > > Exactly. > >> * support manual "override" of the SCN for slaves, this would allow the >> slaves to inherit the SCN from the master per transaction making >> slave->master promotion easier. > > This just adds to the confusion, since "faking" consistency can be a good > source > of problems. The problem is that you assume a total order, but that is not > necessary. > > Use a vector instead (called "version vector" by some), where you have a > partial > order, but not a total order on the elements. I have attached a short > presentation I did for the replication team at MySQL a few years ago, which > describes this approach. > >> * determine SCN from a central source which uses consensus/paxos, etc > > That just adds a single point of failure, which of course can be used, but > does > not scale very well, since you then need to replicate the SCN central source > as > well, and then you need to have an SCN for every SCN distributed (assuming you > use the same scheme for checking consistency of the SCN server), which is kind > of recursive. > >> Does the idea of supporting something like a SCN and exposing it in the >> protocol to support use case such as above seem valid? > > Absolutely. > > However, it would be nice to be able to handle a branching temporal model as > well, not just a linear temporal model. There were some mention that slaves in > an "alternate future" frequently is a problem, so right now I am thinking of > ways to handle that without introducing an unnecessary overhead. > > Just my few cents, > Mats Kindahl > > > ------------------------------------------------------------------------ > > _______________________________________________ > Mailing list: https://launchpad.net/~drizzle-discuss > Post to : [email protected] > Unsubscribe : https://launchpad.net/~drizzle-discuss > More help : https://help.launchpad.net/ListHelp -- Mats Kindahl Senior Software Engineer Database Technology Group Sun Microsystems _______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
