I'll second that. We could use this yesterday. :-) Jeremy
On Thu, May 7, 2009 at 5:19 PM, Arjen Lentz <[email protected]> wrote: > Hi Clint! > > > > On 08/05/2009, at 1:54 AM, Clint Byrum wrote: > >> I've been thinking a lot about replication since the conference. Patrick >> Galbraith and I sat in the back of the room at the BOF on Wednesday night >> and worked on DBD::drizzle, but I was sort of half listening to the ideas >> being presented. If this one already was presented, then great, we'll have >> confirmation that I, indeed, have a subconscious that was listening, and it >> is working properly to bubble ideas up to my conscious mind ;). If not, then >> I'd like to get peoples' opinion on this before wandering into the code and >> working on it. >> >> >> The way MySQL's binlog works, it is basically a FIFO, in completion order. >> Because we don't know what operations were dependent on others that were >> completed before them, we have to perform them one by one in a single >> thread. This can get really ugly if you have a really high concurrency write >> laden master.. slaves might never catch up. >> >> After looking at google's work on a global transaction ID, I had a >> thought. What if every statement took note of the last comitted transaction >> ID when it started. When it completes and needs to be replicated, it records >> the end transaction ID as well. This gives us a window for each operation in >> the binlog. >> >> So, on a slave we read these items, which basically look like this (I'm >> writing statement based, but row based would be no different as long as >> single transactions' row operations are wrapped in transaction IDs the same >> way): >> >> >> StartTXID: 1 >> INSERT INTO t1 (id,x,desc) VALUES (1,3,'foo') >> EndTXID: 2 >> >> StartTXID: 1 >> INSERT INTO t2 (id, z) VALUES('a','bc') >> EndTXID: 3 >> >> StartTXID: 2 >> UPDATE t1 SET x = 9 WHERE ID=1 >> EndTXID: 4 >> >> StartTXID: 2 >> UPDATE t2 SET desc = 'the quick brown fox jumped over the lazy brown dog' >> WHERE id = 'a' >> EndTXID: 5 >> >> StartTXID: 1 >> UPDATE t2 SET archiveme=1 WHERE date <= '2008-01-01' >> EndTXID: 6 >> >> >> The slave can read ahead, performing each statement without committing it, >> until it encouters a StartTXID >= the lowest current EndTXID. This would >> result above in running the two inserts, then seeing that next StartTXID: 2 >> is >= 2, it must not start those statements until the other two have been >> committed. However, it can continue on in the log to the archival UPDATE and >> run it right now. As long as we have a transactional storage engine, we just >> have to commit them in the same order as they were committed on the master >> so that we don't present a view that never existed on the master. But much >> of the work can be done in parallel just like it was done on the master. >> >> It might even make sense to offer the user a choice between concurrency >> and absolute consistency by saying that statements can be comitted in any >> order as long as they're not dependent on one another. That would allow us >> to also apply this same mode of operation to non-transactional storage >> engines (I think..). >> >> This should allow us to take advantage of almost the same level of >> concurrency that the master has available to it. >> >> Thoughts? >> > > > I like. > > > Cheers, > Arjen. > -- > Arjen Lentz, Director @ Open Query (http://openquery.com) > Affordable Training and ProActive Support for MySQL & related technologies > > Follow our blog at http://openquery.com/blog/ > OurDelta: free enhanced builds for MySQL @ http://ourdelta.org > > > > _______________________________________________ > Mailing list: > https://launchpad.net/~drizzle-discuss<https://launchpad.net/%7Edrizzle-discuss> > Post to : [email protected] > Unsubscribe : > https://launchpad.net/~drizzle-discuss<https://launchpad.net/%7Edrizzle-discuss> > More help : https://help.launchpad.net/ListHelp >
_______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
