Thanks JD for the response... I was just wondering if issues have ever been seen with regards to moving over a large number of WAL(s) entirely from one region server to another since that would double the replication related load on the one server which takes over.
Another side question: After the WAL has been replicated - is it purged immediately or soonish from the zookeeper ? Thanks Varun On Mon, May 20, 2013 at 9:57 AM, Jean-Daniel Cryans <[email protected]>wrote: > On Mon, May 20, 2013 at 12:35 AM, Varun Sharma <[email protected]> > wrote: > > Hi Lars, > > > > Thanks for the response. > > > > Regarding #2 again, so if RS1 failed, then the following happens... > > 1) RS2 takes over its logs... > > 2) Master renames the log containing directory to have a -splitting in > the > > path > > 3) Does RS2 already know about the "-splitting" path ? > > It will look at all the possible locations. See > ReplicationSource.openReader > > > > > Also on a related note, was there a reason that we have all region > servers > > watching all other region server's queue of logs. Otherwise, couldn't the > > master have done the reassignment of outstanding logs to other region > > servers more fairly upon failure ? > > I think I did it like that because it was easier since the region > server has to be told to grab the queue(s) anyway. > > > > > Thanks > > Varun > > > > > > On Sun, May 19, 2013 at 8:49 PM, lars hofhansl <[email protected]> wrote: > > > >> #1 yes > >> #2 no > >> > >> :) > >> > >> Now, there are scenarios where inconsistencies can happen. The edits are > >> not necessarily shipped in order when there are failures. > >> So it is possible to have some Puts at T1 and some Deletes at T2 (T1 < > >> T2), and end up with the deletes shipped first. > >> Now imagine a compaction happens at the slave after the Deletes are > >> shipped to the slave, but before the Puts are shipped... The Puts will > >> reappear. > >> > >> -- Lars > >> > >> > >> > >> ________________________________ > >> From: Varun Sharma <[email protected]> > >> To: [email protected] > >> Sent: Sunday, May 19, 2013 12:13 PM > >> Subject: Questions about HBase replication > >> > >> > >> Hi, > >> > >> I have a couple of questions about HBase replication... > >> > >> 1) When we ship edits to slave cluster - do we retain the timestamps in > the > >> edits - if we don't, I can imagine hitting some inconsistencies ? > >> > >> 2) When a region server fails, the master renames the directory > containing > >> WAL(s). Does this impact reading of those logs for replication ? > >> > >> Thanks > >> Varun > >> >
