Thanks J-D. With cleanly, I meant that it did not produce any errors other than the YouAreDeadException. When a very long pause causes this I often also see HDFS client related errors because of leases expiring, but this time that was not the case. It's not "cleanly" in the sense that I issued the shutdown myself.
Friso ________________________________________ Van: [email protected] [[email protected]] namens Jean-Daniel Cryans [[email protected]] Verzonden: vrijdag 21 januari 2011 18:32 Aan: [email protected] Onderwerp: Re: Data loss on clean RS shutdown without WAL? If the region servers gets YouAreDeadException, it does an "abort" and won't flush the data since another region server could already be serving the region. If you're not writing to the WAL, then yes it's data loss. Not sure what you mean by "shuts down cleanly" in your case, if you see a log that starts with "Aborting region server" then it's not really "clean". J-D On Fri, Jan 21, 2011 at 2:38 AM, Friso van Vollenhoven <[email protected]> wrote: > Hi all, > > Question: when a regionserver shuts down cleanly after a YouAreDeadException > and the master nicely reassigns the regions, will you loose any data that was > written to the memstore of the dead RS when not using WAL? > > There was no hard crash and not a single error in any of the logs (except for > the FATAL: YouAreDeadException). The RS lost its zookeeper session after a > timeout, probably GC combined with some other starvation on heavy load. I > think the memstore flushes on shutdown, but I am not entirely sure what > happens in the situation where regions are already opened by other > regionservers when the dying executes the shutdown code. Can I assume that > the RS that gets reassigned a region creates a new HFile and that this will > be compacted together with the one left by the dead RS at the next compaction > run? > > > Thanks, > Friso > >
