On Wed, Jul 21, 2010 at 11:45 PM, Brian Bockelman <[email protected]> wrote: > Hi Bobby, > > We keep 2 or so replicas here at Nebraska. We have about 800TB of raw space. > > As a rule of thumb, we: > 1) Increase the replication of extremely important files. We are a site for > the LHC, so a large part of our data is stored on tape elsewhere (but not > everything!). It's an operational pain to re-download a few tens of TB, but > not the end of the world. > 2) Estimate that we will lose 1 file per month due to disk corruption and > loss. > 3) Make sure our management understands the risks due to (2) and what would > occur during a double-node failure. > > There are 2 other similarly sized sites that roughly follow the same rules. > > We haven't discovered any fatal software bugs that cause data loss since the > various ones in 0.19 were ironed out. > > Brian > > On Jul 21, 2010, at 8:29 PM, Bobby Dennett wrote: > >> The team that manages our Hadoop clusters is currently being pressured >> to reduce block replication from 3 to 2 in our production cluster. This >> request is for various reasons -- particularly the reduction of used >> space in the cluster and potential of reduced write operations -- but >> from what I've read previously, it seems to be strongly discouraged. >> >> Of course I can't find it now, but I recall seeing a post that Doug >> Cutting was involved with stating that having replication 3 is something >> like 100 times "safer" than replication 2. If I remember correctly, >> there was mention of potential NameNode bugs that could introduce >> undetected corrupted/missing replicas so the idea was that if more >> replicas are created, the chance of this type of bug is much less. On a >> related note, it seems that the companies using a reduced replication >> factor (e.g. Facebook) have also built an application layer on top of >> Hadoop to perform exception handling, corruption issues, etc. >> Unfortunately, we do not currently have the resources to do something >> similar. >> >> For anyone currently using a replication of 2 in production, can you >> please share your experience and any issues you may have encountered? >> Also, I would appreciate any thoughts about whether a replication factor >> of 2 can be considered "safe". >> >> Thanks in advance, >> -Bobby > >
+1 On the above. We use replication of 2. I have not seen a specific problem from it. When you get a nagios alert while at the beach on a Saturday, that you lost a node, you sweat a little more and it is not from the sun.
