On Oct 22, 2009, at 9:37 AM, <[email protected]> wrote:
As with Dhruba's comment, so long as it is just the namenode that is
running on a networked file system everything should be chill. The
namenode
keeps all of its working metadata in main mem, and it only
occasionally
pushes a log file out to hard storage (and if I remember correctly
you can
adjust this time window in one of the site files).
Actually it pushes out the update logs on each and every update
synchronously.
The checkpoint however is pushed out periodically.
Also, at yahoo, we push out NN state to multiple disks and one of the
"disks" is a nfs filer. This is configurable.
sanjay
However, you are going to run into huge performance issues running
datanodes over a networked storage system. Having to push that many
file
requests over a network for a respectable mapreduce job is going to
kill
your equipment.
- Grant
On Oct 21 2009, Jonathan Seidman wrote:
>Apologies if this has been answered previously, but I'm unable to
find
>anything that seems to cover this.
>
> It's clear that datanodes require local storage for Hadoop to
function
> efficiently, but is there any significant disadvantage to using
external
> storage for namenodes? We're exploring the possibility of using a
> different class of hardware for our namenodes with attached
storage and
> little or no internal storage. Some of the benefits this would
provide us
> are: 1) allowing our sysadmins to deploy hardware that they're
familiar
> with and already have considerable experience keeping up in a
production
> environment. 2) no namenode downtime to replace a failed disk.
>
>We don't anticipate that this approach would cause any significant
>degradation to performance, but let me know if there's something
we're not
>considering.
>
>Thanks.
>
>Jonathan
>
--
--
Grant Mackey
PhD student Computer Engineering
University of Central Florida
Rm 231 cube 5 (321) 960-8851
