Allen Wittenauer wrote:
On 9/21/08 2:51 PM, "Dmitry Pushkarev" <[EMAIL PROTECTED]> wrote:
Speaking about NFS-backup idea:
If I have secure nfs storage which is much slower than network (3MB/d vs
100MB/s network we use between nodes) will it adversely affect performance,
or I can rely on NFS caching to do the job?
I think Konstantin has some benchmarks in a JIRA somewhere that shows
that the current bottleneck isn't the fsimage/edits writes.
HADOOP-3860 has name-node benchmark numbers.
It concludes that for the name-node operations the bottleneck is exactly the
edits writes.
But another conclusion is that real-world clusters do not provide enough load
on the
name-node so that it could reach that bottleneck.
Particularly for NFS I found out that although it slows down the name-node
but the slow down is less than 5%.
And if nfs share dies, will it shutdown the namenode as well?
In our experiences, the name node continues. But be warned that it will
only put a message in the name node log that the NFS mount became
unwritable. There is a JIRA open to fix this though.
Name-node treats NFS shares the same as local ones, it does not distinguish
between
different storage directories. The name-node will continue to run until there
is at
least one storage directory available. So if you have one NFS share and one
local
and NFS fails the name-node will continue to run. But if NFS was the only
storage
directory the name-node will shut down.
--Konstantin
