> How much memory are you giving the NameNode? and the SecondaryNameNode?
We give the NN 4 GB and the 2NN the default 1 GB. Technically according to the Hadoop manual (which suggests the 2NN's task is as resource intensive as the NN's) this is wrong, but with the HA configuration of the NN, in our setup the 2NN is not critical, and it functions well enough. I'm not even sure we need it. Also given the current number of files in the filesystem, not all of the 4 GB heap allocated to the NN is actually required. > but do they take a lot of CPU? Because everything falls apart if HDFS falls apart, the NN deserves special consideration. It depends on the particulars of your workload but in general an environment which includes HBase will be more taxing on the balance. I think RAM is the critical resource for the NN. For example, to my understanding, Facebook runs at least one cluster with >= 20 GB heap for the NN. It obviously tracks the block locations for millions of files. Give your NN a lot of RAM in the beginning and there will be plenty of headroom to scale up into -- you can add more datanodes over time in a seamless manner and won't need to bring down HDFS to upgrade RAM on the NN. > if i ignore HA could they share a box with other services? If you ignore HA, my advice is to run the NN and the HBase Master on the same node. The Master spends most of its time suspended waiting for work, so this would be a good match. I also run a DataNode in addition to the NN and Master on one node on my test cluster and have never had an incident. Your mileage may vary. Something like this is suitable for testing only. - Andy ________________________________ From: Fernando Padilla <[email protected]> To: [email protected] Sent: Friday, July 17, 2009 7:37:04 PM Subject: Re: hbase/zookeeper Ok.. so it seems like ZK and TT can be smaller than we thought.. at least it's an option. :) How much memory are you giving the NameNode? and the SecondaryNameNode? It looks like those are beefy on your setup for HA purposes.. but do they take a lot of CPU? if i ignore HA could they share a box with other services? Andrew Purtell wrote: > That looks good to me, in line with the best practices that are gelling as > we collectively gain operational experience. > This is how we allocate RAM on our 8GB worker nodes: > > Hadoop > DataNode - 1 GB TaskTracker - 256 MB (JVM default) > map/reduce tasks - 200 MB (Hadoop default) > > HBase > ZK - 256 MB (JVM default) > Master - 1 GB (HBase default, but actual use is < 500MB) > RegionServer - 4 GB > > We have a Master and hot spare Master each running on one of the workers. > Our workers are dual quad core so we have them configured for maximum > concurrent task execution of 4 mappers and 2 reducers and we run the > TaskTracker (therefore, also the tasks) with niceness +10 to hint to > the OS the importance of scheduling the DataNodes, ZK quorum peers, or > RegionServers ahead of them. > Note that the Hadoop NameNode is a special case which runs the NN in a > standalone configuration with block device level replication to a hot > spare configured in the typical HA fashion: heartbeat monitoring, > fencing via power control operations, virtual IP address and L3 fail > over, etc. > Also, not all nodes participate in the ZK ensemble. Some 2N+1 subset is > reasonable: 3, 5, 7, or 9. I expect that a 7 or 9 node ensemble can > handle 1000s of clients, if the quorum peers are running on dedicated > hardware. We are considering this type of deployment for the future. > However, for now we colocate ZK quorum peers with (some) HBase > regionservers. > Our next generation will use 32GB. This can support aggressive caching > and in memory tables. > - Andy > > > > > ________________________________ > From: Fernando Padilla <[email protected]> > To: [email protected] > Sent: Friday, July 17, 2009 10:30:52 AM > Subject: Re: hbase/zookeeper > > thank you! > > I'll pay attention to the CPU load then. Any tips about the memory > distribution? This is what I'm expecting, but I'm a newb. :) > > DataNode - 1.5G > TaskTracker - .5G > Zookeeper - .5G > RegionServer - 2G > M/R - 2G > > >
