On Sun, Jan 16, 2011 at 8:39 AM, Mag Gam <magaw...@gmail.com> wrote: > I am curious now... > > If you have a cluster the size of 10, what should the heartbeat be set > as? What about 100, 1000? >
The heartbeat interval auto-tunes based on size of the cluster. I've never seen anyone tune these settings (yet) > > > I too am interested in tuning documentation. For example, how much > memory should we allocate to JVM? How much memory for namenode? etc... > > My rough guide (with a decent fudge factor built in) is 1GB RAM on NN per million files. This is obviously a rule of thumb since the amount of RAM taken by a file depends on a lot of factors (number of blocks, length of filename, etc) but as a rough guide it should give you a decent idea. Konstantin Shvachko did some better analysis on this a few months back: http://www.usenix.org/publications/login/2010-04/openpdfs/shvachko.pdf His number is 0.6GB per 1M files, but not sure if that includes a "safety factor". -Todd -- Todd Lipcon Software Engineer, Cloudera
