Suresh had made an spreadsheet for memory consumption.. will check.
A large portion of NN memory is taken by references. I would expect
memory savings to be very substantial (same as going from 64bit to
32bit), could be on the order of 40%.
The last I heard from Sun was that compressed pointers will be in very
near future JVM (certainly JDK 1.6_x). It can use compressed pointers
upto 32GB of heap.
I would expect runtime over head on NN would be minimal in practice.
Raghu.
Steve Loughran wrote:
does anyone have any up to date data on the memory consumption per
block/file on the NN on a 64-bit JVM with compressed pointers?
The best documentation on consumption is
http://issues.apache.org/jira/browse/HADOOP-1687 -I'm just wondering if
anyone has looked at the memory footprint on the latest Hadoop releases,
on those latest JVMs? -and which JVM the numbers from HADOOP-1687 came
from?
Those compressed pointers (which BEA JRockit had for a while) save RAM
when the pointer references are within a couple of GB of the other refs,
and which are discussed in some papers
http://rappist.elis.ugent.be/~leeckhou/papers/cgo06.pdf
http://www.elis.ugent.be/~kvenster/papers/VenstermansKris_ORA.pdf
sun's commentary is up here
http://wikis.sun.com/display/HotSpotInternals/CompressedOops
I'm just not sure what it means for the NameNode, and as there is no
sizeof() operator in Java, something that will take a bit of effort to
work out. From what I read of the Sun wiki, when you go compressed,
while your heap is <3-4GB, there is no decompress operation; once you go
above that there is a shift and an add, which is probably faster than
fetching another 32 bits from $L2 or main RAM. The result could be
-could be- that your NN takes up much less space on 64 bit JVMs than it
did before, but is no slower.
Has anyone worked out the numbers yet?
-steve
