Hello,
Let me clarify. The Hadoop software works
flawlessly out of the box but those of us
without a cluster (few machines) need a way
to maximize hardware resources.
My machine is 64-bit and a single-node
Hadoop job can take up to 1GB resident
private memory (RPRVT) on a one box job.
Possibly this is because I do String
concatenations within a single reduce instead
of running multiple identity mappers and
additional reduce passes.
Now, I'd like to run HBase on a single node pulling
megabytes of data in and out and imagine that memory
resources would be quickly exhausted. The idea was
to grow the JVM by using -Xms1g -Xmx4g.
Since I'd probably need to run solaris to try this
the regular Hadoop configuration should be fine.
Peter W.
On Sep 7, 2007, at 10:28 AM, Peter W. wrote:
Lance,
Allocating more than 2GB to the JVM requires 64-bit.
Regards,
Peter W.
Peter W. wrote:
Hi,
My only experience is using Hadoop and HBase shell on OSX Tiger.
It seems that if you are going to store large amounts of data
w/HBase allocating more than 2GB to the JVM would be helpful.
Not sure about about your HDFS compatibility questions.
Regards,
Peter W.
Lance Boomerang wrote:
So are we assuming that all the HDFS nodes are on 64bit hosts?
Any idea if using the Hadoop Java API on 32bit clients would work
to read / write files into a 64 bit HDFS setup over the network
(HTTP)?
Thanks,
Lance
