CPU Usage = 95-100% Memory = 650-850 MB (2GB Ram) Baran
2011/5/2 James Seigel <[email protected]> > If you have windows and cygwin you probably don't have a lot if memory > left at 2 gig. > > Pull up system monitor on the data nodes and check for free memory > when you have you jobs running. I bet it is quite low. > > I am not a windows guy so I can't take you much farther. > > James > > Sent from my mobile. Please excuse the typos. > > On 2011-05-02, at 8:32 AM, baran cakici <[email protected]> wrote: > > > yes, I am running under cygwin on my datanodes too. OS of Datanodes are > > Windows as well. > > > > What can I do exactly for a better Performance. I changed > > mapred.child.java.opts to default value.How can I solve this "swapping" > > problem? > > > > PS: I dont have a chance to get Slaves(Celeron 2GHz) with Liniux OS. > > > > thanks, both of you > > > > Regards, > > > > Baran > > 2011/5/2 Richard Nadeau <[email protected]> > > > >> Are you running under cygwin on your data nodes as well? That is certain > to > >> cause performance problems. As James suggested, swapping to disk is > going > >> to > >> be a killer, running on Windows with Celeron processors only compounds > the > >> problem. The Celeron processor is also sub-optimal for CPU intensive > tasks > >> > >> Rick > >> > >> On Apr 28, 2011 9:22 AM, "baran cakici" <[email protected]> wrote: > >>> Hi Everyone, > >>> > >>> I have a Cluster with one Master(JobTracker and NameNode - Intel > Core2Duo > >> 2 > >>> GB Ram) and four Slaves(Datanode and Tasktracker - Celeron 2 GB Ram). > My > >>> Inputdata are between 2GB-10GB and I read Inputdata in MapReduce line > by > >>> line. Now, I try to accelerate my System(Benchmark), but I'm not sure, > if > >> my > >>> Configuration is correctly. Can you please just look, if it is ok? > >>> > >>> -mapred-site.xml > >>> > >>> <property> > >>> <name>mapred.job.tracker</name> > >>> <value>apple:9001</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.child.java.opts</name> > >>> <value>-Xmx512m -server</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.job.tracker.handler.count</name> > >>> <value>2</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.local.dir</name> > >>> > >> > <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/local</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.map.tasks</name> > >>> <value>1</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.reduce.tasks</name> > >>> <value>4</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.submit.replication</name> > >>> <value>2</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.system.dir</name> > >>> > >> > >> > <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/system</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.tasktracker.indexcache.mb</name> > >>> <value>10</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.tasktracker.map.tasks.maximum</name> > >>> <value>1</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.tasktracker.reduce.tasks.maximum</name> > >>> <value>4</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.temp.dir</name> > >>> > >> > <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/temp</value> > >>> </property> > >>> > >>> <property> > >>> <name>webinterface.private.actions</name> > >>> <value>true</value> > >>> </property> > >>> > >>> <property> > >>> <name>mapred.reduce.slowstart.completed.maps</name> > >>> <value>0.01</value> > >>> </property> > >>> > >>> -hdfs-site.xml > >>> > >>> <property> > >>> <name>dfs.block.size</name> > >>> <value>268435456</value> > >>> </property> > >>> PS: I extended dfs.block.size, because I won 50% better performance > with > >>> this change. > >>> > >>> I am waiting for your comments... > >>> > >>> Regards, > >>> > >>> Baran > >> >
