without job; CPU Usage = 0% Memory = 585 MB (2GB Ram)
Baran 2011/5/2 baran cakici <[email protected]> > CPU Usage = 95-100% > Memory = 650-850 MB (2GB Ram) > > Baran > > > 2011/5/2 James Seigel <[email protected]> > >> If you have windows and cygwin you probably don't have a lot if memory >> left at 2 gig. >> >> Pull up system monitor on the data nodes and check for free memory >> when you have you jobs running. I bet it is quite low. >> >> I am not a windows guy so I can't take you much farther. >> >> James >> >> Sent from my mobile. Please excuse the typos. >> >> On 2011-05-02, at 8:32 AM, baran cakici <[email protected]> wrote: >> >> > yes, I am running under cygwin on my datanodes too. OS of Datanodes are >> > Windows as well. >> > >> > What can I do exactly for a better Performance. I changed >> > mapred.child.java.opts to default value.How can I solve this "swapping" >> > problem? >> > >> > PS: I dont have a chance to get Slaves(Celeron 2GHz) with Liniux OS. >> > >> > thanks, both of you >> > >> > Regards, >> > >> > Baran >> > 2011/5/2 Richard Nadeau <[email protected]> >> > >> >> Are you running under cygwin on your data nodes as well? That is >> certain to >> >> cause performance problems. As James suggested, swapping to disk is >> going >> >> to >> >> be a killer, running on Windows with Celeron processors only compounds >> the >> >> problem. The Celeron processor is also sub-optimal for CPU intensive >> tasks >> >> >> >> Rick >> >> >> >> On Apr 28, 2011 9:22 AM, "baran cakici" <[email protected]> wrote: >> >>> Hi Everyone, >> >>> >> >>> I have a Cluster with one Master(JobTracker and NameNode - Intel >> Core2Duo >> >> 2 >> >>> GB Ram) and four Slaves(Datanode and Tasktracker - Celeron 2 GB Ram). >> My >> >>> Inputdata are between 2GB-10GB and I read Inputdata in MapReduce line >> by >> >>> line. Now, I try to accelerate my System(Benchmark), but I'm not sure, >> if >> >> my >> >>> Configuration is correctly. Can you please just look, if it is ok? >> >>> >> >>> -mapred-site.xml >> >>> >> >>> <property> >> >>> <name>mapred.job.tracker</name> >> >>> <value>apple:9001</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.child.java.opts</name> >> >>> <value>-Xmx512m -server</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.job.tracker.handler.count</name> >> >>> <value>2</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.local.dir</name> >> >>> >> >> >> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/local</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.map.tasks</name> >> >>> <value>1</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.reduce.tasks</name> >> >>> <value>4</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.submit.replication</name> >> >>> <value>2</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.system.dir</name> >> >>> >> >> >> >> >> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/system</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.tasktracker.indexcache.mb</name> >> >>> <value>10</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.tasktracker.map.tasks.maximum</name> >> >>> <value>1</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.tasktracker.reduce.tasks.maximum</name> >> >>> <value>4</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.temp.dir</name> >> >>> >> >> >> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/temp</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>webinterface.private.actions</name> >> >>> <value>true</value> >> >>> </property> >> >>> >> >>> <property> >> >>> <name>mapred.reduce.slowstart.completed.maps</name> >> >>> <value>0.01</value> >> >>> </property> >> >>> >> >>> -hdfs-site.xml >> >>> >> >>> <property> >> >>> <name>dfs.block.size</name> >> >>> <value>268435456</value> >> >>> </property> >> >>> PS: I extended dfs.block.size, because I won 50% better performance >> with >> >>> this change. >> >>> >> >>> I am waiting for your comments... >> >>> >> >>> Regards, >> >>> >> >>> Baran >> >> >> > >
