<property> <name>mapreduce.reduce.memory.mb</name>
<value>4096</value> </property> change this to 8 G On Wed, Oct 14, 2015 at 12:52 AM, Ranjana Rajendran < ranjana.rajend...@gmail.com> wrote: > Here is Altiscale's documentation about the topic. Do let me know if you > have any more questions. > > http://documentation.altiscale.com/heapsize-for-mappers-and-reducers > > On Tue, Oct 13, 2015 at 9:31 AM, Mich Talebzadeh <m...@peridale.co.uk> > wrote: > >> Hi, >> >> >> >> I have been having some issues with loading data into hive from one table >> to another for 1,767,886 rows. I was getting the following error >> >> >> >> Task with the most failures(4): >> >> ----- >> >> Task ID: >> >> task_1444731612741_0001_r_000000 >> >> >> >> URL: >> >> >> http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1444731612741_0001&tipid=task_1444731612741_0001_r_000000 >> >> ----- >> >> Diagnostic Messages for this Task: >> >> Container [pid=16238,containerID=container_1444731612741_0001_01_000019] is >> running beyond physical memory limits. Current usage: 2.0 GB of 2 GB >> physical memory used; 6.6 GB of 8 GB virtual memory used. Killing >> container. >> >> >> >> >> >> Changed parameters in yarn-site.xml and mapred-site.xml files few times >> but no joy. >> >> >> >> Finally the following changes in mapred-site.xml worked for me >> >> >> >> <property> >> >> <name>mapreduce.job.tracker.reserved.physicalmemory.mb</name> >> >> <value>1024</value> >> >> </property> >> >> >> >> <property> >> >> <name>mapreduce.map.memory.mb</name> >> >> <value>4096</value> >> >> </property> >> >> >> >> <property> >> >> <name>mapreduce.reduce.memory.mb</name> >> >> <value>4096</value> >> >> </property> >> >> >> >> <property> >> >> <name>mapreduce.map.java.opts</name> >> >> <value>-Xmx3072m</value> >> >> </property> >> >> >> >> <property> >> >> <name>mapreduce.reduce.java.opts</name> >> >> <value>-Xmx6144m</value> >> >> </property> >> >> >> >> And the following changes to yarn-site.xml >> >> >> >> <property> >> >> <name>yarn.nodemanager.vmem-check-enabled</name> >> >> <value>false</value> >> >> </property> >> >> <property> >> >> <name>yarn.nodemanager.resource.memory-mb</name> >> >> <value>8192</value> >> >> <description>Amount of physical memory, in MB, that can be allocated for >> containers.</description> >> >> </property> >> >> <property> >> >> <name>yarn.nodemanager.vmem-pmem-ratio</name> >> >> <value>4</value> >> >> <description>Ratio between virtual memory to physical memory when >> setting memory limits for containers</description> >> >> </property> >> >> >> >> I did a lot of web search but most resolution to this issue seems to be >> cryptic or anecdotal. Anyone has better explanation I would be interested. >> >> >> >> Mich Talebzadeh >> >> >> >> *Sybase ASE 15 Gold Medal Award 2008* >> >> A Winning Strategy: Running the most Critical Financial Data on ASE 15 >> >> >> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf >> >> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE >> 15", ISBN 978-0-9563693-0-7*. >> >> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN >> 978-0-9759693-0-4* >> >> *Publications due shortly:* >> >> *Complex Event Processing in Heterogeneous Environments*, ISBN: >> 978-0-9563693-3-8 >> >> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume >> one out shortly >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> NOTE: The information in this email is proprietary and confidential. This >> message is for the designated recipient only, if you are not the intended >> recipient, you should destroy it immediately. Any information in this >> message shall not be understood as given or endorsed by Peridale Technology >> Ltd, its subsidiaries or their employees, unless expressly so stated. It is >> the responsibility of the recipient to ensure that this email is virus >> free, therefore neither Peridale Ltd, its subsidiaries nor their employees >> accept any responsibility. >> >> >> > >
