Hi,
I have been having some issues with loading data into hive from one table to another for 1,767,886 rows. I was getting the following error Task with the most failures(4): ----- Task ID: task_1444731612741_0001_r_000000 URL: http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1444731612741_0001&tipid=task_ 1444731612741_0001_r_000000 ----- Diagnostic Messages for this Task: Container [pid=16238,containerID=container_1444731612741_0001_01_000019] is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 6.6 GB of 8 GB virtual memory used. Killing container. Changed parameters in yarn-site.xml and mapred-site.xml files few times but no joy. Finally the following changes in mapred-site.xml worked for me <property> <name>mapreduce.job.tracker.reserved.physicalmemory.mb</name> <value>1024</value> </property> <property> <name>mapreduce.map.memory.mb</name> <value>4096</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>4096</value> </property> <property> <name>mapreduce.map.java.opts</name> <value>-Xmx3072m</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value>-Xmx6144m</value> </property> And the following changes to yarn-site.xml <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>8192</value> <description>Amount of physical memory, in MB, that can be allocated for containers.</description> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>4</value> <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description> </property> I did a lot of web search but most resolution to this issue seems to be cryptic or anecdotal. Anyone has better explanation I would be interested. Mich Talebzadeh Sybase ASE 15 Gold Medal Award 2008 A Winning Strategy: Running the most Critical Financial Data on ASE 15 <http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908 .pdf> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908. pdf Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4 Publications due shortly: Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8 Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly <http://talebzadehmich.wordpress.com/> http://talebzadehmich.wordpress.com NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
