Liyin Liang created MAPREDUCE-5799:
--------------------------------------
Summary: add default value of MR_AM_ADMIN_USER_ENV
Key: MAPREDUCE-5799
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5799
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.3.0
Reporter: Liyin Liang
Priority: Minor
Submit a 1 map + 1 reduce sleep job with the following config:
{code}
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapreduce.job.ubertask.enable</name>
<value>true</value>
</property>
{code}
And the LinuxContainerExecutor is enable on NodeManager.
This job will fail with the following error:
{code}
2014-03-18 21:28:20,153 FATAL [uber-SubtaskRunner]
org.apache.hadoop.mapred.LocalContainerLauncher: Error running local (uberized)
'child' : java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
Method)
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1583)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1462)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1990)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:774)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at
org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:317)
at
org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:232)
at java.lang.Thread.run(Thread.java:662)
{code}
When create a ContainerLaunchContext for task in
TaskAttemptImpl.createCommonContainerLaunchContext(), the
DEFAULT_MAPRED_ADMIN_USER_ENV which is
"LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native" is added to the environment.
Where when create a ContainerLaunchContext for mrappmaster in
YARNRunner.createApplicationSubmissionContext(), there is no default
environment. So the ubermode job fails to find native lib.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
