[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085602#comment-15085602
]
Jason Lowe commented on MAPREDUCE-6577:
---------------------------------------
Thanks, Sangjin! Could you also commit this to branch-2.8? I find it odd that
the fix is important enough to backport to 2.6 and 2.7 but not important enough
for 2.8. Users updating from those older releases to 2.8 will accidentally
drop this fix.
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
> Affects Versions: 2.6.0
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Priority: Critical
> Fix For: 2.7.3, 2.9.0, 2.6.4
>
> Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch,
> MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will
> fail. For example, an uber-AM with lz4 compression for the mapper task will
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not
> available
> at
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
