[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6577: -- Fix Version/s: (was: 2.9.0) > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Fix For: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Fix Version/s: 2.8.0 > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.4 > > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6577: -- Target Version/s: 2.6.4 > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Fix For: 2.8.0, 2.7.3, 2.9.0, 2.6.4 > > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.6.4 2.9.0 2.7.3 Status: Resolved (was: Patch Available) Committed the patch to trunk, branch-2, branch-2.7, and branch-2.6. Thanks [~jlowe] and [~cnauroth] for your reviews and suggestions! > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Fix For: 2.7.3, 2.9.0, 2.6.4 > > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Attachment: MAPREDUCE-6577.04.patch Posted patch v.4. This keeps the Windows behavior identical to the current trunk. As a result, some of the code changes are slightly uglier. :) Another interesting change: as part of testing this pretty extensively, I stumbled onto the fact that HADOOP_COMMON_HOME was not being set in the hadoop unit tests. There are a lot of places where HADOOP_COMMON_HOME is used (including this one), and I suppose this problem was masked because most paths were being inherited from the node manager (mini-YARN cluster). > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch, MAPREDUCE-6577.04.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Attachment: MAPREDUCE-6577.03.patch Posted patch v.3. Fixed TestMiniMRChildTask#testMapRedExecutionEnv(). Removed the first part of the test as it is not meaningful to test the HADOOP_COMMON_HOME substitution as it is no longer substituted with the late binding. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch, > MAPREDUCE-6577.03.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Attachment: MAPREDUCE-6577.02.patch Posted patch v.2 that addresses Jason's review comments. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch, MAPREDUCE-6577.02.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Status: Patch Available (was: Open) Posted patch v.1. Added the default for the AM admin user env (which has the same value as the non-AM admin user env). Also did a small amount of yak-shaving. I suspect that the lib path handling on Windows is not quite right (PWD was not being added for Windows). I tried to make it a little more cross-platform there. Folks who use Windows ([~cnauroth]?), I'd greatly appreciate it if you could review the patch from that perspective. > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated MAPREDUCE-6577: --- Attachment: MAPREDUCE-6577.01.patch > MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set > > > Key: MAPREDUCE-6577 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.6.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: MAPREDUCE-6577.01.patch > > > If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not > configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library: > {noformat} > 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > {noformat} > As a result, any code that needs the hadoop native library in the MR AM will > fail. For example, an uber-AM with lz4 compression for the mapper task will > fail: > {noformat} > 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] > org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local > (uberized) 'child' : java.lang.RuntimeException: native lz4 library not > available > at > org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114) > at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195) > at > org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)