[
https://issues.apache.org/jira/browse/YARN-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-7433:
-----------------------------
Environment: (was: From centos6.5 upgrade centos7, hadoop
version(2.7.1) is compiled on centos6.5 and Support snappy, the copy runs on
centos7.
but yarn task error ，yarn task errorlog：
(hadoop Native Libraries Whether to recompile based on centos7???)
Error: java.lang.RuntimeException: native snappy library not available: this
version of libhadoop was built without snappy support.
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1856)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1511)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.io.IOException: Spill failed)
Description:
>From centos6.5 upgrade centos7, hadoop version(2.7.1) is compiled on
>centos6.5 and Support snappy, the copy runs on centos7.
but yarn task error ，yarn task errorlog：
(hadoop Native Libraries Whether to recompile based on centos7???)
Error: java.lang.RuntimeException: native snappy library not available: this
version of libhadoop was built without snappy support.
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1856)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1511)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.io.IOException: Spill failed
JIRA is for reporting bugs or tracking features rather than general user
support. The [mailing lists|http://hadoop.apache.org/mailing_lists.html] would
be more appropriate for this.
There are two possible reasons why this error is occurring. Either the task
cannot find libhadoop.so or libhadoop.so was compiled without Snappy support.
For the former, see if the message "Unable to load native-hadoop library for
your platform" appears in the task logs. If it does then the task is not
loading libhadoop.so and Snappy cannot work. Check the LD_LIBRARY_PATH
environment variable setting to make sure the task can locate libhadoop.so. If
it is loading libhadoop.so properly then the error means the native library was
compiled without Snappy support and needs to be rebuilt with
-Drequire.snappy=true
> java.lang.RuntimeException: native snappy library not available: this version
> of libhadoop was built without snappy support.
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-7433
> URL: https://issues.apache.org/jira/browse/YARN-7433
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: gehaijiang
>
> From centos6.5 upgrade centos7, hadoop version(2.7.1) is compiled on
> centos6.5 and Support snappy, the copy runs on centos7.
> but yarn task error ，yarn task errorlog：
> (hadoop Native Libraries Whether to recompile based on centos7???)
> Error: java.lang.RuntimeException: native snappy library not available: this
> version of libhadoop was built without snappy support.
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
> at
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1856)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1511)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Error: java.io.IOException: Spill failed
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
