kylin.hbase.default.compression.codec=lzo
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
<description></description>
</property>
<property>
<name>mapreduce.output.fileoutputformat.compress.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
<description></description>
</property>
??????????????????
------------------ ???????? ------------------
??????: "ShaoFeng Shi";<[email protected]>;
????????: 2017??4??19??(??????) ????3:40
??????: "dev"<[email protected]>;
????: Re: ????: ??????????: Error: java.io.IOException: Unable to initialize
any output collector
To change use lzo:
1) Hive/MR compression: search "SnappyCodec" in conf/*.xml file, replace
them with lzo's codec
2) HBase compression: in conf/kylin.properties, set
"kylin.hbase.default.compression.codec" to "lzo"
2017-04-19 10:56 GMT+08:00 35925138 <[email protected]>:
> ????????????????????hive-site.xml????hive.aux.jars.path????hadoop-lzo????????jar??????????????????????????build
> cube??????????????????????????????
> yarn????????????????
> Error: java.io.IOException: Unable to initialize any output collector at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:412)
> at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1628) at org.apache.hadoop.mapred.
> YarnChild.main(YarnChild.java:158) Container killed by the
> ApplicationMaster. Container killed on request. Exit code is 143 Container
> exited with a non-zero exit code 143
>
> ??????????????????????????????
>
> 2017-04-19 10:48:15,875 WARN [main] org.apache.hadoop.mapred.MapTask:
> Unable to initialize MapOutputCollector org.apache.hadoop.mapred.
> MapTask$MapOutputBuffer
> java.lang.IllegalArgumentException: Compression codec
> com.hadoop.compression.lzo.LzoCodec was not found.
> at org.apache.hadoop.mapred.JobConf.getMapOutputCompressorClass(
> JobConf.java:798)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(
> MapTask.java:1019)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(
> MapTask.java:401)
> at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<
> init>(MapTask.java:695)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1628)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.ClassNotFoundException: Class
> com.hadoop.compression.lzo.LzoCodec not found
> at org.apache.hadoop.conf.Configuration.getClassByName(
> Configuration.java:1980)
> at org.apache.hadoop.mapred.JobConf.getMapOutputCompressorClass(
> JobConf.java:796)
> ... 11 more
>
> ????hadoop????????hadoop/bin/hadoop checknative -a
> ??????????????
> 17/04/19 09:55:10 WARN bzip2.Bzip2Factory: Failed to load/initialize
> native-bzip2 library system-native, will use pure-Java version
> 17/04/19 09:55:10 INFO zlib.ZlibFactory: Successfully loaded & initialized
> native-zlib library
> Native library checking:
> hadoop: true /home/hadooper/hadoop/lib/native/libhadoop.so.1.0.0
> zlib: true /lib64/libz.so.1
> snappy: false
> lz4: true revision:99
> bzip2: false
> openssl: true /usr/lib64/libcrypto.so
>
> lzo??????????????????????????????????????????????????
>
>
> ------------------ ???????? ------------------
> *??????:* "35925138";<[email protected]>;
> *????????:* 2017??4??14??(??????) ????4:25
> *??????:* "dev"<[email protected]>;
> *????:* ??????????: ??????????: Error: java.io.IOException: Unable to
> initialize any
> output collector
>
> ????????????????????hadoop??????????????????????2047??????????2048
>
>
> ------------------ ???????? ------------------
> *??????:* "roger shi";<[email protected]>;
> *????????:* 2017??4??13??(??????) ????4:30
> *??????:* "dev"<[email protected]>;
> *????:* ????: ??????????: Error: java.io.IOException: Unable to initialize any
> output collector
>
> It seems the hadoop environment configuration issue.
>
>
> One possible reason is the configuration "mapreduce.task.io.sort.mb" of
> this failed job is too large. For the detailed information please refer to
> https://issues.apache.org/jira/browse/MAPREDUCE-6194.
>
> Bubble up final exception in failures during creation of ...<
> https://issues.apache.org/jira/browse/MAPREDUCE-6194>
> issues.apache.org
> MAPREDUCE-5974 added in ability to instantiate multiple OCs, but if none
> of them are able to load it "throws" only a final a generic message:
> "Unable to initialize ...
>
>
>
> ________________________________
> ??????: 35925138 <[email protected]>
> ????????: 2017??4??13?? 15:13:45
> ??????: dev
> ????: ??????????: Error: java.io.IOException: Unable to initialize any output
> collector
>
> ????????????????????????????????????
>
>
>
>
> ------------------ ???????? ------------------
> ??????: "roger shi";<[email protected]>;
> ????????: 2017??4??13??(??????) ????2:51
> ??????: "dev"<[email protected]>;
>
> ????: ????: Error: java.io.IOException: Unable to initialize any output
> collector
>
>
>
> Could you please attach the complete stack trace of the error?
>
> ________________________________
> ??????: 35925138 <[email protected]>
> ????????: 2017??4??13?? 13:45:09
> ??????: dev
> ????: Error: java.io.IOException: Unable to initialize any output collector
>
> ????kylin??????1.6.0 hadoop??????2.6.0??build cube????????Step Name:
> Redistribute Flat
> Hive Table ??????????????????
> kylin??????????????????
> no counters for job job_1492049582227_0011
>
>
> ????hadoop????userlog??????????????????????
> 2017-04-13 11:23:33,758 ERROR [IPC Server handler 1 on 43772]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1492049582227_0011_m_000000_0 - exited : java.io.IOException:
> Unable to initialize any output collector
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(
> MapTask.java:412)
> at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<
> init>(MapTask.java:695)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1628)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
>
> ??????????????????????????????????????????
>
>
--
Best regards,
Shaofeng Shi ??????
