??????????????????????????????????hadoop2.6.0??????mr????????????????classpath??????????????hadoop_lzo.jar ??????????yarn-site.xml????????????????mapreduce.application.classpath??????????????classpath,????????????????hadoop_lzo.jar??????$HADOOP_HOME/share/hadoop/mapreduce/lib/????????????????mapreduce.application.classpath??????????????
------------------ ???????? ------------------ ??????: "ShaoFeng Shi";<[email protected]>; ????????: 2017??4??20??(??????) ????9:08 ??????: "dev"<[email protected]>; ????: Re: ????: ??????????: Error: java.io.IOException: Unable to initialize any output collector Hive is Kylin's source, not computing engine; Kylin starts its own MR jobs to build the cube. If need specify additional parameters into MR jobs, add them in conf/kylin_job*.xml, or you can customize them at cube level: https://kylin.apache.org/docs20/install/advance_settings.html 2017-04-19 16:24 GMT+08:00 35925138 <[email protected]>: > kylin.hbase.default.compression.codec=lzo > > <property> > <name>mapreduce.map.output.compress.codec</name> > <value>com.hadoop.compression.lzo.LzoCodec</value> > <description></description> > </property> > > <property> > <name>mapreduce.output.fileoutputformat.compress.codec</name> > <value>com.hadoop.compression.lzo.LzoCodec</value> > <description></description> > </property> > > > > ?????????????????? > > > > > ------------------ ???????? ------------------ > ??????: "ShaoFeng Shi";<[email protected]>; > ????????: 2017??4??19??(??????) ????3:40 > ??????: "dev"<[email protected]>; > > ????: Re: ????: ??????????: Error: java.io.IOException: Unable to initialize > any > output collector > > > > To change use lzo: > 1) Hive/MR compression: search "SnappyCodec" in conf/*.xml file, replace > them with lzo's codec > 2) HBase compression: in conf/kylin.properties, set > "kylin.hbase.default.compression.codec" to "lzo" > > 2017-04-19 10:56 GMT+08:00 35925138 <[email protected]>: > > > ????????????????????hive-site.xml????hive.aux.jars.path????hadoop- > lzo????????jar??????????????????????????build > > cube?????????????????????????????? > > yarn???????????????? > > Error: java.io.IOException: Unable to initialize any output collector at > > org.apache.hadoop.mapred.MapTask.createSortingCollector( > MapTask.java:412) > > at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81) at > > org.apache.hadoop.mapred.MapTask$NewOutputCollector.< > init>(MapTask.java:695) > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) at > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at > > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at > > java.security.AccessController.doPrivileged(Native Method) at > > javax.security.auth.Subject.doAs(Subject.java:422) at > > org.apache.hadoop.security.UserGroupInformation.doAs( > > UserGroupInformation.java:1628) at org.apache.hadoop.mapred. > > YarnChild.main(YarnChild.java:158) Container killed by the > > ApplicationMaster. Container killed on request. Exit code is 143 > Container > > exited with a non-zero exit code 143 > > > > ?????????????????????????????? > > > > 2017-04-19 10:48:15,875 WARN [main] org.apache.hadoop.mapred.MapTask: > > Unable to initialize MapOutputCollector org.apache.hadoop.mapred. > > MapTask$MapOutputBuffer > > java.lang.IllegalArgumentException: Compression codec > > com.hadoop.compression.lzo.LzoCodec was not found. > > at org.apache.hadoop.mapred.JobConf.getMapOutputCompressorClass( > > JobConf.java:798) > > at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init( > > MapTask.java:1019) > > at org.apache.hadoop.mapred.MapTask.createSortingCollector( > > MapTask.java:401) > > at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81) > > at org.apache.hadoop.mapred.MapTask$NewOutputCollector.< > > init>(MapTask.java:695) > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask. > java:767) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at org.apache.hadoop.security.UserGroupInformation.doAs( > > UserGroupInformation.java:1628) > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > > Caused by: java.lang.ClassNotFoundException: Class > > com.hadoop.compression.lzo.LzoCodec not found > > at org.apache.hadoop.conf.Configuration.getClassByName( > > Configuration.java:1980) > > at org.apache.hadoop.mapred.JobConf.getMapOutputCompressorClass( > > JobConf.java:796) > > ... 11 more > > > > ????hadoop????????hadoop/bin/hadoop checknative -a > > ?????????????? > > 17/04/19 09:55:10 WARN bzip2.Bzip2Factory: Failed to load/initialize > > native-bzip2 library system-native, will use pure-Java version > > 17/04/19 09:55:10 INFO zlib.ZlibFactory: Successfully loaded & > initialized > > native-zlib library > > Native library checking: > > hadoop: true /home/hadooper/hadoop/lib/native/libhadoop.so.1.0.0 > > zlib: true /lib64/libz.so.1 > > snappy: false > > lz4: true revision:99 > > bzip2: false > > openssl: true /usr/lib64/libcrypto.so > > > > lzo?????????????????????????????????????????????????? > > > > > > ------------------ ???????? ------------------ > > *??????:* "35925138";<[email protected]>; > > *????????:* 2017??4??14??(??????) ????4:25 > > *??????:* "dev"<[email protected]>; > > *????:* ??????????: ??????????: Error: java.io.IOException: Unable to > > initialize any > > output collector > > > > ????????????????????hadoop??????????????????????2047??????????2048 > > > > > > ------------------ ???????? ------------------ > > *??????:* "roger shi";<[email protected]>; > > *????????:* 2017??4??13??(??????) ????4:30 > > *??????:* "dev"<[email protected]>; > > *????:* ????: ??????????: Error: java.io.IOException: Unable to initialize > > any > > output collector > > > > It seems the hadoop environment configuration issue. > > > > > > One possible reason is the configuration "mapreduce.task.io.sort.mb" of > > this failed job is too large. For the detailed information please refer > to > > https://issues.apache.org/jira/browse/MAPREDUCE-6194. > > > > Bubble up final exception in failures during creation of ...< > > https://issues.apache.org/jira/browse/MAPREDUCE-6194> > > issues.apache.org > > MAPREDUCE-5974 added in ability to instantiate multiple OCs, but if none > > of them are able to load it "throws" only a final a generic message: > > "Unable to initialize ... > > > > > > > > ________________________________ > > ??????: 35925138 <[email protected]> > > ????????: 2017??4??13?? 15:13:45 > > ??????: dev > > ????: ??????????: Error: java.io.IOException: Unable to initialize any > > output > > collector > > > > ???????????????????????????????????? > > > > > > > > > > ------------------ ???????? ------------------ > > ??????: "roger shi";<[email protected]>; > > ????????: 2017??4??13??(??????) ????2:51 > > ??????: "dev"<[email protected]>; > > > > ????: ????: Error: java.io.IOException: Unable to initialize any output > > collector > > > > > > > > Could you please attach the complete stack trace of the error? > > > > ________________________________ > > ??????: 35925138 <[email protected]> > > ????????: 2017??4??13?? 13:45:09 > > ??????: dev > > ????: Error: java.io.IOException: Unable to initialize any output collector > > > > ????kylin??????1.6.0 hadoop??????2.6.0??build cube????????Step Name: > > Redistribute Flat > > Hive Table ?????????????????? > > kylin?????????????????? > > no counters for job job_1492049582227_0011 > > > > > > ????hadoop????userlog?????????????????????? > > 2017-04-13 11:23:33,758 ERROR [IPC Server handler 1 on 43772] > > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > > attempt_1492049582227_0011_m_000000_0 - exited : java.io.IOException: > > Unable to initialize any output collector > > at org.apache.hadoop.mapred.MapTask.createSortingCollector( > > MapTask.java:412) > > at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81) > > at org.apache.hadoop.mapred.MapTask$NewOutputCollector.< > > init>(MapTask.java:695) > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask. > java:767) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at org.apache.hadoop.security.UserGroupInformation.doAs( > > UserGroupInformation.java:1628) > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > > > > > > ?????????????????????????????????????????? > > > > > > > -- > Best regards, > > Shaofeng Shi ?????? > -- Best regards, Shaofeng Shi ??????
