[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061501#comment-17061501
]
kangjunxiang commented on KYLIN-3970:
-------------------------------------
@[~hit_lacus]
*environment：*
HDP-3.1.0.0
Apache kylin 3.0.1
Kafka 2.0.0
A new error occurred after using "try-2",The following information：
--------------------------------------------------------------------------------------
org.apache.kylin.engine.mr.exception.MapReduceException: Counters: 12
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=286636
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=71659
Total vcore-milliseconds taken by all map tasks=71659
Total megabyte-milliseconds taken by all map tasks=97742876
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Job Diagnostics:Task failed task_1584424619465_0008_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error: java.lang.ClassNotFoundException:
org.apache.kafka.clients.consumer.Consumer
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at
org.apache.kylin.source.kafka.hadoop.KafkaInputFormat.createRecordReader(KafkaInputFormat.java:107)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:527)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:223)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kyl
> NRT Streaming cannot work at HDP3.0
> -----------------------------------
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
> Affects Versions: v2.6.1
> Reporter: Xiaoxiang Yu
> Assignee: Xiaoxiang Yu
> Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h3. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
> h3. *Error message*
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
