[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:48 AM:
--------------------------------------------------------------
*Problem:* *Kafka jar conflict*
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.<init>(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
h3.
was (Author: hit_lacus):
h3. *First problem:* *Kafka jar conflict*
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
h3. *Second problem: Hive null value*
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0007Job Diagnostics:Task failed
task_1555653800318_0007_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error: java.lang.IllegalArgumentException: there is no valid date pattern for:\N
at
org.apache.kylin.common.util.DateFormat.stringToMillis(DateFormat.java:124)
at
org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:154)
at
org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:148)
at
org.apache.kylin.metadata.datatype.DataTypeOrder.compare(DataTypeOrder.java:95)
at
org.apache.kylin.metadata.datatype.DataTypeOrder.max(DataTypeOrder.java:90)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:194)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
> NRT Streaming cannot work at HDP3.0
> -----------------------------------
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
> Affects Versions: v2.6.1
> Reporter: XiaoXiang Yu
> Assignee: XiaoXiang Yu
> Priority: Major
> Labels: HDP, Hadoop3
> Fix For: v3.0.0, v2.6.2
>
> Attachments: image-2019-04-23-14-25-51-066.png,
> image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
