[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:55 AM:
--------------------------------------------------------------
h2. *Problem:* *Kafka jar conflict*
* when click build buttom
{code:java}
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.<init>(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
{code}
* First step of build segment
{code:java}
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
h3.
was (Author: hit_lacus):
*Problem:* *Kafka jar conflict*
{code}
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.<init>(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
{code}
{code}
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
h3.
> NRT Streaming cannot work at HDP3.0
> -----------------------------------
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
> Affects Versions: v2.6.1
> Reporter: XiaoXiang Yu
> Assignee: XiaoXiang Yu
> Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-23-14-25-51-066.png,
> image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
