Hi Sir, If I make no mistake, from the logs provided by you, it is clear that the workaround/steps provided by me in my later mail has FIXED the previous error. The previous error was occured at Step1(Collect data from kafka), and it is a class conflict problem, because we can see the text like " Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;". And the later error is occured at Step4(Fact Distinct Column), because we can see the text like " org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper". I think it is a totally NEW problem(has no relation with KYLIN-3970 because it didn't use kafka client), and it may be caused by Mapreduce Framework. You should use less data and try again or modify the map reduce configuration (by check http://kylin.apache.org/docs/install/configuration.html#mr-config-override). If I make any mistake or you have more findings , please let me know. Thank you.
---------------- Best wishes, Xiaoxiang Yu 在 2019/10/24 19:10，“tianjiazheng”<[email protected]> 写入: yes，I think this is the same problem，can you tell me the ways? -- 发自邮洽 _____________________________________________ 发件人： [email protected] 发送时间： 2019-10-24 18:58:53 收件人： [email protected] 主题： Re: kylin2.6.4流式构建不能与hdp3.1.4.0使用(issue3970) Does your error log like this KYLIN-3970 ? If yes, there're some ways to solve this problem, it may be helpful. 在 2019/10/24 17:30，“田家铮”<[email protected]> 写入: I followed your steps and found the error report again. Can you help me fix this problem? ———————————————————————————————————————————————————— 2019-10-24 05:25:11,171 ERROR [pool-11-thread-2] threadpool.DefaultScheduler:116 : ExecuteException job:9722333e-789b-1f30-50e6-38e9905802c0 org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1571906689528_0004Job Diagnostics:Task failed task_1571906689528_0004_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: [2019-10-24 05:25:00.454]Exception from container-launch. Container id: container_e01_1571906689528_0004_01_000005 Exit code: 255 [2019-10-24 05:25:00.456]Container exited with a non-zero exit code 255. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. [2019-10-24 05:25:00.457]Container exited with a non-zero exit code 255. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:182) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1571906689528_0004Job Diagnostics:Task failed task_1571906689528_0004_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: [2019-10-24 05:25:00.454]Exception from container-launch. Container id: container_e01_1571906689528_0004_01_000005 Exit code: 255 [2019-10-24 05:25:00.456]Container exited with a non-zero exit code 255. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. [2019-10-24 05:25:00.457]Container exited with a non-zero exit code 255. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:182) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) ... 4 more Caused by: org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1571906689528_0004Job Diagnostics:Task failed task_1571906689528_0004_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: [2019-10-24 05:25:00.454]Exception from container-launch. Container id: container_e01_1571906689528_0004_01_000005 Exit code: 255 [2019-10-24 05:25:00.456]Container exited with a non-zero exit code 255. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : log4j:WARN No appenders could be found for logger (org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. ———————————————————————————————————————————————————— ------------------ Original ------------------ From: "Xiaoxiang Yu"<[email protected]>; Date: Thu, Oct 24, 2019 03:57 PM To: "[email protected]"<[email protected]>; "[email protected]"<[email protected]>; Subject: Re: kylin2.6.4流式构建不能与hdp3.1.4.0使用(issue3970) Hi Sir, Please retry with the following steps, if it still not works, please let us know. I have verified it at HDP 3.1. 1. export KAFKA_HOME before you start kylin process (better with equal or higher than 1.0) export KAFKA_HOME= XXX sh bin/kylin.sh start 2. copy mapreduce.tar.gz from hdfs to local hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . 3. remove kafka-clients- 0.8.2.1.jar in mapreduce.tar.gz and repackage tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ 4 . put it back to original place hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce 5. drop and restart the build job and see if it works Note: You should change to your specific version, for example: 3.0.1.0 -187 and 0.8.2.1. ---------------- Best wishes, Xiaoxiang Yu 在2019/10/24 14:24，“田家铮”<[email protected]>写入: The error reported in the log is the same as the error reported in the issue, and the scheme provided by the issue has been tried, but it still cannot be solved, and the reason is unknown. This is the logs , please take a look ,thanks ————————————————————————————————————————————— ——————————————————— 2019-10-09 23:58:44,395 ERROR [pool-11-thread-1] threadpool.DefaultScheduler:116 : ExecuteException job:dab5ba1f-e1ed-c8fa-409c-29f71715e8e0 org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1570617080701_0005Job Diagnostics:Task failed task_1570617080701_0005_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps: 0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java: 182) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.engine.mr .exception.MapReduceException: no counters for job job_1570617080701_0005Job Diagnostics:Task failed task_1570617080701_0005_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:182) at org.apache.kylin .job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) ... 4 more Caused by: org.apache.kylin. engine.mr.exception.MapReduceException: no counters for job job_1570617080701_0005Job Diagnostics:Task failed task_1570617080701_0005_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache .kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) ... 6 more 2019-10-09 23:58:44,396 INFO [FetcherRunner 1609136206-49] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others 2019-10-09 23:58:44,400 DEBUG [http-nio-7070-exec-7] cachesync.Broadcaster: 246 : Broadcasting UPDATE, execute_output, dab5ba1f-e1ed-c8fa-409c-29f71715e8e0 2019-10-09 23:58:44,401 DEBUG [http-nio-7070-exec-7] cachesync.Broadcaster:280 : Done broadcasting UPDATE, execute_output, dab5ba1f-e1ed-c8fa-409c-29f71715e8e0 2019-10-09 23: 59:12,827 INFO [FetcherRunner 1609136206-49] threadpool.DefaultFetcherRunner:85 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others ————— —————————————————————————————————————————————————— ————————— ------------------ Original ------------------ From: "Xiaoxiang Yu"<[email protected]>; Date: Thu, Oct 24, 2019 01:32 PM To: "[email protected]"<[email protected]>; "tianjiazheng@tuzhanai. com"<[email protected]>; Subject: Re: kylin2.6.4流式构建不能与hdp3.1.4.0使用(issue3970) Hi Sir, I cannot see your screenshot, could you please paste the stack trace (in kylin.log) or provided more description to us? And have you ever try use the workaround which was provided in KYLIN-3970 to fix your problem? ---------------- Best wishes, Xiaoxiang Yu 发件人:田家铮<tianjiazheng@ tuzhanai.com> 答复: "[email protected]" <[email protected]> 日期: 2019年10月24日星期四11:44 收件人: dev <[email protected]> 主题: kylin2.6.4流式构建不能与hdp3.1.4.0使用(issue3970) 环境配置： kylin-2.6.4 HDP-3.1.4。0 issue: 3970 https://issues.apache.org/jira/browse/KYLIN-3970?jql=text%20~%20%22partition.assignment.strategy%22<https://issues.apache.org/jira/browse/KYLIN -3970?jql=text%20%7E%20%22partition.assignment.strategy%22> 问题： 流式构建读取kafka数据时，无法构建，但是hive构建没问题 [cid:[email protected]] [ cid:[email protected]]
