[
https://issues.apache.org/jira/browse/KYLIN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shiyu resolved KYLIN-4675.
--------------------------
Fix Version/s: v3.1.0
Reviewers: shiyu
Resolution: Fixed
> apache-kylin-3.1.0 source run in IDEA to build a cube
> -----------------------------------------------------
>
> Key: KYLIN-4675
> URL: https://issues.apache.org/jira/browse/KYLIN-4675
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v3.1.0
> Environment: Windows7 旗舰版
> IntelliJ IDEA 2018.3.6 x64
> Java version "1.8.0_71"
> Scala version 2.11.12
> Hadoop version hadoop-3.0.2
> Spark version 2.4.5
> CHD 6.0.1
> Kylin src 3.1.0
> Maven 3.5.4
> Git version 2.14.1.windows.1
> Nodejs version 6.9.0
> Bower version 1.8.8
> Reporter: shiyu
> Priority: Major
> Fix For: v3.1.0
>
> Attachments: 1.png, 8.png, 9.png, build_cube_error_step2.png,
> image-2020-08-04-17-37-01-331.png
>
>
> apache-kylin-3.1.0 source run in IDEA to build a cube:
> 2020-08-03 11:42:04,859 WARN [Scheduler 910302043 Job
> ff0a61d0-1a6e-91c4-179a-3472d0676e16-127] common.HadoopCmdOutput:124 :
> Failure task Diagnostics:2020-08-03 11:42:04,859 WARN [Scheduler 910302043
> Job ff0a61d0-1a6e-91c4-179a-3472d0676e16-127] common.HadoopCmdOutput:124 :
> Failure task Diagnostics:2020-08-03 11:42:04,859 WARN [Scheduler 910302043
> Job ff0a61d0-1a6e-91c4-179a-3472d0676e16-127] common.HadoopCmdOutput:127 :
> {color:#FF0000}Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper not found
> at{color}
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2506) at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:187)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:759) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)Caused by:
> java.lang.ClassNotFoundException: Class
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper not found at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2410)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2504) ...
> 8 more
> 2020-08-03 11:42:04,904 INFO [Scheduler 910302043 Job
> ff0a61d0-1a6e-91c4-179a-3472d0676e16-127] execution.ExecutableManager:479 :
> job id:ff0a61d0-1a6e-91c4-179a-3472d0676e16-01 from RUNNING to
> ERROR2020-08-03 11:42:04,909 ERROR [Scheduler 910302043 Job
> ff0a61d0-1a6e-91c4-179a-3472d0676e16-127] execution.AbstractExecutable:180 :
> error running Executable: CubingJob\{id=ff0a61d0-1a6e-91c4-179a-3472d0676e16,
> name=BUILD CUBE - book_c - FULL_BUILD - GMT+08:00 2020-08-03 10:55:28,
> state=RUNNING}2020-08-03 11:42:04,924 DEBUG [pool-5-thread-1]
> cachesync.Broadcaster:119 : Servers in the cluster:
> [localhost:7070]2020-08-03 11:42:04,924 DEBUG [pool-5-thread-1]
> cachesync.Broadcaster:129 : Announcing new broadcast to all:
> BroadcastEvent\{entity=execute_output, event=update,
> cacheKey=ff0a61d0-1a6e-91c4-179a-3472d0676e16}2020-08-03 11:42:04,928 DEBUG
> [http-bio-7070-exec-1] cachesync.Broadcaster:267 : Broadcasting UPDATE,
> execute_output, ff0a61d0-1a6e-91c4-179a-3472d0676e162020-08-03 11:42:04,933
> DEBUG [http-bio-7070-exec-1] cachesync.Broadcaster:301 : Done broadcasting
> UPDATE, execute_output, ff0a61d0-1a6e-91c4-179a-3472d0676e162020-08-03
> 11:42:04,935 INFO [Scheduler 910302043 Job
> ff0a61d0-1a6e-91c4-179a-3472d0676e16-127] execution.ExecutableManager:479 :
> job id:ff0a61d0-1a6e-91c4-179a-3472d0676e16 from RUNNING to ERROR2020-08-03
> 11:42:04,935 DEBUG [Scheduler 910302043 Job
> ff0a61d0-1a6e-91c4-179a-3472d0676e16-127] execution.AbstractExecutable:331 :
> no need to send email, user list is empty2020-08-03 11:42:04,935 DEBUG
> [pool-5-thread-1] cachesync.Broadcaster:119 : Servers in the cluster:
> [localhost:7070]2020-08-03 11:42:04,936 DEBUG [pool-5-thread-1]
> cachesync.Broadcaster:129 : Announcing new broadcast to all:
> BroadcastEvent\{entity=execute_output, event=update,
> cacheKey=ff0a61d0-1a6e-91c4-179a-3472d0676e16}2020-08-03 11:42:04,940 ERROR
> [pool-12-thread-2] threadpool.DefaultScheduler:116 : ExecuteException
> job:ff0a61d0-1a6e-91c4-179a-3472d0676e16org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.engine.mr.exception.MapReduceException: Counters: 10 Job
> Counters Failed map tasks=2 Killed reduce tasks=7 Launched map tasks=2 Other
> local map tasks=1 Data-local map tasks=1 Total time spent by all maps in
> occupied slots (ms)=5022 Total time spent by all reduces in occupied slots
> (ms)=0 Total time spent by all map tasks (ms)=5022 Total vcore-seconds taken
> by all map tasks=5022 Total megabyte-seconds taken by all map
> tasks=5142528Job Diagnostics:Task failed task_1595310265660_3263_m_000000Job
> failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
> killedReduces: 0
> Failure task Diagnostics:Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper not found at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2506) at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:187)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:759) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)Caused by:
> java.lang.ClassNotFoundException: Class
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper not found at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2410)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2504) ...
> 8 more
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:193)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)Caused by:
> org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.engine.mr.exception.MapReduceException: Counters: 10 Job
> Counters Failed map tasks=2 Killed reduce tasks=7 Launched map tasks=2 Other
> local map tasks=1 Data-local map tasks=1 Total time spent by all maps in
> occupied slots (ms)=5022 Total time spent by all reduces in occupied slots
> (ms)=0 Total time spent by all map tasks (ms)=5022 Total vcore-seconds taken
> by all map tasks=5022 Total megabyte-seconds taken by all map
> tasks=5142528Job Diagnostics:Task failed task_1595310265660_3263_m_000000Job
> failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
> killedReduces: 0
> Failure task Diagnostics:Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper not found at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2506) at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:187)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:759) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)Caused by:
> java.lang.ClassNotFoundException: Class
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper not found at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2410)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2504) ...
> 8 more
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:193)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> ... 4 moreCaused by:
> org.apache.kylin.engine.mr.exception.MapReduceException: Counters: 10 Job
> Counters Failed map tasks=2 Killed reduce tasks=7 Launched map tasks=2 Other
> local map tasks=1 Data-local map tasks=1 Total time spent by all maps in
> occupied slots (ms)=5022 Total time spent by all reduces in occupied slots
> (ms)=0 Total time spent by all map tasks (ms)=5022 Total vcore-seconds taken
> by all map tasks=5022 Total megabyte-seconds taken by all map
> tasks=5142528Job Diagnostics:Task failed task_1595310265660_3263_m_000000Job
> failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
> killedReduces: 0
> Failure task Diagnostics:Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper not found at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2506) at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:187)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:759) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)Caused by:
> java.lang.ClassNotFoundException: Class
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper not found at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2410)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2504) ...
> 8 more
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:223)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> ... 6 more2020-08-03 11:42:04,941 INFO [FetcherRunner 555305792-83]
> threadpool.DefaultFetcherRunner:115 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 0 already succeed, 3 error, 0 discarded, 0
> others2020-08-03 11:42:04,942 DEBUG [http-bio-7070-exec-1]
> cachesync.Broadcaster:267 : Broadcasting UPDATE, execute_output,
> ff0a61d0-1a6e-91c4-179a-3472d0676e162020-08-03 11:42:04,947 DEBUG
> [http-bio-7070-exec-1] cachesync.Broadcaster:301 : Done broadcasting UPDATE,
> execute_output, ff0a61d0-1a6e-91c4-179a-3472d0676e162020-08-03 11:42:06,065
> INFO [FetcherRunner 555305792-83] threadpool.DefaultFetcherRunner:115 : Job
> Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already
> succeed, 3 error, 0 discarded, 0 others2020-08-03 11:42:06,581 DEBUG
> [streaming_job_submitter-thread-1] coordinate.BuildJobSubmitter:142 :
> =========================================================================
> 102020-08-03 11:42:06,581 DEBUG [streaming_job_submitter-thread-1]
> coordinator.ZookeeperStreamMetadataStore:673 : [StreamMetadataStoreStats]
> read : 2 ; write: 1 ; read failed: 0 ; write failed: 0 .2020-08-03
> 11:42:36,063 INFO [FetcherRunner 555305792-83]
> threadpool.DefaultFetcherRunner:115 : Job Fetcher: 0 should running, 0 actual
> running, 0 stopped, 0 ready, 0 already succeed, 3 error, 0 discarded, 0 others
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
