Hi feng, your map reduce classpath might not be correctly configured; Kylin will read the ³mapreduce.application.classpath² from default job configuration; if not found that, it will run ³mapred classpath² command to get the classpath, and then append hive/hbase dependencies; Please check kylin.log to see whether the final classpath includes the jar for this missing class;
The message in kylin.log is as below, you can search it: Hadoop job classpath is: On 9/7/15, 10:18 PM, "yu feng" <[email protected]> wrote: >After submit a mapreduce job, we get job status,But it tells the job >failed! we check this application on RM website(xxx:8088/cluster/apps), >we >find those log : > > Application application_1418904565842_3597024 failed 2 times due to AM >Container for appattempt_1418904565842_3597024_000002 exited with >exitCode: >1 due to: Exception from container-launch: >org.apache.hadoop.util.Shell$ExitCodeException: >at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) >at org.apache.hadoop.util.Shell.run(Shell.java:379) >at >org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) >at >org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchCon >tainer(LinuxContainerExecutor.java:252) >at >org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contai >nerLaunch.call(ContainerLaunch.java:283) >at >org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contai >nerLaunch.call(ContainerLaunch.java:79) >at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >at java.util.concurrent.FutureTask.run(FutureTask.java:138) >at >java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor. >java:895) >at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java >:918) >at java.lang.Thread.run(Thread.java:662) >main : command provided 1 > >we check yarn log with this command : yarn logs -applicationId >application_1418904565842_3597024,get those log : >Container: container_1418904565842_3597024_01_000001 on >hadoop88.photo.163.org_56708 >========================================================================== >============ >LogType: stderr >LogLength: 664 >Log Contents: >Exception in thread "main" java.lang.NoClassDefFoundError: >org/apache/hadoop/mapreduce/v2/app/MRAppMaster >Caused by: java.lang.ClassNotFoundException: >org.apache.hadoop.mapreduce.v2.app.MRAppMaster >at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >at java.security.AccessController.doPrivileged(Native Method) >at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >Could not find the main class: >org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. > > >I thinks this means the task can not find the jar files, So I upload all >the jars that kylin dependent to HDFS, and before submit this job*(in >AbstractHadoopJob.attachKylinPropsAndMetadata function )* I set "tmpjars" >to those files located on HDFS(this way can avoid uploading all files when >submit every mapreduce job). > >This measure works in kylin-0.7.2, I get the same error in kylin-1.0 and I >guess this measure will work in kylin-1.0 too, But I do not think this is >a >good idea, > >It will be highly appreciated if you have some good idea or some >suggestion. Thanks...
