as the previous reply, currently after changing the pom for job.jar as mentioned, the error changes as below: Task failed task_1425461099463_0009_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0
another thing is that the same error occurs when I use the tgz file from official build. 2015-03-04 20:40 GMT+08:00 dong wang <[email protected]>: > CDH 5.3.1 + java version "1.7.0_75" > > 2015-03-04 20:34 GMT+08:00 Zhou, Qianhao <[email protected]>: > >> Hi dong >> Can you please tell us what exactly is your java version of your >> sandbox. >> According to >> http://stackoverflow.com/questions/8302022/maven-shade-jar-throw-exception >> The root cause maybe the java version >> >> Best Regard >> Zhou QianHao >> >> >> >> >> >> On 3/4/15, 7:08 PM, "dong wang" <[email protected]> wrote: >> >> >I built the jar from the source, as checked, only, there is a file named >> >*.SF under job/target/kylin-job-0.7.1-SNAPSHOT-job.jar, then I tried to >> >update the pom.xml with adding "<exclude>META-INF/*.SF</exclude>" >> >mentioned >> >above under "job" directory for the job.jar, then build again, however, >> >currently, there is stranger problem with few job info: >> > >> >Task failed task_1425461099463_0005_m_000000 >> >Job failed as tasks failed. failedMaps:1 failedReduces:0 >> > >> > >> > >> >2015-03-04 18:03 GMT+08:00 Zhou, Qianhao <[email protected]>: >> > >> >> @Dong >> >> Are you running kylin from source code or kylin-0.7.1-SNAPSHOT.tgz? >> >> >> >> If from source code, you will see >> >> job/target/kylin-job-0.7.1-SNAPSHOT-job.jar >> >> storage/target/kylin-storage-0.7.1-SNAPSHOT-coprocessor.jar >> >> >> >> If from tgz >> >> lib/kylin-job-0.7.1-SNAPSHOT.jar >> >> lib/kylin-coprocessor-0.7.1-SNAPSHOT.jar >> >> >> >> Please check if there are *.SF *.RSA *.DSA files in these two jars. >> >>If >> >> they do exist, as a workaround, you can remove these files from the >> jar. >> >> Please let us know if it works, and then we will fix it. >> >> >> >> Best Regard >> >> Zhou QianHao >> >> >> >> >> >> >> >> >> >> >> >> On 3/4/15, 5:48 PM, "dong wang" <[email protected]> wrote: >> >> >> >> >I'm not sure whether the problem is similar to: >> >> >http://www.cnblogs.com/fuxinci/p/3356087.html >> >> > >> >> >2015-03-04 17:38 GMT+08:00 dong wang <[email protected]>: >> >> > >> >> >> As checked, it says: >> >> >> >> >> >> Job init failed : java.lang.SecurityException: Invalid signature >> file >> >> >> digest for Manifest main attributes >> >> >> at >> >> >> >> >> >> >> >>>>sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifi >> >>>>er >> >> >>.java:286) >> >> >> at >> >> >> >> >> >> >> >>>>sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.j >> >>>>av >> >> >>a:239) >> >> >> at java.util.jar.JarVerifier.processEntry(JarVerifier.java:317) >> >> >> at java.util.jar.JarVerifier.update(JarVerifier.java:228) >> >> >> at java.util.jar.JarFile.initializeVerifier(JarFile.java:348) >> >> >> at java.util.jar.JarFile.getInputStream(JarFile.java:415) >> >> >> at >> >> >> >> >>>>sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775) >> >> >> at sun.misc.Resource.cachedInputStream(Resource.java:77) >> >> >> at sun.misc.Resource.getByteBuffer(Resource.java:160) >> >> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:436) >> >> >> at java.net.URLClassLoader.access$100(URLClassLoader.java:71) >> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> >> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> >> >> at java.lang.Class.forName0(Native Method) >> >> >> at java.lang.Class.forName(Class.java:191) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.isChainJob(JobImpl. >> >>>>ja >> >> >>va:1305) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.makeUberDecision(Jo >> >>>>bI >> >> >>mpl.java:1247) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$3800(JobImpl >> >>>>.j >> >> >>ava:139) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.tran >> >>>>si >> >> >>tion(JobImpl.java:1451) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.tran >> >>>>si >> >> >>tion(JobImpl.java:1388) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.do >> >>>>Tr >> >> >>ansition(StateMachineFactory.java:385) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMach >> >>>>in >> >> >>eFactory.java:302) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachin >> >>>>eF >> >> >>actory.java:46) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.d >> >>>>oT >> >> >>ransition(StateMachineFactory.java:448) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java >> >>>>:9 >> >> >>96) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java >> >>>>:1 >> >> >>38) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handl >> >>>>e( >> >> >>MRAppMaster.java:1272) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster >> >>>>.j >> >> >>ava:1045) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.service.AbstractService.start(AbstractService.java:19 >> >>>>3) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1 >> >>>>47 >> >> >>8) >> >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> >> at javax.security.auth.Subject.doAs(Subject.java:415) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati >> >>>>on >> >> >>.java:1642) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MR >> >>>>Ap >> >> >>pMaster.java:1474) >> >> >> at >> >> >> >> >> >> >> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:14 >> >>>>07 >> >> >>) >> >> >> >> >> >> 2015-03-04 17:03 GMT+08:00 Zhou, Qianhao <[email protected]>: >> >> >> >> >> >>> @Dong >> >> >>> The map-reduce job may encounter some error, can you please >> look >> >> >>>into >> >> >>> the job detail. >> >> >>> Just FYI, you can use http://$job history >> >> >>> server$/jobhistory/job/job_1425448727678_0003/ to see what is >> really >> >> >>>going >> >> >>> on. >> >> >>> >> >> >>> Best Regard >> >> >>> Zhou QianHao >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> On 3/4/15, 3:21 PM, "dong wang" <[email protected]> wrote: >> >> >>> >> >> >>> >more error info: >> >> >>> > >> >> >>> > >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:41,023][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.c >> >>>>>>he >> >> >>>>ckS >> >> >>> >tatus(HadoopStatusChecker.java:57)] >> >> >>> >- State of Hadoop job: job_1425448727678_0003:ACCEPTED-UNDEFINED >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:41,030][DEBUG][org.apache.kylin.common.persistence.ResourceStor >> >>>>>>e. >> >> >>>>put >> >> >>> >Resource(ResourceStore.java:171)] >> >> >>> >- Saving resource >> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 >> >> >>> >(Store kylin_metadata@hbase) >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:51,046][WARN][org.apache.commons.httpclient.HttpMethodBase.getR >> >>>>>>es >> >> >>>>pon >> >> >>> >seBody(HttpMethodBase.java:682)] >> >> >>> >- Going to buffer response body of large or unknown size. Using >> >> >>> >getResponseBodyAsStream instead is recommended. >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >>>>>>14:19:51,047][DEBUG][ >> org.apache.kylin.job.tools.HadoopStatusGetter.ge >> >>>>>>tH >> >> >>>>ttp >> >> >>> >Response(HadoopStatusGetter.java:90)] >> >> >>> >- Job job_1425448727678_0003 get status check result. >> >> >>> > >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.c >> >>>>>>he >> >> >>>>ckS >> >> >>> >tatus(HadoopStatusChecker.java:57)] >> >> >>> >- State of Hadoop job: job_1425448727678_0003:FINISHED-FAILED >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:51,058][DEBUG][org.apache.kylin.common.persistence.ResourceStor >> >>>>>>e. >> >> >>>>put >> >> >>> >Resource(ResourceStore.java:171)] >> >> >>> >- Saving resource >> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 >> >> >>> >(Store kylin_metadata@hbase) >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:51,068][DEBUG][org.apache.kylin.common.persistence.ResourceStor >> >>>>>>e. >> >> >>>>put >> >> >>> >Resource(ResourceStore.java:171)] >> >> >>> >- Saving resource >> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 >> >> >>> >(Store kylin_metadata@hbase) >> >> >>> >2015-03-04 14:19:51,074 INFO [pool-7-thread-3] >> >> >>> >mapred.ClientServiceDelegate: Application state is completed. >> >> >>> >FinalApplicationStatus=FAILED. Redirecting to job history server >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:51,135][WARN][org.apache.kylin.job.common.HadoopCmdOutput.updat >> >>>>>>eJ >> >> >>>>obC >> >> >>> >ounter(HadoopCmdOutput.java:89)] >> >> >>> >- no counters for job job_1425448727678_0003 >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:51,141][DEBUG][org.apache.kylin.common.persistence.ResourceStor >> >>>>>>e. >> >> >>>>put >> >> >>> >Resource(ResourceStore.java:171)] >> >> >>> >- Saving resource >> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 >> >> >>> >(Store kylin_metadata@hbase) >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:51,149][DEBUG][org.apache.kylin.common.persistence.ResourceStor >> >>>>>>e. >> >> >>>>put >> >> >>> >Resource(ResourceStore.java:171)] >> >> >>> >- Saving resource >> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 >> >> >>> >(Store kylin_metadata@hbase) >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:51,162][DEBUG][org.apache.kylin.common.persistence.ResourceStor >> >>>>>>e. >> >> >>>>put >> >> >>> >Resource(ResourceStore.java:171)] >> >> >>> >- Saving resource >> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 >> >> >>> >(Store kylin_metadata@hbase) >> >> >>> >[pool-7-thread-3]:[2015-03-04 >> >> >>> >> >> >>> >> >> >> >> >>>>>>14:19:51,165][INFO][org.apache.kylin.job.manager.ExecutableManager.up >> >>>>>>da >> >> >>>>teJ >> >> >>> >obOutput(ExecutableManager.java:222)] >> >> >>> >- job id:99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 from RUNNING to >> >>ERROR >> >> >>> > >> >> >>> >2015-03-04 14:42 GMT+08:00 dong wang <[email protected]>: >> >> >>> > >> >> >>> >> when check the log shown in web GUI of this step: >> >> >>> >> >> >> >>> >> 2015-03-03 22:39:39.495 - State of Hadoop job: >> >> >>> >> job_1425448727678_0004:ACCEPTED - UNDEFINED >> >> >>> >> 2015-03-03 22:39:49.518 - State of Hadoop job: >> >> >>> >> job_1425448727678_0004:FINISHED - FAILED >> >> >>> >> no counters for job job_1425448727678_0004 >> >> >>> >> >> >> >>> >> >> >> >>> >> with the same cube, it can be built successfully with kylin-0.6* >> >> >>>before >> >> >>> >> >> >> >>> >> >> >>> >> >> >> >> >> >> >> >> >> >
