Also, here is yarn logs output from one of the failing reducer : $ yarn logs -applicationId application_1512700237856_1192 -containerId > container_e181_1512700237856_1192_01_001172 -nodeAddress > pc1udatahad08.x.y:8041 >
> LogType:stderr > Log Upload Time:Tue Jan 02 12:34:51 -0700 2018 > LogLength:46 > Log Contents: > Error: Could not find or load main class 8192 > > LogType:stdout > Log Upload Time:Tue Jan 02 12:34:51 -0700 2018 > LogLength:0 > Log Contents: So there is nothing meaningful except that "could not find or load main class 8192". Any ideas? -- Ruslan Dautkhanov On Tue, Jan 2, 2018 at 10:31 PM, Ruslan Dautkhanov <[email protected]> wrote: > ShaoFeng, I've uploaded full log as https://github.com/Tagar/ > dropbox/blob/master/application_1512700237856_1192.2.log.tgz > Sorry it's not small log as the application was running for 45 minutes and > all mappers have completed before reducers started to fail. > None of the reducers were successful. > I can not spot a problem there.. any hints would be extremely helpful. > > > > -- > Ruslan Dautkhanov > > On Tue, Jan 2, 2018 at 8:08 PM, ShaoFeng Shi <[email protected]> > wrote: > >> There should be more logs for this job. You can try "yarn logs" command >> to retrieve it. As-is log doesn't have enough information. >> >> 2018-01-03 3:52 GMT+08:00 Ruslan Dautkhanov <[email protected]>: >> >>> Happy New Year ! >>> >>> Bouncing at the users list if this is a known issue to someone: >>> >>> Cube build consistently fails on Step 3 "Extract Fact Table Distinct >>> Columns". >>> First two steps complete just fine. >>> Mappers work fine, it only the reducers that fail - many of them with >>> the same error [1]. >>> Also [2] shows stdout/stderr from one of the reducers attempts. >>> >>> CDH 5.12.1. >>> Kylin 2.2. >>> >>> Has anyone seen this issue? >>> >>> >>> [1] >>> >>> Exception from container-launch. >>>> Container id: container_e181_1512700237856_1192_01_001227 >>>> Exit code: 1 >>>> Stack trace: ExitCodeException exitCode=1: >>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:601) >>>> at org.apache.hadoop.util.Shell.run(Shell.java:504) >>>> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Sh >>>> ell.java:786) >>>> at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExec >>>> utor.launchContainer(LinuxContainerExecutor.java:373) >>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.l >>>> auncher.ContainerLaunch.call(ContainerLaunch.java:302) >>>> at org.apache.hadoop.yarn.server.nodemanager.containermanager.l >>>> auncher.ContainerLaunch.call(ContainerLaunch.java:82) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>>> Executor.java:1149) >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>>> lExecutor.java:624) >>>> at java.lang.Thread.run(Thread.java:748) >>>> Shell output: main : command provided 1 main : >>>> run as user is rdautkha main : requested yarn user is rdautkha >>>> Writing to tmp file /hdfs02/yarn/nm/nmPrivate/appl >>>> ication_1512700237856_1192/container_e181_1512700237856_1192 >>>> _01_001227/container_e181_1512700237856_1192_01_001227.pid.tmp >>>> Container exited with a non-zero exit code 1 >>> >>> >>> >>> [2] >>> >>> Log Type: stderr >>>> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018 >>>> Log Length: 46 >>>> Error: Could not find or load main class 8192 >>> >>> >>> Log Type: stdout >>>> Log Upload Time: Tue Jan 02 12:34:51 -0700 2018 >>>> Log Length: 0 >>> >>> >>> >> >> >> -- >> Best regards, >> >> Shaofeng Shi 史少锋 >> >> >
