Hi,
It seems the problem is following
"60505 [dispatcher-event-loop-6] ERROR
org.apache.spark.scheduler.cluster.YarnScheduler - Lost executor 1 on
*********: Container marked as failed:"
It usually comes out with not enough memory for your yarn so that yarn
container is closed because of lack of memory , you can go to yarn resource
manager web page to see more details with yarn log.
If it's the memory issue, you can try to allocate more memory for spark
yarn executor by change the following configuration item in
"$KYLIN_HOME/conf/kylin.properties"
kylin.engine.spark-conf.spark.yarn.executor.memoryOverhead=384
-------------------
Best wishes,
Rupeng Wang
在 2019/8/29 14:57，“Gourav Gupta”<[email protected]> 写入:
Hi Sir,
I have installed and configured Apache Kylin 2.4 on Cloudera Platform for
creating the Cube.
I have been able to create a cube in MapReduce mode but getting the
below-mentioned caveat while executes on spark mode. I have had followed
all the steps and tried many remedies for debugging the problem.
Please let me know how to resolve this bug. Thanks in Advance.
1091 [main] ERROR org.apache.spark.SparkContext - Error adding jar
(java.lang.IllegalArgumentException: requirement failed: JAR
kylin-job-2.4.0.jar already registered.), was the --addJars option used?
[Stage 0:> (0 + 0)
/ 2]
[Stage 0:> (0 + 2)
/ 2]
60505 [dispatcher-event-loop-6] ERROR
org.apache.spark.scheduler.cluster.YarnScheduler - Lost executor 1 on **
*******: Container marked as failed:
container_e62_1566915974858_6628_01_000003 on host: *******. Exit status:
50. Diagnostics: Exception from container-launch.
Container id: container_e62_1566915974858_6628_01_000003
Exit code: 50
Stack trace: ExitCodeException exitCode=50:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
at org.apache.hadoop.util.Shell.run(Shell.java:507)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 50
82664 [dispatcher-event-loop-5] ERROR
org.apache.spark.scheduler.cluster.YarnScheduler
- Lost executor 2 on *******: Container marked as failed:
container_e62_1566915974858_6628_01_000004 on host: *******. Exit status:
50. Diagnostics: Exception from container-launch.
Container id: container_e62_1566915974858_6628_01_000004
Exit code: 50
Stack trace: ExitCodeException exitCode=50:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:604)
at org.apache.hadoop.util.Shell.run(Shell.java:507)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.
launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.
launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.
launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 50
The command is:
export HADOOP_CONF_DIR=/etc/hadoop/conf && /usr/lib/spark/bin/spark-submit
--class org.apache.kylin.common.util.SparkEntry --conf
spark.executor.instances=1 --conf spark.yarn.archive=hdfs://
namenode:8020/kylin/spark/spark-libs.jar --conf spark.yarn.queue=default
--conf spark.yarn.am.extraJavaOptions=-Dhdp.version=current --conf
spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf
spark.driver.extraJavaOptions=-Dhdp.version=current --conf
spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
--conf spark.master=yarn --conf
spark.executor.extraJavaOptions=-Dhdp.version=current
--conf spark.hadoop.yarn.timeline-service.enabled=false --conf
spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf
spark.eventLog.dir=hdfs:///kylin/spark-history --conf
spark.executor.cores=2 --conf spark.submit.deployMode=cluster --jars
/opt/apache-kylin-2.4.0-bin-cdh57/lib/kylin-job-2.4.0.jar
/opt/apache-kylin-2.4.0-bin-cdh57/lib/kylin-job-2.4.0.jar -className
org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable
default.kylin_intermediate_kylin_sales_cube_c1526d16_9719_4dec_be41_346f43654e02
-input hdfs://nameservice1/kylin/kylin_metadata/kylin-2159d40b-
f14e-4500-af95-1fbfd5a4073f/kylin_intermediate_kylin_
sales_cube_c1526d16_9719_4dec_be41_346f43654e02 -segmentId
c1526d16-9719-4dec-be41-346f43654e02 -metaUrl kylin_metadata@hdfs,path=hdfs:
//nameservice1/kylin/kylin_metadata/kylin-2159d40b-f14e-
4500-af95-1fbfd5a4073f/kylin_sales_cube/metadata -output
hdfs://nameservice1/kylin/kylin_metadata/kylin-2159d40b-
f14e-4500-af95-1fbfd5a4073f/kylin_sales_cube/cuboid/ -cubename
kylin_sales_cube
