Hi Sir, I have installed and configured Apache Kylin 2.4 on Cloudera Platform for creating the Cube.
I have been able to create a cube in MapReduce mode but getting the below-mentioned caveat while executes on spark mode. I have had followed all the steps and tried many remedies for debugging the problem. Please let me know how to resolve this bug. Thanks in Advance. 1091 [main] ERROR org.apache.spark.SparkContext - Error adding jar (java.lang.IllegalArgumentException: requirement failed: JAR kylin-job-2.4.0.jar already registered.), was the --addJars option used? [Stage 0:> (0 + 0) / 2] [Stage 0:> (0 + 2) / 2] 60505 [dispatcher-event-loop-6] ERROR org.apache.spark.scheduler.cluster.YarnScheduler - Lost executor 1 on ** *******: Container marked as failed: container_e62_1566915974858_6628_01_000003 on host: *******. Exit status: 50. Diagnostics: Exception from container-launch. Container id: container_e62_1566915974858_6628_01_000003 Exit code: 50 Stack trace: ExitCodeException exitCode=50: at org.apache.hadoop.util.Shell.runCommand(Shell.java:604) at org.apache.hadoop.util.Shell.run(Shell.java:507) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 50 82664 [dispatcher-event-loop-5] ERROR org.apache.spark.scheduler.cluster.YarnScheduler - Lost executor 2 on *******: Container marked as failed: container_e62_1566915974858_6628_01_000004 on host: *******. Exit status: 50. Diagnostics: Exception from container-launch. Container id: container_e62_1566915974858_6628_01_000004 Exit code: 50 Stack trace: ExitCodeException exitCode=50: at org.apache.hadoop.util.Shell.runCommand(Shell.java:604) at org.apache.hadoop.util.Shell.run(Shell.java:507) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor. launchContainer(DefaultContainerExecutor.java:213) at org.apache.hadoop.yarn.server.nodemanager.containermanager. launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager. launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 50 The command is: export HADOOP_CONF_DIR=/etc/hadoop/conf && /usr/lib/spark/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=1 --conf spark.yarn.archive=hdfs:// namenode:8020/kylin/spark/spark-libs.jar --conf spark.yarn.queue=default --conf spark.yarn.am.extraJavaOptions=-Dhdp.version=current --conf spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf spark.driver.extraJavaOptions=-Dhdp.version=current --conf spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec --conf spark.master=yarn --conf spark.executor.extraJavaOptions=-Dhdp.version=current --conf spark.hadoop.yarn.timeline-service.enabled=false --conf spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs:///kylin/spark-history --conf spark.executor.cores=2 --conf spark.submit.deployMode=cluster --jars /opt/apache-kylin-2.4.0-bin-cdh57/lib/kylin-job-2.4.0.jar /opt/apache-kylin-2.4.0-bin-cdh57/lib/kylin-job-2.4.0.jar -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable default.kylin_intermediate_kylin_sales_cube_c1526d16_9719_4dec_be41_346f43654e02 -input hdfs://nameservice1/kylin/kylin_metadata/kylin-2159d40b- f14e-4500-af95-1fbfd5a4073f/kylin_intermediate_kylin_ sales_cube_c1526d16_9719_4dec_be41_346f43654e02 -segmentId c1526d16-9719-4dec-be41-346f43654e02 -metaUrl kylin_metadata@hdfs,path=hdfs: //nameservice1/kylin/kylin_metadata/kylin-2159d40b-f14e- 4500-af95-1fbfd5a4073f/kylin_sales_cube/metadata -output hdfs://nameservice1/kylin/kylin_metadata/kylin-2159d40b- f14e-4500-af95-1fbfd5a4073f/kylin_sales_cube/cuboid/ -cubename kylin_sales_cube
