Hi Yanghong, There had upload all logs file in attach, and I found a NPE exception at spark cube building time. Like below hope it can help you.
18/04/03 15:13:22 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:636) Caused by: java.lang.NullPointerException at org.apache.kylin.engine.mr.common.CubeStatsReader.estimateLayerSize(CubeStatsReader.java:281) at org.apache.kylin.engine.spark.SparkCubingByLayer.estimateRDDPartitionNum(SparkCubingByLayer.java:218) at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:189) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) Thanks. 2018-04-03 2:21 GMT+08:00 Zhong, Yanghong <[email protected]>: > Hi, > > Could you attach the app master log? For system cube, it's better to use > MR mode with INMEM algorithm. However, we should figure out why spark > cubing not works. > > Best regards, > Yanghong Zhong > > Email: [email protected] > > On 4/2/18, 10:35 AM, "凡梦星尘" <[email protected]> wrote: > > So no one can help me about this trouble? > > 2018-03-28 19:12 GMT+08:00 凡梦星尘 <[email protected]>: > > > Hi guys: > > > > Is there someone use newest Kylin and try open dashboard? I had try > this > > new feature, by the default the system cube use MR mode, build and > refresh > > action no any problems. But when I change to spark mode not success. > The > > error logs see below: > > > > > > OS command error exit with return code: 1, error message: log4j:WARN > No > > appenders could be found for logger (org.apache.hadoop.util.Shell). > > log4j:WARN Please initialize the log4j system properly. > > log4j:WARN See https://na01.safelinks.protection.outlook.com/?url= > http%3A%2F%2Flogging.apache.org%2Flog4j%2F1.2%2Ffaq.html% > 23noconfig&data=02%7C01%7Cyangzhong%40ebay.com% > 7C56cb5aff32af4699415b08d598426d17%7C46326bff992841a0baca17c16c94 > ea99%7C0%7C1%7C636582333428432372&sdata=O74xeYvbdtaJ2BtDNvBkX0AinRnOob > c0LvgdxG9%2Fgx4%3D&reserved=0 for > > more info. > > Using Spark's default log4j profile: org/apache/spark/log4j- > > defaults.properties > > 18/03/28 18:03:30 INFO Client: Requesting a new application from > cluster > > with 3 NodeManagers > > 18/03/28 18:03:30 INFO Client: Verifying our application has not > requested > > more than the maximum memory capability of the cluster (8192 MB per > > container) > > 18/03/28 18:03:30 INFO Client: Will allocate AM container, with 1408 > MB > > memory including 384 MB overhead > > 18/03/28 18:03:30 INFO Client: Setting up container launch context > for our > > AM > > 18/03/28 18:03:30 INFO Client: Setting up the launch environment for > our > > AM container > > 18/03/28 18:03:30 INFO Client: Preparing resources for our AM > container > > 18/03/28 18:03:32 WARN Client: Neither spark.yarn.jars nor > > spark.yarn.archive is set, falling back to uploading libraries under > > SPARK_HOME. > > 18/03/28 18:03:34 INFO Client: Uploading resource > > file:/tmp/spark-e9a0636e-6d6e-476a-ad62-dfcb1e4d6102/__spark_libs__ > 437322364481459969.zip > > -> hdfs://xxx:8020/user/hdfs/.sparkStaging/application_ > > 1522141683864_0091/__spark_libs__437322364481459969.zip > > 18/03/28 18:03:38 INFO Client: Uploading resource > > file:/opt/apache-kylin-2.3.0/lib/kylin-job-2.3.0.jar -> > > hdfs://xxx:8020/user/hdfs/.sparkStaging/application_ > > 1522141683864_0091/kylin-job-2.3.0.jar > > 18/03/28 18:03:39 INFO Client: Uploading resource > > file:/data/hdp/2.6.4.0-91/hbase/lib/htrace-core-3.1.0-incubating.jar > -> > > hdfs://xxx:8020/user/hdfs/.sparkStaging/application_ > > 1522141683864_0091/htrace-core-3.1.0-incubating.jar > > 18/03/28 18:03:39 INFO Client: Uploading resource > > file:/data/hdp/2.6.4.0-91/hbase/lib/metrics-core-2.2.0.jar -> > > hdfs://xxx:8020/user/hdfs/.sparkStaging/application_ > > 1522141683864_0091/metrics-core-2.2.0.jar > > 18/03/28 18:03:39 INFO Client: Uploading resource > > file:/data/hdp/2.6.4.0-91/hbase/lib/guava-12.0.1.jar -> > > hdfs://xxx:8020/user/hdfs/.sparkStaging/application_ > > 1522141683864_0091/guava-12.0.1.jar > > 18/03/28 18:03:40 INFO Client: Uploading resource > > file:/tmp/spark-e9a0636e-6d6e-476a-ad62-dfcb1e4d6102/__spark_conf__ > 5932514761984685807.zip > > -> hdfs://xxx:8020/user/hdfs/.sparkStaging/application_ > > 1522141683864_0091/__spark_conf__.zip > > 18/03/28 18:03:40 WARN Client: spark.yarn.am.extraJavaOptions will > not > > take effect in cluster mode > > 18/03/28 18:03:40 INFO SecurityManager: Changing view acls to: hdfs > > 18/03/28 18:03:40 INFO SecurityManager: Changing modify acls to: hdfs > > 18/03/28 18:03:40 INFO SecurityManager: Changing view acls groups to: > > 18/03/28 18:03:40 INFO SecurityManager: Changing modify acls groups > to: > > 18/03/28 18:03:40 INFO SecurityManager: SecurityManager: > authentication > > disabled; ui acls disabled; users with view permissions: Set(hdfs); > groups > > with view permissions: Set(); users with modify permissions: > Set(hdfs); > > groups with modify permissions: Set() > > 18/03/28 18:03:40 INFO Client: Submitting application > > application_1522141683864_0091 to ResourceManager > > 18/03/28 18:03:40 INFO YarnClientImpl: Submitted application > > application_1522141683864_0091 > > 18/03/28 18:03:41 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:41 INFO Client: > > client token: N/A > > diagnostics: AM container is launched, waiting for AM container to > > Register with RM > > ApplicationMaster host: N/A > > ApplicationMaster RPC port: -1 > > queue: default > > start time: 1522231420490 > > final status: UNDEFINED > > tracking URL: http://xxx:8088/proxy/application_1522141683864_0091/ > > user: hdfs > > 18/03/28 18:03:42 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:43 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:44 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:45 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:46 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:47 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:48 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:49 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:50 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:51 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:52 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:03:53 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:03:53 INFO Client: > > client token: N/A > > diagnostics: N/A > > ApplicationMaster host: xxx > > ApplicationMaster RPC port: 0 > > queue: default > > start time: 1522231420490 > > final status: UNDEFINED > > tracking URL: http://xxx:8088/proxy/application_1522141683864_0091/ > > user: hdfs > > 18/03/28 18:03:54 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:03:55 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:03:56 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:03:57 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:03:58 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:03:59 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:00 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:01 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:02 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:03 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:04 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:05 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:06 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:07 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:08 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:09 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:10 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:11 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:12 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:13 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:14 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:15 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:16 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:16 INFO Client: > > client token: N/A > > diagnostics: AM container is launched, waiting for AM container to > > Register with RM > > ApplicationMaster host: N/A > > ApplicationMaster RPC port: -1 > > queue: default > > start time: 1522231420490 > > final status: UNDEFINED > > tracking URL: http://xxx:8088/proxy/application_1522141683864_0091/ > > user: hdfs > > 18/03/28 18:04:17 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:18 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:19 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:20 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:21 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:22 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:23 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:25 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:26 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:27 INFO Client: Application report for > > application_1522141683864_0091 (state: ACCEPTED) > > 18/03/28 18:04:28 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:28 INFO Client: > > client token: N/A > > diagnostics: N/A > > ApplicationMaster host: xxx > > ApplicationMaster RPC port: 0 > > queue: default > > start time: 1522231420490 > > final status: UNDEFINED > > tracking URL: http://xxx:8088/proxy/application_1522141683864_0091/ > > user: hdfs > > 18/03/28 18:04:29 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:30 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:31 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:32 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:33 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:34 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:35 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:36 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:37 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:38 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:39 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:40 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:41 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:42 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:43 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:44 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:45 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:46 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:47 INFO Client: Application report for > > application_1522141683864_0091 (state: RUNNING) > > 18/03/28 18:04:48 INFO Client: Application report for > > application_1522141683864_0091 (state: FINISHED) > > 18/03/28 18:04:48 INFO Client: > > client token: N/A > > diagnostics: User class threw exception: java.lang.RuntimeException: > error > > execute org.apache.kylin.engine.spark.SparkCubingByLayer > > ApplicationMaster host: xxx > > ApplicationMaster RPC port: 0 > > queue: default > > start time: 1522231420490 > > final status: FAILED > > tracking URL: http://xxx:8088/proxy/application_1522141683864_0091/ > > user: hdfs > > Exception in thread "main" org.apache.spark.SparkException: > Application > > application_1522141683864_0091 finished with failed status > > at org.apache.spark.deploy.yarn.Client.run(Client.scala:1180) > > at org.apache.spark.deploy.yarn.Client\$.main(Client.scala:1226) > > at org.apache.spark.deploy.yarn.Client.main(Client.scala) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java:62) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at org.apache.spark.deploy.SparkSubmit\$.org\$apache\$ > > spark\$deploy\$SparkSubmit\$\$runMain(SparkSubmit.scala:744) > > at org.apache.spark.deploy.SparkSubmit\$.doRunMain\$1( > > SparkSubmit.scala:187) > > at org.apache.spark.deploy.SparkSubmit\$.submit( > SparkSubmit.scala:212) > > at org.apache.spark.deploy.SparkSubmit\$.main(SparkSubmit.scala:126) > > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > 18/03/28 18:04:48 INFO ShutdownHookManager: Shutdown hook called > > 18/03/28 18:04:48 INFO ShutdownHookManager: Deleting directory > > /tmp/spark-e9a0636e-6d6e-476a-ad62-dfcb1e4d6102 > > The command is: > > export HADOOP_CONF_DIR=/opt/apache-kylin-2.3.0/hadoop-conf && > > /opt/apache-kylin-2.3.0/spark/bin/spark-submit --class > > org.apache.kylin.common.util.SparkEntry --conf > > spark.executor.instances=1 --conf spark.yarn.queue=default --conf > > spark.yarn.am.extraJavaOptions=-Dhdp.version=current --conf > > spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf > > spark.driver.extraJavaOptions=-Dhdp.version=current --conf > > spark.io.compression.codec=org.apache.spark.io. > SnappyCompressionCodec > > --conf spark.master=yarn --conf spark.executor. > extraJavaOptions=-Dhdp.version=current > > --conf spark.hadoop.yarn.timeline-service.enabled=true --conf > > spark.executor.memory=1G --conf spark.eventLog.enabled=true --conf > > spark.eventLog.dir=hdfs:///kylin/spark-history --conf > > spark.executor.cores=2 --conf spark.submit.deployMode=cluster > --jars > > /data/hdp/2.6.4.0-91/hbase/lib/htrace-core-3.1.0- > > incubating.jar,/data/hdp/2.6.4.0-91/hbase/lib/metrics-core- > > 2.2.0.jar,/data/hdp/2.6.4.0-91/hbase/lib/guava-12.0.1.jar, > > /opt/apache-kylin-2.3.0/lib/kylin-job-2.3.0.jar -className > > org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable > > kylin_tmp.kylin_intermediate_kylin_hive_metrics_job_qa_ > > 912467dc_7a1c_42ce_8ff5_60c2a5b11441 -output > hdfs://xxx:8020/kylin/kylin_ > > metadata/kylin-f943d829-3436-4816-bfa6-da2234401862/KYLIN_ > HIVE_METRICS_JOB_QA/cuboid/ > > -segmentId 912467dc-7a1c-42ce-8ff5-60c2a5b11441 -metaUrl > > kylin_metadata@hdfs,path=hdfs://xxx:8020/kylin/kylin_ > > metadata/metadata/912467dc-7a1c-42ce-8ff5-60c2a5b11441 -cubename > > KYLIN_HIVE_METRICS_JOB_QA > > > > I not sure whether is my HDP environment problem, but I can sure the > > sample cube running spark mode well. And there had someone bring > > forward same issues but not earn resolve. So please tell me how to > do? > > > > PS: > > > > I can't find more detail info about this exception. Only cat from > > kylin.log file. If you need more things, tell what steps. > > > > Thanks. > > > > > > >
kylin.log.tar.gz
Description: application/gzip
yarn.log.tar.gz
Description: application/gzip
