Sure, no problem. Here it is.
OS command error exit with 1 -- export HADOOP_CONF_DIR=/data/kylin/kylin/hadoop-conf-dir && /data/kylin/kylin/spark/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=1 --conf spark.yarn.jar=hdfs://nsha/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar --conf spark.yarn.queue=default --conf spark.history.fs.logDirectory=hdfs:///ahz/var/spark-logs --conf spark.master=yarn --conf spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs:///ahz/var/spark-logs --conf spark.executor.cores=2 --conf spark.submit.deployMode=cluster --files /etc/hbase/conf.cloudera.hbase/hbase-site.xml --jars /data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/htrace-core-3.2.0-incubating.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/metrics-core-2.2.0.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar, /data/kylin/kylin/lib/kylin-job-2.0.0.jar -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6 -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube SparkEntry args:-className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6 -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube Abstract Application args:-hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6 -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 17/05/17 09:13:36 INFO SparkContext: Running Spark version 1.6.3 17/05/17 09:13:37 INFO SecurityManager: Changing view acls to: kylin 17/05/17 09:13:37 INFO SecurityManager: Changing modify acls to: kylin 17/05/17 09:13:37 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kylin); users with modify permissions: Set(kylin) 17/05/17 09:13:37 INFO Utils: Successfully started service 'sparkDriver' on port 42715. 17/05/17 09:13:37 INFO Slf4jLogger: Slf4jLogger started 17/05/17 09:13:37 INFO Remoting: Starting remoting 17/05/17 09:13:38 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 39514. 17/05/17 09:13:38 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:39514] 17/05/17 09:13:38 INFO SparkEnv: Registering MapOutputTracker 17/05/17 09:13:38 INFO SparkEnv: Registering BlockManagerMaster 17/05/17 09:13:38 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-6a57441d-d256-4ade-acf0-af6985732211 17/05/17 09:13:38 INFO MemoryStore: MemoryStore started with capacity 511.1 MB 17/05/17 09:13:38 INFO SparkEnv: Registering OutputCommitCoordinator 17/05/17 09:13:38 INFO Utils: Successfully started service 'SparkUI' on port 4040. 17/05/17 09:13:38 INFO SparkUI: Started SparkUI at http://10.24.26.191:4040 17/05/17 09:13:38 INFO HttpFileServer: HTTP File server directory is /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd-7d7509ef-786d-4fa4-a75e-bb3cb1015c77 17/05/17 09:13:38 INFO HttpServer: Starting HTTP Server 17/05/17 09:13:38 INFO Utils: Successfully started service 'HTTP file server' on port 34111. 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar at http://10.24.26.191:34111/jars/spark-assembly-1.6.3-hadoop2.6.0.jar with timestamp 1494983618934 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/htrace-core-3.2.0-incubating.jar at http://10.24.26.191:34111/jars/htrace-core-3.2.0-incubating.jar with timestamp 1494983618938 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-cdh5.8.4.jar at http://10.24.26.191:34111/jars/hbase-client-1.2.0-cdh5.8.4.jar with timestamp 1494983618942 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar at http://10.24.26.191:34111/jars/hbase-common-1.2.0-cdh5.8.4.jar with timestamp 1494983618943 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.0-cdh5.8.4.jar at http://10.24.26.191:34111/jars/hbase-protocol-1.2.0-cdh5.8.4.jar with timestamp 1494983618953 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/metrics-core-2.2.0.jar at http://10.24.26.191:34111/jars/metrics-core-2.2.0.jar with timestamp 1494983618953 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar at http://10.24.26.191:34111/jars/guava-12.0.1.jar with timestamp 1494983618957 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/data/kylin/kylin/lib/kylin-job-2.0.0.jar at http://10.24.26.191:34111/jars/kylin-job-2.0.0.jar with timestamp 1494983618968 17/05/17 09:13:39 INFO Client: Requesting a new application from cluster with 4 NodeManagers 17/05/17 09:13:39 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (18852 MB per container) 17/05/17 09:13:39 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 17/05/17 09:13:39 INFO Client: Setting up container launch context for our AM 17/05/17 09:13:39 INFO Client: Setting up the launch environment for our AM container 17/05/17 09:13:39 INFO Client: Preparing resources for our AM container 17/05/17 09:13:39 INFO YarnSparkHadoopUtil: getting token for namenode: hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568 17/05/17 09:13:40 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1549 for kylin on ha-hdfs:nsha 17/05/17 09:13:40 INFO metastore: Trying to connect to metastore with URI thrift://fonova-ahz-cdh03:9083 17/05/17 09:13:40 INFO metastore: Connected to metastore. 17/05/17 09:13:41 INFO Client: Source and destination file systems are the same. Not copying hdfs://nsha/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar 17/05/17 09:13:41 INFO Client: Uploading resource file:/etc/hbase/conf.cloudera.hbase/hbase-site.xml -> hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568/hbase-site.xml 17/05/17 09:13:41 INFO Client: Uploading resource file:/tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/__spark_conf__5160745721183215779.zip -> hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568/__spark_conf__5160745721183215779.zip 17/05/17 09:13:41 INFO SecurityManager: Changing view acls to: kylin 17/05/17 09:13:41 INFO SecurityManager: Changing modify acls to: kylin 17/05/17 09:13:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kylin); users with modify permissions: Set(kylin) 17/05/17 09:13:41 INFO Client: Submitting application 568 to ResourceManager 17/05/17 09:13:41 INFO YarnClientImpl: Submitted application application_1493867056374_0568 17/05/17 09:13:42 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED) 17/05/17 09:13:42 INFO Client: client token: Token { kind: YARN_CLIENT_TOKEN, service: } diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.users.kylin start time: 1494983621477 final status: UNDEFINED tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568/ user: kylin 17/05/17 09:13:43 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED) 17/05/17 09:13:44 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED) 17/05/17 09:13:45 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED) 17/05/17 09:13:45 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null) 17/05/17 09:13:45 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> fonova-ahz-cdh03,fonova-ahz-cdh04, PROXY_URI_BASES -> http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568,http://fonova-ahz-cdh04:8088/proxy/application_1493867056374_0568), /proxy/application_1493867056374_0568 17/05/17 09:13:45 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 17/05/17 09:13:46 INFO Client: Application report for application_1493867056374_0568 (state: RUNNING) 17/05/17 09:13:46 INFO Client: client token: Token { kind: YARN_CLIENT_TOKEN, service: } diagnostics: N/A ApplicationMaster host: 10.28.30.122 ApplicationMaster RPC port: 0 queue: root.users.kylin start time: 1494983621477 final status: UNDEFINED tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568/ user: kylin 17/05/17 09:13:46 INFO YarnClientSchedulerBackend: Application application_1493867056374_0568 has started running. 17/05/17 09:13:46 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44037. 17/05/17 09:13:46 INFO NettyBlockTransferService: Server created on 44037 17/05/17 09:13:46 INFO BlockManagerMaster: Trying to register BlockManager 17/05/17 09:13:46 INFO BlockManagerMasterEndpoint: Registering block manager 10.24.26.191:44037 with 511.1 MB RAM, BlockManagerId(driver, 10.24.26.191, 44037) 17/05/17 09:13:46 INFO BlockManagerMaster: Registered BlockManager 17/05/17 09:13:46 INFO EventLoggingListener: Logging events to hdfs:///ahz/var/spark-logs/application_1493867056374_0568 17/05/17 09:13:51 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (fonova-ahz-cdh34:36522) with ID 1 17/05/17 09:13:51 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 17/05/17 09:13:51 INFO ClassUtil: Adding path /data/kylin/kylin/conf to class path 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-tools-log4j.properties to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-tools-log4j.properties 17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin-tools-log4j.properties at http://10.24.26.191:34111/files/kylin-tools-log4j.properties with timestamp 1494983631827 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_hive_conf.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_hive_conf.xml 17/05/17 09:13:51 INFO BlockManagerMasterEndpoint: Registering block manager fonova-ahz-cdh34:34665 with 2.7 GB RAM, BlockManagerId(1, fonova-ahz-cdh34, 34665) 17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin_hive_conf.xml at http://10.24.26.191:34111/files/kylin_hive_conf.xml with timestamp 1494983631843 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-server-log4j.properties to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-server-log4j.properties 17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin-server-log4j.properties at http://10.24.26.191:34111/files/kylin-server-log4j.properties with timestamp 1494983631859 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin.properties to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin.properties 17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin.properties at http://10.24.26.191:34111/files/kylin.properties with timestamp 1494983631873 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_job_conf.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf.xml 17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin_job_conf.xml at http://10.24.26.191:34111/files/kylin_job_conf.xml with timestamp 1494983631884 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-kafka-consumer.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-kafka-consumer.xml 17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin-kafka-consumer.xml at http://10.24.26.191:34111/files/kylin-kafka-consumer.xml with timestamp 1494983631898 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_job_conf_inmem.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf_inmem.xml 17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin_job_conf_inmem.xml at http://10.24.26.191:34111/files/kylin_job_conf_inmem.xml with timestamp 1494983631904 17/05/17 09:13:51 INFO KylinConfig: Use KYLIN_CONF=/data/kylin/kylin/conf 17/05/17 09:13:51 INFO KylinConfig: Initialized a new KylinConfig from getInstanceFromEnv : 1577864993 17/05/17 09:13:52 INFO HiveContext: Initializing execution hive, version 1.2.1 17/05/17 09:13:52 INFO ClientWrapper: Inspected Hadoop version: 2.6.0 17/05/17 09:13:52 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 17/05/17 09:13:52 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 17/05/17 09:13:52 INFO ObjectStore: ObjectStore, initialize called 17/05/17 09:13:52 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 17/05/17 09:13:52 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored 17/05/17 09:13:54 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 17/05/17 09:13:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 17/05/17 09:13:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 17/05/17 09:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 17/05/17 09:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 17/05/17 09:13:56 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 17/05/17 09:13:56 INFO ObjectStore: Initialized ObjectStore 17/05/17 09:13:56 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 17/05/17 09:13:57 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException 17/05/17 09:13:57 INFO HiveMetaStore: Added admin role in metastore 17/05/17 09:13:57 INFO HiveMetaStore: Added public role in metastore 17/05/17 09:13:57 INFO HiveMetaStore: No user is added in admin role, since config is empty 17/05/17 09:13:57 INFO SessionState: Created local directory: /tmp/4878224d-23a8-4f7e-adb2-0c261830abe1_resources 17/05/17 09:13:57 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1 17/05/17 09:13:57 INFO SessionState: Created local directory: /tmp/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1 17/05/17 09:13:57 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1/_tmp_space.db 17/05/17 09:13:57 INFO HiveContext: default warehouse location is /user/hive/warehouse 17/05/17 09:13:57 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. 17/05/17 09:13:57 INFO ClientWrapper: Inspected Hadoop version: 2.6.0 17/05/17 09:13:57 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 17/05/17 09:13:58 INFO metastore: Trying to connect to metastore with URI thrift://fonova-ahz-cdh03:9083 17/05/17 09:13:58 INFO metastore: Connected to metastore. 17/05/17 09:13:58 INFO SessionState: Created local directory: /tmp/30eaa1b9-5883-42b8-9b8b-ac844590cac1_resources 17/05/17 09:13:58 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1 17/05/17 09:13:58 INFO SessionState: Created local directory: /tmp/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1 17/05/17 09:13:58 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1/_tmp_space.db 17/05/17 09:13:58 INFO CubeManager: Initializing CubeManager with config kylin_metadata@hbase 17/05/17 09:13:58 INFO ResourceStore: Using metadata url kylin_metadata@hbase for resource store 17/05/17 09:13:58 INFO HBaseConnection: connection is null or closed, creating a new one 17/05/17 09:13:58 INFO RecoverableZooKeeper: Process identifier=hconnection-0x5d01a2eb connecting to ZooKeeper ensemble=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181 17/05/17 09:13:58 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 17/05/17 09:13:58 INFO ZooKeeper: Client environment:host.name=fonova-app-gw01 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.version=1.8.0_91 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_91/jre 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.class.path=/data/kylin/kylin/spark/conf/:/data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar:/data/kylin/kylin/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/kylin/kylin/spark/lib/datanucleus-core-3.2.10.jar:/data/kylin/kylin/spark/lib/datanucleus-rdbms-3.2.9.jar:/data/kylin/kylin/hadoop-conf-dir/ 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.library.path=:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/hbase/bin/../lib/native/Linux-amd64-64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.compiler=<NA> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.name=Linux 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.arch=amd64 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.version=3.10.0-514.6.2.el7.x86_64 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.name=kylin 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.home=/data/kylin 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.dir=/data/kylin/kylin/bin 17/05/17 09:13:58 INFO ZooKeeper: Initiating client connection, connectString=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181 sessionTimeout=60000 watcher=hconnection-0x5d01a2eb0x0, quorum=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181, baseZNode=/ahz/hbase 17/05/17 09:13:58 INFO ClientCnxn: Opening socket connection to server fonova-ahz-cdh04/10.47.54.56:2181. Will not attempt to authenticate using SASL (unknown error) 17/05/17 09:13:58 INFO ClientCnxn: Socket connection established to fonova-ahz-cdh04/10.47.54.56:2181, initiating session 17/05/17 09:13:58 INFO ClientCnxn: Session establishment complete on server fonova-ahz-cdh04/10.47.54.56:2181, sessionid = 0x25bd16992de5eac, negotiated timeout = 60000 17/05/17 09:13:59 INFO deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available 17/05/17 09:13:59 INFO CubeManager: Loading Cube from folder kylin_metadata(key='/cube')@kylin_metadata@hbase 17/05/17 09:14:00 INFO CubeDescManager: Initializing CubeDescManager with config kylin_metadata@hbase 17/05/17 09:14:00 INFO CubeDescManager: Reloading Cube Metadata from folder kylin_metadata(key='/cube_desc')@kylin_metadata@hbase 17/05/17 09:14:00 INFO ProjectManager: Initializing ProjectManager with metadata url kylin_metadata@hbase 17/05/17 09:14:00 INFO MeasureTypeFactory: Checking custom measure types from kylin config 17/05/17 09:14:00 INFO MeasureTypeFactory: registering COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory 17/05/17 09:14:00 INFO MeasureTypeFactory: registering COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory 17/05/17 09:14:00 INFO MeasureTypeFactory: registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory 17/05/17 09:14:00 INFO MeasureTypeFactory: registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory 17/05/17 09:14:00 INFO MeasureTypeFactory: registering EXTENDED_COLUMN(extendedcolumn), class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory 17/05/17 09:14:00 INFO MeasureTypeFactory: registering PERCENTILE(percentile), class org.apache.kylin.measure.percentile.PercentileMeasureType$Factory 17/05/17 09:14:00 INFO CubeDescManager: Loaded 3 Cube(s) 17/05/17 09:14:00 INFO CubeManager: Reloaded cube kylin_sales_cube being CUBE[name=kylin_sales_cube] having 2 segments 17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test being CUBE[name=yuyuan_flow_test] having 1 segments 17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test_spark being CUBE[name=yuyuan_flow_test_spark] having 1 segments 17/05/17 09:14:00 INFO CubeManager: Loaded 3 cubes, fail on 0 cubes 17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 95.7 KB, free 511.0 MB) 17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 9.1 KB, free 511.0 MB) 17/05/17 09:14:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.24.26.191:44037 (size: 9.1 KB, free: 511.1 MB) 17/05/17 09:14:00 INFO SparkContext: Created broadcast 0 from broadcast at SparkCubingByLayer.java:166 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 33.5 KB, free 511.0 MB) 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.5 KB, free 511.0 MB) 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.24.26.191:44037 (size: 3.5 KB, free: 511.1 MB) 17/05/17 09:14:01 INFO SparkContext: Created broadcast 1 from broadcast at SparkCubingByLayer.java:167 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/LEAF_CATEG_ID/3b10e09c-0ee7-48db-9f83-1f21552e0113.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/META_CATEG_NAME/579dd544-fbca-4595-a13d-b76acec741ff.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/CATEG_LVL2_NAME/669d36f7-bb09-4a0e-8239-84efd806ec0d.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/CATEG_LVL3_NAME/a89ec9e5-edef-4c1a-b50f-09a6f768b3f2.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_BUYER_LEVEL/61bd0c4f-bd22-4f38-99ce-1ffe5fa779d3.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_SELLER_LEVEL/e4f67b28-afb9-4dca-9989-c1393424a0da.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_COUNTRY/97c0a5cb-31d1-47fc-80ee-b9b07379ea1a.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_COUNTRY/NAME/9b410ddc-9004-4253-b39c-630954c0c4d0.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/LSTG_FORMAT_NAME/77b68501-3dce-4f7d-b860-652d2a8c2daa.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/SITE_ID/caf25161-7de0-4f8c-8a4c-1a91c92977fc.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_USER_ID/5f3ef593-0157-46d6-a700-e609f4ec7733.dict 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_REGION/6e20015c-7249-4436-807e-b260361279dd.dict 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 96.1 KB, free 510.9 MB) 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.2 KB, free 510.9 MB) 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.24.26.191:44037 (size: 9.2 KB, free: 511.1 MB) 17/05/17 09:14:01 INFO SparkContext: Created broadcast 2 from broadcast at SparkCubingByLayer.java:170 17/05/17 09:14:01 INFO SparkCubingByLayer: All measure are normal (agg on all cuboids) ? : true 17/05/17 09:14:01 INFO deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 616.1 KB, free 510.3 MB) 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 46.3 KB, free 510.2 MB) 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.24.26.191:44037 (size: 46.3 KB, free: 511.1 MB) 17/05/17 09:14:01 INFO SparkContext: Created broadcast 3 from javaRDD at SparkCubingByLayer.java:193 17/05/17 09:14:01 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1550 for kylin on ha-hdfs:nsha 17/05/17 09:14:01 INFO TokenCache: Got dt for hdfs://nsha; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN token 1550 for kylin) 17/05/17 09:14:01 INFO FileInputFormat: Total input paths to process : 1 17/05/17 09:14:01 INFO SparkCubingByLayer: encodedBaseRDD partition number: 1 17/05/17 09:14:01 INFO CubeStatsReader: Estimating size for layer 0, all cuboids are 262143, total size is 0.24426913261413574 17/05/17 09:14:01 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 17/05/17 09:14:02 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1551 for kylin on ha-hdfs:nsha 17/05/17 09:14:02 INFO TokenCache: Got dt for hdfs://nsha; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN token 1551 for kylin) 17/05/17 09:14:02 INFO SparkContext: Starting job: saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288 17/05/17 09:14:02 INFO DAGScheduler: Registering RDD 4 (mapToPair at SparkCubingByLayer.java:193) 17/05/17 09:14:02 INFO DAGScheduler: Got job 0 (saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288) with 1 output partitions 17/05/17 09:14:02 INFO DAGScheduler: Final stage: ResultStage 1 (saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288) 17/05/17 09:14:02 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 17/05/17 09:14:02 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0) 17/05/17 09:14:02 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[4] at mapToPair at SparkCubingByLayer.java:193), which has no missing parents 17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 59.9 KB, free 510.2 MB) 17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 25.6 KB, free 510.2 MB) 17/05/17 09:14:02 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.24.26.191:44037 (size: 25.6 KB, free: 511.0 MB) 17/05/17 09:14:02 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006 17/05/17 09:14:02 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[4] at mapToPair at SparkCubingByLayer.java:193) 17/05/17 09:14:02 INFO YarnScheduler: Adding task set 0.0 with 1 tasks 17/05/17 09:14:02 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes) 17/05/17 09:14:06 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on fonova-ahz-cdh34:34665 (size: 25.6 KB, free: 2.7 GB) 17/05/17 09:14:07 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on fonova-ahz-cdh34:34665 (size: 46.3 KB, free: 2.7 GB) 17/05/17 09:14:57 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370) at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298) at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109) at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81) at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109) at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119) at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467) at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48) at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 17/05/17 09:14:57 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes) 17/05/17 09:15:45 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370) at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298) at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109) at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81) at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109) at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119) at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467) at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48) at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 17/05/17 09:15:45 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 2, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes) 17/05/17 09:16:33 WARN TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370) at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298) at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109) at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81) at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109) at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119) at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467) at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48) at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 17/05/17 09:16:33 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes) 17/05/17 09:17:22 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370) at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298) at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109) at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81) at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109) at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119) at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467) at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48) at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 17/05/17 09:17:22 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job 17/05/17 09:17:22 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 17/05/17 09:17:22 INFO YarnScheduler: Cancelling stage 0 17/05/17 09:17:22 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at SparkCubingByLayer.java:193) failed in 199.860 s 17/05/17 09:17:22 INFO DAGScheduler: Job 0 failed: saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288, took 200.053479 s Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370) at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298) at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109) at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81) at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109) at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119) at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467) at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48) at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1144) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1074) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:994) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:985) at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(JavaPairRDD.scala:800) at org.apache.kylin.engine.spark.SparkCubingByLayer.saveToHDFS(SparkCubingByLayer.java:288) at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:257) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) ... 10 more Caused by: java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370) at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298) at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109) at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81) at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109) at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119) at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467) at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48) at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x25bd16992de5eac 17/05/17 09:17:22 INFO SparkContext: Invoking stop() from shutdown hook 17/05/17 09:17:22 INFO ZooKeeper: Session: 0x25bd16992de5eac closed 17/05/17 09:17:22 INFO ClientCnxn: EventThread shut down 17/05/17 09:17:22 INFO SparkUI: Stopped Spark web UI at http://10.24.26.191:4040 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Shutting down all executors 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Interrupting monitor thread 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Asking each executor to shut down 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Stopped 17/05/17 09:17:22 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/05/17 09:17:22 INFO MemoryStore: MemoryStore cleared 17/05/17 09:17:22 INFO BlockManager: BlockManager stopped 17/05/17 09:17:22 INFO BlockManagerMaster: BlockManagerMaster stopped 17/05/17 09:17:22 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/05/17 09:17:22 INFO SparkContext: Successfully stopped SparkContext 17/05/17 09:17:22 INFO ShutdownHookManager: Shutdown hook called 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-3e8cd18d-d4d0-4180-9c29-18d5814b8439 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd-7d7509ef-786d-4fa4-a75e-bb3cb1015c77 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc Thanks, Zhuoran 发件人: ShaoFeng Shi [mailto:[email protected]] 发送时间: 2017年5月17日 13:36 收件人: dev 抄送: [email protected] 主题: Re: 答复: A problem in cube building with SPARK The mailing list filters attachement. Could you please paste it directly, or share it with gist? 2017-05-17 12:17 GMT+08:00 吕卓然 <[email protected]<mailto:[email protected]>>: Hi Shaofeng, I’ve attached the error log in the attachment.. 发件人: ShaoFeng Shi [mailto:[email protected]<mailto:[email protected]>] 发送时间: 2017年5月17日 10:38 收件人: user 抄送: [email protected]<mailto:[email protected]> 主题: Re: A problem in cube building with SPARK Hi zhuoran, is there any more messages before this error? This error is not the root cause. 2017-05-17 10:27 GMT+08:00 吕卓然 <[email protected]<mailto:[email protected]>>: Hi all, Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use MapReduce engine. However, when I try to use spark engine to build cube, it fails at step 7: Build Cube with Spark. Here is the log info: 17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99) at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110) at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370) at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298) at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109) at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81) at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109) at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119) at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467) at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48) at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205) at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Any suggestions would help. Thanks, Zhuoran -- Best regards, Shaofeng Shi 史少锋 -- Best regards, Shaofeng Shi 史少锋
