Hi Zuoran, did you check Spark history server to see what's wrong in the executors? Is your kylin running it on a cluster and some nodes couldn't connect with hbase, or the "hbase-site.xml" uses address that couldn't visit from external nodes? Please do a check.
2017-05-17 13:40 GMT+08:00 吕卓然 <[email protected]>: > Sure, no problem. > > Here it is. > > OS command error exit with 1 -- export > HADOOP_CONF_DIR=/data/kylin/kylin/hadoop-conf-dir > && /data/kylin/kylin/spark/bin/spark-submit --class > org.apache.kylin.common.util.SparkEntry --conf > spark.executor.instances=1 --conf spark.yarn.jar=hdfs://nsha/ > kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar --conf > spark.yarn.queue=default --conf > spark.history.fs.logDirectory=hdfs:///ahz/var/spark-logs > --conf spark.master=yarn --conf spark.executor.memory=4G --conf > spark.eventLog.enabled=true --conf > spark.eventLog.dir=hdfs:///ahz/var/spark-logs > --conf spark.executor.cores=2 --conf spark.submit.deployMode=cluster > --files /etc/hbase/conf.cloudera.hbase/hbase-site.xml --jars > /data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2. > 6.0.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/ > jars/htrace-core-3.2.0-incubating.jar,/opt/cloudera/ > parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0- > cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4. > p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar,/opt/cloudera/ > parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2. > 0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4. > p0.5/jars/metrics-core-2.2.0.jar,/opt/cloudera/parcels/CDH- > 5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar, > /data/kylin/kylin/lib/kylin-job-2.0.0.jar > -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable > default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 > -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07- > 4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId > 2ffd651b-cdd9-469f-a957-67aee260abd6 > -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube > SparkEntry args:-className org.apache.kylin.engine.spark.SparkCubingByLayer > -hiveTable > default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 > -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07- > 4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId > 2ffd651b-cdd9-469f-a957-67aee260abd6 > -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube > Abstract Application args:-hiveTable default.kylin_intermediate_ > kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output > hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07- > 4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId > 2ffd651b-cdd9-469f-a957-67aee260abd6 > -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube > Using Spark's default log4j profile: org/apache/spark/log4j- > defaults.properties > 17/05/17 09:13:36 INFO SparkContext: Running Spark version 1.6.3 > 17/05/17 09:13:37 INFO SecurityManager: Changing view acls to: kylin > 17/05/17 09:13:37 INFO SecurityManager: Changing modify acls to: kylin > 17/05/17 09:13:37 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(kylin); users > with modify permissions: Set(kylin) > 17/05/17 09:13:37 INFO Utils: Successfully started service 'sparkDriver' > on port 42715. > 17/05/17 09:13:37 INFO Slf4jLogger: Slf4jLogger started > 17/05/17 09:13:37 INFO Remoting: Starting remoting > 17/05/17 09:13:38 INFO Utils: Successfully started service > 'sparkDriverActorSystem' on port 39514. > 17/05/17 09:13:38 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://[email protected]:39514] > 17/05/17 09:13:38 INFO SparkEnv: Registering MapOutputTracker > 17/05/17 09:13:38 INFO SparkEnv: Registering BlockManagerMaster > 17/05/17 09:13:38 INFO DiskBlockManager: Created local directory at > /tmp/blockmgr-6a57441d-d256-4ade-acf0-af6985732211 > 17/05/17 09:13:38 INFO MemoryStore: MemoryStore started with capacity > 511.1 MB > 17/05/17 09:13:38 INFO SparkEnv: Registering OutputCommitCoordinator > 17/05/17 09:13:38 INFO Utils: Successfully started service 'SparkUI' on > port 4040. > 17/05/17 09:13:38 INFO SparkUI: Started SparkUI at > http://10.24.26.191:4040 > 17/05/17 09:13:38 INFO HttpFileServer: HTTP File server directory is > /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd- > 7d7509ef-786d-4fa4-a75e-bb3cb1015c77 > 17/05/17 09:13:38 INFO HttpServer: Starting HTTP Server > 17/05/17 09:13:38 INFO Utils: Successfully started service 'HTTP file > server' on port 34111. > 17/05/17 09:13:38 INFO SparkContext: Added JAR > file:/data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar at > http://10.24.26.191:34111/jars/spark-assembly-1.6.3-hadoop2.6.0.jar with > timestamp 1494983618934 > 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/ > CDH-5.8.4-1.cdh5.8.4.p0.5/jars/htrace-core-3.2.0-incubating.jar at > http://10.24.26.191:34111/jars/htrace-core-3.2.0-incubating.jar with > timestamp 1494983618938 > 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/ > CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-cdh5.8.4.jar at > http://10.24.26.191:34111/jars/hbase-client-1.2.0-cdh5.8.4.jar with > timestamp 1494983618942 > 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/ > CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar at > http://10.24.26.191:34111/jars/hbase-common-1.2.0-cdh5.8.4.jar with > timestamp 1494983618943 > 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/ > CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.0-cdh5.8.4.jar at > http://10.24.26.191:34111/jars/hbase-protocol-1.2.0-cdh5.8.4.jar with > timestamp 1494983618953 > 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/ > CDH-5.8.4-1.cdh5.8.4.p0.5/jars/metrics-core-2.2.0.jar at > http://10.24.26.191:34111/jars/metrics-core-2.2.0.jar with timestamp > 1494983618953 > 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/ > CDH-5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar at > http://10.24.26.191:34111/jars/guava-12.0.1.jar with timestamp > 1494983618957 > 17/05/17 09:13:38 INFO SparkContext: Added JAR > file:/data/kylin/kylin/lib/kylin-job-2.0.0.jar > at http://10.24.26.191:34111/jars/kylin-job-2.0.0.jar with timestamp > 1494983618968 > 17/05/17 09:13:39 INFO Client: Requesting a new application from cluster > with 4 NodeManagers > 17/05/17 09:13:39 INFO Client: Verifying our application has not requested > more than the maximum memory capability of the cluster (18852 MB per > container) > 17/05/17 09:13:39 INFO Client: Will allocate AM container, with 896 MB > memory including 384 MB overhead > 17/05/17 09:13:39 INFO Client: Setting up container launch context for our > AM > 17/05/17 09:13:39 INFO Client: Setting up the launch environment for our > AM container > 17/05/17 09:13:39 INFO Client: Preparing resources for our AM container > 17/05/17 09:13:39 INFO YarnSparkHadoopUtil: getting token for namenode: > hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568 > 17/05/17 09:13:40 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1549 > for kylin on ha-hdfs:nsha > 17/05/17 09:13:40 INFO metastore: Trying to connect to metastore with URI > thrift://fonova-ahz-cdh03:9083 > 17/05/17 09:13:40 INFO metastore: Connected to metastore. > 17/05/17 09:13:41 INFO Client: Source and destination file systems are the > same. Not copying hdfs://nsha/kylin/spark/spark- > assembly-1.6.3-hadoop2.6.0.jar > 17/05/17 09:13:41 INFO Client: Uploading resource > file:/etc/hbase/conf.cloudera.hbase/hbase-site.xml -> > hdfs://nsha/user/kylin/.sparkStaging/application_ > 1493867056374_0568/hbase-site.xml > 17/05/17 09:13:41 INFO Client: Uploading resource > file:/tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/__spark_conf__5160745721183215779.zip > -> hdfs://nsha/user/kylin/.sparkStaging/application_ > 1493867056374_0568/__spark_conf__5160745721183215779.zip > 17/05/17 09:13:41 INFO SecurityManager: Changing view acls to: kylin > 17/05/17 09:13:41 INFO SecurityManager: Changing modify acls to: kylin > 17/05/17 09:13:41 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(kylin); users > with modify permissions: Set(kylin) > 17/05/17 09:13:41 INFO Client: Submitting application 568 to > ResourceManager > 17/05/17 09:13:41 INFO YarnClientImpl: Submitted application > application_1493867056374_0568 > 17/05/17 09:13:42 INFO Client: Application report for > application_1493867056374_0568 (state: ACCEPTED) > 17/05/17 09:13:42 INFO Client: > client token: Token { kind: YARN_CLIENT_TOKEN, service: } > diagnostics: N/A > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: root.users.kylin > start time: 1494983621477 > final status: UNDEFINED > tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_ > 1493867056374_0568/ > user: kylin > 17/05/17 09:13:43 INFO Client: Application report for > application_1493867056374_0568 (state: ACCEPTED) > 17/05/17 09:13:44 INFO Client: Application report for > application_1493867056374_0568 (state: ACCEPTED) > 17/05/17 09:13:45 INFO Client: Application report for > application_1493867056374_0568 (state: ACCEPTED) > 17/05/17 09:13:45 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster registered as NettyRpcEndpointRef(null) > 17/05/17 09:13:45 INFO YarnClientSchedulerBackend: Add WebUI Filter. > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, > Map(PROXY_HOSTS -> fonova-ahz-cdh03,fonova-ahz-cdh04, PROXY_URI_BASES -> > http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568,http:// > fonova-ahz-cdh04:8088/proxy/application_1493867056374_0568), > /proxy/application_1493867056374_0568 > 17/05/17 09:13:45 INFO JettyUtils: Adding filter: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter > 17/05/17 09:13:46 INFO Client: Application report for > application_1493867056374_0568 (state: RUNNING) > 17/05/17 09:13:46 INFO Client: > client token: Token { kind: YARN_CLIENT_TOKEN, service: } > diagnostics: N/A > ApplicationMaster host: 10.28.30.122 > ApplicationMaster RPC port: 0 > queue: root.users.kylin > start time: 1494983621477 > final status: UNDEFINED > tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_ > 1493867056374_0568/ > user: kylin > 17/05/17 09:13:46 INFO YarnClientSchedulerBackend: Application > application_1493867056374_0568 has started running. > 17/05/17 09:13:46 INFO Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44037. > 17/05/17 09:13:46 INFO NettyBlockTransferService: Server created on 44037 > 17/05/17 09:13:46 INFO BlockManagerMaster: Trying to register BlockManager > 17/05/17 09:13:46 INFO BlockManagerMasterEndpoint: Registering block > manager 10.24.26.191:44037 with 511.1 MB RAM, BlockManagerId(driver, > 10.24.26.191, 44037) > 17/05/17 09:13:46 INFO BlockManagerMaster: Registered BlockManager > 17/05/17 09:13:46 INFO EventLoggingListener: Logging events to > hdfs:///ahz/var/spark-logs/application_1493867056374_0568 > 17/05/17 09:13:51 INFO YarnClientSchedulerBackend: Registered executor > NettyRpcEndpointRef(null) (fonova-ahz-cdh34:36522) with ID 1 > 17/05/17 09:13:51 INFO YarnClientSchedulerBackend: SchedulerBackend is > ready for scheduling beginning after reached minRegisteredResourcesRatio: > 0.8 > 17/05/17 09:13:51 INFO ClassUtil: Adding path /data/kylin/kylin/conf to > class path > 17/05/17 09:13:51 INFO Utils: Copying > /data/kylin/kylin/conf/kylin-tools-log4j.properties > to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles- > c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-tools-log4j.properties > 17/05/17 09:13:51 INFO SparkContext: Added file > /data/kylin/kylin/conf/kylin-tools-log4j.properties at > http://10.24.26.191:34111/files/kylin-tools-log4j.properties with > timestamp 1494983631827 > 17/05/17 09:13:51 INFO Utils: Copying > /data/kylin/kylin/conf/kylin_hive_conf.xml > to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles- > c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_hive_conf.xml > 17/05/17 09:13:51 INFO BlockManagerMasterEndpoint: Registering block > manager fonova-ahz-cdh34:34665 with 2.7 GB RAM, BlockManagerId(1, > fonova-ahz-cdh34, 34665) > 17/05/17 09:13:51 INFO SparkContext: Added file > /data/kylin/kylin/conf/kylin_hive_conf.xml at http://10.24.26.191:34111/ > files/kylin_hive_conf.xml with timestamp 1494983631843 > 17/05/17 09:13:51 INFO Utils: Copying > /data/kylin/kylin/conf/kylin-server-log4j.properties > to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles- > c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-server-log4j.properties > 17/05/17 09:13:51 INFO SparkContext: Added file > /data/kylin/kylin/conf/kylin-server-log4j.properties at > http://10.24.26.191:34111/files/kylin-server-log4j.properties with > timestamp 1494983631859 > 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin.properties > to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles- > c691c4b5-c867-4678-9b68-feaf9fce4277/kylin.properties > 17/05/17 09:13:51 INFO SparkContext: Added file > /data/kylin/kylin/conf/kylin.properties at http://10.24.26.191:34111/ > files/kylin.properties with timestamp 1494983631873 > 17/05/17 09:13:51 INFO Utils: Copying > /data/kylin/kylin/conf/kylin_job_conf.xml > to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles- > c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf.xml > 17/05/17 09:13:51 INFO SparkContext: Added file > /data/kylin/kylin/conf/kylin_job_conf.xml at http://10.24.26.191:34111/ > files/kylin_job_conf.xml with timestamp 1494983631884 > 17/05/17 09:13:51 INFO Utils: Copying > /data/kylin/kylin/conf/kylin-kafka-consumer.xml > to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles- > c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-kafka-consumer.xml > 17/05/17 09:13:51 INFO SparkContext: Added file > /data/kylin/kylin/conf/kylin-kafka-consumer.xml at > http://10.24.26.191:34111/files/kylin-kafka-consumer.xml with timestamp > 1494983631898 > 17/05/17 09:13:51 INFO Utils: Copying > /data/kylin/kylin/conf/kylin_job_conf_inmem.xml > to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles- > c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf_inmem.xml > 17/05/17 09:13:51 INFO SparkContext: Added file > /data/kylin/kylin/conf/kylin_job_conf_inmem.xml at > http://10.24.26.191:34111/files/kylin_job_conf_inmem.xml with timestamp > 1494983631904 > 17/05/17 09:13:51 INFO KylinConfig: Use KYLIN_CONF=/data/kylin/kylin/conf > 17/05/17 09:13:51 INFO KylinConfig: Initialized a new KylinConfig from > getInstanceFromEnv : 1577864993 > 17/05/17 09:13:52 INFO HiveContext: Initializing execution hive, version > 1.2.1 > 17/05/17 09:13:52 INFO ClientWrapper: Inspected Hadoop version: 2.6.0 > 17/05/17 09:13:52 INFO ClientWrapper: Loaded > org.apache.hadoop.hive.shims.Hadoop23Shims > for Hadoop version 2.6.0 > 17/05/17 09:13:52 INFO HiveMetaStore: 0: Opening raw store with > implemenation class:org.apache.hadoop.hive.metastore.ObjectStore > 17/05/17 09:13:52 INFO ObjectStore: ObjectStore, initialize called > 17/05/17 09:13:52 INFO Persistence: Property > hive.metastore.integral.jdo.pushdown > unknown - will be ignored > 17/05/17 09:13:52 INFO Persistence: Property datanucleus.cache.level2 > unknown - will be ignored > 17/05/17 09:13:54 INFO ObjectStore: Setting MetaStore object pin classes > with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo, > Partition,Database,Type,FieldSchema,Order" > 17/05/17 09:13:55 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" > is tagged as "embedded-only" so does not have its own datastore table. > 17/05/17 09:13:55 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" > is tagged as "embedded-only" so does not have its own datastore table. > 17/05/17 09:13:56 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" > is tagged as "embedded-only" so does not have its own datastore table. > 17/05/17 09:13:56 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" > is tagged as "embedded-only" so does not have its own datastore table. > 17/05/17 09:13:56 INFO MetaStoreDirectSql: Using direct SQL, underlying DB > is DERBY > 17/05/17 09:13:56 INFO ObjectStore: Initialized ObjectStore > 17/05/17 09:13:56 WARN ObjectStore: Version information not found in > metastore. hive.metastore.schema.verification is not enabled so recording > the schema version 1.2.0 > 17/05/17 09:13:57 WARN ObjectStore: Failed to get database default, > returning NoSuchObjectException > 17/05/17 09:13:57 INFO HiveMetaStore: Added admin role in metastore > 17/05/17 09:13:57 INFO HiveMetaStore: Added public role in metastore > 17/05/17 09:13:57 INFO HiveMetaStore: No user is added in admin role, > since config is empty > 17/05/17 09:13:57 INFO SessionState: Created local directory: > /tmp/4878224d-23a8-4f7e-adb2-0c261830abe1_resources > 17/05/17 09:13:57 INFO SessionState: Created HDFS directory: > /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1 > 17/05/17 09:13:57 INFO SessionState: Created local directory: > /tmp/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1 > 17/05/17 09:13:57 INFO SessionState: Created HDFS directory: > /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1/_tmp_space.db > 17/05/17 09:13:57 INFO HiveContext: default warehouse location is > /user/hive/warehouse > 17/05/17 09:13:57 INFO HiveContext: Initializing HiveMetastoreConnection > version 1.2.1 using Spark classes. > 17/05/17 09:13:57 INFO ClientWrapper: Inspected Hadoop version: 2.6.0 > 17/05/17 09:13:57 INFO ClientWrapper: Loaded > org.apache.hadoop.hive.shims.Hadoop23Shims > for Hadoop version 2.6.0 > 17/05/17 09:13:58 INFO metastore: Trying to connect to metastore with URI > thrift://fonova-ahz-cdh03:9083 > 17/05/17 09:13:58 INFO metastore: Connected to metastore. > 17/05/17 09:13:58 INFO SessionState: Created local directory: > /tmp/30eaa1b9-5883-42b8-9b8b-ac844590cac1_resources > 17/05/17 09:13:58 INFO SessionState: Created HDFS directory: > /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1 > 17/05/17 09:13:58 INFO SessionState: Created local directory: > /tmp/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1 > 17/05/17 09:13:58 INFO SessionState: Created HDFS directory: > /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1/_tmp_space.db > 17/05/17 09:13:58 INFO CubeManager: Initializing CubeManager with config > kylin_metadata@hbase > 17/05/17 09:13:58 INFO ResourceStore: Using metadata url > kylin_metadata@hbase for resource store > 17/05/17 09:13:58 INFO HBaseConnection: connection is null or closed, > creating a new one > 17/05/17 09:13:58 INFO RecoverableZooKeeper: Process > identifier=hconnection-0x5d01a2eb connecting to ZooKeeper > ensemble=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181 > 17/05/17 09:13:58 INFO ZooKeeper: Client > environment:zookeeper.version=3.4.6-1569965, > built on 02/20/2014 09:09 GMT > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:host.name=fonova- > app-gw01 > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.version=1.8.0_91 > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.vendor=Oracle > Corporation > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.home=/usr/ > java/jdk1.8.0_91/jre > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.class.path=/ > data/kylin/kylin/spark/conf/:/data/kylin/kylin/spark/lib/ > spark-assembly-1.6.3-hadoop2.6.0.jar:/data/kylin/kylin/ > spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/kylin/kylin/ > spark/lib/datanucleus-core-3.2.10.jar:/data/kylin/kylin/ > spark/lib/datanucleus-rdbms-3.2.9.jar:/data/kylin/kylin/hadoop-conf-dir/ > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.library.path= > :/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/ > hadoop/lib/native:/opt/cloudera/parcels/CDH-5.8.4-1. > cdh5.8.4.p0.5/lib/hbase/bin/../lib/native/Linux-amd64-64:/ > usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.compiler=<NA> > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.name=Linux > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.arch=amd64 > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.version=3.10.0- > 514.6.2.el7.x86_64 > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.name=kylin > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.home=/data/kylin > 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.dir=/data/ > kylin/kylin/bin > 17/05/17 09:13:58 INFO ZooKeeper: Initiating client connection, > connectString=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181 > sessionTimeout=60000 watcher=hconnection-0x5d01a2eb0x0, > quorum=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181, > baseZNode=/ahz/hbase > 17/05/17 09:13:58 INFO ClientCnxn: Opening socket connection to server > fonova-ahz-cdh04/10.47.54.56:2181. Will not attempt to authenticate using > SASL (unknown error) > 17/05/17 09:13:58 INFO ClientCnxn: Socket connection established to > fonova-ahz-cdh04/10.47.54.56:2181, initiating session > 17/05/17 09:13:58 INFO ClientCnxn: Session establishment complete on > server fonova-ahz-cdh04/10.47.54.56:2181, sessionid = 0x25bd16992de5eac, > negotiated timeout = 60000 > 17/05/17 09:13:59 INFO deprecation: hadoop.native.lib is deprecated. > Instead, use io.native.lib.available > 17/05/17 09:13:59 INFO CubeManager: Loading Cube from folder > kylin_metadata(key='/cube')@kylin_metadata@hbase > 17/05/17 09:14:00 INFO CubeDescManager: Initializing CubeDescManager with > config kylin_metadata@hbase > 17/05/17 09:14:00 INFO CubeDescManager: Reloading Cube Metadata from > folder kylin_metadata(key='/cube_desc')@kylin_metadata@hbase > 17/05/17 09:14:00 INFO ProjectManager: Initializing ProjectManager with > metadata url kylin_metadata@hbase > 17/05/17 09:14:00 INFO MeasureTypeFactory: Checking custom measure types > from kylin config > 17/05/17 09:14:00 INFO MeasureTypeFactory: registering > COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc. > HLLCMeasureType$Factory > 17/05/17 09:14:00 INFO MeasureTypeFactory: registering > COUNT_DISTINCT(bitmap), class org.apache.kylin.measure. > bitmap.BitmapMeasureType$Factory > 17/05/17 09:14:00 INFO MeasureTypeFactory: registering TOP_N(topn), class > org.apache.kylin.measure.topn.TopNMeasureType$Factory > 17/05/17 09:14:00 INFO MeasureTypeFactory: registering RAW(raw), class > org.apache.kylin.measure.raw.RawMeasureType$Factory > 17/05/17 09:14:00 INFO MeasureTypeFactory: registering > EXTENDED_COLUMN(extendedcolumn), > class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$ > Factory > 17/05/17 09:14:00 INFO MeasureTypeFactory: registering > PERCENTILE(percentile), class org.apache.kylin.measure.percentile. > PercentileMeasureType$Factory > 17/05/17 09:14:00 INFO CubeDescManager: Loaded 3 Cube(s) > 17/05/17 09:14:00 INFO CubeManager: Reloaded cube kylin_sales_cube being > CUBE[name=kylin_sales_cube] having 2 segments > 17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test being > CUBE[name=yuyuan_flow_test] having 1 segments > 17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test_spark > being CUBE[name=yuyuan_flow_test_spark] having 1 segments > 17/05/17 09:14:00 INFO CubeManager: Loaded 3 cubes, fail on 0 cubes > 17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0 stored as values in > memory (estimated size 95.7 KB, free 511.0 MB) > 17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0_piece0 stored as > bytes in memory (estimated size 9.1 KB, free 511.0 MB) > 17/05/17 09:14:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in > memory on 10.24.26.191:44037 (size: 9.1 KB, free: 511.1 MB) > 17/05/17 09:14:00 INFO SparkContext: Created broadcast 0 from broadcast at > SparkCubingByLayer.java:166 > 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1 stored as values in > memory (estimated size 33.5 KB, free 511.0 MB) > 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1_piece0 stored as > bytes in memory (estimated size 3.5 KB, free 511.0 MB) > 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_1_piece0 in > memory on 10.24.26.191:44037 (size: 3.5 KB, free: 511.1 MB) > 17/05/17 09:14:01 INFO SparkContext: Created broadcast 1 from broadcast at > SparkCubingByLayer.java:167 > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_ > GROUPINGS/LEAF_CATEG_ID/3b10e09c-0ee7-48db-9f83-1f21552e0113.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_ > GROUPINGS/META_CATEG_NAME/579dd544-fbca-4595-a13d-b76acec741ff.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_ > GROUPINGS/CATEG_LVL2_NAME/669d36f7-bb09-4a0e-8239-84efd806ec0d.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_ > GROUPINGS/CATEG_LVL3_NAME/a89ec9e5-edef-4c1a-b50f-09a6f768b3f2.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ > ACCOUNT_BUYER_LEVEL/61bd0c4f-bd22-4f38-99ce-1ffe5fa779d3.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ > ACCOUNT_SELLER_LEVEL/e4f67b28-afb9-4dca-9989-c1393424a0da.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ > ACCOUNT_COUNTRY/97c0a5cb-31d1-47fc-80ee-b9b07379ea1a.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_COUNTRY/ > NAME/9b410ddc-9004-4253-b39c-630954c0c4d0.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/ > LSTG_FORMAT_NAME/77b68501-3dce-4f7d-b860-652d2a8c2daa.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_ > GROUPINGS/SITE_ID/caf25161-7de0-4f8c-8a4c-1a91c92977fc.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_ > USER_ID/5f3ef593-0157-46d6-a700-e609f4ec7733.dict > 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) > loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_ > REGION/6e20015c-7249-4436-807e-b260361279dd.dict > 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2 stored as values in > memory (estimated size 96.1 KB, free 510.9 MB) > 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2_piece0 stored as > bytes in memory (estimated size 9.2 KB, free 510.9 MB) > 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_2_piece0 in > memory on 10.24.26.191:44037 (size: 9.2 KB, free: 511.1 MB) > 17/05/17 09:14:01 INFO SparkContext: Created broadcast 2 from broadcast at > SparkCubingByLayer.java:170 > 17/05/17 09:14:01 INFO SparkCubingByLayer: All measure are normal (agg on > all cuboids) ? : true > 17/05/17 09:14:01 INFO deprecation: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3 stored as values in > memory (estimated size 616.1 KB, free 510.3 MB) > 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3_piece0 stored as > bytes in memory (estimated size 46.3 KB, free 510.2 MB) > 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_3_piece0 in > memory on 10.24.26.191:44037 (size: 46.3 KB, free: 511.1 MB) > 17/05/17 09:14:01 INFO SparkContext: Created broadcast 3 from javaRDD at > SparkCubingByLayer.java:193 > 17/05/17 09:14:01 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1550 > for kylin on ha-hdfs:nsha > 17/05/17 09:14:01 INFO TokenCache: Got dt for hdfs://nsha; Kind: > HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN > token 1550 for kylin) > 17/05/17 09:14:01 INFO FileInputFormat: Total input paths to process : 1 > 17/05/17 09:14:01 INFO SparkCubingByLayer: encodedBaseRDD partition > number: 1 > 17/05/17 09:14:01 INFO CubeStatsReader: Estimating size for layer 0, all > cuboids are 262143, total size is 0.24426913261413574 > 17/05/17 09:14:01 INFO deprecation: mapred.output.dir is deprecated. > Instead, use mapreduce.output.fileoutputformat.outputdir > 17/05/17 09:14:02 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1551 > for kylin on ha-hdfs:nsha > 17/05/17 09:14:02 INFO TokenCache: Got dt for hdfs://nsha; Kind: > HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN > token 1551 for kylin) > 17/05/17 09:14:02 INFO SparkContext: Starting job: saveAsNewAPIHadoopFile > at SparkCubingByLayer.java:288 > 17/05/17 09:14:02 INFO DAGScheduler: Registering RDD 4 (mapToPair at > SparkCubingByLayer.java:193) > 17/05/17 09:14:02 INFO DAGScheduler: Got job 0 (saveAsNewAPIHadoopFile at > SparkCubingByLayer.java:288) with 1 output partitions > 17/05/17 09:14:02 INFO DAGScheduler: Final stage: ResultStage 1 > (saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288) > 17/05/17 09:14:02 INFO DAGScheduler: Parents of final stage: > List(ShuffleMapStage 0) > 17/05/17 09:14:02 INFO DAGScheduler: Missing parents: List(ShuffleMapStage > 0) > 17/05/17 09:14:02 INFO DAGScheduler: Submitting ShuffleMapStage 0 > (MapPartitionsRDD[4] at mapToPair at SparkCubingByLayer.java:193), which > has no missing parents > 17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4 stored as values in > memory (estimated size 59.9 KB, free 510.2 MB) > 17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4_piece0 stored as > bytes in memory (estimated size 25.6 KB, free 510.2 MB) > 17/05/17 09:14:02 INFO BlockManagerInfo: Added broadcast_4_piece0 in > memory on 10.24.26.191:44037 (size: 25.6 KB, free: 511.0 MB) > 17/05/17 09:14:02 INFO SparkContext: Created broadcast 4 from broadcast at > DAGScheduler.scala:1006 > 17/05/17 09:14:02 INFO DAGScheduler: Submitting 1 missing tasks from > ShuffleMapStage 0 (MapPartitionsRDD[4] at mapToPair at > SparkCubingByLayer.java:193) > 17/05/17 09:14:02 INFO YarnScheduler: Adding task set 0.0 with 1 tasks > 17/05/17 09:14:02 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID > 0, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes) > 17/05/17 09:14:06 INFO BlockManagerInfo: Added broadcast_4_piece0 in > memory on fonova-ahz-cdh34:34665 (size: 25.6 KB, free: 2.7 GB) > 17/05/17 09:14:07 INFO BlockManagerInfo: Added broadcast_3_piece0 in > memory on fonova-ahz-cdh34:34665 (size: 46.3 KB, free: 2.7 GB) > 17/05/17 09:14:57 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, > fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > at org.apache.kylin.common.persistence.ResourceStore. > createResourceStore(ResourceStore.java:99) > at org.apache.kylin.common.persistence.ResourceStore. > getStore(ResourceStore.java:110) > at org.apache.kylin.cube.CubeDescManager.getStore( > CubeDescManager.java:370) > at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc( > CubeDescManager.java:298) > at org.apache.kylin.cube.CubeDescManager.<init>( > CubeDescManager.java:109) > at org.apache.kylin.cube.CubeDescManager.getInstance( > CubeDescManager.java:81) > at org.apache.kylin.cube.CubeInstance.getDescriptor( > CubeInstance.java:109) > at org.apache.kylin.cube.CubeSegment.getCubeDesc( > CubeSegment.java:119) > at org.apache.kylin.cube.CubeSegment.isEnableSharding( > CubeSegment.java:467) > at org.apache.kylin.cube.kv.RowKeyEncoder.<init>( > RowKeyEncoder.java:48) > at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance( > AbstractRowKeyEncoder.java:48) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:205) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:193) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at org.apache.spark.util.collection.ExternalSorter. > insertAll(ExternalSorter.scala:191) > at org.apache.spark.shuffle.sort.SortShuffleWriter.write( > SortShuffleWriter.scala:64) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:73) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 17/05/17 09:14:57 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID > 1, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes) > 17/05/17 09:15:45 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1, > fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > at org.apache.kylin.common.persistence.ResourceStore. > createResourceStore(ResourceStore.java:99) > at org.apache.kylin.common.persistence.ResourceStore. > getStore(ResourceStore.java:110) > at org.apache.kylin.cube.CubeDescManager.getStore( > CubeDescManager.java:370) > at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc( > CubeDescManager.java:298) > at org.apache.kylin.cube.CubeDescManager.<init>( > CubeDescManager.java:109) > at org.apache.kylin.cube.CubeDescManager.getInstance( > CubeDescManager.java:81) > at org.apache.kylin.cube.CubeInstance.getDescriptor( > CubeInstance.java:109) > at org.apache.kylin.cube.CubeSegment.getCubeDesc( > CubeSegment.java:119) > at org.apache.kylin.cube.CubeSegment.isEnableSharding( > CubeSegment.java:467) > at org.apache.kylin.cube.kv.RowKeyEncoder.<init>( > RowKeyEncoder.java:48) > at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance( > AbstractRowKeyEncoder.java:48) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:205) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:193) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at org.apache.spark.util.collection.ExternalSorter. > insertAll(ExternalSorter.scala:191) > at org.apache.spark.shuffle.sort.SortShuffleWriter.write( > SortShuffleWriter.scala:64) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:73) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 17/05/17 09:15:45 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID > 2, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes) > 17/05/17 09:16:33 WARN TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2, > fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > at org.apache.kylin.common.persistence.ResourceStore. > createResourceStore(ResourceStore.java:99) > at org.apache.kylin.common.persistence.ResourceStore. > getStore(ResourceStore.java:110) > at org.apache.kylin.cube.CubeDescManager.getStore( > CubeDescManager.java:370) > at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc( > CubeDescManager.java:298) > at org.apache.kylin.cube.CubeDescManager.<init>( > CubeDescManager.java:109) > at org.apache.kylin.cube.CubeDescManager.getInstance( > CubeDescManager.java:81) > at org.apache.kylin.cube.CubeInstance.getDescriptor( > CubeInstance.java:109) > at org.apache.kylin.cube.CubeSegment.getCubeDesc( > CubeSegment.java:119) > at org.apache.kylin.cube.CubeSegment.isEnableSharding( > CubeSegment.java:467) > at org.apache.kylin.cube.kv.RowKeyEncoder.<init>( > RowKeyEncoder.java:48) > at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance( > AbstractRowKeyEncoder.java:48) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:205) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:193) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at org.apache.spark.util.collection.ExternalSorter. > insertAll(ExternalSorter.scala:191) > at org.apache.spark.shuffle.sort.SortShuffleWriter.write( > SortShuffleWriter.scala:64) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:73) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 17/05/17 09:16:33 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID > 3, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes) > 17/05/17 09:17:22 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3, > fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > at org.apache.kylin.common.persistence.ResourceStore. > createResourceStore(ResourceStore.java:99) > at org.apache.kylin.common.persistence.ResourceStore. > getStore(ResourceStore.java:110) > at org.apache.kylin.cube.CubeDescManager.getStore( > CubeDescManager.java:370) > at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc( > CubeDescManager.java:298) > at org.apache.kylin.cube.CubeDescManager.<init>( > CubeDescManager.java:109) > at org.apache.kylin.cube.CubeDescManager.getInstance( > CubeDescManager.java:81) > at org.apache.kylin.cube.CubeInstance.getDescriptor( > CubeInstance.java:109) > at org.apache.kylin.cube.CubeSegment.getCubeDesc( > CubeSegment.java:119) > at org.apache.kylin.cube.CubeSegment.isEnableSharding( > CubeSegment.java:467) > at org.apache.kylin.cube.kv.RowKeyEncoder.<init>( > RowKeyEncoder.java:48) > at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance( > AbstractRowKeyEncoder.java:48) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:205) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:193) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at org.apache.spark.util.collection.ExternalSorter. > insertAll(ExternalSorter.scala:191) > at org.apache.spark.shuffle.sort.SortShuffleWriter.write( > SortShuffleWriter.scala:64) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:73) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 17/05/17 09:17:22 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 > times; aborting job > 17/05/17 09:17:22 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks > have all completed, from pool > 17/05/17 09:17:22 INFO YarnScheduler: Cancelling stage 0 > 17/05/17 09:17:22 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at > SparkCubingByLayer.java:193) failed in 199.860 s > 17/05/17 09:17:22 INFO DAGScheduler: Job 0 failed: saveAsNewAPIHadoopFile > at SparkCubingByLayer.java:288, took 200.053479 s > Exception in thread "main" java.lang.RuntimeException: error execute > org.apache.kylin.engine.spark.SparkCubingByLayer > at org.apache.kylin.common.util.AbstractApplication.execute( > AbstractApplication.java:42) > at org.apache.kylin.common.util.SparkEntry.main(SparkEntry. > java:44) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1( > SparkSubmit.scala:181) > at org.apache.spark.deploy.SparkSubmit$.submit( > SparkSubmit.scala:206) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit. > scala:121) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: org.apache.spark.SparkException: Job aborted due to stage > failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task > 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34): > java.lang.IllegalArgumentException: > Failed to find metadata store by url: kylin_metadata@hbase > at org.apache.kylin.common.persistence.ResourceStore. > createResourceStore(ResourceStore.java:99) > at org.apache.kylin.common.persistence.ResourceStore. > getStore(ResourceStore.java:110) > at org.apache.kylin.cube.CubeDescManager.getStore( > CubeDescManager.java:370) > at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc( > CubeDescManager.java:298) > at org.apache.kylin.cube.CubeDescManager.<init>( > CubeDescManager.java:109) > at org.apache.kylin.cube.CubeDescManager.getInstance( > CubeDescManager.java:81) > at org.apache.kylin.cube.CubeInstance.getDescriptor( > CubeInstance.java:109) > at org.apache.kylin.cube.CubeSegment.getCubeDesc( > CubeSegment.java:119) > at org.apache.kylin.cube.CubeSegment.isEnableSharding( > CubeSegment.java:467) > at org.apache.kylin.cube.kv.RowKeyEncoder.<init>( > RowKeyEncoder.java:48) > at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance( > AbstractRowKeyEncoder.java:48) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:205) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:193) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at org.apache.spark.util.collection.ExternalSorter. > insertAll(ExternalSorter.scala:191) > at org.apache.spark.shuffle.sort.SortShuffleWriter.write( > SortShuffleWriter.scala:64) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:73) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Driver stacktrace: > at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$ > scheduler$DAGScheduler$$failJobAndIndependentStages( > DAGScheduler.scala:1431) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > abortStage$1.apply(DAGScheduler.scala:1419) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > abortStage$1.apply(DAGScheduler.scala:1418) > at scala.collection.mutable.ResizableArray$class.foreach( > ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach( > ArrayBuffer.scala:47) > at org.apache.spark.scheduler.DAGScheduler.abortStage( > DAGScheduler.scala:1418) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > handleTaskSetFailed$1.apply(DAGScheduler.scala:799) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > handleTaskSetFailed$1.apply(DAGScheduler.scala:799) > at scala.Option.foreach(Option.scala:236) > at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed( > DAGScheduler.scala:799) > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > doOnReceive(DAGScheduler.scala:1640) > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > onReceive(DAGScheduler.scala:1599) > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > onReceive(DAGScheduler.scala:1588) > at org.apache.spark.util.EventLoop$$anon$1.run( > EventLoop.scala:48) > at org.apache.spark.scheduler.DAGScheduler.runJob( > DAGScheduler.scala:620) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922) > at org.apache.spark.rdd.PairRDDFunctions$$anonfun$ > saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1144) > at org.apache.spark.rdd.PairRDDFunctions$$anonfun$ > saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074) > at org.apache.spark.rdd.PairRDDFunctions$$anonfun$ > saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074) > at org.apache.spark.rdd.RDDOperationScope$.withScope( > RDDOperationScope.scala:150) > at org.apache.spark.rdd.RDDOperationScope$.withScope( > RDDOperationScope.scala:111) > at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) > at org.apache.spark.rdd.PairRDDFunctions. > saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1074) > at org.apache.spark.rdd.PairRDDFunctions$$anonfun$ > saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:994) > at org.apache.spark.rdd.PairRDDFunctions$$anonfun$ > saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985) > at org.apache.spark.rdd.PairRDDFunctions$$anonfun$ > saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985) > at org.apache.spark.rdd.RDDOperationScope$.withScope( > RDDOperationScope.scala:150) > at org.apache.spark.rdd.RDDOperationScope$.withScope( > RDDOperationScope.scala:111) > at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) > at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile( > PairRDDFunctions.scala:985) > at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile( > JavaPairRDD.scala:800) > at org.apache.kylin.engine.spark.SparkCubingByLayer.saveToHDFS( > SparkCubingByLayer.java:288) > at org.apache.kylin.engine.spark.SparkCubingByLayer.execute( > SparkCubingByLayer.java:257) > at org.apache.kylin.common.util.AbstractApplication.execute( > AbstractApplication.java:37) > ... 10 more > Caused by: java.lang.IllegalArgumentException: Failed to find metadata > store by url: kylin_metadata@hbase > at org.apache.kylin.common.persistence.ResourceStore. > createResourceStore(ResourceStore.java:99) > at org.apache.kylin.common.persistence.ResourceStore. > getStore(ResourceStore.java:110) > at org.apache.kylin.cube.CubeDescManager.getStore( > CubeDescManager.java:370) > at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc( > CubeDescManager.java:298) > at org.apache.kylin.cube.CubeDescManager.<init>( > CubeDescManager.java:109) > at org.apache.kylin.cube.CubeDescManager.getInstance( > CubeDescManager.java:81) > at org.apache.kylin.cube.CubeInstance.getDescriptor( > CubeInstance.java:109) > at org.apache.kylin.cube.CubeSegment.getCubeDesc( > CubeSegment.java:119) > at org.apache.kylin.cube.CubeSegment.isEnableSharding( > CubeSegment.java:467) > at org.apache.kylin.cube.kv.RowKeyEncoder.<init>( > RowKeyEncoder.java:48) > at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance( > AbstractRowKeyEncoder.java:48) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:205) > at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call( > SparkCubingByLayer.java:193) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at org.apache.spark.util.collection.ExternalSorter. > insertAll(ExternalSorter.scala:191) > at org.apache.spark.shuffle.sort.SortShuffleWriter.write( > SortShuffleWriter.scala:64) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:73) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation: > Closing master protocol: MasterService > 17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation: > Closing zookeeper sessionid=0x25bd16992de5eac > 17/05/17 09:17:22 INFO SparkContext: Invoking stop() from shutdown hook > 17/05/17 09:17:22 INFO ZooKeeper: Session: 0x25bd16992de5eac closed > 17/05/17 09:17:22 INFO ClientCnxn: EventThread shut down > 17/05/17 09:17:22 INFO SparkUI: Stopped Spark web UI at > http://10.24.26.191:4040 > 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Shutting down all > executors > 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Interrupting monitor > thread > 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Asking each executor to > shut down > 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Stopped > 17/05/17 09:17:22 INFO MapOutputTrackerMasterEndpoint: > MapOutputTrackerMasterEndpoint stopped! > 17/05/17 09:17:22 INFO MemoryStore: MemoryStore cleared > 17/05/17 09:17:22 INFO BlockManager: BlockManager stopped > 17/05/17 09:17:22 INFO BlockManagerMaster: BlockManagerMaster stopped > 17/05/17 09:17:22 INFO OutputCommitCoordinator$ > OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! > 17/05/17 09:17:22 INFO SparkContext: Successfully stopped SparkContext > 17/05/17 09:17:22 INFO ShutdownHookManager: Shutdown hook called > 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory > /tmp/spark-3e8cd18d-d4d0-4180-9c29-18d5814b8439 > 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory > /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd- > 7d7509ef-786d-4fa4-a75e-bb3cb1015c77 > 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: > Shutting down remote daemon. > 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Remote > daemon shut down; proceeding with flushing remote transports. > 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: > Remoting shut down. > 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory > /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc > > > > Thanks, > Zhuoran > > > 发件人: ShaoFeng Shi [mailto:[email protected]] > 发送时间: 2017年5月17日 13:36 > 收件人: dev > 抄送: [email protected] > 主题: Re: 答复: A problem in cube building with SPARK > > The mailing list filters attachement. Could you please paste it directly, > or share it with gist? > > 2017-05-17 12:17 GMT+08:00 吕卓然 <[email protected]<mailto:lv > [email protected]>>: > Hi Shaofeng, > > I’ve attached the error log in the attachment.. > > > > > > > > > 发件人: ShaoFeng Shi [mailto:[email protected]<mailto: > [email protected]>] > 发送时间: 2017年5月17日 10:38 > 收件人: user > 抄送: [email protected]<mailto:[email protected]> > 主题: Re: A problem in cube building with SPARK > > Hi zhuoran, is there any more messages before this error? This error is > not the root cause. > > 2017-05-17 10:27 GMT+08:00 吕卓然 <[email protected]<mailto:lv > [email protected]>>: > Hi all, > > Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use > MapReduce engine. However, when I try to use spark engine to build cube, it > fails at step 7: Build Cube with Spark. Here is the log info: > > 17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, > fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > at org.apache.kylin.common.persistence.ResourceStore. > createResourceStore(ResourceStore.java:99) > at org.apache.kylin.common.persistence.ResourceStore. > getStore(ResourceStore.java:110) > at org.apache.kylin.cube.CubeDescManager.getStore( > CubeDescManager.java:370) > at org.apache.kylin.cube.CubeDescManager. > reloadAllCubeDesc(CubeDescManager.java:298) > at org.apache.kylin.cube.CubeDescManager.<init>( > CubeDescManager.java:109) > at org.apache.kylin.cube.CubeDescManager.getInstance( > CubeDescManager.java:81) > at org.apache.kylin.cube.CubeInstance.getDescriptor( > CubeInstance.java:109) > at org.apache.kylin.cube.CubeSegment.getCubeDesc( > CubeSegment.java:119) > at org.apache.kylin.cube.CubeSegment.isEnableSharding( > CubeSegment.java:467) > at org.apache.kylin.cube.kv.RowKeyEncoder.<init>( > RowKeyEncoder.java:48) > at org.apache.kylin.cube.kv.AbstractRowKeyEncoder. > createInstance(AbstractRowKeyEncoder.java:48) > at org.apache.kylin.engine.spark. > SparkCubingByLayer$2.call(SparkCubingByLayer.java:205) > at org.apache.kylin.engine.spark. > SparkCubingByLayer$2.call(SparkCubingByLayer.java:193) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at org.apache.spark.api.java.JavaPairRDD$$anonfun$ > pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018) > at scala.collection.Iterator$$ > anon$11.next(Iterator.scala:328) > at org.apache.spark.util.collection.ExternalSorter. > insertAll(ExternalSorter.scala:191) > at org.apache.spark.shuffle.sort. > SortShuffleWriter.write(SortShuffleWriter.scala:64) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:73) > at org.apache.spark.scheduler.ShuffleMapTask.runTask( > ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:227) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Any suggestions would help. > > Thanks, > Zhuoran > > > > -- > Best regards, > > Shaofeng Shi 史少锋 > > > > > -- > Best regards, > > Shaofeng Shi 史少锋 > > -- Best regards, Shaofeng Shi 史少锋
