Hi, Did the previous building steps successfully create the Hive intermediate tables in Hive?
Thanks, Silas On 30 Jan 2018, at 3:09 AM, macwork <[email protected]> wrote: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop/tmp/nm-local-dir/filecache/13/spark-libs.jar/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop/software/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/01/30 19:05:28 INFO util.SignalUtils: Registered signal handler for TERM 18/01/30 19:05:28 INFO util.SignalUtils: Registered signal handler for HUP 18/01/30 19:05:28 INFO util.SignalUtils: Registered signal handler for INT 18/01/30 19:05:29 INFO yarn.ApplicationMaster: Preparing Local resources 18/01/30 19:05:30 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1516010594436_0185_000002 18/01/30 19:05:30 INFO spark.SecurityManager: Changing view acls to: hadoop 18/01/30 19:05:30 INFO spark.SecurityManager: Changing modify acls to: hadoop 18/01/30 19:05:30 INFO spark.SecurityManager: Changing view acls groups to: 18/01/30 19:05:30 INFO spark.SecurityManager: Changing modify acls groups to: 18/01/30 19:05:30 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set() 18/01/30 19:05:30 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread 18/01/30 19:05:30 INFO yarn.ApplicationMaster: Waiting for spark context initialization... 18/01/30 19:05:30 INFO spark.SparkContext: Running Spark version 2.1.2 18/01/30 19:05:30 INFO spark.SecurityManager: Changing view acls to: hadoop 18/01/30 19:05:30 INFO spark.SecurityManager: Changing modify acls to: hadoop 18/01/30 19:05:30 INFO spark.SecurityManager: Changing view acls groups to: 18/01/30 19:05:30 INFO spark.SecurityManager: Changing modify acls groups to: 18/01/30 19:05:30 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set() 18/01/30 19:05:31 INFO util.Utils: Successfully started service 'sparkDriver' on port 42339. 18/01/30 19:05:31 INFO spark.SparkEnv: Registering MapOutputTracker 18/01/30 19:05:31 INFO spark.SparkEnv: Registering BlockManagerMaster 18/01/30 19:05:31 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 18/01/30 19:05:31 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 18/01/30 19:05:31 INFO storage.DiskBlockManager: Created local directory at /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/blockmgr-e9044b90-ff87-4ad3-9ae7-cf85a131ba4e 18/01/30 19:05:31 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB 18/01/30 19:05:31 INFO spark.SparkEnv: Registering OutputCommitCoordinator 18/01/30 19:05:31 INFO util.log: Logging initialized @3601ms 18/01/30 19:05:31 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 18/01/30 19:05:31 INFO server.Server: jetty-9.2.z-SNAPSHOT 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1727271{/jobs,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5a8c27d3{/jobs/json,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@201b73c{/jobs/job,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6dc9732e{/jobs/job/json,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7f3a751c{/stages,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@44336797{/stages/json,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4b84807a{/stages/stage,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6d72da27{/stages/stage/json,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@41ccc000{/stages/pool,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1e1c0871{/stages/pool/json,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@71f4dc98{/storage,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@a2c9363{/storage/json,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2d479ee6{/storage/rdd,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@53e654bf{/storage/rdd/json,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4215247{/environment,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6af87e2b{/environment/json,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7bd97d3c{/executors,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1afb9d54{/executors/json,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@14c64726{/executors/threadDump,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3de67ab3{/executors/threadDump/json,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@d66c0ae{/static,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4bb1ed6b{/,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6de87c4c{/api,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6355a0a0{/jobs/job/kill,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@580d6fd4{/stages/stage/kill,null,AVAILABLE,@Spark} 18/01/30 19:05:31 INFO server.ServerConnector: Started Spark@36b4f4c5{HTTP/1.1}{0.0.0.0:37939} 18/01/30 19:05:31 INFO server.Server: Started @3740ms 18/01/30 19:05:31 INFO util.Utils: Successfully started service 'SparkUI' on port 37939. 18/01/30 19:05:31 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.184:37939 18/01/30 19:05:31 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler 18/01/30 19:05:31 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1516010594436_0185 and attemptId Some(appattempt_1516010594436_0185_000002) 18/01/30 19:05:31 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36754. 18/01/30 19:05:31 INFO netty.NettyBlockTransferService: Server created on 192.168.1.184:36754 18/01/30 19:05:31 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 18/01/30 19:05:31 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.1.184, 36754, None) 18/01/30 19:05:31 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.1.184:36754 with 366.3 MB RAM, BlockManagerId(driver, 192.168.1.184, 36754, None) 18/01/30 19:05:31 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.1.184, 36754, None) 18/01/30 19:05:31 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.1.184, 36754, None) 18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@57ef634b{/metrics/json,null,AVAILABLE,@Spark} 18/01/30 19:05:32 INFO scheduler.EventLoggingListener: Logging events to hdfs:///kylin/spark-history/application_1516010594436_0185_2 18/01/30 19:05:32 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://[email protected]:42339) 18/01/30 19:05:32 INFO yarn.ApplicationMaster: =============================================================================== YARN executor launch context: env: CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/* SPARK_YARN_STAGING_DIR -> hdfs://192.168.1.171:9000/user/hadoop/.sparkStaging/application_1516010594436_0185 SPARK_USER -> hadoop SPARK_YARN_MODE -> true command: {{JAVA_HOME}}/bin/java \ -server \ -Xmx1024m \ '-Dhdp.version=current' \ -Djava.io.tmpdir={{PWD}}/tmp \ -Dspark.yarn.app.container.log.dir=<LOG_DIR> \ -XX:OnOutOfMemoryError='kill %p' \ org.apache.spark.executor.CoarseGrainedExecutorBackend \ --driver-url \ spark://[email protected]:42339 \ --executor-id \ <executorId> \ --hostname \ <hostname> \ --cores \ 2 \ --app-id \ application_1516010594436_0185 \ --user-class-path \ file:$PWD/__app__.jar \ --user-class-path \ file:$PWD/htrace-core-3.1.0-incubating.jar \ --user-class-path \ file:$PWD/metrics-core-2.2.0.jar \ --user-class-path \ file:$PWD/guava-12.0.1.jar \ 1><LOG_DIR>/stdout \ 2><LOG_DIR>/stderr resources: htrace-core-3.1.0-incubating.jar -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000 file: "/user/hadoop/.sparkStaging/application_1516010594436_0185/htrace-core-3.1.0-incubating.jar" } size: 1475955 timestamp: 1517310297655 type: FILE visibility: PRIVATE __app__.jar -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000 file: "/user/hadoop/.sparkStaging/application_1516010594436_0185/kylin-job-2.2.0.jar" } size: 16695259 timestamp: 1517310297574 type: FILE visibility: PRIVATE guava-12.0.1.jar -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000 file: "/user/hadoop/.sparkStaging/application_1516010594436_0185/guava-12.0.1.jar" } size: 1795932 timestamp: 1517310297780 type: FILE visibility: PRIVATE __spark_conf__ -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000 file: "/user/hadoop/.sparkStaging/application_1516010594436_0185/__spark_conf__.zip" } size: 86203 timestamp: 1517310297898 type: ARCHIVE visibility: PRIVATE metrics-core-2.2.0.jar -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000 file: "/user/hadoop/.sparkStaging/application_1516010594436_0185/metrics-core-2.2.0.jar" } size: 82123 timestamp: 1517310297714 type: FILE visibility: PRIVATE __spark_libs__ -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000 file: "/kylin/spark/spark-libs.jar" } size: 200781646 timestamp: 1517295215668 type: ARCHIVE visibility: PUBLIC =============================================================================== 18/01/30 19:05:32 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.171:8030 18/01/30 19:05:32 INFO yarn.YarnRMClient: Registering the ApplicationMaster 18/01/30 19:05:32 INFO yarn.YarnAllocator: Will request 1 executor container(s), each with 2 core(s) and 1408 MB memory (including 384 MB of overhead) 18/01/30 19:05:32 INFO yarn.YarnAllocator: Submitted 1 unlocalized container requests. 18/01/30 19:05:32 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals 18/01/30 19:05:32 INFO impl.AMRMClientImpl: Received new token for : sx-hadoop-dn02:32911 18/01/30 19:05:32 INFO yarn.YarnAllocator: Launching container container_1516010594436_0185_02_000002 on host sx-hadoop-dn02 18/01/30 19:05:32 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. 18/01/30 19:05:32 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 18/01/30 19:05:32 INFO impl.ContainerManagementProtocolProxy: Opening proxy : sx-hadoop-dn02:32911 18/01/30 19:05:36 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.1.181:54402) with ID 1 18/01/30 19:05:36 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 18/01/30 19:05:36 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 18/01/30 19:05:36 INFO storage.BlockManagerMasterEndpoint: Registering block manager sx-hadoop-dn02:42882 with 366.3 MB RAM, BlockManagerId(1, sx-hadoop-dn02, 42882, None) 18/01/30 19:05:36 INFO common.AbstractHadoopJob: Ready to load KylinConfig from uri: kylin_metadata@hdfs,path=hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a 18/01/30 19:05:36 INFO cube.CubeManager: Initializing CubeManager with config kylin_metadata@hdfs,path=hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a 18/01/30 19:05:36 INFO persistence.ResourceStore: Using metadata url kylin_metadata@hdfs,path=hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a for resource store 18/01/30 19:05:36 INFO hdfs.HDFSResourceStore: hdfs meta path : hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a 18/01/30 19:05:36 INFO cube.CubeManager: Loading Cube from folder hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a/cube 18/01/30 19:05:37 INFO cube.CubeDescManager: Initializing CubeDescManager with config kylin_metadata@hdfs,path=hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a 18/01/30 19:05:37 INFO cube.CubeDescManager: Reloading Cube Metadata from folder hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a/cube_desc 18/01/30 19:05:37 INFO project.ProjectManager: Initializing ProjectManager with metadata url kylin_metadata@hdfs,path=hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a 18/01/30 19:05:37 INFO measure.MeasureTypeFactory: Checking custom measure types from kylin config 18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory 18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory 18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory 18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory 18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering EXTENDED_COLUMN(extendedcolumn), class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory 18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering PERCENTILE(percentile), class org.apache.kylin.measure.percentile.PercentileMeasureType$Factory 18/01/30 19:05:37 INFO metadata.MetadataManager: Reloading data model at /model_desc/kylin_sales_model.json 18/01/30 19:05:37 INFO cube.CubeDescManager: Loaded 1 Cube(s) 18/01/30 19:05:37 INFO cube.CubeManager: Reloaded cube kylin_sales_cube_clone being CUBE[name=kylin_sales_cube_clone] having 2 segments 18/01/30 19:05:37 INFO cube.CubeManager: Loaded 1 cubes, fail on 0 cubes 18/01/30 19:05:37 INFO spark.SparkCubingByLayer: RDD Output path: hdfs://192.168.1.171:9000/kylin/kylin_metadata/kylin-5b2617ee-90ec-4c4b-86ff-699d0f14223e/kylin_sales_cube_clone/cuboid/ 18/01/30 19:05:37 INFO spark.SparkCubingByLayer: All measure are normal (agg on all cuboids) ? : true 18/01/30 19:05:38 INFO internal.SharedState: Warehouse path is 'file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/container_1516010594436_0185_02_000001/spark-warehouse'. 18/01/30 19:05:38 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@75fae875{/SQL,null,AVAILABLE,@Spark} 18/01/30 19:05:38 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3785cc60{/SQL/json,null,AVAILABLE,@Spark} 18/01/30 19:05:38 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6299f0a5{/SQL/execution,null,AVAILABLE,@Spark} 18/01/30 19:05:38 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@578d227{/SQL/execution/json,null,AVAILABLE,@Spark} 18/01/30 19:05:38 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@29f2a18{/static/sql,null,AVAILABLE,@Spark} 18/01/30 19:05:38 INFO hive.HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. 18/01/30 19:05:38 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 18/01/30 19:05:39 INFO metastore.ObjectStore: ObjectStore, initialize called 18/01/30 19:05:39 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 18/01/30 19:05:39 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 18/01/30 19:05:45 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 18/01/30 19:05:47 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 18/01/30 19:05:47 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 18/01/30 19:05:51 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 18/01/30 19:05:51 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 18/01/30 19:05:52 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 18/01/30 19:05:52 INFO metastore.ObjectStore: Initialized ObjectStore 18/01/30 19:05:52 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 18/01/30 19:05:52 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException 18/01/30 19:05:53 INFO metastore.HiveMetaStore: Added admin role in metastore 18/01/30 19:05:53 INFO metastore.HiveMetaStore: Added public role in metastore 18/01/30 19:05:53 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty 18/01/30 19:05:53 INFO metastore.HiveMetaStore: 0: get_all_databases 18/01/30 19:05:53 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_all_databases 18/01/30 19:05:53 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=* 18/01/30 19:05:53 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_functions: db=default pat=* 18/01/30 19:05:53 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table. 18/01/30 19:05:54 INFO session.SessionState: Created local directory: /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/container_1516010594436_0185_02_000001/tmp/hadoop 18/01/30 19:05:54 INFO session.SessionState: Created local directory: /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/container_1516010594436_0185_02_000001/tmp/4990ced4-7b5c-46d6-a0f6-9ecfdcbc22c5_resources 18/01/30 19:05:54 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/4990ced4-7b5c-46d6-a0f6-9ecfdcbc22c5 18/01/30 19:05:54 INFO session.SessionState: Created local directory: /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/container_1516010594436_0185_02_000001/tmp/hadoop/4990ced4-7b5c-46d6-a0f6-9ecfdcbc22c5 18/01/30 19:05:54 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/4990ced4-7b5c-46d6-a0f6-9ecfdcbc22c5/_tmp_space.db 18/01/30 19:05:54 INFO client.HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/container_1516010594436_0185_02_000001/spark-warehouse 18/01/30 19:05:54 INFO metastore.HiveMetaStore: 0: get_database: default 18/01/30 19:05:54 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_database: default 18/01/30 19:05:54 INFO metastore.HiveMetaStore: 0: get_database: global_temp 18/01/30 19:05:54 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_database: global_temp 18/01/30 19:05:54 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException 18/01/30 19:05:54 INFO execution.SparkSqlParser: Parsing command: default.kylin_intermediate_kylin_sales_cube_clone_c3468331_ab55_4368_b691_bf08e2b5fd2a 18/01/30 19:05:55 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=kylin_intermediate_kylin_sales_cube_clone_c3468331_ab55_4368_b691_bf08e2b5fd2a 18/01/30 19:05:55 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_table : db=default tbl=kylin_intermediate_kylin_sales_cube_clone_c3468331_ab55_4368_b691_bf08e2b5fd2a 18/01/30 19:05:55 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:636) Caused by: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'kylin_intermediate_kylin_sales_cube_clone_c3468331_ab55_4368_b691_bf08e2b5fd2a' not found in database 'default'; at org.apache.spark.sql.hive.client.HiveClient$$anonfun$getTable$1.apply(HiveClient.scala:74) at org.apache.spark.sql.hive.client.HiveClient$$anonfun$getTable$1.apply(HiveClient.scala:74) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:74) at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:78) at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$org$apache$spark$sql$hive$HiveExternalCatalog$$getRawTable$1.apply(HiveExternalCatalog.scala:118) at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$org$apache$spark$sql$hive$HiveExternalCatalog$$getRawTable$1.apply(HiveExternalCatalog.scala:118) at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97) at org.apache.spark.sql.hive.HiveExternalCatalog.org$apache$spark$sql$hive$HiveExternalCatalog$$getRawTable(HiveExternalCatalog.scala:117) at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:628) at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:628) at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97) at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:627) at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:124) at org.apache.spark.sql.hive.HiveSessionCatalog.lookupRelation(HiveSessionCatalog.scala:70) at org.apache.spark.sql.SparkSession.table(SparkSession.scala:586) at org.apache.spark.sql.SparkSession.table(SparkSession.scala:582) at org.apache.spark.sql.SQLContext.table(SQLContext.scala:708) at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:167) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) ... 6 more 18/01/30 19:05:55 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer) 18/01/30 19:05:55 INFO spark.SparkContext: Invoking stop() from shutdown hook 18/01/30 19:05:55 INFO server.ServerConnector: Stopped Spark@36b4f4c5{HTTP/1.1}{0.0.0.0:0} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@580d6fd4{/stages/stage/kill,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6355a0a0{/jobs/job/kill,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6de87c4c{/api,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4bb1ed6b{/,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@d66c0ae{/static,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3de67ab3{/executors/threadDump/json,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@14c64726{/executors/threadDump,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1afb9d54{/executors/json,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7bd97d3c{/executors,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6af87e2b{/environment/json,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4215247{/environment,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@53e654bf{/storage/rdd/json,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2d479ee6{/storage/rdd,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@a2c9363{/storage/json,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@71f4dc98{/storage,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1e1c0871{/stages/pool/json,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@41ccc000{/stages/pool,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6d72da27{/stages/stage/json,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4b84807a{/stages/stage,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@44336797{/stages/json,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7f3a751c{/stages,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6dc9732e{/jobs/job/json,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@201b73c{/jobs/job,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5a8c27d3{/jobs/json,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1727271{/jobs,null,UNAVAILABLE,@Spark} 18/01/30 19:05:55 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.1.184:37939 18/01/30 19:05:55 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s). 18/01/30 19:05:55 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors 18/01/30 19:05:55 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down 18/01/30 19:05:55 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 18/01/30 19:05:55 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 18/01/30 19:05:55 INFO memory.MemoryStore: MemoryStore cleared 18/01/30 19:05:55 INFO storage.BlockManager: BlockManager stopped 18/01/30 19:05:55 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 18/01/30 19:05:55 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 18/01/30 19:05:55 INFO spark.SparkContext: Successfully stopped SparkContext 18/01/30 19:05:55 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer) 18/01/30 19:05:55 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered. 18/01/30 19:05:55 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://192.168.1.171:9000/user/hadoop/.sparkStaging/application_1516010594436_0185 18/01/30 19:05:55 INFO util.ShutdownHookManager: Shutdown hook called 18/01/30 19:05:55 INFO util.ShutdownHookManager: Deleting directory /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/spark-502e0062-d08e-47a0-9663-4b284f141d3f
