>From the error it tried to seek the file from local file system, instead of HDFS. Please check whethere "fs.DefaultFS" in your environment was set to local file by mistake.
2017-02-04 22:58 GMT+08:00 ? ? <[email protected]>: > Hi Sir, > > When I built the sample cube (and my own cube), I met the error as below. > It seems the issue is at the moving step, because I confirmed there was > ‘hdfs://sandbox.hortonworks.com:8020/kylin/kylin_metadata/ > kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count/000000_0’, but Kylin > was trying to move is hdfs://sandbox.hortonworks. > com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf- > 1ce44302c394/row_count/.hive-staging_hive_2017-02-04_09-38- > 21_841_1217104325959051054-5/-ext-10000. > > [root@ip-10-9-255-49 ec2-user]# hdfs dfs -ls hdfs://sandbox.hortonworks. > com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf- > 1ce44302c394/row_count > Found 1 items > -rwxr-xr-x 1 hive hdfs 3 2017-02-04 09:38 hdfs:// > sandbox.hortonworks.com:8020/kylin/kylin_metadata/ > kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count/000000_0 > > Environment: > Kylin 1.6.0 + HDP 2.5 > I am not sure if HDP 2.5 is a too high version, because I only see HDP 2.4 > is referred in Kylin doc. > > > Build error: > 2017-02-04 09:38:17,947 INFO [pool-5-thread-2] > manager.ExecutableManager:292 : job id:6a392cfd-a903-4763-89cf-1ce44302c394-01 > from READY to RUNNING > 2017-02-04 09:38:17,947 INFO [pool-5-thread-2] hive.HiveCmdBuilder:81 : > The statements to execute in beeline: > USE default; > SET hive.exec.compress.output=true; > SET hive.auto.convert.join.noconditionaltask=true; > SET hive.auto.convert.join.noconditionaltask.size=100000000; > SET mapreduce.output.fileoutputformat.compress.type=BLOCK; > SET mapreduce.job.split.metainfo.maxsize=-1; > > set hive.exec.compress.output=false; > > set hive.exec.compress.output=false; > INSERT OVERWRITE DIRECTORY '/kylin/kylin_metadata/kylin- > 6a392cfd-a903-4763-89cf-1ce44302c394/row_count' SELECT count(*) FROM > kylin_intermediate_kylin_sales_cube_desc_0eb9faec_6a22_ > 4b49_889f_c283d82d72dd; > > > 2017-02-04 09:38:17,947 DEBUG [pool-5-thread-2] hive.HiveCmdBuilder:83 : > The SQL to execute in beeline: > > 2017-02-04 09:38:17,947 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : Compute row count of flat hive table, > cmd: > 2017-02-04 09:38:17,947 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : beeline -n root -u 'jdbc:hive2:// > sandbox.hortonworks.com:2181/;serviceDiscoveryMode= > zooKeeper;zooKeeperNamespace=hiveserver2' -f > /root/apache-kylin-1.6.0-hbase1.x-bin/bin/../tomcat/temp/beeline_3013188987573907903.hql;rm > -f /root/apache-kylin-1.6.0-hbase1.x-bin/bin/../tomcat/temp/beeline_ > 3013188987573907903.hql > 2017-02-04 09:38:19,832 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch YARN > applications. > 2017-02-04 09:38:20,342 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : Connecting to jdbc:hive2://sandbox. > hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace= > hiveserver2 > 2017-02-04 09:38:21,629 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : Connected to: Apache Hive (version > 1.2.1000.2.5.0.0-1245) > 2017-02-04 09:38:21,630 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : Driver: Hive JDBC (version > 1.2.1.2.3.2.0-2950) > 2017-02-04 09:38:21,630 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : Transaction isolation: > TRANSACTION_REPEATABLE_READ > 2017-02-04 09:38:21,679 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> USE default; > 2017-02-04 09:38:21,732 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : No rows affected (0.051 seconds) > 2017-02-04 09:38:21,746 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> SET hive.exec.compress.output=true; > 2017-02-04 09:38:21,757 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : No rows affected (0.011 seconds) > 2017-02-04 09:38:21,766 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> SET hive.auto.convert.join.noconditionaltask=true; > 2017-02-04 09:38:21,768 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : No rows affected (0.002 seconds) > 2017-02-04 09:38:21,777 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> SET hive.auto.convert.join.noconditionaltask.size= > 100000000; > 2017-02-04 09:38:21,779 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : No rows affected (0.002 seconds) > 2017-02-04 09:38:21,787 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> SET mapreduce.output.fileoutputformat.compress. > type=BLOCK; > 2017-02-04 09:38:21,790 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : No rows affected (0.003 seconds) > 2017-02-04 09:38:21,796 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> SET mapreduce.job.split.metainfo.maxsize=-1; > 2017-02-04 09:38:21,801 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : No rows affected (0.005 seconds) > 2017-02-04 09:38:21,804 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> > 2017-02-04 09:38:21,808 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> set hive.exec.compress.output=false; > 2017-02-04 09:38:21,811 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : No rows affected (0.003 seconds) > 2017-02-04 09:38:21,812 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> > 2017-02-04 09:38:21,816 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> set hive.exec.compress.output=false; > 2017-02-04 09:38:21,818 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : No rows affected (0.002 seconds) > 2017-02-04 09:38:21,833 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> INSERT OVERWRITE DIRECTORY > '/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1c > e44302c394/row_count' SELECT count(*) FROM k > 2017-02-04 09:38:21,840 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : ylin_intermediate_kylin_sales_ > cube_desc_0eb9faec_6a22_4b49_889f_c283d82d72dd; > 2017-02-04 09:38:30,452 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : Tez session hasn't been created > yet. Opening session > 2017-02-04 09:38:30,452 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : Dag name: INSERT OVERWRITE > DIRE...49_889f_c283d82d72dd(Stage-1) > 2017-02-04 09:38:30,452 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : > 2017-02-04 09:38:30,452 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : > 2017-02-04 09:38:30,452 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : Status: Running (Executing on > YARN cluster with App id application_1486198519344_0004) > 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : > 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : Map 1: -/- Reducer 2: 0/1 > 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : Map 1: 0/1 Reducer 2: 0/1 > 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : Map 1: 0(+1)/1 Reducer > 2: 0/1 > 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : Map 1: 1/1 Reducer 2: 0/1 > 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : Map 1: 1/1 Reducer 2: 0(+1)/1 > 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : Map 1: 1/1 Reducer 2: 1/1 > 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : INFO : Moving data to directory > /kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count > from hdfs://sandbox.h > ortonworks.com:8020/kylin/kylin_metadata/kylin-6a392cfd- > a903-4763-89cf-1ce44302c394/row_count/.hive-staging_hive_ > 2017-02-04_09-38-21_841_1217104325959051054-5/-ext-10000 > 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : No rows affected (8.613 seconds) > 2017-02-04 09:38:30,457 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> > 2017-02-04 09:38:30,457 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > hortonworks.com:2181/> > 2017-02-04 09:38:30,458 INFO [pool-5-thread-2] > execution.AbstractExecutable:36 : Closing: 0: jdbc:hive2://sandbox. > hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace= > hiveserver2 > 2017-02-04 09:38:30,573 ERROR [pool-5-thread-2] > execution.AbstractExecutable:370 : job:6a392cfd-a903-4763-89cf-1ce44302c394-01 > execute finished with exception > java.io.FileNotFoundException: File /kylin/kylin_metadata/kylin- > 6a392cfd-a903-4763-89cf-1ce44302c394/row_count does not exist > at org.apache.hadoop.fs.RawLocalFileSystem.listStatus( > RawLocalFileSystem.java:429) > at org.apache.hadoop.fs.FileSystem.listStatus( > FileSystem.java:1515) > at org.apache.hadoop.fs.FileSystem.listStatus( > FileSystem.java:1555) > at org.apache.hadoop.fs.ChecksumFileSystem.listStatus( > ChecksumFileSystem.java:574) > at org.apache.kylin.source.hive.HiveMRInput$ > RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:338) > at org.apache.kylin.job.execution.AbstractExecutable. > execute(AbstractExecutable.java:113) > at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork( > DefaultChainedExecutable.java:57) > at org.apache.kylin.job.execution.AbstractExecutable. > execute(AbstractExecutable.java:113) > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$ > JobRunner.run(DefaultScheduler.java:136) > > > > Thx > Lei Wang > > -- Best regards, Shaofeng Shi 史少锋
