Hi Sir,
When I built the sample cube (and my own cube), I met the error as below. It
seems the issue is at the moving step, because I confirmed there was
‘hdfs://sandbox.hortonworks.com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count/000000_0’,
but Kylin was trying to move is
hdfs://sandbox.hortonworks.com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count/.hive-staging_hive_2017-02-04_09-38-21_841_1217104325959051054-5/-ext-10000.
[root@ip-10-9-255-49 ec2-user]# hdfs dfs -ls
hdfs://sandbox.hortonworks.com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count
Found 1 items
-rwxr-xr-x 1 hive hdfs 3 2017-02-04 09:38
hdfs://sandbox.hortonworks.com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count/000000_0
Environment:
Kylin 1.6.0 + HDP 2.5
I am not sure if HDP 2.5 is a too high version, because I only see HDP 2.4 is
referred in Kylin doc.
Build error:
2017-02-04 09:38:17,947 INFO [pool-5-thread-2] manager.ExecutableManager:292 :
job id:6a392cfd-a903-4763-89cf-1ce44302c394-01 from READY to RUNNING
2017-02-04 09:38:17,947 INFO [pool-5-thread-2] hive.HiveCmdBuilder:81 : The
statements to execute in beeline:
USE default;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=100000000;
SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
SET mapreduce.job.split.metainfo.maxsize=-1;
set hive.exec.compress.output=false;
set hive.exec.compress.output=false;
INSERT OVERWRITE DIRECTORY
'/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count'
SELECT count(*) FROM
kylin_intermediate_kylin_sales_cube_desc_0eb9faec_6a22_4b49_889f_c283d82d72dd;
2017-02-04 09:38:17,947 DEBUG [pool-5-thread-2] hive.HiveCmdBuilder:83 : The
SQL to execute in beeline:
2017-02-04 09:38:17,947 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: Compute row count of flat hive table, cmd:
2017-02-04 09:38:17,947 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: beeline -n root -u
'jdbc:hive2://sandbox.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2'
-f
/root/apache-kylin-1.6.0-hbase1.x-bin/bin/../tomcat/temp/beeline_3013188987573907903.hql;rm
-f
/root/apache-kylin-1.6.0-hbase1.x-bin/bin/../tomcat/temp/beeline_3013188987573907903.hql
2017-02-04 09:38:19,832 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: WARNING: Use "yarn jar" to launch YARN applications.
2017-02-04 09:38:20,342 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: Connecting to
jdbc:hive2://sandbox.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
2017-02-04 09:38:21,629 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: Connected to: Apache Hive (version 1.2.1000.2.5.0.0-1245)
2017-02-04 09:38:21,630 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: Driver: Hive JDBC (version 1.2.1.2.3.2.0-2950)
2017-02-04 09:38:21,630 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: Transaction isolation: TRANSACTION_REPEATABLE_READ
2017-02-04 09:38:21,679 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/> USE default;
2017-02-04 09:38:21,732 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: No rows affected (0.051 seconds)
2017-02-04 09:38:21,746 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/> SET
hive.exec.compress.output=true;
2017-02-04 09:38:21,757 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: No rows affected (0.011 seconds)
2017-02-04 09:38:21,766 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/> SET
hive.auto.convert.join.noconditionaltask=true;
2017-02-04 09:38:21,768 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: No rows affected (0.002 seconds)
2017-02-04 09:38:21,777 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/> SET
hive.auto.convert.join.noconditionaltask.size=100000000;
2017-02-04 09:38:21,779 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: No rows affected (0.002 seconds)
2017-02-04 09:38:21,787 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/> SET
mapreduce.output.fileoutputformat.compress.type=BLOCK;
2017-02-04 09:38:21,790 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: No rows affected (0.003 seconds)
2017-02-04 09:38:21,796 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/> SET
mapreduce.job.split.metainfo.maxsize=-1;
2017-02-04 09:38:21,801 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: No rows affected (0.005 seconds)
2017-02-04 09:38:21,804 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/>
2017-02-04 09:38:21,808 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/> set
hive.exec.compress.output=false;
2017-02-04 09:38:21,811 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: No rows affected (0.003 seconds)
2017-02-04 09:38:21,812 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/>
2017-02-04 09:38:21,816 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/> set
hive.exec.compress.output=false;
2017-02-04 09:38:21,818 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: No rows affected (0.002 seconds)
2017-02-04 09:38:21,833 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/> INSERT OVERWRITE DIRECTORY
'/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1c
e44302c394/row_count' SELECT count(*) FROM k
2017-02-04 09:38:21,840 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: ylin_intermediate_kylin_sales_cube_desc_0eb9faec_6a22_4b49_889f_c283d82d72dd;
2017-02-04 09:38:30,452 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO : Tez session hasn't been created yet. Opening session
2017-02-04 09:38:30,452 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO : Dag name: INSERT OVERWRITE DIRE...49_889f_c283d82d72dd(Stage-1)
2017-02-04 09:38:30,452 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO :
2017-02-04 09:38:30,452 INFO [pool-5-thread-2] execution.AbstractExecutable:36
:
2017-02-04 09:38:30,452 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO : Status: Running (Executing on YARN cluster with App id
application_1486198519344_0004)
2017-02-04 09:38:30,453 INFO [pool-5-thread-2] execution.AbstractExecutable:36
:
2017-02-04 09:38:30,453 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO : Map 1: -/- Reducer 2: 0/1
2017-02-04 09:38:30,453 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO : Map 1: 0/1 Reducer 2: 0/1
2017-02-04 09:38:30,453 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO : Map 1: 0(+1)/1 Reducer 2: 0/1
2017-02-04 09:38:30,453 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO : Map 1: 1/1 Reducer 2: 0/1
2017-02-04 09:38:30,453 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO : Map 1: 1/1 Reducer 2: 0(+1)/1
2017-02-04 09:38:30,453 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO : Map 1: 1/1 Reducer 2: 1/1
2017-02-04 09:38:30,453 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: INFO : Moving data to directory
/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count from
hdfs://sandbox.h
ortonworks.com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count/.hive-staging_hive_2017-02-04_09-38-21_841_1217104325959051054-5/-ext-10000
2017-02-04 09:38:30,453 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: No rows affected (8.613 seconds)
2017-02-04 09:38:30,457 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/>
2017-02-04 09:38:30,457 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: 0: jdbc:hive2://sandbox.hortonworks.com:2181/>
2017-02-04 09:38:30,458 INFO [pool-5-thread-2] execution.AbstractExecutable:36
: Closing: 0:
jdbc:hive2://sandbox.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
2017-02-04 09:38:30,573 ERROR [pool-5-thread-2]
execution.AbstractExecutable:370 : job:6a392cfd-a903-4763-89cf-1ce44302c394-01
execute finished with exception
java.io.FileNotFoundException: File
/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count does
not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:429)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555)
at
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:574)
at
org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:338)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
Thx
Lei Wang
