Yes copy from the hadoop environment. We haven't tested on HDP 2.5 so I couldn't ensure that's the problem; If you're new to Kylin, suggest you downgrade to HDP 2.4 or even HDP 2.2, which are much easier. Besides, you can open a JIRA for this problem ( https://issues.apache.org/jira/secure/Dashboard.jspa), we will investigate it.
2017-02-05 22:00 GMT+08:00 磊 王 <[email protected]>: > Thanks your reply again. > > I will follow your suggestion to try it. Some discussion- > I did not touch core-site.xml under Kylin conf folder, so it should be the > default. I do not think the customization is always needed to make kylin > work, then why this issue occurred in my environment? > > Where can I get the correct core-site.xml, can I copy it from my Hadoop > environment? Or what specific conf items you specially care about to judge > its correctness, say 'fs.defaultFS'? > > > 发自我的 iPhone > > > 在 2017年2月5日，21:38，ShaoFeng Shi <[email protected]> 写道： > > > > No, from the output of "hdfs fs -ls" the file does exist on HDFS; In > hive's > > context its default fs is HDFS, so it works as expected. The behavior of > > Kylin is wrong as it goes to "RawLocalFileSystem". Usually this is > because > > the core-site.xml is wrongly configured or it is absent from Kylin's > > classpath I think. A quick workaround can be manually finding the right > > core-site.xml and then copy it to KYLIN_HOME/conf, restart and then > resume > > the job. Please take a try and let us know whether it can solve. (we > > haven't tried HDP 2.5) > > > > 2017-02-05 21:22 GMT+08:00 磊 王 <[email protected]>: > > > >> Thanks for your reply. > >> I see the log shows it firstly moved that file from hdfs to local, and > >> then read from local. But the issue is the file to move in hdfs does not > >> exist at the step of move, so the local file is not available to read at > >> the step of read. > >> Is my understanding correct? > >> > >> 发自我的 iPhone > >> > >>> 在 2017年2月5日，20:54，ShaoFeng Shi <[email protected]> 写道： > >>> > >>> From the error it tried to seek the file from local file system, > instead > >> of > >>> HDFS. Please check whethere "fs.DefaultFS" in your environment was set > to > >>> local file by mistake. > >>> > >>> > >>> 2017-02-04 22:58 GMT+08:00 ? ? <[email protected]>: > >>> > >>>> Hi Sir, > >>>> > >>>> When I built the sample cube (and my own cube), I met the error as > >> below. > >>>> It seems the issue is at the moving step, because I confirmed there > was > >>>> ‘hdfs://sandbox.hortonworks.com:8020/kylin/kylin_metadata/ > >>>> kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count/000000_0’, but > >> Kylin > >>>> was trying to move is hdfs://sandbox.hortonworks. > >>>> com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf- > >>>> 1ce44302c394/row_count/.hive-staging_hive_2017-02-04_09-38- > >>>> 21_841_1217104325959051054-5/-ext-10000. > >>>> > >>>> [root@ip-10-9-255-49 ec2-user]# hdfs dfs -ls > >> hdfs://sandbox.hortonworks. > >>>> com:8020/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf- > >>>> 1ce44302c394/row_count > >>>> Found 1 items > >>>> -rwxr-xr-x 1 hive hdfs 3 2017-02-04 09:38 hdfs:// > >>>> sandbox.hortonworks.com:8020/kylin/kylin_metadata/ > >>>> kylin-6a392cfd-a903-4763-89cf-1ce44302c394/row_count/000000_0 > >>>> > >>>> Environment: > >>>> Kylin 1.6.0 + HDP 2.5 > >>>> I am not sure if HDP 2.5 is a too high version, because I only see HDP > >> 2.4 > >>>> is referred in Kylin doc. > >>>> > >>>> > >>>> Build error: > >>>> 2017-02-04 09:38:17,947 INFO [pool-5-thread-2] > >>>> manager.ExecutableManager:292 : job id:6a392cfd-a903-4763-89cf- > >> 1ce44302c394-01 > >>>> from READY to RUNNING > >>>> 2017-02-04 09:38:17,947 INFO [pool-5-thread-2] > hive.HiveCmdBuilder:81 : > >>>> The statements to execute in beeline: > >>>> USE default; > >>>> SET hive.exec.compress.output=true; > >>>> SET hive.auto.convert.join.noconditionaltask=true; > >>>> SET hive.auto.convert.join.noconditionaltask.size=100000000; > >>>> SET mapreduce.output.fileoutputformat.compress.type=BLOCK; > >>>> SET mapreduce.job.split.metainfo.maxsize=-1; > >>>> > >>>> set hive.exec.compress.output=false; > >>>> > >>>> set hive.exec.compress.output=false; > >>>> INSERT OVERWRITE DIRECTORY '/kylin/kylin_metadata/kylin- > >>>> 6a392cfd-a903-4763-89cf-1ce44302c394/row_count' SELECT count(*) FROM > >>>> kylin_intermediate_kylin_sales_cube_desc_0eb9faec_6a22_ > >>>> 4b49_889f_c283d82d72dd; > >>>> > >>>> > >>>> 2017-02-04 09:38:17,947 DEBUG [pool-5-thread-2] > hive.HiveCmdBuilder:83 : > >>>> The SQL to execute in beeline: > >>>> > >>>> 2017-02-04 09:38:17,947 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : Compute row count of flat hive > table, > >>>> cmd: > >>>> 2017-02-04 09:38:17,947 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : beeline -n root -u 'jdbc:hive2:// > >>>> sandbox.hortonworks.com:2181/;serviceDiscoveryMode= > >>>> zooKeeper;zooKeeperNamespace=hiveserver2' -f > >>>> /root/apache-kylin-1.6.0-hbase1.x-bin/bin/../tomcat/temp/beeline_ > >> 3013188987573907903.hql;rm > >>>> -f /root/apache-kylin-1.6.0-hbase1.x-bin/bin/../tomcat/temp/beeline_ > >>>> 3013188987573907903.hql > >>>> 2017-02-04 09:38:19,832 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch > >> YARN > >>>> applications. > >>>> 2017-02-04 09:38:20,342 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : Connecting to jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/;serviceDiscoveryMode= > >> zooKeeper;zooKeeperNamespace= > >>>> hiveserver2 > >>>> 2017-02-04 09:38:21,629 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : Connected to: Apache Hive (version > >>>> 1.2.1000.2.5.0.0-1245) > >>>> 2017-02-04 09:38:21,630 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : Driver: Hive JDBC (version > >>>> 1.2.1.2.3.2.0-2950) > >>>> 2017-02-04 09:38:21,630 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : Transaction isolation: > >>>> TRANSACTION_REPEATABLE_READ > >>>> 2017-02-04 09:38:21,679 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> USE default; > >>>> 2017-02-04 09:38:21,732 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : No rows affected (0.051 seconds) > >>>> 2017-02-04 09:38:21,746 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> SET hive.exec.compress.output=true; > >>>> 2017-02-04 09:38:21,757 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : No rows affected (0.011 seconds) > >>>> 2017-02-04 09:38:21,766 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> SET hive.auto.convert.join. > >> noconditionaltask=true; > >>>> 2017-02-04 09:38:21,768 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : No rows affected (0.002 seconds) > >>>> 2017-02-04 09:38:21,777 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> SET hive.auto.convert.join. > >> noconditionaltask.size= > >>>> 100000000; > >>>> 2017-02-04 09:38:21,779 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : No rows affected (0.002 seconds) > >>>> 2017-02-04 09:38:21,787 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> SET mapreduce.output. > fileoutputformat.compress. > >>>> type=BLOCK; > >>>> 2017-02-04 09:38:21,790 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : No rows affected (0.003 seconds) > >>>> 2017-02-04 09:38:21,796 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> SET mapreduce.job.split.metainfo.maxsize=-1; > >>>> 2017-02-04 09:38:21,801 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : No rows affected (0.005 seconds) > >>>> 2017-02-04 09:38:21,804 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> > >>>> 2017-02-04 09:38:21,808 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> set hive.exec.compress.output=false; > >>>> 2017-02-04 09:38:21,811 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : No rows affected (0.003 seconds) > >>>> 2017-02-04 09:38:21,812 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> > >>>> 2017-02-04 09:38:21,816 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> set hive.exec.compress.output=false; > >>>> 2017-02-04 09:38:21,818 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : No rows affected (0.002 seconds) > >>>> 2017-02-04 09:38:21,833 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> INSERT OVERWRITE DIRECTORY > >>>> '/kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf-1c > >>>> e44302c394/row_count' SELECT count(*) FROM k > >>>> 2017-02-04 09:38:21,840 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : ylin_intermediate_kylin_sales_ > >>>> cube_desc_0eb9faec_6a22_4b49_889f_c283d82d72dd; > >>>> 2017-02-04 09:38:30,452 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : Tez session hasn't been > >> created > >>>> yet. Opening session > >>>> 2017-02-04 09:38:30,452 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : Dag name: INSERT OVERWRITE > >>>> DIRE...49_889f_c283d82d72dd(Stage-1) > >>>> 2017-02-04 09:38:30,452 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : > >>>> 2017-02-04 09:38:30,452 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : > >>>> 2017-02-04 09:38:30,452 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : Status: Running (Executing > on > >>>> YARN cluster with App id application_1486198519344_0004) > >>>> 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : > >>>> 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : Map 1: -/- Reducer 2: > 0/1 > >>>> 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : Map 1: 0/1 Reducer 2: > 0/1 > >>>> 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : Map 1: 0(+1)/1 > Reducer > >>>> 2: 0/1 > >>>> 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : Map 1: 1/1 Reducer 2: > 0/1 > >>>> 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : Map 1: 1/1 Reducer 2: > >> 0(+1)/1 > >>>> 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : Map 1: 1/1 Reducer 2: > 1/1 > >>>> 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : INFO : Moving data to directory > >>>> /kylin/kylin_metadata/kylin-6a392cfd-a903-4763-89cf- > >> 1ce44302c394/row_count > >>>> from hdfs://sandbox.h > >>>> ortonworks.com:8020/kylin/kylin_metadata/kylin-6a392cfd- > >>>> a903-4763-89cf-1ce44302c394/row_count/.hive-staging_hive_ > >>>> 2017-02-04_09-38-21_841_1217104325959051054-5/-ext-10000 > >>>> 2017-02-04 09:38:30,453 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : No rows affected (8.613 seconds) > >>>> 2017-02-04 09:38:30,457 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> > >>>> 2017-02-04 09:38:30,457 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/> > >>>> 2017-02-04 09:38:30,458 INFO [pool-5-thread-2] > >>>> execution.AbstractExecutable:36 : Closing: 0: jdbc:hive2://sandbox. > >>>> hortonworks.com:2181/;serviceDiscoveryMode= > >> zooKeeper;zooKeeperNamespace= > >>>> hiveserver2 > >>>> 2017-02-04 09:38:30,573 ERROR [pool-5-thread-2] > >>>> execution.AbstractExecutable:370 : job:6a392cfd-a903-4763-89cf- > >> 1ce44302c394-01 > >>>> execute finished with exception > >>>> java.io.FileNotFoundException: File /kylin/kylin_metadata/kylin- > >>>> 6a392cfd-a903-4763-89cf-1ce44302c394/row_count does not exist > >>>> at org.apache.hadoop.fs.RawLocalFileSystem.listStatus( > >>>> RawLocalFileSystem.java:429) > >>>> at org.apache.hadoop.fs.FileSystem.listStatus( > >>>> FileSystem.java:1515) > >>>> at org.apache.hadoop.fs.FileSystem.listStatus( > >>>> FileSystem.java:1555) > >>>> at org.apache.hadoop.fs.ChecksumFileSystem.listStatus( > >>>> ChecksumFileSystem.java:574) > >>>> at org.apache.kylin.source.hive.HiveMRInput$ > >>>> RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:338) > >>>> at org.apache.kylin.job.execution.AbstractExecutable. > >>>> execute(AbstractExecutable.java:113) > >>>> at org.apache.kylin.job.execution.DefaultChainedExecutable. > >> doWork( > >>>> DefaultChainedExecutable.java:57) > >>>> at org.apache.kylin.job.execution.AbstractExecutable. > >>>> execute(AbstractExecutable.java:113) > >>>> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$ > >>>> JobRunner.run(DefaultScheduler.java:136) > >>>> > >>>> > >>>> > >>>> Thx > >>>> Lei Wang > >>>> > >>>> > >>> > >>> > >>> -- > >>> Best regards, > >>> > >>> Shaofeng Shi 史少锋 > >> > > > > > > > > -- > > Best regards, > > > > Shaofeng Shi 史少锋 > -- Best regards, Shaofeng Shi 史少锋
