other idea: Can be a problem with permissions?: the user that execute Kylin can't read data generated by YARN check if Kylin user can read your folder /young/kylin_test/ Which Hadoop user are executing Kylin?
(no more ideas, Good Luck) 2016-12-20 7:51 GMT+01:00 雨日听风 <[email protected]>: > Thank you! > We checked the yarn and hard disk. But not found any error. Hard disk > space and memory and so on is working well. > Last time its error code was "unknownhost clusterB",now in new server env > it cant find clusterB(hbase only). but cant find rowCount file. > === > the follow command runs ok: > hdfs dfs -mkdir /young/kylin_test/kylin_metadata_nokia/ > kylin-678c15ba-5375-4f80-831e-1ae0af8ed576/row_count/tmp > And "ls" cant find file "000000_0" which it said "file does not exist". > > ------------------ 原始邮件 ------------------ > *发件人:* "Alberto Ramón";<[email protected]>; > *发送时间:* 2016年12月19日(星期一) 晚上9:13 > *收件人:* "user"<[email protected]>; > *主题:* Re: Error when #2 Step: Redistribute Flat Hive Table - File does > not exist > > i think i had this error last nigth :) > (go to yarn to find detailed error & find on internet) > in my case was free space less than 10% of hard disk. Check this please > > El 19/12/2016 11:35, "雨日听风" <[email protected]> escribió: > >> When I build a cube in kylin1.6, I get error in step #2: Redistribute >> Flat Hive Table >> >> Please help! Thank you very much! >> >> env: kylin1.6 is in a independent server, and have 2 other server >> cluster: clusterA(hive only) and clusterB(hbase only). >> Error is: >> >> 2016-12-19 10:28:00,641 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Compute row count of flat hive table, >> cmd: >> 2016-12-19 10:28:00,642 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : hive -e "USE boco; >> SET dfs.replication=2; >> SET hive.exec.compress.output=true; >> SET hive.auto.convert.join.noconditionaltask=true; >> SET hive.auto.convert.join.noconditionaltask.size=100000000; >> SET mapreduce.output.fileoutputformat.compress.type=BLOCK; >> SET mapreduce.job.split.metainfo.maxsize=-1; >> SET mapreduce.job.queuename=young; >> SET tez.queue.name=young; >> >> set hive.exec.compress.output=false; >> >> set hive.exec.compress.output=false; >> INSERT OVERWRITE DIRECTORY '/young/kylin_test/kylin_metad >> ata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count' SELECT >> count(*) FROM kylin_intermediate_hbase_in_testCluster_CUBE_f9468805_eabf_ >> 4b54_bf2b_182e4c86214a; >> >> " >> 2016-12-19 10:28:03,277 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch YARN >> applications. >> 2016-12-19 10:28:04,444 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : >> 2016-12-19 10:28:04,445 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Logging initialized using >> configuration in file:/etc/hive/conf/hive-log4j.properties >> 2016-12-19 10:28:14,700 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : OK >> 2016-12-19 10:28:14,703 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Time taken: 0.935 seconds >> 2016-12-19 10:28:15,559 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Query ID = >> young_20161219102814_a7104fd4-ba83-47fc-ac0b-0c9bef4e1969 >> 2016-12-19 10:28:15,560 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Total jobs = 1 >> 2016-12-19 10:28:15,575 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Launching Job 1 out of 1 >> 2016-12-19 10:28:22,842 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : >> 2016-12-19 10:28:22,842 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : >> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Status: Running (Executing on YARN >> cluster with App id application_1473415773736_1063281) >> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : >> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Map 1: -/- Reducer 2: 0/1 >> 2016-12-19 10:28:23,307 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1 >> 2016-12-19 10:28:26,363 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1 >> 2016-12-19 10:28:26,567 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Map 1: 0(+1)/2 Reducer 2: 0/1 >> 2016-12-19 10:28:26,596 INFO [pool-7-thread-1] >> threadpool.DefaultScheduler:118 : Job Fetcher: 1 should running, 1 >> actual running, 0 ready, 0 already succeed, 3 error, 1 discarded, 0 others >> 2016-12-19 10:28:26,769 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1 >> 2016-12-19 10:28:29,810 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1 >> 2016-12-19 10:28:30,217 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Map 1: 1(+1)/2 Reducer 2: 0(+1)/1 >> 2016-12-19 10:28:30,826 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 0(+1)/1 >> 2016-12-19 10:28:31,232 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 1/1 >> 2016-12-19 10:28:31,319 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Moving data to: >> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e- >> 48b4-bdb5-6e578320375a/row_count >> 2016-12-19 10:28:31,406 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : OK >> 2016-12-19 10:28:31,454 INFO [pool-8-thread-7] >> execution.AbstractExecutable:36 : Time taken: 16.701 seconds >> 2016-12-19 10:28:35,074 ERROR [pool-8-thread-7] >> execution.AbstractExecutable:357 : >> job:678266c0-ba0e-48b4-bdb5-6e578320375a-01 >> execute finished with exception >> java.io.FileNotFoundException: File does not exist: >> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e- >> 48b4-bdb5-6e578320375a/row_count/000000_0 >> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo >> deFile.java:71) >> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo >> deFile.java:61) >> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >> LocationsInt(FSNamesystem.java:1820) >> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >> Locations(FSNamesystem.java:1791) >> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >> Locations(FSNamesystem.java:1704) >> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get >> BlockLocations(NameNodeRpcServer.java:587) >> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ >> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS >> erverSideTranslatorPB.java:365) >> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol >> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam >> enodeProtocolProtos.java) >> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn >> voker.call(ProtobufRpcEngine.java:616) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >> upInformation.java:1657) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075) >> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native >> ConstructorAccessorImpl.java:57) >> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De >> legatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >> at org.apache.hadoop.ipc.RemoteException.instantiateException(R >> emoteException.java:106) >> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException( >> RemoteException.java:73) >> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl >> ient.java:1242) >> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient. >> java:1227) >> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient. >> java:1215) >> at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndG >> etLastBlockLength(DFSInputStream.java:303) >> at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStrea >> m.java:269) >> at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:261) >> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540) >> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall( >> DistributedFileSystem.java:303) >> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall( >> DistributedFileSystem.java:299) >> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst >> emLinkResolver.java:81) >> at org.apache.hadoop.hdfs.DistributedFileSystem.open(Distribute >> dFileSystem.java:299) >> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) >> at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv >> eTableStep.readRowCountFromFile(HiveMRInput.java:294) >> at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv >> eTableStep.doWork(HiveMRInput.java:337) >> at org.apache.kylin.job.execution.AbstractExecutable.execute( >> AbstractExecutable.java:113) >> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWo >> rk(DefaultChainedExecutable.java:57) >> at org.apache.kylin.job.execution.AbstractExecutable.execute( >> AbstractExecutable.java:113) >> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRun >> ner.run(DefaultScheduler.java:136) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1145) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: >> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): >> File does not exist: /young/kylin_test/kylin_metada >> ta_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0 >> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo >> deFile.java:71) >> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo >> deFile.java:61) >> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >> LocationsInt(FSNamesystem.java:1820) >> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >> Locations(FSNamesystem.java:1791) >> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >> Locations(FSNamesystem.java:1704) >> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get >> BlockLocations(NameNodeRpcServer.java:587) >> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ >> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS >> erverSideTranslatorPB.java:365) >> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol >> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam >> enodeProtocolProtos.java) >> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn >> voker.call(ProtobufRpcEngine.java:616) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >> upInformation.java:1657) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1427) >> at org.apache.hadoop.ipc.Client.call(Client.java:1358) >> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke( >> ProtobufRpcEngine.java:229) >> at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source) >> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran >> slatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >> ssorImpl.java:57) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth >> od(RetryInvocationHandler.java:252) >> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret >> ryInvocationHandler.java:104) >> at com.sun.proxy.$Proxy26.getBlockLocations(Unknown Source) >> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl >> ient.java:1240) >> ... 20 more >> 2016-12-19 10:28:35,078 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : >> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01 >> 2016-12-19 10:28:35,082 DEBUG [pool-8-thread-7] >> hbase.HBaseResourceStore:262 : Update row >> /execute_output/678266c0-ba0e-48b4-bdb5-6e578320375a-01 >> from oldTs: 1482114480638, to newTs: 1482114515078, operation result: true >> 2016-12-19 10:28:35,085 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : >> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01 >> >
