Hi Nan, This error indicates that the hadoop configuration of the HBase cluster has overwriten the configuration of the default hadoop cluster. As you know, Kylin uses the output of "hbase classpath" as the classpath to startup; so it is possible that the hbase cluster's "core-site.xml", "hdfs-site.xml" are on the classpath's starting position. Please locate them, take backup, and then replace them with the hive/mapreduce cluster's config file; only left "hbase-site.xml" with the configurations to the dedicated hbase cluster.
After doing that, restart Kylin, discard the error job, and resubmit a build. 2017-01-09 23:12 GMT+08:00 柯南 <[email protected]>: > hi,all： > I want to deploy apache kylin with standalone HBase cluster,and also > refer offical doc(http://kylin.apache.org/blog/2016/06/10/standalone- > hbase-cluster) to update the config kylin.hbase.cluster.fs in > kylin.properties, but when I build new cube in step2(Redistribute Flat Hive > Table), it is using hadoop client whch hbase cluster is dependent on, but > not main cluster's. Thus error that file in hdfs couldn't be find occur. > The program get hadoop configuration using > HadoopUtil.getCurrentConfiguration() > in CreateFlatHiveTableStep class. > The version of Kylin upgade 1.6.0 is the same. > > > > > 2017-01-04 18:42:43,459 ERROR [pool-7-thread-3] > hive.CreateFlatHiveTableStep:114 : job:3889515b-0054-4b71-9db0-615b1ceab3bc-01 > execute finished with exception > java.io.FileNotFoundException: File does not exist: > /user/kylin/kylin_metadata/kylin-3889515b-0054-4b71-9db0- > 615b1ceab3bc/row_count/000000_0 > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > INodeFile.java:65) > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > INodeFile.java:55) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > getBlockLocationsUpdateTimes(FSNamesystem.java:1879) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > getBlockLocationsInt(FSNamesystem.java:1820) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > getBlockLocations(FSNamesystem.java:1800) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > getBlockLocations(FSNamesystem.java:1772) > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. > getBlockLocations(NameNodeRpcServer.java:527) > at org.apache.hadoop.hdfs.server.namenode. > AuthorizationProviderProxyClientProtocol.getBlockLocations( > AuthorizationProviderProxyClientProtocol.java:85) > at org.apache.hadoop.hdfs.protocolPB. > ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations( > ClientNamenodeProtocolServerSideTranslatorPB.java:356) > at org.apache.hadoop.hdfs.protocol.proto. > ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod( > ClientNamenodeProtocolProtos.java) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ > ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1642) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:57) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at org.apache.hadoop.ipc.RemoteException.instantiateException( > RemoteException.java:106) > at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException( > RemoteException.java:73) > at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations( > DFSClient.java:1171) > at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks( > DFSClient.java:1159) > at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks( > DFSClient.java:1149) > at org.apache.hadoop.hdfs.DFSInputStream. > fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270) > at org.apache.hadoop.hdfs.DFSInputStream.openInfo( > DFSInputStream.java:237) > at org.apache.hadoop.hdfs.DFSInputStream.<init>( > DFSInputStream.java:230) > at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1448) > at org.apache.hadoop.hdfs.DistributedFileSystem$3. > doCall(DistributedFileSystem.java:301) > at org.apache.hadoop.hdfs.DistributedFileSystem$3. > doCall(DistributedFileSystem.java:297) > at org.apache.hadoop.fs.FileSystemLinkResolver.resolve( > FileSystemLinkResolver.java:81) > at org.apache.hadoop.hdfs.DistributedFileSystem.open( > DistributedFileSystem.java:297) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) > at org.apache.kylin.source.hive.CreateFlatHiveTableStep. > readRowCountFromFile(CreateFlatHiveTableStep.java:51) > at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork( > CreateFlatHiveTableStep.java:103) > at org.apache.kylin.job.execution.AbstractExecutable. > execute(AbstractExecutable.java:112) > at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork( > DefaultChainedExecutable.java:57) > at org.apache.kylin.job.execution.AbstractExecutable. > execute(AbstractExecutable.java:112) > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$ > JobRunner.run(DefaultScheduler.java:127) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): > File does not exist: /user/kylin/kylin_metadata/ > kylin-3889515b-0054-4b71-9db0-615b1ceab3bc/row_count/000000_0 > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > INodeFile.java:65) > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > INodeFile.java:55) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > getBlockLocationsUpdateTimes(FSNamesystem.java:1879) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > getBlockLocationsInt(FSNamesystem.java:1820) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > getBlockLocations(FSNamesystem.java:1800) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > getBlockLocations(FSNamesystem.java:1772) > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. > getBlockLocations(NameNodeRpcServer.java:527) > at org.apache.hadoop.hdfs.server.namenode. > AuthorizationProviderProxyClientProtocol.getBlockLocations( > AuthorizationProviderProxyClientProtocol.java:85) > at org.apache.hadoop.hdfs.protocolPB. > ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations( > ClientNamenodeProtocolServerSideTranslatorPB.java:356) > at org.apache.hadoop.hdfs.protocol.proto. > ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod( > ClientNamenodeProtocolProtos.java) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ > ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1642) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > > > at org.apache.hadoop.ipc.Client.call(Client.java:1411) > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy23.getBlockLocations(Unknown Source) > at org.apache.hadoop.hdfs.protocolPB. > ClientNamenodeProtocolTranslatorPB.getBlockLocations( > ClientNamenodeProtocolTranslatorPB.java:246) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod( > RetryInvocationHandler.java:187) > at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy24.getBlockLocations(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations( > DFSClient.java:1169) > ... 20 more -- Best regards, Shaofeng Shi 史少锋
