Thank you! It works,and the file "yarn-site.xml" and "mapred-site.xml" also should be replaced.
------------------ ???????? ------------------ ??????: "ShaoFeng Shi";<[email protected]>; ????????: 2017??1??10??(??????) ????6:05 ??????: "dev"<[email protected]>; ????: Re: Error in kylin with standalone HBase cluster Hi, Firstly you need install Kylin in a dedicated machine (pure client machine to run Kylin, not a hadoop job node nor HBase RS), so that replace the "core-site.xml" or "hdfs-site.xml" will not impact on other services. Secondly, Kylin's communication with HBase all goes to HBase API/RPC, that means Kylin only need the "hbase-site.xml". So replace the "core-site.xml" or "hdfs-site.xml" in HBase client folder (only this machine) doesn't impact. Thirdly, configure "kylin.hbase.cluster.fs" in kylin.properties is telling Kylin that HBase is using another HDFS (not the default one) ; so after the cube be calculated in the default cluster, Kylin will write the HFile to this HDFS. After all these done, Kylin will use HBase's API to do bulkload; HBase region server will load the files from their own HDFS directly, which are already in place. Hope this helps. 2017-01-10 14:10 GMT+08:00 ???? <[email protected]>: > hi Shao feng, > > Thank you for your advice. > As you said, if I replace the "core-site.xml" and "hdfs-site.xml" with the > hive/mapreduce cluster's config file, it means that the hbase and > hive/mapreduce cluster's are all dependent on the same hadoop client, but > not deploying with standalone HBase cluster. When they dependent on the > same hadoop client, I know it works well. > On the other hand, the property "kylin.hbase.cluster.fs" in > kylin.properties,when to use for? > If I need to dependent hbase and hive/mapreduce cluster's on diffent hdfs, > counld kylin support that? Or may be in the future? > > > ------------------ ???????? ------------------ > ??????: "ShaoFeng Shi";<[email protected]>; > ????????: 2017??1??10??(??????) ????10:35 > ??????: "dev"<[email protected]>; > > ????: Re: Error in kylin with standalone HBase cluster > > > > Hi Nan, > > This error indicates that the hadoop configuration of the HBase cluster has > overwriten the configuration of the default hadoop cluster. As you know, > Kylin uses the output of "hbase classpath" as the classpath to startup; so > it is possible that the hbase cluster's "core-site.xml", "hdfs-site.xml" > are on the classpath's starting position. Please locate them, take backup, > and then replace them with the hive/mapreduce cluster's config file; only > left "hbase-site.xml" with the configurations to the dedicated hbase > cluster. > > After doing that, restart Kylin, discard the error job, and resubmit a > build. > > > > 2017-01-09 23:12 GMT+08:00 ???? <[email protected]>: > > > hi,all?? > > I want to deploy apache kylin with standalone HBase cluster,and also > > refer offical doc(http://kylin.apache.org/blog/2016/06/10/standalone- > > hbase-cluster) to update the config kylin.hbase.cluster.fs in > > kylin.properties, but when I build new cube in step2(Redistribute Flat > Hive > > Table), it is using hadoop client whch hbase cluster is dependent on, but > > not main cluster's. Thus error that file in hdfs couldn't be find occur. > > The program get hadoop configuration using HadoopUtil. > getCurrentConfiguration() > > in CreateFlatHiveTableStep class. > > The version of Kylin upgade 1.6.0 is the same. > > > > > > > > > > 2017-01-04 18:42:43,459 ERROR [pool-7-thread-3] > > hive.CreateFlatHiveTableStep:114 : job:3889515b-0054-4b71-9db0- > 615b1ceab3bc-01 > > execute finished with exception > > java.io.FileNotFoundException: File does not exist: > > /user/kylin/kylin_metadata/kylin-3889515b-0054-4b71-9db0- > > 615b1ceab3bc/row_count/000000_0 > > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > > INodeFile.java:65) > > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > > INodeFile.java:55) > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > getBlockLocationsUpdateTimes(FSNamesystem.java:1879) > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > getBlockLocationsInt(FSNamesystem.java:1820) > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > getBlockLocations(FSNamesystem.java:1800) > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > getBlockLocations(FSNamesystem.java:1772) > > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. > > getBlockLocations(NameNodeRpcServer.java:527) > > at org.apache.hadoop.hdfs.server.namenode. > > AuthorizationProviderProxyClientProtocol.getBlockLocations( > > AuthorizationProviderProxyClientProtocol.java:85) > > at org.apache.hadoop.hdfs.protocolPB. > > ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations( > > ClientNamenodeProtocolServerSideTranslatorPB.java:356) > > at org.apache.hadoop.hdfs.protocol.proto. > > ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2. > callBlockingMethod( > > ClientNamenodeProtocolProtos.java) > > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ > > ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at org.apache.hadoop.security.UserGroupInformation.doAs( > > UserGroupInformation.java:1642) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > > > > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > Method) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance( > > NativeConstructorAccessorImpl.java:57) > > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > > DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java: > 526) > > at org.apache.hadoop.ipc.RemoteException.instantiateException( > > RemoteException.java:106) > > at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException( > > RemoteException.java:73) > > at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations( > > DFSClient.java:1171) > > at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks( > > DFSClient.java:1159) > > at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks( > > DFSClient.java:1149) > > at org.apache.hadoop.hdfs.DFSInputStream. > > fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270) > > at org.apache.hadoop.hdfs.DFSInputStream.openInfo( > > DFSInputStream.java:237) > > at org.apache.hadoop.hdfs.DFSInputStream.<init>( > > DFSInputStream.java:230) > > at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1448) > > at org.apache.hadoop.hdfs.DistributedFileSystem$3. > > doCall(DistributedFileSystem.java:301) > > at org.apache.hadoop.hdfs.DistributedFileSystem$3. > > doCall(DistributedFileSystem.java:297) > > at org.apache.hadoop.fs.FileSystemLinkResolver.resolve( > > FileSystemLinkResolver.java:81) > > at org.apache.hadoop.hdfs.DistributedFileSystem.open( > > DistributedFileSystem.java:297) > > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) > > at org.apache.kylin.source.hive.CreateFlatHiveTableStep. > > readRowCountFromFile(CreateFlatHiveTableStep.java:51) > > at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork( > > CreateFlatHiveTableStep.java:103) > > at org.apache.kylin.job.execution.AbstractExecutable. > > execute(AbstractExecutable.java:112) > > at org.apache.kylin.job.execution.DefaultChainedExecutable. > doWork( > > DefaultChainedExecutable.java:57) > > at org.apache.kylin.job.execution.AbstractExecutable. > > execute(AbstractExecutable.java:112) > > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$ > > JobRunner.run(DefaultScheduler.java:127) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1145) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: org.apache.hadoop.ipc.RemoteException(java.io. > FileNotFoundException): > > File does not exist: /user/kylin/kylin_metadata/ > > kylin-3889515b-0054-4b71-9db0-615b1ceab3bc/row_count/000000_0 > > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > > INodeFile.java:65) > > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > > INodeFile.java:55) > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > getBlockLocationsUpdateTimes(FSNamesystem.java:1879) > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > getBlockLocationsInt(FSNamesystem.java:1820) > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > getBlockLocations(FSNamesystem.java:1800) > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > getBlockLocations(FSNamesystem.java:1772) > > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. > > getBlockLocations(NameNodeRpcServer.java:527) > > at org.apache.hadoop.hdfs.server.namenode. > > AuthorizationProviderProxyClientProtocol.getBlockLocations( > > AuthorizationProviderProxyClientProtocol.java:85) > > at org.apache.hadoop.hdfs.protocolPB. > > ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations( > > ClientNamenodeProtocolServerSideTranslatorPB.java:356) > > at org.apache.hadoop.hdfs.protocol.proto. > > ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2. > callBlockingMethod( > > ClientNamenodeProtocolProtos.java) > > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ > > ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at org.apache.hadoop.security.UserGroupInformation.doAs( > > UserGroupInformation.java:1642) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > > > > > > at org.apache.hadoop.ipc.Client.call(Client.java:1411) > > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > > at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > > invoke(ProtobufRpcEngine.java:206) > > at com.sun.proxy.$Proxy23.getBlockLocations(Unknown Source) > > at org.apache.hadoop.hdfs.protocolPB. > > ClientNamenodeProtocolTranslatorPB.getBlockLocations( > > ClientNamenodeProtocolTranslatorPB.java:246) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java:57) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at org.apache.hadoop.io.retry.RetryInvocationHandler. > invokeMethod( > > RetryInvocationHandler.java:187) > > at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > > RetryInvocationHandler.java:102) > > at com.sun.proxy.$Proxy24.getBlockLocations(Unknown Source) > > at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations( > > DFSClient.java:1169) > > ... 20 more > > > > > -- > Best regards, > > Shaofeng Shi ?????? > -- Best regards, Shaofeng Shi ??????
