Good; enjoy! 2017-01-11 18:01 GMT+08:00 柯南 <[email protected]>:
> Thank you! It works,and the file "yarn-site.xml" and "mapred-site.xml" > also should be replaced. > > > > > ------------------ 原始邮件 ------------------ > 发件人: "ShaoFeng Shi";<[email protected]>; > 发送时间: 2017年1月10日(星期二) 晚上6:05 > 收件人: "dev"<[email protected]>; > > 主题: Re: Error in kylin with standalone HBase cluster > > > > Hi, > > Firstly you need install Kylin in a dedicated machine (pure client machine > to run Kylin, not a hadoop job node nor HBase RS), so that replace the > "core-site.xml" or "hdfs-site.xml" will not impact on other services. > > Secondly, Kylin's communication with HBase all goes to HBase API/RPC, that > means Kylin only need the "hbase-site.xml". So replace the "core-site.xml" > or "hdfs-site.xml" in HBase client folder (only this machine) doesn't > impact. > > Thirdly, configure "kylin.hbase.cluster.fs" in kylin.properties is telling > Kylin that HBase is using another HDFS (not the default one) ; so after the > cube be calculated in the default cluster, Kylin will write the HFile to > this HDFS. > > After all these done, Kylin will use HBase's API to do bulkload; HBase > region server will load the files from their own HDFS directly, which are > already in place. > > Hope this helps. > > 2017-01-10 14:10 GMT+08:00 柯南 <[email protected]>: > > > hi Shao feng, > > > > Thank you for your advice. > > As you said, if I replace the "core-site.xml" and "hdfs-site.xml" with > the > > hive/mapreduce cluster's config file, it means that the hbase and > > hive/mapreduce cluster's are all dependent on the same hadoop client, but > > not deploying with standalone HBase cluster. When they dependent on the > > same hadoop client, I know it works well. > > On the other hand, the property "kylin.hbase.cluster.fs" in > > kylin.properties,when to use for? > > If I need to dependent hbase and hive/mapreduce cluster's on diffent > hdfs, > > counld kylin support that? Or may be in the future? > > > > > > ------------------ 原始邮件 ------------------ > > 发件人: "ShaoFeng Shi";<[email protected]>; > > 发送时间: 2017年1月10日(星期二) 上午10:35 > > 收件人: "dev"<[email protected]>; > > > > 主题: Re: Error in kylin with standalone HBase cluster > > > > > > > > Hi Nan, > > > > This error indicates that the hadoop configuration of the HBase cluster > has > > overwriten the configuration of the default hadoop cluster. As you know, > > Kylin uses the output of "hbase classpath" as the classpath to startup; > so > > it is possible that the hbase cluster's "core-site.xml", "hdfs-site.xml" > > are on the classpath's starting position. Please locate them, take > backup, > > and then replace them with the hive/mapreduce cluster's config file; only > > left "hbase-site.xml" with the configurations to the dedicated hbase > > cluster. > > > > After doing that, restart Kylin, discard the error job, and resubmit a > > build. > > > > > > > > 2017-01-09 23:12 GMT+08:00 柯南 <[email protected]>: > > > > > hi,all： > > > I want to deploy apache kylin with standalone HBase cluster,and > also > > > refer offical doc(http://kylin.apache.org/blog/2016/06/10/standalone- > > > hbase-cluster) to update the config kylin.hbase.cluster.fs in > > > kylin.properties, but when I build new cube in step2(Redistribute Flat > > Hive > > > Table), it is using hadoop client whch hbase cluster is dependent on, > but > > > not main cluster's. Thus error that file in hdfs couldn't be find > occur. > > > The program get hadoop configuration using HadoopUtil. > > getCurrentConfiguration() > > > in CreateFlatHiveTableStep class. > > > The version of Kylin upgade 1.6.0 is the same. > > > > > > > > > > > > > > > 2017-01-04 18:42:43,459 ERROR [pool-7-thread-3] > > > hive.CreateFlatHiveTableStep:114 : job:3889515b-0054-4b71-9db0- > > 615b1ceab3bc-01 > > > execute finished with exception > > > java.io.FileNotFoundException: File does not exist: > > > /user/kylin/kylin_metadata/kylin-3889515b-0054-4b71-9db0- > > > 615b1ceab3bc/row_count/000000_0 > > > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > > > INodeFile.java:65) > > > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > > > INodeFile.java:55) > > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > > getBlockLocationsUpdateTimes(FSNamesystem.java:1879) > > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > > getBlockLocationsInt(FSNamesystem.java:1820) > > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > > getBlockLocations(FSNamesystem.java:1800) > > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > > getBlockLocations(FSNamesystem.java:1772) > > > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. > > > getBlockLocations(NameNodeRpcServer.java:527) > > > at org.apache.hadoop.hdfs.server.namenode. > > > AuthorizationProviderProxyClientProtocol.getBlockLocations( > > > AuthorizationProviderProxyClientProtocol.java:85) > > > at org.apache.hadoop.hdfs.protocolPB. > > > ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations( > > > ClientNamenodeProtocolServerSideTranslatorPB.java:356) > > > at org.apache.hadoop.hdfs.protocol.proto. > > > ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2. > > callBlockingMethod( > > > ClientNamenodeProtocolProtos.java) > > > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ > > > ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java: > 2013) > > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java: > 2009) > > > at java.security.AccessController.doPrivileged(Native Method) > > > at javax.security.auth.Subject.doAs(Subject.java:415) > > > at org.apache.hadoop.security.UserGroupInformation.doAs( > > > UserGroupInformation.java:1642) > > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > > > > > > > > > at sun.reflect.NativeConstructorAccessorImpl. > newInstance0(Native > > > Method) > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance( > > > NativeConstructorAccessorImpl.java:57) > > > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > > > DelegatingConstructorAccessorImpl.java:45) > > > at java.lang.reflect.Constructor.newInstance(Constructor.java: > > 526) > > > at org.apache.hadoop.ipc.RemoteException.instantiateException( > > > RemoteException.java:106) > > > at org.apache.hadoop.ipc.RemoteException. > unwrapRemoteException( > > > RemoteException.java:73) > > > at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations( > > > DFSClient.java:1171) > > > at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks( > > > DFSClient.java:1159) > > > at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks( > > > DFSClient.java:1149) > > > at org.apache.hadoop.hdfs.DFSInputStream. > > > fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270) > > > at org.apache.hadoop.hdfs.DFSInputStream.openInfo( > > > DFSInputStream.java:237) > > > at org.apache.hadoop.hdfs.DFSInputStream.<init>( > > > DFSInputStream.java:230) > > > at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1448) > > > at org.apache.hadoop.hdfs.DistributedFileSystem$3. > > > doCall(DistributedFileSystem.java:301) > > > at org.apache.hadoop.hdfs.DistributedFileSystem$3. > > > doCall(DistributedFileSystem.java:297) > > > at org.apache.hadoop.fs.FileSystemLinkResolver.resolve( > > > FileSystemLinkResolver.java:81) > > > at org.apache.hadoop.hdfs.DistributedFileSystem.open( > > > DistributedFileSystem.java:297) > > > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) > > > at org.apache.kylin.source.hive.CreateFlatHiveTableStep. > > > readRowCountFromFile(CreateFlatHiveTableStep.java:51) > > > at org.apache.kylin.source.hive.CreateFlatHiveTableStep. > doWork( > > > CreateFlatHiveTableStep.java:103) > > > at org.apache.kylin.job.execution.AbstractExecutable. > > > execute(AbstractExecutable.java:112) > > > at org.apache.kylin.job.execution.DefaultChainedExecutable. > > doWork( > > > DefaultChainedExecutable.java:57) > > > at org.apache.kylin.job.execution.AbstractExecutable. > > > execute(AbstractExecutable.java:112) > > > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$ > > > JobRunner.run(DefaultScheduler.java:127) > > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > > ThreadPoolExecutor.java:1145) > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:745) > > > Caused by: org.apache.hadoop.ipc.RemoteException(java.io. > > FileNotFoundException): > > > File does not exist: /user/kylin/kylin_metadata/ > > > kylin-3889515b-0054-4b71-9db0-615b1ceab3bc/row_count/000000_0 > > > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > > > INodeFile.java:65) > > > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf( > > > INodeFile.java:55) > > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > > getBlockLocationsUpdateTimes(FSNamesystem.java:1879) > > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > > getBlockLocationsInt(FSNamesystem.java:1820) > > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > > getBlockLocations(FSNamesystem.java:1800) > > > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem. > > > getBlockLocations(FSNamesystem.java:1772) > > > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. > > > getBlockLocations(NameNodeRpcServer.java:527) > > > at org.apache.hadoop.hdfs.server.namenode. > > > AuthorizationProviderProxyClientProtocol.getBlockLocations( > > > AuthorizationProviderProxyClientProtocol.java:85) > > > at org.apache.hadoop.hdfs.protocolPB. > > > ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations( > > > ClientNamenodeProtocolServerSideTranslatorPB.java:356) > > > at org.apache.hadoop.hdfs.protocol.proto. > > > ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2. > > callBlockingMethod( > > > ClientNamenodeProtocolProtos.java) > > > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ > > > ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) > > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java: > 2013) > > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java: > 2009) > > > at java.security.AccessController.doPrivileged(Native Method) > > > at javax.security.auth.Subject.doAs(Subject.java:415) > > > at org.apache.hadoop.security.UserGroupInformation.doAs( > > > UserGroupInformation.java:1642) > > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) > > > > > > > > > at org.apache.hadoop.ipc.Client.call(Client.java:1411) > > > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > > > at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > > > invoke(ProtobufRpcEngine.java:206) > > > at com.sun.proxy.$Proxy23.getBlockLocations(Unknown Source) > > > at org.apache.hadoop.hdfs.protocolPB. > > > ClientNamenodeProtocolTranslatorPB.getBlockLocations( > > > ClientNamenodeProtocolTranslatorPB.java:246) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java:57) > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:606) > > > at org.apache.hadoop.io.retry.RetryInvocationHandler. > > invokeMethod( > > > RetryInvocationHandler.java:187) > > > at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > > > RetryInvocationHandler.java:102) > > > at com.sun.proxy.$Proxy24.getBlockLocations(Unknown Source) > > > at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations( > > > DFSClient.java:1169) > > > ... 20 more > > > > > > > > > > -- > > Best regards, > > > > Shaofeng Shi 史少锋 > > > > > > -- > Best regards, > > Shaofeng Shi 史少锋 > -- Best regards, Shaofeng Shi 史少锋
