hi Shao feng,
Thank you for your advice.
As you said, if I replace the "core-site.xml" and "hdfs-site.xml" with the
hive/mapreduce cluster's config file, it means that the hbase and
hive/mapreduce cluster's are all dependent on the same hadoop client, but not
deploying with standalone HBase cluster. When they dependent on the same hadoop
client, I know it works well.
On the other hand, the property "kylin.hbase.cluster.fs" in
kylin.properties,when to use for?
If I need to dependent hbase and hive/mapreduce cluster's on diffent hdfs,
counld kylin support that? Or may be in the future?
------------------ ???????? ------------------
??????: "ShaoFeng Shi";<[email protected]>;
????????: 2017??1??10??(??????) ????10:35
??????: "dev"<[email protected]>;
????: Re: Error in kylin with standalone HBase cluster
Hi Nan,
This error indicates that the hadoop configuration of the HBase cluster has
overwriten the configuration of the default hadoop cluster. As you know,
Kylin uses the output of "hbase classpath" as the classpath to startup; so
it is possible that the hbase cluster's "core-site.xml", "hdfs-site.xml"
are on the classpath's starting position. Please locate them, take backup,
and then replace them with the hive/mapreduce cluster's config file; only
left "hbase-site.xml" with the configurations to the dedicated hbase
cluster.
After doing that, restart Kylin, discard the error job, and resubmit a
build.
2017-01-09 23:12 GMT+08:00 ???? <[email protected]>:
> hi,all??
> I want to deploy apache kylin with standalone HBase cluster,and also
> refer offical doc(http://kylin.apache.org/blog/2016/06/10/standalone-
> hbase-cluster) to update the config kylin.hbase.cluster.fs in
> kylin.properties, but when I build new cube in step2(Redistribute Flat Hive
> Table), it is using hadoop client whch hbase cluster is dependent on, but
> not main cluster's. Thus error that file in hdfs couldn't be find occur.
> The program get hadoop configuration using
> HadoopUtil.getCurrentConfiguration()
> in CreateFlatHiveTableStep class.
> The version of Kylin upgade 1.6.0 is the same.
>
>
>
>
> 2017-01-04 18:42:43,459 ERROR [pool-7-thread-3]
> hive.CreateFlatHiveTableStep:114 : job:3889515b-0054-4b71-9db0-615b1ceab3bc-01
> execute finished with exception
> java.io.FileNotFoundException: File does not exist:
> /user/kylin/kylin_metadata/kylin-3889515b-0054-4b71-9db0-
> 615b1ceab3bc/row_count/000000_0
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> INodeFile.java:65)
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> INodeFile.java:55)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocationsUpdateTimes(FSNamesystem.java:1879)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocationsInt(FSNamesystem.java:1820)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocations(FSNamesystem.java:1800)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocations(FSNamesystem.java:1772)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> getBlockLocations(NameNodeRpcServer.java:527)
> at org.apache.hadoop.hdfs.server.namenode.
> AuthorizationProviderProxyClientProtocol.getBlockLocations(
> AuthorizationProviderProxyClientProtocol.java:85)
> at org.apache.hadoop.hdfs.protocolPB.
> ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(
> ClientNamenodeProtocolServerSideTranslatorPB.java:356)
> at org.apache.hadoop.hdfs.protocol.proto.
> ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
> ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1642)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(
> RemoteException.java:106)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(
> RemoteException.java:73)
> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(
> DFSClient.java:1171)
> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(
> DFSClient.java:1159)
> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(
> DFSClient.java:1149)
> at org.apache.hadoop.hdfs.DFSInputStream.
> fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270)
> at org.apache.hadoop.hdfs.DFSInputStream.openInfo(
> DFSInputStream.java:237)
> at org.apache.hadoop.hdfs.DFSInputStream.<init>(
> DFSInputStream.java:230)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1448)
> at org.apache.hadoop.hdfs.DistributedFileSystem$3.
> doCall(DistributedFileSystem.java:301)
> at org.apache.hadoop.hdfs.DistributedFileSystem$3.
> doCall(DistributedFileSystem.java:297)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
> FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(
> DistributedFileSystem.java:297)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
> at org.apache.kylin.source.hive.CreateFlatHiveTableStep.
> readRowCountFromFile(CreateFlatHiveTableStep.java:51)
> at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(
> CreateFlatHiveTableStep.java:103)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:112)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:57)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:112)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:127)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException):
> File does not exist: /user/kylin/kylin_metadata/
> kylin-3889515b-0054-4b71-9db0-615b1ceab3bc/row_count/000000_0
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> INodeFile.java:65)
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> INodeFile.java:55)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocationsUpdateTimes(FSNamesystem.java:1879)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocationsInt(FSNamesystem.java:1820)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocations(FSNamesystem.java:1800)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocations(FSNamesystem.java:1772)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> getBlockLocations(NameNodeRpcServer.java:527)
> at org.apache.hadoop.hdfs.server.namenode.
> AuthorizationProviderProxyClientProtocol.getBlockLocations(
> AuthorizationProviderProxyClientProtocol.java:85)
> at org.apache.hadoop.hdfs.protocolPB.
> ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(
> ClientNamenodeProtocolServerSideTranslatorPB.java:356)
> at org.apache.hadoop.hdfs.protocol.proto.
> ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
> ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1642)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.getBlockLocations(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.
> ClientNamenodeProtocolTranslatorPB.getBlockLocations(
> ClientNamenodeProtocolTranslatorPB.java:246)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:187)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy24.getBlockLocations(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(
> DFSClient.java:1169)
> ... 20 more
--
Best regards,
Shaofeng Shi ??????
