Hi Andras, Did you try this? https://kylin.apache.org/docs/howto/howto_update_coprocessor.html
Best regards, Shaofeng Shi 史少锋 Apache Kylin PMC Email: [email protected] Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html Join Kylin user mail group: [email protected] Join Kylin dev mail group: [email protected] Andras Nagy <[email protected]> 于2019年6月26日周三 下午10:05写道： > Greetings, > > I'm testing a setup where HBase is running on AWS EMR and HBase data is > stored on S3. It's working fine so far, but when I terminate the EMR > cluster and recreate it with the same S3 location for HBase, HBase won't > start up properly. Before shutting down, I did execute the > disable_all_tables.sh script to flush HBase state to S3. > > Actually the issue is that RegionServers don't start up. Maybe I'm missing > something in the EMR setup and not in Kylin setup, but the exceptions I get > in the RegionServer's log point at Kylin's CubeVisitService coprocessor, > which is still trying to connect to the old HBase master on the old EMR > cluster's master node and fails with: "coprocessor.CoprocessorHost: The > coprocessor > org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService > threw java.net.NoRouteToHostException: No Route to Host from > ip-172-35-5-11/172.35.5.11 to > ip-172-35-7-125.us-west-2.compute.internal:8020 failed on socket timeout > exception: java.net.NoRouteToHostException: No route to host; " > > (Here, ip-172-35-7-125 was the old clusters' master node.) > > Does anyone have any idea what I'm doing wrong here? > The HBase master node's address seems to be cached somewhere, and when > starting up HBase on the new cluster with the same S3 location for HFiles, > this old address is used still. > Is there anything specific I have missed to get this scenario to work > properly? > > This is the full stacktrace: > > 2019-06-26 12:33:53,352 ERROR [RS_OPEN_REGION-ip-172-35-5-11:16020-1] > coprocessor.CoprocessorHost: The coprocessor > org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService > threw java.net.NoRouteToHostException: No Route to Host from > ip-172-35-5-11/172.35.5.11 to > ip-172-35-7-125.us-west-2.compute.internal:8020 failed on socket timeout > exception: java.net.NoRouteToHostException: No route to host; For more > details see: http://wiki.apache.org/hadoop/NoRouteToHost > java.net.NoRouteToHostException: No Route to Host from ip-172-35-5-11/ > 172.35.5.11 to ip-172-35-7-125.us-west-2.compute.internal:8020 failed on > socket timeout exception: java.net.NoRouteToHostException: No route to > host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:758) > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1493) > at org.apache.hadoop.ipc.Client.call(Client.java:1435) > at org.apache.hadoop.ipc.Client.call(Client.java:1345) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy36.getFileInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:796) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) > at com.sun.proxy.$Proxy37.getFileInfo(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1649) > at > org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1440) > at > org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1437) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1452) > at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1466) > at > org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:264) > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:214) > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:188) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:376) > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:238) > at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:802) > at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:710) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6716) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7020) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6992) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6948) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6899) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) > at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685) > at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788) > at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1550) > at org.apache.hadoop.ipc.Client.call(Client.java:1381) > ... 43 more > > Many thanks, > Andras >
