If HBase RS is running together with YARN NodeManager, when building Cube, MR may impact HBase's stability.
To ensure the query's availability and performance, it is recommended to separate HBase RS and NodeManager; further more you can setup a dedicated HBase cluster for query, check this: https://kylin.apache.org/blog/2016/06/10/standalone-hbase-cluster/ 2017-03-14 21:37 GMT+08:00 rahulsingh <[email protected]>: > Hi all, > > I am facing this exception while building the cube. Whenever I build the > cube the region server goes down. I am not understanding this exception. > Please explain is anyone know about this exception. > > Thank You, > Rahul Singh > > java.net.SocketTimeoutException: callTimeout=60000, callDuration=87372: > row > '/execute_output/266df361-5b32-4007-8164-4d8c1972616f-02' on table > 'kylin_metadata' at > region=kylin_metadata,,1487943010804.31525a8d9285b5a15424c6f698838c10., > hostname=hrdn3,60020,1489490883750, seqNum=14429 > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries( > RpcRetryingCaller.java:159) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:935) > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:901) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable( > HBaseResourceStore.java:332) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable( > HBaseResourceStore.java:312) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl( > HBaseResourceStore.java:224) > at > org.apache.kylin.common.persistence.ResourceStore. > getResource(ResourceStore.java:140) > at > org.apache.kylin.job.dao.ExecutableDao.readJobOutputResource( > ExecutableDao.java:93) > at > org.apache.kylin.job.dao.ExecutableDao.getJobOutput( > ExecutableDao.java:186) > at > org.apache.kylin.job.manager.ExecutableManager.getOutput( > ExecutableManager.java:123) > at > org.apache.kylin.job.execution.AbstractExecutable.isDiscarded( > AbstractExecutable.java:369) > at > org.apache.kylin.engine.mr.common.MapReduceExecutable. > doWork(MapReduceExecutable.java:148) > at > org.apache.kylin.job.execution.AbstractExecutable. > execute(AbstractExecutable.java:113) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork( > DefaultChainedExecutable.java:57) > at > org.apache.kylin.job.execution.AbstractExecutable. > execute(AbstractExecutable.java:113) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run( > DefaultScheduler.java:136) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server > is > in the failed servers list: HRDN1/192.168.101.34:60020 > at > org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection. > setupIOstreams(RpcClientImpl.java:708) > at > org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection. > writeRequest(RpcClientImpl.java:907) > at > org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest( > RpcClientImpl.java:874) > at org.apache.hadoop.hbase.ipc.RpcClientImpl.call( > RpcClientImpl.java:1243) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod( > AbstractRpcClient.java:227) > at > org.apache.hadoop.hbase.ipc.AbstractRpcClient$ > BlockingRpcChannelImplementation.callBlockingMethod( > AbstractRpcClient.java:336) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$ > BlockingStub.get(ClientProtos.java:34070) > at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil. > java:1589) > at > org.apache.hadoop.hbase.client.ConnectionManager$ > HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1398) > at > org.apache.hadoop.hbase.client.ConnectionManager$ > HConnectionImplementation.locateRegion(ConnectionManager.java:1199) > at > org.apache.hadoop.hbase.client.ConnectionManager$ > HConnectionImplementation.relocateRegion(ConnectionManager.java:1166) > at > org.apache.hadoop.hbase.client.ConnectionManager$ > HConnectionImplementation.relocateRegion(ConnectionManager.java:1150) > at > org.apache.hadoop.hbase.client.ConnectionManager$ > HConnectionImplementation.getRegionLocation(ConnectionManager.java:971) > at > org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation( > HRegionLocator.java:83) > at > org.apache.hadoop.hbase.client.RegionServerCallable. > prepare(RegionServerCallable.java:79) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries( > RpcRetryingCaller.java:124) > ... 18 more > > -- > View this message in context: http://apache-kylin.74782.x6. > nabble.com/Hbase-Region-server-goes-down-tp7437.html > Sent from the Apache Kylin mailing list archive at Nabble.com. > -- Best regards, Shaofeng Shi 史少锋
