[
https://issues.apache.org/jira/browse/KYLIN-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225926#comment-17225926
]
ASF GitHub Bot commented on KYLIN-4711:
---------------------------------------
guangxuCheng commented on pull request #1369:
URL: https://github.com/apache/kylin/pull/1369#issuecomment-721589488
@hit-lacus can you take a look again? This PR will be very effective when
hbase encounters an exception
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Change default value to 3 for kylin.metadata.hbase-client-retries-number
> ------------------------------------------------------------------------
>
> Key: KYLIN-4711
> URL: https://issues.apache.org/jira/browse/KYLIN-4711
> Project: Kylin
> Issue Type: Improvement
> Affects Versions: v3.1.0
> Reporter: Guangxu Cheng
> Assignee: Guangxu Cheng
> Priority: Major
>
> ```shell
> java.lang.RuntimeException:
> org.apache.kylin.job.exception.PersistentException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=1, exceptions:
> Thu Aug 20 21:06:01 GMT+08:00 2020, RpcRetryingCaller
> {globalStartTime=1597928761253, pause=1000, retries=1}
> , org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> kylin_production_metadata,/execute_output/3adc92f2-edcd-2705-5a9c-ad0afe4a0808-01,1594348337103.48b9e5e9c3c7891750236fcec84b38d5.
> is not online on xxx.xxx.xxx.xxx,16031,1558009276096
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3033)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1110)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2064)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33857)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2189)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> on xxx.xxx.xxx.xxx,16031,1558009276096
> at
> org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:174)
> at
> org.apache.kylin.job.execution.AbstractExecutable.getOutput(AbstractExecutable.java:450)
> at
> org.apache.kylin.job.execution.AbstractExecutable.isDiscarded(AbstractExecutable.java:561)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:165)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
> at
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> ```
> Recently, our build job failed occasionally. After analysis, it was found
> that the reason for the failure was due to abnormal access to the MetaStore.
> We use HBase as MetaStore.
> When accessing HBase, the client will cache the region information of the
> table in the client. When the region was moved, client will not actively
> update the information in the cache. So the client will receive a
> NotServingRegionException, the client will update the cache information when
> retrying. But the number of retries in kylin is 1, which means that the
> clinet will not try again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
