Try to increase number of retries form 1 to something higher in hbase
client config, located at hbase-site.xml (in my case, at
$KYLIN_HOME/hadoop-conf/hbase-site.xml)
<property>
<name>hbase.client.retries.number</name>
<value>35</value>
</property>
El lun., 22 abr. 2019 a las 8:45, 蒋慧明 (<[email protected]>) escribió:
> Hi，
>
> adjust "kylin.metadata.hbase-rpc-timeout"
>
> This method does work.
>
>
>
> Another problem: When I tried to run kylin job, I encountered below problem.
>
> I tried to check consistancy of kylin_metadata with "hbase hbck -details
> kylin_metadata". The result is "0 inconsistency"
>
> org.apache.kylin.engine.mr.exception.HadoopShellException:
> java.lang.RuntimeException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=1, exceptions:
> Mon Apr 22 14:30:23 GMT+08:00 2019,
> RpcRetryingCaller{globalStartTime=1555914623296, pause=100, retries=1},
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> kylin_metadata,/dict/FACT/DIM_TB/8b5cdf3e-8aa3-5c70-44bd-fffdc6ed4d1a.dict,1555653527986.9fbc862f521b93968a3299c9d853992e.
> is not online on ip-109-105-1-504.compute.internal,16020,1555901919971
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3008)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1144)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2476)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
>
> Does anyone knows how to solve this problem?
>
> Thanks a lot!
>
>
>
>
>
> --------- *Original Message* ---------
>
> *Sender* : JiaTao Tao <[email protected]>
>
> *Date* : 2019-04-19 10:54 (GMT+9)
>
> *Title* : Re: KYLIN timeout problem
>
>
> Hi
>
> You can adjust "kylin.metadata.hbase-rpc-timeout" to a larger value. And
> then run metadata/StorageCleanup, It will reduce the data in Hbase.
>
> --
>
>
> Regards!
>
> Aron Tao
>
> 蒋慧明 <[email protected]> 于2019年4月18日周四 下午2:35写道：
>
>> Hi,
>>
>> I met below error when I run cube in Kylin. It happened in the "#4 Step
>> Name: Build Dimension Dictionary"
>>
>>
>>
>> Tue Apr 16 14:18:06 GMT+08:00 2019,
>> RpcRetryingCaller{globalStartTime=1555395481041, pause=100, retries=1},
>> java.io.IOException: Call to [HBASE URL] failed on local exception:
>> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=15589,
>> waitTime=5001, operationTimeout=5000 expired.
>>
>> at
>> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
>> at
>> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
>> ... 3 more
>> Caused by: java.io.IOException: Call to
>> ip-10-10-110-102.cn-north-1.compute.internal/10.10.110.102:16020 failed
>> on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
>> id=15589, waitTime=5001, operationTimeout=5000 expired.
>> at
>> org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:292)
>> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1274)
>> at
>> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
>> at
>> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
>> at
>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:35396)
>> at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:224)
>> at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:65)
>> at
>> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
>>
>>
>>
>>
>> The normal dimension of this cube is about 25.
>>
>> Then I created a new cube with 3 normal dimensions and run it. The job is
>> successful.
>>
>>
>>
>> When I tried to do metadata backup and metadata clean wich cmd
>> "metastore.sh", I also met the error:
>> "org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=15589,
>> waitTime=5001, operationTimeout=5000 expired".
>>
>>
>>
>> Does anyone know about the root cause of this problem? And how to fix it?
>> Thanks a lot!
>>
>>
>>
>>
>>
>>
>
>
>
--
Iñigo Martínez
Systems Manager
[email protected]
[image: Telecoming - Make it digital]
[image: 5000_empresas]
[image: 1000_empresas]
Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain |
telecoming.com <http://www.telecoming.com/>
Este correo electrónico y sus archivos adjuntos están dirigidos
únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter
confidencial, personal e intransferible del mismo está protegido
legalmente. Cualquier publicación, reproducción, distribución o
retransmisión no autorizada, ya sea completa o en parte, se encuentra
prohibida. Si ha recibido este mensaje por equivocación, notifíquelo
inmediatamente a la persona que lo ha enviado y borre el mensaje original
junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.
