hi,Prasanna,
maybe it is beause few free memory left in your system. I find this on
your log :
------------------ Original ------------------
From: "Prasanna";<[email protected]>;
Date: Mon, Jan 8, 2018 02:09 PM
To: "user"<[email protected]>;
Subject: If hbase region server is down,then will down or not?
Hi all,
I have small doubt regarding kylin ,If my hbase region server is down then
kylin service will down automatically or it will work. I am facing this issue
from longback.Sometimes in my cluster regionserver is getting down then my
kylin service is getting down.I am attaching the logs,can you please confirm
what is route cause.
Logs: kylin.log
----------------------
Thu Jan 04 14:27:44 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047253026, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
id=475450, waitTime=11172, operationTimeout=10000 expired.
2018-01-04 11:58:03,998 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1019 : Opening socket connection to server
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:2181. Will not attempt to authenticate
using SASL (unknown error)
2018-01-04 11:58:03,999 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:864 : Socket connection established to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:2181, initiating session
2018-01-04 11:58:19,376 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from
server in 15377ms for sessionid 0x360bc7adc000011, closing socket connection
and attempting reconnect
2018-01-04 11:58:19,376 ERROR [pool-8-thread-1] dao.ExecutableDao:155 : error
get all Jobs:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
id=475451, waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Call to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1261)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:372)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:356)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:330)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
... 14 more
2018-01-04 11:58:19,377 ERROR [pool-8-thread-1] execution.ExecutableManager:269
: error get All Job Ids
org.apache.kylin.job.exception.PersistentException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
id=475451, waitTime=11282, operationTimeout=5000 expired.
at
org.apache.kylin.job.dao.ExecutableDao.getJobIds(ExecutableDao.java:156)
at
org.apache.kylin.job.execution.ExecutableManager.getAllJobIds(ExecutableManager.java:267)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:85)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=1, exceptions:
Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
id=475451, waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
... 3 more
Caused by: java.io.IOException: Call to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1261)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:372)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:356)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:330)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
... 14 more
2018-01-04 11:58:19,377 WARN [pool-8-thread-1] threadpool.DefaultScheduler:127
: Job Fetcher caught a exception java.lang.RuntimeException:
org.apache.kylin.job.exception.PersistentException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
id=475451, waitTime=11282, operationTimeout=5000 expired.
2018-01-04 11:58:31,002 INFO [BadQueryDetector] service.BadQueryDetector:160 :
System free memory less than 100 MB. 0 queries running.
2018-01-04 11:58:39,176 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1019 : Opening socket connection to server
ICCC-THBDPS01.EYEWAY.local/10.82.0.17:2181. Will not attempt to authenticate
using SASL (unknown error)
2018-01-04 11:58:39,488 ERROR [pool-8-thread-1] dao.ExecutableDao:155 : error
get all Jobs:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Thu Jan 04 14:28:39 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047299715, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
id=475452, waitTime=8173, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Call to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475452,
waitTime=8173, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1261)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:372)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:356)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:330)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475452,
waitTime=8173, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
... 14 more
2018-01-04 11:59:49,777 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPSM01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from
server in 40161ms for sessionid 0x360bc7adc000012, closing socket connection
and attempting reconnect
2018-01-04 11:59:31,385 INFO [BadQueryDetector] service.BadQueryDetector:160 :
System free memory less than 100 MB. 0 queries running.
2018-01-04 11:58:39,488 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:864 : Socket connection established to
ICCC-THBDPS01.EYEWAY.local/10.82.0.17:2181, initiating session
