Hi all,
I have small doubt regarding kylin ,If my hbase region server is down then
kylin service will down automatically or it will work. I am facing this
issue from longback.Sometimes in my cluster regionserver is getting down
then my kylin service is getting down.I am attaching the logs,can you please
confirm what is route cause.
Logs: kylin.log
----------------------
Thu Jan 04 14:27:44 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047253026, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475450, waitTime=11172, operationTimeout=10000 expired.
2018-01-04 11:58:03,998 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1019 : Opening socket connection to server
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:2181. Will not attempt to authenticate
using SASL (unknown error)
2018-01-04 11:58:03,999 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:864 : Socket connection established to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:2181, initiating session
2018-01-04 11:58:19,376 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from
server in 15377ms for sessionid 0x360bc7adc000011, closing socket connection
and attempting reconnect
2018-01-04 11:58:19,376 ERROR [pool-8-thread-1] dao.ExecutableDao:155 :
error get all Jobs:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475451, waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:147)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture
.run(ResultBoundedCompletionService.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Call to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1
261)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpc
Client.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementati
on.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$Blocki
ngStub.scan(ClientProtos.java:32651)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.j
ava:372)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199
)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetry
ingCaller.java:200)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:356)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:330)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:126)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
... 14 more
2018-01-04 11:58:19,377 ERROR [pool-8-thread-1]
execution.ExecutableManager:269 : error get All Job Ids
org.apache.kylin.job.exception.PersistentException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475451, waitTime=11282, operationTimeout=5000 expired.
at
org.apache.kylin.job.dao.ExecutableDao.getJobIds(ExecutableDao.java:156)
at
org.apache.kylin.job.execution.ExecutableManager.getAllJobIds(ExecutableMana
ger.java:267)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(Defa
ultScheduler.java:85)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=1, exceptions:
Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475451, waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:147)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture
.run(ResultBoundedCompletionService.java:65)
... 3 more
Caused by: java.io.IOException: Call to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1
261)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpc
Client.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementati
on.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$Blocki
ngStub.scan(ClientProtos.java:32651)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.j
ava:372)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199
)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetry
ingCaller.java:200)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:356)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:330)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:126)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
... 14 more
2018-01-04 11:58:19,377 WARN [pool-8-thread-1]
threadpool.DefaultScheduler:127 : Job Fetcher caught a exception
java.lang.RuntimeException:
org.apache.kylin.job.exception.PersistentException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475451, waitTime=11282, operationTimeout=5000 expired.
2018-01-04 11:58:31,002 INFO [BadQueryDetector]
service.BadQueryDetector:160 : System free memory less than 100 MB. 0
queries running.
2018-01-04 11:58:39,176 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1019 : Opening socket connection to server
ICCC-THBDPS01.EYEWAY.local/10.82.0.17:2181. Will not attempt to authenticate
using SASL (unknown error)
2018-01-04 11:58:39,488 ERROR [pool-8-thread-1] dao.ExecutableDao:155 :
error get all Jobs:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Thu Jan 04 14:28:39 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047299715, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475452, waitTime=8173, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:147)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture
.run(ResultBoundedCompletionService.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Call to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475452,
waitTime=8173, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1
261)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpc
Client.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementati
on.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$Blocki
ngStub.scan(ClientProtos.java:32651)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.j
ava:372)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199
)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetry
ingCaller.java:200)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:356)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:330)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:126)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475452,
waitTime=8173, operationTimeout=5000 expired.
at
org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
... 14 more
2018-01-04 11:59:49,777 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPSM01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from
server in 40161ms for sessionid 0x360bc7adc000012, closing socket connection
and attempting reconnect
2018-01-04 11:59:31,385 INFO [BadQueryDetector]
service.BadQueryDetector:160 : System free memory less than 100 MB. 0
queries running.
2018-01-04 11:58:39,488 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:864 : Socket connection established to
ICCC-THBDPS01.EYEWAY.local/10.82.0.17:2181, initiating session
