zhangsong created KUDU-1542:
-------------------------------
Summary: kudu-client will hang in some corner case(bad network or
busy kudu server)
Key: KUDU-1542
URL: https://issues.apache.org/jira/browse/KUDU-1542
Project: Kudu
Issue Type: Bug
Components: client
Affects Versions: 0.9.1
Reporter: zhangsong
Priority: Minor
when using java-client loading data into kudu, there are rare cases where
kudu-client hang there "forever" , these some java exceptions when "hang"
happened:
" thread pool-5-thread-1 -> wakeup thread pool-5-thread-1 -> wakeup thread
pool-5-thread-1 -> wakeup thread pool-5-thread-1 -> wakeup thread
pool-5-thread-1 -> wakeup thread pool-5-thread-1 -> wakeup thread
pool-5-thread-1 -> wakeup thread pool-5-thread-1 -> wakeup thread pool-5-thread
-1 -> wakeup thread pool-5-thread-1 -> wakeup thread pool-5-thread-1 -> wakeup
thread pool-5-thread-1 -> wakeup thread pool-5-thread-1)
at com.stumbleupon.async.Deferred.doJoin(Deferred.java:1183)
at com.stumbleupon.async.Deferred.join(Deferred.java:1042)
at org.kududb.client.KuduSession.apply(KuduSession.java:79)
at
com.jd.kafka.service.jdq.KududbLoaderDbHandler.doOptKudu(KududbLoaderDbHandler.java:190)
at
com.jd.kafka.service.jdq.KududbLoaderDbHandler.doUpdate(KududbLoaderDbHandler.java:277)
at
com.jd.kafka.service.jdq.KududbLoaderDbHandler.doOpt(KududbLoaderDbHandler.java:174)
at com.jd.kafka.service.jdq.KududbLoaderDbHandler.doMessageHandle
r(KududbLoaderDbHandler.java:153)
at
com.jd.kafka.service.jdq.KududbLoaderDbHandler.doMessageHandler(KududbLoaderDbHandler.java:40)
at
com.jd.bdp.jdq.consumer.zk.JDQConsumerTask.run(JDQConsumerTask.java:65)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)"
After examine code of KuduSession.java , it seems that the "while(true)" clause
in "apply" method of KuduSession cause the hang. In current logic, if there is
an exception happened during executing "session.apply" such as "timeout
exception" , apply method will delay for a while and try again and again.
However , kudu-client should return error or throw exception out to it caller
instead of trying forever. In my case , it re-connect/re-create kudu-client and
re-apply will always succeed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
