[
https://issues.apache.org/jira/browse/KUDU-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350469#comment-17350469
]
ASF subversion and git services commented on KUDU-3277:
-------------------------------------------------------
Commit 07a283ee1c990ec9639622cf621f2d36385941d5 in kudu's branch
refs/heads/branch-1.15.x from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=07a283e ]
KUDU-3277 fix concurrent session flush issue in Java client
This patch fixes the issue reported by KUDU-3277. The bug manifested
itself in rare cases when a session running in the AUTO_FLUSH_BACKGROUND
mode was either being flushed explicitly or implicitly (i.e. upon
closing the session) while the AUTO_FLUSH_BACKGROUND session's logic was
flushing its data buffers concurrently as well.
This patch also adds a reproduction scenario for KUDU-3277. The newly
introduced test scenario was reliably failing before the fix:
* 'java.lang.AssertionError: This Deferred was already called'
messages were encountered in the log multiple times with the stack
exactly as described in KUDU-3277
* some flusher threads were unable to join since KuduSession.flush()
would hang (i.e. would not return)
Change-Id: If6aaccc06abf1a2673620ab7c649f51f91999ad9
Reviewed-on: http://gerrit.cloudera.org:8080/17486
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Grant Henke <[email protected]>
(cherry picked from commit d1969e2623e30bfd62f2463f15e0bf02422d1d84)
Reviewed-on: http://gerrit.cloudera.org:8080/17493
Reviewed-by: Bankim Bhavsar <[email protected]>
> unexpected exception from downstream with kudu client 1.10
> -----------------------------------------------------------
>
> Key: KUDU-3277
> URL: https://issues.apache.org/jira/browse/KUDU-3277
> Project: Kudu
> Issue Type: Bug
> Components: client, java
> Affects Versions: 1.10.0, 1.11.0, 1.12.0, 1.11.1, 1.13.0, 1.14.0
> Environment: flink-connector-kudu
> Reporter: wangkang
> Assignee: Alexey Serbin
> Priority: Major
> Labels: client
> Fix For: 1.15.0
>
>
> flink-connector-kudu:
> {code:java}
> 2021-04-20 00:19:02,451 ERROR org.apache.kudu.client.Connection - [peer
> 26a2b2540158413ebf18206e5864a770(kudu-tserver:7050)] unexpected exception
> from downstream on [id: 0x9309d12d, /.....:60166 => ....:7050]
> java.lang.AssertionError: This Deferred was already called! New result=null,
> this=Deferred@75271701(state=DONE, result=null, callback=<none>,
> errback=<none>) at com.stumbleupon.async.Deferred.callback(Deferred.java:979)
> at
> org.apache.kudu.client.AsyncKuduSession$Buffer.callbackFlushNotification(AsyncKuduSession.java:858)
> at
> org.apache.kudu.client.AsyncKuduSession.queueBuffer(AsyncKuduSession.java:311)
> at
> org.apache.kudu.client.AsyncKuduSession.access$900(AsyncKuduSession.java:114)
> at
> org.apache.kudu.client.AsyncKuduSession$TabletLookupCB$1.call(AsyncKuduSession.java:406)
> at
> org.apache.kudu.client.AsyncKuduSession$TabletLookupCB$1.call(AsyncKuduSession.java:403)
> at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) at
> com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) at
> com.stumbleupon.async.Deferred.callback(Deferred.java:1002) at
> com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:173) at
> com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:142)
> at com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36) at
> com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:82) at
> com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) at
> com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) at
> com.stumbleupon.async.Deferred.callback(Deferred.java:1002) at
> org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275) at
> org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:322) at
> org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:296) at
> org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59) at
> org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149) at
> org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145) at
> org.apache.kudu.client.Connection.messageReceived(Connection.java:390) at
> org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) at
> org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
> at
> org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
> at
> org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at
> org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> one of flink task manager node occur with above exception, it didn't cause
> flink job fail,but cause whole job stopped writting to kudu server.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
