[
https://issues.apache.org/jira/browse/IGNITE-22207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847308#comment-17847308
]
Kirill Sizov commented on IGNITE-22207:
----------------------------------------
There is a deadlock between {{ReplicaService.pendingInvokes}} and
{{ReplicaManager.replicas}}
See threaddump:
{noformat}
"%sqllogic0%low-watermark-updater-0" #273 prio=5 os_prio=0 cpu=1503.23ms
elapsed=2391.82s tid=0x00007fd3492b1000 nid=0x2527f6 waiting for monitor entry
[0x00007fd289870000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
java.util.concurrent.ConcurrentHashMap.replaceNode([email protected]/ConcurrentHashMap.java:1122)
- waiting to lock <0x000000071ab00178> (a
java.util.concurrent.ConcurrentHashMap$ReservationNode)
at
java.util.concurrent.ConcurrentHashMap.remove([email protected]/ConcurrentHashMap.java:1552)
at
org.apache.ignite.internal.replicator.ReplicaService.lambda$sendToReplica$3(ReplicaService.java:167)
at
org.apache.ignite.internal.replicator.ReplicaService$$Lambda$3914/0x000000080117c840.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniHandle([email protected]/CompletableFuture.java:930)
at
java.util.concurrent.CompletableFuture$UniHandle.tryFire([email protected]/CompletableFuture.java:907)
at
java.util.concurrent.CompletableFuture.postComplete([email protected]/CompletableFuture.java:506)
at
java.util.concurrent.CompletableFuture.complete([email protected]/CompletableFuture.java:2073)
at
org.apache.ignite.internal.network.DefaultMessagingService.onInvokeResponse(DefaultMessagingService.java:524)
at
org.apache.ignite.internal.network.DefaultMessagingService.send0(DefaultMessagingService.java:240)
at
org.apache.ignite.internal.network.DefaultMessagingService.respond(DefaultMessagingService.java:183)
at
org.apache.ignite.internal.network.DefaultMessagingService.respond(DefaultMessagingService.java:196)
at
org.apache.ignite.internal.network.MessagingService.respond(MessagingService.java:142)
at
org.apache.ignite.internal.replicator.ReplicaManager.lambda$handleReplicaRequest$2(ReplicaManager.java:301)
at
org.apache.ignite.internal.replicator.ReplicaManager$$Lambda$3898/0x0000000801179040.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniWhenComplete([email protected]/CompletableFuture.java:859)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire([email protected]/CompletableFuture.java:837)
at
java.util.concurrent.CompletableFuture.postComplete([email protected]/CompletableFuture.java:506)
at
java.util.concurrent.CompletableFuture.completeExceptionally([email protected]/CompletableFuture.java:2088)
at
org.apache.ignite.internal.replicator.ReplicaManager.lambda$stopReplicaInternal$17(ReplicaManager.java:630)
at
org.apache.ignite.internal.replicator.ReplicaManager$$Lambda$4502/0x0000000801292040.apply(Unknown
Source)
at
java.util.concurrent.ConcurrentHashMap.compute([email protected]/ConcurrentHashMap.java:1932)
- locked <0x0000000719f589f8> (a
java.util.concurrent.ConcurrentHashMap$Node)
at
org.apache.ignite.internal.replicator.ReplicaManager.lambda$stopReplicaInternal$18(ReplicaManager.java:624)
at
org.apache.ignite.internal.replicator.ReplicaManager$$Lambda$4501/0x0000000801291c40.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniWhenComplete([email protected]/CompletableFuture.java:859)
at
java.util.concurrent.CompletableFuture.uniWhenCompleteStage([email protected]/CompletableFuture.java:883)
at
java.util.concurrent.CompletableFuture.whenComplete([email protected]/CompletableFuture.java:2251)
at
org.apache.ignite.internal.replicator.ReplicaManager.stopReplicaInternal(ReplicaManager.java:612)
at
org.apache.ignite.internal.replicator.ReplicaManager.stopReplica(ReplicaManager.java:595)
at
org.apache.ignite.internal.table.distributed.TableManager.stopPartition(TableManager.java:2413)
at
org.apache.ignite.internal.table.distributed.TableManager.destroyTableLocally(TableManager.java:1519)
at
org.apache.ignite.internal.table.distributed.TableManager.lambda$onLwmChanged$22(TableManager.java:807)
at
org.apache.ignite.internal.table.distributed.TableManager$$Lambda$3174/0x0000000800ffc040.apply(Unknown
Source)
at
java.util.stream.ReferencePipeline$3$1.accept([email protected]/ReferencePipeline.java:195)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining([email protected]/ArrayList.java:1655)
at
java.util.stream.AbstractPipeline.copyInto([email protected]/AbstractPipeline.java:484)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto([email protected]/AbstractPipeline.java:474)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential([email protected]/ReduceOps.java:913)
at
java.util.stream.AbstractPipeline.evaluate([email protected]/AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect([email protected]/ReferencePipeline.java:578)
at
org.apache.ignite.internal.table.distributed.TableManager.lambda$onLwmChanged$25(TableManager.java:808)
at
org.apache.ignite.internal.table.distributed.TableManager$$Lambda$3173/0x0000000800ffb440.get(Unknown
Source)
at
org.apache.ignite.internal.util.IgniteUtils.inBusyLockAsync(IgniteUtils.java:890)
at
org.apache.ignite.internal.table.distributed.TableManager.onLwmChanged(TableManager.java:803)
at
org.apache.ignite.internal.table.distributed.TableManager.lambda$startAsync$5(TableManager.java:631)
at
org.apache.ignite.internal.table.distributed.TableManager$$Lambda$1978/0x0000000800c24040.notify(Unknown
Source)
at
org.apache.ignite.internal.event.AbstractEventProducer.fireEvent(AbstractEventProducer.java:88)
at
org.apache.ignite.internal.lowwatermark.LowWatermarkImpl.lambda$updateAndNotify$8(LowWatermarkImpl.java:306)
at
org.apache.ignite.internal.lowwatermark.LowWatermarkImpl$$Lambda$3166/0x0000000800fecc40.apply(Unknown
Source)
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire([email protected]/CompletableFuture.java:1072)
at
java.util.concurrent.CompletableFuture$Completion.run([email protected]/CompletableFuture.java:478)
at
java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:515)
at
java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run([email protected]/ScheduledThreadPoolExecutor.java:304)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at java.lang.Thread.run([email protected]/Thread.java:834)
{noformat}
and this:
{noformat}
"%sqllogic0%partition-operations-4" #485 prio=5 os_prio=0 cpu=9301.10ms
elapsed=2387.08s tid=0x00007fd35899c000 nid=0x2528cc waiting for monitor entry
[0x00007fd278711000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
java.util.concurrent.ConcurrentHashMap.compute([email protected]/ConcurrentHashMap.java:1923)
- waiting to lock <0x0000000719f589f8> (a
java.util.concurrent.ConcurrentHashMap$Node)
at
org.apache.ignite.internal.replicator.ReplicaManager.handleReplicaRequest(ReplicaManager.java:293)
at
org.apache.ignite.internal.replicator.ReplicaManager.onReplicaMessageReceived(ReplicaManager.java:255)
at
org.apache.ignite.internal.replicator.ReplicaManager$$Lambda$751/0x00000008004fe840.onReceived(Unknown
Source)
at
org.apache.ignite.internal.network.DefaultMessagingService.sendToSelf(DefaultMessagingService.java:354)
at
org.apache.ignite.internal.network.DefaultMessagingService.invoke0(DefaultMessagingService.java:287)
at
org.apache.ignite.internal.network.DefaultMessagingService.invoke(DefaultMessagingService.java:215)
at
org.apache.ignite.internal.network.wrapper.JumpToExecutorByConsistentIdAfterSend.invoke(JumpToExecutorByConsistentIdAfterSend.java:97)
at
org.apache.ignite.internal.network.MessagingService.invoke(MessagingService.java:198)
at
org.apache.ignite.internal.replicator.ReplicaService.lambda$sendToReplica$1(ReplicaService.java:158)
at
org.apache.ignite.internal.replicator.ReplicaService$$Lambda$3884/0x0000000801172c40.apply(Unknown
Source)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent([email protected]/ConcurrentHashMap.java:1705)
- locked <0x000000071ab00178> (a
java.util.concurrent.ConcurrentHashMap$ReservationNode)
at
org.apache.ignite.internal.replicator.ReplicaService.lambda$sendToReplica$4(ReplicaService.java:151)
at
org.apache.ignite.internal.replicator.ReplicaService$$Lambda$3872/0x000000080116d440.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniWhenComplete([email protected]/CompletableFuture.java:859)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire([email protected]/CompletableFuture.java:837)
at
java.util.concurrent.CompletableFuture.postComplete([email protected]/CompletableFuture.java:506)
at
java.util.concurrent.CompletableFuture.complete([email protected]/CompletableFuture.java:2073)
at
org.apache.ignite.internal.network.DefaultMessagingService.onInvokeResponse(DefaultMessagingService.java:524)
at
org.apache.ignite.internal.network.DefaultMessagingService.send0(DefaultMessagingService.java:240)
at
org.apache.ignite.internal.network.DefaultMessagingService.respond(DefaultMessagingService.java:183)
at
org.apache.ignite.internal.network.DefaultMessagingService.respond(DefaultMessagingService.java:196)
at
org.apache.ignite.internal.network.MessagingService.respond(MessagingService.java:142)
at
org.apache.ignite.internal.replicator.ReplicaManager.sendReplicaUnavailableErrorResponse(ReplicaManager.java:813)
at
org.apache.ignite.internal.replicator.ReplicaManager.handleReplicaRequest(ReplicaManager.java:326)
at
org.apache.ignite.internal.replicator.ReplicaManager.lambda$onReplicaMessageReceived$1(ReplicaManager.java:253)
at
org.apache.ignite.internal.replicator.ReplicaManager$$Lambda$3871/0x000000080116d040.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at java.lang.Thread.run([email protected]/Thread.java:834)
{noformat}
> Handling of AwaitPrimaryReplica request maight lead to the dedalock
> -------------------------------------------------------------------
>
> Key: IGNITE-22207
> URL: https://issues.apache.org/jira/browse/IGNITE-22207
> Project: Ignite
> Issue Type: Bug
> Reporter: Vladislav Pyatkov
> Priority: Major
> Labels: ignite-3
>
> h3. Motivation
> We handle ReplicaUnavailableException in ReplicaService to handle request
> transparency even if the replica had not been read at the time the request
> was sent. To do this, we send AwaitReplicaRequest as a handling of the
> exception and then process the origin request when the replica is guaranteed
> ready.
> In the event that the AwaitReplicaRequest is handled locally, we will do it
> synchronously in pendingInvokes.computeIfAbsent:
> {code}
> CompletableFuture<NetworkMessage> awaitReplicaFut =
> pendingInvokes.computeIfAbsent(
> targetNodeConsistentId,
> consistentId -> {
> AwaitReplicaRequest awaitReplicaReq =
> REPLICA_MESSAGES_FACTORY.awaitReplicaRequest()
> .groupId(req.groupId())
> .build();
> return messagingService.invoke(
> targetNodeConsistentId,
> awaitReplicaReq,
> replicationConfiguration.rpcTimeout().value()
> );
> }
> );
> {code}
> and then hands on handling of AwaitReplicaRequest, due to removing from
> pendingInvokes:
> {code}
> awaitReplicaFut.handle((response0, throwable0) -> {
> pendingInvokes.remove(targetNodeConsistentId, awaitReplicaFut);
> ...
> {code}
> h3. Definition of doe
> Take out of the network invokation from the map compute closure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
