[
https://issues.apache.org/jira/browse/RATIS-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17987055#comment-17987055
]
Tian Jiang commented on RATIS-2314:
-----------------------------------
Unfortunately, queue.offer(..) is not run by an appendEntries thread but the
SegmentedRaftLogWorker thread in this issue. Let me explain it in detail:
# When the SegmentedRaftLogWorker thread finishes a task, it calls task.done()
!image-2025-07-01-10-44-53-302.png!
# In task.done(), future.complete() is called
!image-2025-07-01-10-46-04-453.png!
# And here is the key, in future.complete(), postComplete() is called, and in
postComplete(), latter stages of this future will be fired, as shown in the
previous stack; !image-2025-07-01-10-47-20-566.png!
!image-2025-07-01-10-47-54-166.png!
!image-2025-07-01-10-48-56-806.png!
4. In this issue, the latter stage is the call of appendEntry() registered by
the append thread by calling future.thenCompose(). Therefore, the
SegmentedRaftLogWorker thread will try to append the entries by itself.
!image-2025-07-01-10-52-05-896.png!
> Deadlock when appending large batches of entries
> ------------------------------------------------
>
> Key: RATIS-2314
> URL: https://issues.apache.org/jira/browse/RATIS-2314
> Project: Ratis
> Issue Type: Bug
> Components: server
> Reporter: Tian Jiang
> Priority: Major
> Attachments: image-2025-06-30-13-25-54-574.png,
> image-2025-06-30-13-33-13-766.png, image-2025-06-30-13-35-09-832.png,
> image-2025-06-30-13-37-28-743.png, image-2025-07-01-10-43-50-136.png,
> image-2025-07-01-10-44-53-302.png, image-2025-07-01-10-46-04-453.png,
> image-2025-07-01-10-47-16-439.png, image-2025-07-01-10-47-20-566.png,
> image-2025-07-01-10-47-54-166.png, image-2025-07-01-10-48-56-806.png,
> image-2025-07-01-10-52-05-896.png
>
>
> Greeting, I found a deadlock when I am using Ratis, which is shown in the
> following stack:
> !image-2025-06-30-13-25-54-574.png!
> The problem is that Ratis uses CompletableFuture to serialize the process of
> appending logs, and below is the timeline:
> # Thread "SegmentedRaftLogWorker" is working on some logs;
> # Thread "SomeClient" tries to append 5000 entries to the log;
> # Because "SegmentedRaftLogWorker" is working, "SomeClient" adds the entries
> to the CompletableFuture; !image-2025-06-30-13-33-13-766.png!
> # When "SegmentedRaftLogWorker" is done with its last task, it checks the
> CompletableFuture for the next task; !image-2025-06-30-13-35-09-832.png!
> # Of course, there is one from "SomeClient", ordering it to append another
> 5000 entries; therefore "SegmentedRaftLogWorker" continues to append the
> entries, and it comes to: !image-2025-06-30-13-37-28-743.png!
> # However, the queue of SegmentedRaftLogWorker has a max capacity of 4096,
> and there are 5000 entries to be appended, so Thread "SegmentedRaftLogWorker"
> waits for the queue to be not full;
> # But who is supposed to consume from the queue? Thread
> "SegmentedRaftLogWorker" itself! As a result, the wait shall never end.
> I notice that Ratis is using many thread-steal tricks to improve thread
> locality, which is appreciated. However, it seems such a technique, in this
> scenario, creates deadlocks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
