[
https://issues.apache.org/jira/browse/KAFKA-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18035626#comment-18035626
]
Colt McNealy commented on KAFKA-19853:
--------------------------------------
I have opened a PR which addresses this specific issue (the transaction timeout
under EOS) and have tested it in our soak test with ~600GB state. Feedback
would be appreciated (: [https://github.com/apache/kafka/pull/20833/files]
> StreamThread blocks on StateUpdater during onAssignment()
> ---------------------------------------------------------
>
> Key: KAFKA-19853
> URL: https://issues.apache.org/jira/browse/KAFKA-19853
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.9.0
> Reporter: Colt McNealy
> Priority: Major
> Attachments: image (3).png, image (4).png, image (5).png
>
>
> We've observed that the `StreamThread` blocks waiting for a `Future` from the
> `StateUpdater` in the `StreamsPartitionAssigner#onAssignment()` method when
> we are moving a task out of the `StateUpdater` and onto the `StreamThread`.
>
> This can cause problems because, during restoration or with warmup replicas,
> the `StateUpdater#runOnce()` method can take a long time (upwards of 20
> seconds) when RocksDB stalls writes to allow compaction to keep up. In EOS
> this blockage may cause the transaction to time out, which is a big mess.
> This is because the `StreamThread` may have an open transaction before the
> `StreamsPartitionAssignor#onAssignment()` method is called.
>
> Some screenshots from the JFR below (credit to [~eduwerc]).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
