Bruno Cadonna created KAFKA-19124:
-------------------------------------
Summary: Calling unsubscribe() on the consumer leads always to a
timeout
Key: KAFKA-19124
URL: https://issues.apache.org/jira/browse/KAFKA-19124
Project: Kafka
Issue Type: Sub-task
Components: streams
Reporter: Bruno Cadonna
Assignee: Bruno Cadonna
With the current implementation of the new Streams reabalance protocol proposed
in KIP-1071, when calling {{mainConsumer.unsubscribe()}} – for example during
handling a {{TaskMigratedException}} – the call times out. The reason is that
{{unsubscribe()}} sends a {{UnsubscribeEvent}} to the background thread and
waits for a given time until the future of the {{UnsubscribeEvent}} completes.
The future of the {{UnsubscribeEvent}} completes when the rebalance listener
callback for releasing the assigned partitions is executed. This callback is
executed when the application thread – i.e., the thread the called
{{unsubscribe()}} – receives the respective event from the background thread.
Thus, when the application thread waits for the future of the
{{UnsubscribeEvent}} to complete it also processes events from the background
thread to ensure that the callback to release the assigned partitions is called.
Streams uses a separate event queue for the background events that request the
execution of the task release callbacks. Within the call to {{unsubscribe()}},
the application thread does not have access to those Streams-specific
background events and cannot process them. That means that the future of the
{{UnsubscribeEvent}} can never complete, because the callback are not executed.
This leads to the situation that the {{unsubscribe()}} call in Streams always
times out.
This is a blocker because Streams needs to unsubscribe and rejoin to recover
from a TaskMigratedException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
