Matthias J. Sax created KAFKA-21488:
---------------------------------------
Summary: StandbyUpdateListener should be called when state-updated
goes into active-restoring mode
Key: KAFKA-21488
URL: https://issues.apache.org/jira/browse/KAFKA-21488
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Matthias J. Sax
The state-updater thread operates in two different modes: active-restoring and
standby-updating. When there is at least one active task that needs to get
restored, the active-restoring mode ensures that we get full bandwidth to get
the active tasks into RUNNING state first, before we spent bandwidth to
maintain standbys. Only if there is no active task that needs restoring, we use
standby-updating state to maintain standby tasks.
After a rebalance, we may transit from standby-updating back to
active-restoring, effectively pausing standby tasks for a potentially longer
period of time. However, the standby-update-listener is not invoked for this
case. The user would only know about this indirectly, if they have a restore
listener; as a side effect, `onBatchLoaded()` would just not be called for a
longer period of time.
Even worse, if we are in active-restoring mode and a rebalance assigns us a new
standby task, we would invoke `onUpdateStart()` right away when initializing
the standby task, even if we put the standby into "paused" mode right away, and
would only effectively start maintaining it, when entering standby-updating
mode.
We should consider to improve this, to give users better visibility, when
standby tasks are effectively maintained vs paused. We might also need a new
"suspend reason" code for the "pause" state. Not sure if we need a "resume"
callback, or if the first `onBatchLoaded()` would be sufficient?
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)