guozhangwang commented on PR #13523: URL: https://github.com/apache/kafka/pull/13523#issuecomment-1500665780
And just to clarify some context here: currently we have a binary state for the main thread regarding `REBALANCING` and `RUNNING`: the thread would either be spending all its time on restoring active tasks as long as there are any, or spend time on processing tasks which would only happen until all tasks have finished restorations. In the past we actually did allow the thread to interleave restoring some tasks while processing some other tasks that have finished restorations, but we found that this interleaving of processing/restoration would make the restoration end-to-end latency much larger, so the rationale behind the current behavior is to reduce our restoration as much as possible, and only start any processing after all restoration are finished. Part of the state-updater's introduction is to resolve that, i.e. the main thread can just do processing, while the restoration can be done by the state-updater thread, and processing / restoration can happen in parallel. But before we baked in the state-updater, i.e. remove the old code and make state-updater always enabled, we decided to still honor the current behavior, i.e. we only transit to `RUNNING` and process any tasks when all restoration tasks have been completed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
