artemlivshits commented on PR #14728: URL: https://github.com/apache/kafka/pull/14728#issuecomment-1809713385
True, using a separate thread pool has its value. Generally, a separate thread pool is used when the patterns are significantly different, e.g. threads that use only compute vs. threads that also do sync IO, or maybe threads that run at a higher priority or etc. In a fully async system, the number of worker threads is just a function of number of CPUs, proliferation of threads is suboptimal because more cycles are spent on context switching. It seems to me that group coordinator is fairly aligned with what other requests do -- it does sync IO on its threads pretty much the same as other requests do. Offset commit is a part of the core data path (especially for EoS). We could say that other requests technically are not on core data path, but I'm not sure what would justify a separate thread pool for those. > Would it be possible to implement a new CoordinatorEventProcessor It should be possible (given that it has just one method), but I'm not sure if this would help with loosening the coupling. E.g. the event is still coupled to the accumulator, has partition granularity, etc. Thread pool and synchronizer, on the other hand, are general utilities and are not coupled to each other. This example illustrates implementing coordinator event processor on request handler threads but it's actually much more powerful: we can as easily change the thread pool without changing the synchronization or change synchronization (e.g. make it group granularity, not partition granularity) without changing the which thread pool to use. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
