Hi Jaxon, The threading model is implemented differently between the Kafka08Fetcher and all other fetcher versions higher than 0.9+ because the Kafka Java clients used between these versions have different abstraction levels.
The Kafka08Fetcher still uses the low-level `SimpleConsumer` API, which only allows connection to a single Kafka broker. Therefore, the Kafka08Fetcher maintains a thread for each broker that it connects to. If a subtask’s assigned partitions all exist on the same Kafka broker, only one thread will be created. On the other hand, in versions 0.9+, the high-level `KafkaConsumer` API is used, which hides away the complexity for per-broker connections. I believe that the `KafkaConsumer` client still creates per-broker threads internally, if its assigned partitions span multiple brokers. Cheers, Gordon On 2 January 2018 at 8:22:22 AM, Timo Walther (twal...@apache.org) wrote: Maybe Gordon (in CC) can answer your question. Am 1/1/18 um 3:36 PM schrieb Jaxon Hu: > In Kafka08Fetcher, it use Map<Node,SimpleConsumerThread> to manage > multi-threads. But I notice in Kafka09Fetcher or Kafka010Fetcher, it's > gone. So how Kafka09Fetcher implements multi-threads read partitions > from kafka?
