Thanks for the response Lianet. This problem happens only in kraft mode brokers. Works fine if we use kafka broker in non-kraft mode. (WIth zookeeper)
>From what I understand: (I may be wrong. this is what I read online) In zookeeper mode, on getting a partitionsFor request, the broker creates a topic if not present and then immediately returns the number of partitions. All this happens synchronously. In kraft mode, the broker creates a topic asynchronously. So, unless u retry, you wont get the total number of partitions. Like I said, this is what I read. On Tue, Feb 17, 2026 at 3:19 AM Lianet Magrans <[email protected]> wrote: > Hi Anand, > > The partitionsFor API does not guarantee that a single call will return the > data if the topic does not exist. It issues a first metadata request to > create the topic, and waits for the api timeout for a response (timeout > param or default.api.timeout.ms config). Then returns whatever it gets > from > the broker in that single call (empty if the topic is not found within the > timeout, described in the java docs > > https://github.com/apache/kafka/blob/e678b4bb7ca99c7d1be0d554ffe7e66f584771d6/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1415 > ). > Also there are old integration tests with the pattern you describe: a first > call to ensure topic is created, then a following one expected to retrieve > the partitions for the topic (e.g, in 3.9 branch > > https://github.com/apache/kafka/blob/5e9866f43ab8e7e41ef39e5584ac50019381328d/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L176 > ) > > That being said, I wonder if the difference you're mentioning is due to > running with Zookeeper in 3.9 vs 4.x with Kraft? Was that the case? > Not sure if tuning some Kraft configs on your cluster could help (I'm not > familiar with the kraft details really), but metadata is eventually > consistent anyways, so for sure it would be advisable to use the > partitionsFor considering that it may require several calls if the topic is > not found. From there, we can still consider improving for sure (e.g, > improve the client to continue attempting to find the topic while there is > time?). If you can file a jira with your setup (broker configs, params > passed to the client API) and behaviour, we can better look into it. > > Thanks! > Lianet > > On Tue, Jan 20, 2026 at 8:03 AM Subra I <[email protected]> wrote: > > > Hello All, > > > > I am using Kafka-clients-4.1.1.jar with a kafka broker that is on version > > 4.0.0. From my Java code, when I call the KafkaConsumer partitionsFor > API, > > I see that it works when the topic is already created in kafka. But if it > > is not yet created, the above API call returns empty - no information. > > > > I noticed that just before I call partitionsFor API, the topic is not > > created. But after I call this API, the topic gets created, but this API > > still returns no information. Of course, if I call the same API again, I > do > > get valid information for this topic. > > > > I used to have kafka-clients-3.9.0.jar earlier. There both these cases > were > > working fine. Is this a new change in behavior in > kafka-clients-4.1.1.jar? > > > > Thanks, > > Anand > > >
