Hi all, Thanks for your questions!
> We’re in the process of upgrading our Kafka clients from version 3.5.1 to 4.0.0 Plugin Discovery is functionality unique to Kafka Connect, and should not affect applications which depend only on kafka-clients. If you're seeing this happen for Kafka brokers or client applications, that is unusual and should be reported. > We were able to work around this issue by explicitly setting: > plugin.discovery = service_load This is the intended long-term solution, not just a temporary workaround. In the future, default will be changed and the other plugin.discovery modes will be deprecated and removed. > 1. What might be causing the longer discovery times in newer versions? We changed the plugin scanning library (used only for only_scan, hybrid_warn, and hybrid_fail modes) from Reflections to Classgraph in 4.0 [1]. We noticed this performance regression during the review [2,3] but I did not consider it a blocker [4]. > 2. Is this expected behavior, or might it be a regression? This is expected behavior. The default plugin discovery method is slow but backwards-compatible, and service_load is the new recommended mode due to performance improvements, but is currently not the default due to backwards incompatibility. The fact that performance degraded 3.9 -> 4.0 is unfortunate, but is ultimately a net benefit to the project given the retirement of the Reflections library and the presence of the service_load alternative. > 3. Are there any risks to relying on service_load for plugin discovery? The risks for using service_load are described in the documentation [5] already linked up-thread. If you run with hybrid_fail in your CI environment, there should be no additional risks to using service_load in a production environment. Hope this helps, Greg [1] https://issues.apache.org/jira/browse/KAFKA-15203 [2] https://github.com/apache/kafka/pull/16604#discussion_r1696105490 [3] https://github.com/apache/kafka/pull/16604#issuecomment-2274110483 [4] https://github.com/apache/kafka/pull/16604#pullrequestreview-2246118001 [5] https://kafka.apache.org/documentation/#connect_plugindiscovery On Fri, Aug 8, 2025 at 12:52 AM Love Almgren <love.almg...@kambi.com> wrote: > Hi, > I ran the same experiment on version 3.7.0 and did not encounter any > issues with long startup delays. > > > From: pritam kumar <kumarpritamm...@gmail.com> > Date: Saturday, 2 August 2025 at 16:20 > To: dev@kafka.apache.org <dev@kafka.apache.org> > Subject: Re: Significant startup delay after upgrading kafka clients from > 3.5.1 to 4.0.0 > > [You don't often get email from kumarpritamm...@gmail.com. Learn why this > is important at https://aka.ms/LearnAboutSenderIdentification ] > > EXTERNAL SENDER. Do not click links or open attachments unless you > recognize the sender and know the content is safe. DO NOT provide your > username or password. > > > Hi, > Is this delay experienced on 3.7 versions also? This plugin discovery is > the connect feature and recently a change was rolled out with different > modes of plugin discovery. Is this because of that change? > > On Sat, 2 Aug 2025 at 7:24 PM, Kirk True <k...@kirktrue.pro> wrote: > > > Hi! > > > > I work on the Kafka Java client and I was not familiar with the concept > of > > plugin discovery, but found a writeup here: > > > > > https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2Fdocumentation%2F%23connect_plugindiscovery&data=05%7C02%7CLove.Almgren%40kambi.com%7C071e458ee8c24fe1a80708ddd1cfb06c%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638897412159675530%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=8wYTAJi%2BFGqzfY4h6ER01YJBnRfIrr4eSeqQWnBucng%3D&reserved=0 > <https://kafka.apache.org/documentation/#connect_plugindiscovery> > > > > From reading through that, plugin discovery looks like a Connect feature. > > Unfortunately, I'm not familiar with that area, so I can't provide any > > immediate advice :( > > > > Are you seeing this slow behavior when using Connect, or do you see it > > when using the AdminClient, Consumer, and/or Producer clients? > > > > I'm sure the developers working on Connect would like to know about this > > performance behavior. Can you provide this information in a bug report at > > > https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira&data=05%7C02%7CLove.Almgren%40kambi.com%7C071e458ee8c24fe1a80708ddd1cfb06c%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638897412159695592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=VVJ0cLHbHPAaP76Md13J2HjllIHh4SQeMKDBM6Y3yOg%3D&reserved=0 > ?<https://issues.apache.org/jira> If so, please use the Jira "Component" > > value of "connect" so that the developers working in that area will be > more > > likely to see the issue. > > > > Thanks, > > Kirk > > > > On Mon, Jul 28, 2025, at 12:10 AM, Love Almgren wrote: > > > Hello Kafka devs, > > > > > > We’re in the process of upgrading our Kafka clients from version 3.5.1 > > to 4.0.0, and have observed a significant increase in startup time > related > > to plugin loading. > > > > > > In version 3.5.1, plugin discovery completed in 30-40 seconds. After > > upgrading to 4.0.0, the same step takes over 5 minutes. We also tested > > version 3.9.0, where the delay was still 40 seconds. > > > > > > We were able to work around this issue by explicitly setting: > > > > > > plugin.discovery = service_load > > > > > > This brought the startup time back to expected levels. Still, we’d like > > to better understand: > > > > > > 1. What might be causing the longer discovery times in newer > versions? > > > 2. Is this expected behavior, or might it be a regression? > > > 3. Are there any risks to relying on service_load for plugin > > discovery? > > > > > > Any insights or suggestions would be greatly appreciated. > > > Thanks in advance, > > > Love Almgren > > > CONFIDENTIALITY NOTICE: This email message (and any attachment) is > > intended only for the individual or entity to which it is addressed. The > > information in this email is confidential and may contain information > that > > is legally privileged or exempt from disclosure under applicable law. If > > you are not the intended recipient, you are strictly prohibited from > > reading, using, publishing or disseminating such information and upon > > receipt, must permanently delete the original and destroy any copies. We > > take steps to protect against viruses and other defects but advise you to > > carry out your own checks and precautions as Kambi does not accept any > > liability for any which remain. Thank you for your co-operation. > > > > > > CONFIDENTIALITY NOTICE: This email message (and any attachment) is > intended only for the individual or entity to which it is addressed. The > information in this email is confidential and may contain information that > is legally privileged or exempt from disclosure under applicable law. If > you are not the intended recipient, you are strictly prohibited from > reading, using, publishing or disseminating such information and upon > receipt, must permanently delete the original and destroy any copies. We > take steps to protect against viruses and other defects but advise you to > carry out your own checks and precautions as Kambi does not accept any > liability for any which remain. Thank you for your co-operation. >
