Hi Jamie, This was an issue we ran into several years ago with the basic auth extension that comes OOTB with Connect. TL;DR: there are currently two endpoints that Connect uses for inter-worker, but intra-communication (i.e., REST requests that are made spontaneously from one worker to another, instead of originating with a user-issued request that gets forwarded from one worker to another), and authentication isn't really possible for them.
For obvious reasons, these requests will never have user-supplied credentials, and so they are impossible to authenticate. We could have added some kind of plugin to tweak REST requests as they're made in order to allow credentials to be injected, but instead, we opted for two things: - We just skip authentication on these internal endpoints (see [1]) - We implemented KIP-507 [2], which adds automatic an authentication check on these internal endpoints using session keys periodically generated and written to the topic automatically by workers You should add a check similar to the one in [1] that just skips auth for these endpoints. It won't present a security gap since Connect workers already take steps to ensure that requests to these endpoints are signed with credentials present in the config topic, and if you're running a secure Connect cluster, you should already have the config topic locked down to prevent malicious users from reading from or writing to it. [1] - https://github.com/apache/kafka/blob/e78e106221e7ce72af90b32fe6d6b3b454b1667b/connect/basic-auth-extension/src/main/java/org/apache/kafka/connect/rest/basic/auth/extension/JaasBasicAuthFilter.java#L91 [2] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints Hope this helps, Chris On Tue, Apr 22, 2025 at 5:57 AM Jamie <jamied...@aol.co.uk.invalid> wrote: > Hi All, > > Has anyone managed to get a REST extension working with multiple connect > instances / workers? > Many Thanks, > Jamie > On Wednesday 16 April 2025 at 10:57:01 BST, Jamie <jamied...@aol.co.uk> > wrote: > > Hi All, > I'm trying to implement a custom REST extension for Kafka Connect that > allows a token to be used for authentication. > When I have a single instance of Kafka connect running in distributed mode > (i.e. 1 worker) this works as expected. However, when I add another > instance (another worker) I get an error when the worker that received the > request has to forward it on to the group leader: > Error IO error forwarding REST request: > (org.apache.kafka.connect.runtime.rest.RestClient:143) > java.util.concurrent.ExecutionException: > org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: > Authentication challenge without WWW-Authenticate header > > I noticed that someone also had the same issue with the basic > authentication extension that comes as part of connect: > https://stackoverflow.com/questions/60977591/kafkaconnect-rest-api-basic-authentication-issue > . > Has anyone been able to get either the basic authentication or another > custom authentication extension working with Kafka connect when using more > than one instance / worker? > > I'm using version 2.7.1 of Kafka and Kafka connect. > > Many Thanks, > > Jamie. >
