Thank you for the explanation, Chris.
In case it helps, what I'm looking for is similar to KIP 577[1]. My specific
example involves a hard-coded key/value pair that needs to be used for
pod-to-pod as I can connect to any worker without that specific header, but
workers cannot communicate among themselves without it.
To clarify, my environment is behind Istio[2], where Egress Traffic can be
created using the following format: `<service name>.<namespace
name>.svc.cluster.local`. For example, a request among workers should be:
curl -H "Host: <service name>.<namespace name>.svc.cluster.local" workerIP:PORT
Regarding temporary solutions, I've explored options like utilizing a proxy but
I am running within containers that can complicate it further, along with the
possibilities of patching, recompiling, or replacing the connect-runtime jar
temporarily. I think that something like this might work but I need to test it
:
private static void addHeadersToRequest(HttpHeaders headers, Request req) {
req.header("Host","<service name>.<namespace name>.svc.cluster.local");
if (headers != null) {
String credentialAuthorization =
headers.getHeaderString(HttpHeaders.AUTHORIZATION);
if (credentialAuthorization != null) {
req.header(HttpHeaders.AUTHORIZATION, credentialAuthorization);
}
}
}
This is of course risky and it would be significantly more convenient if this
functionality is integrated into Kafka Connect itself
[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP+577%3A+Allow+HTTP+Response+Headers+to+be+Configured+for+Kafka+Connect
[2] https://istio.io/
---- On Sat, 07 Oct 2023 02:05:14 -0400 Chris Egerton <fearthecel...@gmail.com>
wrote ---
Hi Yeikel,
Neat question! And thanks for the link to the RestClient code; very helpful.
I don't believe there's a way to configure Kafka Connect to add these
headers to forwarded requests right now. You may be able to do some kind of
out-of-band proxy magic to intercept forwarded requests and insert the
proper headers there.
I don't see a reason for Kafka Connect to only forward authorization
headers, even after examining the PR [1] and corresponding Jira ticket [2]
that altered the RestClient class to begin including authorization headers
in forwarded REST requests. We may be able to tweak the RestClient to
include all headers instead of just the authorization header. I know that
this doesn't help your immediate situation, but if other committers and
contributors agree that the change would be beneficial, we may be able to
include it in the next release (which may be 3.7.0, or a patch release for
3.4, 3.5, or 3.6). Alternatively, we may have to gate such a change behind
a feature flag (either a coarse-grained boolean that enables/disables
forwarding of all non-authorization headers, or more fine-grained logic
such as include/exclude lists or even regexes), which would require a KIP
and may take longer to release.
I've CC'd the dev list to gather their perspective on this potential
change, and to solicit their input on possible workarounds that may be
useful to you sooner than the next release takes place.
[1] - https://github.com/apache/kafka/pull/6791
[2] - https://issues.apache.org/jira/browse/KAFKA-8404
Cheers,
Chris
On Fri, Oct 6, 2023 at 10:14 PM Yeikel Santana <mailto:em...@yeikel.com> wrote:
> Hello everyone,
>
> I'm currently running Kafka Connect behind a firewall that mandates the
> inclusion of a specific header. This situation becomes particularly
> challenging when forwarding requests among multiple workers, as it appears
> that only the Authorization header is included in the request.
>
> I'm wondering if there's a way to customize the headers of Kafka Connect
> before they are forwarded between workers. From my observations, it seems
> that this capability may not be available[1], and only the response headers
> can be customized.
>
> I'd appreciate any realistic alternatives or suggestions you may have in
> mind.
>
> Thanks!
>
>
>
>
>
>
> [1]
> https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestClient.java#L191-L198
