[
https://issues.apache.org/jira/browse/KAFKA-21143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
akshatha ramesh updated KAFKA-21143:
------------------------------------
Description:
h2. Problem
A broker configured with an external _SASL_SSL/OAUTHBEARER_ listener can fail
during startup when the broker is intended to act only as an OAuth token
validator and the JWKS endpoint requires a private CA.
h3. Topology
{code}
Inter-broker traffic:
PLAINTEXT
External listener:
SASL_SSL + OAUTHBEARER
External clients:
obtain JWTs from an external IdP and present them to Kafka
Kafka broker:
validates incoming JWTs using the IdP JWKS endpoint
does not need to obtain an OAuth token for itself
{code}
The JWKS endpoint uses HTTPS with a private CA, so trust configuration is
supplied for the OAUTHBEARER listener:
{code:properties}
listener.name.oam.oauthbearer.sasl.jaas.config=
org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required
ssl.truststore.location="/path/to/idp-ca.crt"
ssl.truststore.type="PEM";
listener.name.oam.oauthbearer.sasl.server.callback.handler.class=
org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallbackHandler
listener.name.oam.oauthbearer.sasl.oauthbearer.jwks.endpoint.url=
https:///.../certs
security.inter.broker.protocol=PLAINTEXT
{code}
h2. Actual behavior
During broker startup, initialization of the external SASL listener reaches
\{{OAuthBearerLoginModule.login()}} and fails:
{code}
kafka.network.DataPlaneAcceptor.configure
...
org.apache.kafka.common.network.SaslChannelBuilder.configure
...
org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule.login
javax.security.auth.login.LoginException:
An internal error occurred while retrieving token from callback handler
{code}
The failure occurs during listener initialization; an external client does not
need to connect for the failure to occur.
h2. Existing tokenless server-side behavior
KAFKA-7462 addressed the case where \{{OAuthBearerLoginModule}} is initialized
on a broker even though OAUTHBEARER is not being used for inter-broker
authentication.
The default \{{OAuthBearerUnsecuredLoginCallbackHandler}} detects the tokenless
server-side case using:
{code:java}
if (moduleOptions.isEmpty()) {
callback.token(null);
return;
}
{code}
{\{OAuthBearerLoginModule}} accepts this tokenless login:
{code:java}
identifyToken();
if (tokenRequiringCommit != null)
identifyExtensions();
else
log.debug("Logged in without a token...");
{code}
Therefore, when there are no JAAS module options, the broker can initialize the
OAUTHBEARER listener without obtaining an OAuth token for itself.
Conceptually:
{code}
OAUTHBEARER listener
+
no broker OAuth client role
+
empty JAAS module options
|
v
callback.token(null)
|
v
OAuthBearerLoginModule accepts tokenless login
|
v
listener starts
{code}
h2. Problem with a private-CA JWKS endpoint
For a validation-only listener whose JWKS endpoint requires a private CA, the
JAAS configuration contains TLS trust options:
{code:properties}
ssl.truststore.location="/path/to/idp-ca.crt"
ssl.truststore.type="PEM"
{code}
Consequently:
{code:java}
moduleOptions.isEmpty() == false
{code}
However, these options are present to establish TLS trust for the JWKS
endpoint. They do not represent a requirement for the broker to obtain an OAuth
access token.
Because \{{moduleOptions}} is no longer empty,
\{{OAuthBearerUnsecuredLoginCallbackHandler}} does not take the KAFKA-7462
tokenless path.
It instead continues into its token construction path. Without the required
unsecured-login principal configuration, token construction fails and the error
is eventually surfaced from \{{OAuthBearerLoginModule}} as:
{code}
javax.security.auth.login.LoginException:
An internal error occurred while retrieving token from callback handler
{code}
h2. Expected behavior
The following topology should be possible without requiring the broker to
obtain an OAuth token for the external listener:
{code}
inter-broker = PLAINTEXT
external listener = SASL_SSL/OAUTHBEARER
broker OAuth role = validate incoming JWTs only
JWKS endpoint = HTTPS with private CA
{code}
Providing TLS trust configuration required to access the JWKS endpoint should
not, by itself, require broker-side OAuth token acquisition.
h2. Possible gap
The current \{{moduleOptions.isEmpty()}} condition appears unable to
distinguish between:
{code}
JAAS options representing broker-side OAuth login/token configuration
{code}
and:
{code}
JAAS options required only for TLS trust when accessing the JWKS endpoint
{code}
As a result, adding the TLS trust configuration needed for a private-CA JWKS
endpoint prevents the existing tokenless server-side path from being taken.
h2. Question
Is there an intended supported configuration that allows JWKS TLS trust options
to be supplied while retaining the tokenless server-side behavior introduced by
KAFKA-7462?
If not, should the tokenless server-side detection distinguish TLS/JWKS
configuration from options that actually configure broker-side OAuth token
creation or acquisition?
h2. Related
Confluent Kafka report:
[https://github.com/confluentinc/kafka/issues/2160]
KAFKA-7462:
https://issues.apache.org/jira/browse/KAFKA-7462
was:
h3. Problem
A broker configured with an external {{SASL_SSL/OAUTHBEARER}} listener can fail
during startup when the broker is intended to act only as an OAuth token
validator and the JWKS endpoint requires a private CA.
Example topology:
{{Inter-broker traffic:
PLAINTEXT
External listener:
SASL_SSL + OAUTHBEARER
External clients:
obtain JWTs from an external IdP and present them to Kafka
Kafka broker:
validates incoming JWTs using the IdP JWKS endpoint
does not need to obtain an OAuth token for itself}}
The JWKS endpoint uses HTTPS with a private CA, so trust configuration is
supplied:
{{listener.name.oam.oauthbearer.sasl.jaas.config=\
org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
ssl.truststore.location="/path/to/idp-ca.crt" \
ssl.truststore.type="PEM";
listener.name.oam.oauthbearer.sasl.server.callback.handler.class=\
org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallbackHandler
listener.name.oam.oauthbearer.sasl.oauthbearer.jwks.endpoint.url=\
https://<idp>/.../certs
security.inter.broker.protocol=PLAINTEXT}}
During broker startup, initialization of the external SASL listener reaches
{{OAuthBearerLoginModule.login()}} and fails:
{{kafka.network.DataPlaneAcceptor.configure
...
org.apache.kafka.common.network.SaslChannelBuilder.configure
...
org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule.login
javax.security.auth.login.LoginException:
An internal error occurred while retrieving token from callback handler}}
h3. Existing tokenless server-side behavior
KAFKA-7462 addressed the case where {{OAuthBearerLoginModule}} is initialized
on a broker even though OAUTHBEARER is not being used for inter-broker
authentication.
The default {{OAuthBearerUnsecuredLoginCallbackHandler}} currently identifies
the tokenless server-side case using:
{{if (moduleOptions.isEmpty()) \{
callback.token(null);
return;
}}}
{{OAuthBearerLoginModule}} accepts this tokenless login:
{{identifyToken();
if (tokenRequiringCommit != null)
identifyExtensions();
else
log.debug("Logged in without a token...");}}
Therefore an OAUTHBEARER broker listener with no broker OAuth client role can
start when the JAAS module options are empty.
h3. Problem with private JWKS CA configuration
For a validation-only listener whose JWKS endpoint requires a private CA, the
JAAS options are not empty:
{{ssl.truststore.location="/path/to/idp-ca.crt"
ssl.truststore.type="PEM"}}
As a result:
{{moduleOptions.isEmpty() == false}}
even though these options configure TLS trust for JWKS validation and do not
indicate that the broker needs to acquire an OAuth token.
The default login callback therefore does not take the KAFKA-7462 tokenless
path. It proceeds into the login/token creation path, and broker startup fails
when no broker OAuth principal/token-acquisition configuration exists.
h3. Expected behavior
A broker should be able to operate with:
{{inter-broker = PLAINTEXT
external listener = SASL_SSL/OAUTHBEARER
broker role = validate incoming JWT only
JWKS endpoint = HTTPS with private CA}}
without requiring the broker to obtain an OAuth token for that listener.
Providing TLS trust configuration required for the JWKS endpoint should not by
itself cause the listener to be treated as requiring broker-side OAuth
login/token acquisition.
h3. Question / Possible gap
Is there an intended supported configuration that allows JWKS TLS trust options
to be supplied while retaining the tokenless server-side behavior introduced by
KAFKA-7462?
If not, the current {{moduleOptions.isEmpty()}} check appears unable to
distinguish:
{{JAAS options representing broker OAuth login configuration}}
from:
{{JAAS options required only for server-side JWKS TLS trust}}
which prevents the tokenless server-only OAUTHBEARER use case when the JWKS
endpoint uses a private CA.
Related report:
[https://github.com/confluentinc/kafka/issues/2160]
> Server-only OAUTHBEARER listener with private JWKS CA triggers token login
> and fails broker startup
> ---------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21143
> URL: https://issues.apache.org/jira/browse/KAFKA-21143
> Project: Kafka
> Issue Type: Bug
> Components: security
> Affects Versions: 4.2.2
> Environment: Apache Kafka 4.2.0
> Reporter: akshatha ramesh
> Priority: Critical
>
> h2. Problem
> A broker configured with an external _SASL_SSL/OAUTHBEARER_ listener can fail
> during startup when the broker is intended to act only as an OAuth token
> validator and the JWKS endpoint requires a private CA.
> h3. Topology
> {code}
> Inter-broker traffic:
> PLAINTEXT
> External listener:
> SASL_SSL + OAUTHBEARER
> External clients:
> obtain JWTs from an external IdP and present them to Kafka
> Kafka broker:
> validates incoming JWTs using the IdP JWKS endpoint
> does not need to obtain an OAuth token for itself
> {code}
> The JWKS endpoint uses HTTPS with a private CA, so trust configuration is
> supplied for the OAUTHBEARER listener:
> {code:properties}
> listener.name.oam.oauthbearer.sasl.jaas.config=
> org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required
> ssl.truststore.location="/path/to/idp-ca.crt"
> ssl.truststore.type="PEM";
> listener.name.oam.oauthbearer.sasl.server.callback.handler.class=
> org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallbackHandler
> listener.name.oam.oauthbearer.sasl.oauthbearer.jwks.endpoint.url=
> https:///.../certs
> security.inter.broker.protocol=PLAINTEXT
> {code}
> h2. Actual behavior
> During broker startup, initialization of the external SASL listener reaches
> \{{OAuthBearerLoginModule.login()}} and fails:
> {code}
> kafka.network.DataPlaneAcceptor.configure
> ...
> org.apache.kafka.common.network.SaslChannelBuilder.configure
> ...
> org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule.login
> javax.security.auth.login.LoginException:
> An internal error occurred while retrieving token from callback handler
> {code}
> The failure occurs during listener initialization; an external client does
> not need to connect for the failure to occur.
> h2. Existing tokenless server-side behavior
> KAFKA-7462 addressed the case where \{{OAuthBearerLoginModule}} is
> initialized on a broker even though OAUTHBEARER is not being used for
> inter-broker authentication.
> The default \{{OAuthBearerUnsecuredLoginCallbackHandler}} detects the
> tokenless server-side case using:
> {code:java}
> if (moduleOptions.isEmpty()) {
> callback.token(null);
> return;
> }
> {code}
> {\{OAuthBearerLoginModule}} accepts this tokenless login:
> {code:java}
> identifyToken();
> if (tokenRequiringCommit != null)
> identifyExtensions();
> else
> log.debug("Logged in without a token...");
> {code}
> Therefore, when there are no JAAS module options, the broker can initialize
> the OAUTHBEARER listener without obtaining an OAuth token for itself.
> Conceptually:
> {code}
> OAUTHBEARER listener
> +
> no broker OAuth client role
> +
> empty JAAS module options
> |
> v
> callback.token(null)
> |
> v
> OAuthBearerLoginModule accepts tokenless login
> |
> v
> listener starts
> {code}
> h2. Problem with a private-CA JWKS endpoint
> For a validation-only listener whose JWKS endpoint requires a private CA, the
> JAAS configuration contains TLS trust options:
> {code:properties}
> ssl.truststore.location="/path/to/idp-ca.crt"
> ssl.truststore.type="PEM"
> {code}
> Consequently:
> {code:java}
> moduleOptions.isEmpty() == false
> {code}
> However, these options are present to establish TLS trust for the JWKS
> endpoint. They do not represent a requirement for the broker to obtain an
> OAuth access token.
> Because \{{moduleOptions}} is no longer empty,
> \{{OAuthBearerUnsecuredLoginCallbackHandler}} does not take the KAFKA-7462
> tokenless path.
> It instead continues into its token construction path. Without the required
> unsecured-login principal configuration, token construction fails and the
> error is eventually surfaced from \{{OAuthBearerLoginModule}} as:
> {code}
> javax.security.auth.login.LoginException:
> An internal error occurred while retrieving token from callback handler
> {code}
> h2. Expected behavior
> The following topology should be possible without requiring the broker to
> obtain an OAuth token for the external listener:
> {code}
> inter-broker = PLAINTEXT
> external listener = SASL_SSL/OAUTHBEARER
> broker OAuth role = validate incoming JWTs only
> JWKS endpoint = HTTPS with private CA
> {code}
> Providing TLS trust configuration required to access the JWKS endpoint should
> not, by itself, require broker-side OAuth token acquisition.
> h2. Possible gap
> The current \{{moduleOptions.isEmpty()}} condition appears unable to
> distinguish between:
> {code}
> JAAS options representing broker-side OAuth login/token configuration
> {code}
> and:
> {code}
> JAAS options required only for TLS trust when accessing the JWKS endpoint
> {code}
> As a result, adding the TLS trust configuration needed for a private-CA JWKS
> endpoint prevents the existing tokenless server-side path from being taken.
> h2. Question
> Is there an intended supported configuration that allows JWKS TLS trust
> options to be supplied while retaining the tokenless server-side behavior
> introduced by KAFKA-7462?
> If not, should the tokenless server-side detection distinguish TLS/JWKS
> configuration from options that actually configure broker-side OAuth token
> creation or acquisition?
> h2. Related
> Confluent Kafka report:
> [https://github.com/confluentinc/kafka/issues/2160]
> KAFKA-7462:
> https://issues.apache.org/jira/browse/KAFKA-7462
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
