[
https://issues.apache.org/jira/browse/KAFKA-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Zeger updated KAFKA-20644:
--------------------------------
Description:
The Kafka Connect REST server lets operators expose admin endpoints on a
separate listener via admin.listeners. RestServer handles
three intended cases:
1. admin.listeners unset (null) → admin resources are served on the main
listeners.
2. admin.listeners set to an empty list → admin resources are disabled.
3. admin.listeners set to values distinct from listeners → admin resources
are served on a separate Jetty connector.
It does not handle a fourth case: when admin.listeners overlaps listeners
(shares a host:port). The code explicitly assumes the two
are different — see the TODOs in RestServer.initializeResources:
//
connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java:245-246
// TODO: we need to check if these listeners are same as 'listeners'
// TODO: the following code assumes that they are different
There is no cross-validation of the two configs. admin.listeners is validated
only by anyNonDuplicateValues
(RestServerConfig.java:148-153, checks for duplicates within itself), and
listeners uses ListenersValidator
(RestServerConfig.java:166, validates URL format only). Neither compares the
two lists.
What happens with overlapping config
Example:
listeners=http://0.0.0.0:8083
admin.listeners=http://0.0.0.0:8083
RestServer.createConnectors (RestServer.java:120-138) unconditionally creates
a Jetty ServerConnector per main listener and a
separate admin ServerConnector per admin listener. With the config above this
produces two connectors bound to the identical
host:port — a main connector named http_0.0.0.08083 (RestServer.java:180) and
an admin connector named Admin (RestServer.java:185).
initializeResources then binds the admin context to the @Admin connector via
virtual host (RestServer.java:270).
Because nothing enables SO_REUSEPORT, two listening sockets on the same
host:port collide. The expected result is that the second
bind() fails at jettyServer.start() (RestServer.java:215) with BindException:
Address already in use, which initializeServer wraps
as ConnectException("Unable to initialize REST server") — i.e. the worker
fails to start, with an error message that gives no hint
the real cause is admin.listeners overlapping listeners.
(Note: the precise runtime manifestation — startup bind failure vs. a running
server with broken admin endpoints — should be
confirmed with a reproduction test as part of the fix. Code analysis points
to a startup BindException. The earlier "404 / orphaned
resources" framing appears inaccurate: the admin connector is created; it's
the bind that conflicts.)
Either way, a plausible misconfiguration (a user wanting admin endpoints on
the main port may set the two equal, not realizing that
leaving admin.listeners unset already does exactly that) yields a confusing
failure with no actionable message.
Proposed solutions
- Option A: treat an overlapping/identical admin.listeners as unset (inherit
admin resources onto the main listeners).
- Option B (recommended): fail fast. Validate early (before any connector is
built) that no admin.listeners entry shares a host:port
with any listeners entry, and throw a ConfigException directing the user to
use distinct host:port(s) or to leave admin.listeners
unset to share the main listener.
Option B is recommended because it is explicit, naturally covers partial
overlap (e.g. listeners=...:8083,...:9083 with
admin.listeners=...:8083), and avoids silently reinterpreting a non-null
config as null. The unset/null path already provides Option
A's intended "admin on the main listener" behavior.
was:
The Connect REST server supports separating admin endpoints (/loggers, etc.)
onto a different listener via admin.listeners. The intended cases:
> admin.listeners not set (null) → admin resources go on the main listeners.
> admin.listeners set to empty list → admin resources disabled.
> admin.listeners set to different values from listeners → admin resources go
> on a separate connector.
The TODOs flag the missing case:
> admin.listeners set to the same values as listeners.
Today the code enters the else \{ else{ ... }} branch, creates a brand-new
ResourceConfig for admin resources, and proceeds - but the new ResourceConfig
is bound to a Jetty connector that won’t be created (because listeners and
admin.listeners resolve to the same Jetty connector). The admin resources
become orphaned. Operators see /loggers return 404 on the listener they
configured for both regular and admin traffic.
Reproduction:
{code:java}
# connect-distributed.properties
listeners=http://0.0.0.0:8083
admin.listeners=http://0.0.0.0:8083 {code}
It’s a low-stakes but deeply confusing failure mode.
The operator: Reads docs that say admin.listeners exposes admin resources on a
separate listener. Configures them to the same value (e.g. because they want
the admin features but don’t want a separate port). Hits 404 with no clear
error.
Either case should be handled gracefully: If the user wants admin resources on
the same port, set admin.listeners to null (or to the same string and have the
code recognise it). The current code’s silent orphaning is the worst possible
outcome.
*Proposed fix*
# Option A: treat matching listeners as null
# Option B: fail fast
*KIP needed?*
No - edge case for already broken flow by a design
> Connect’s RestServer assumes admin.listeners differ from listeners, silently
> orphaning admin resources when they match
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20644
> URL: https://issues.apache.org/jira/browse/KAFKA-20644
> Project: Kafka
> Issue Type: Bug
> Components: connect
> Reporter: Pavel Zeger
> Assignee: Pavel Zeger
> Priority: Major
>
> The Kafka Connect REST server lets operators expose admin endpoints on a
> separate listener via admin.listeners. RestServer handles
> three intended cases:
> 1. admin.listeners unset (null) → admin resources are served on the main
> listeners.
> 2. admin.listeners set to an empty list → admin resources are disabled.
> 3. admin.listeners set to values distinct from listeners → admin resources
> are served on a separate Jetty connector.
> It does not handle a fourth case: when admin.listeners overlaps listeners
> (shares a host:port). The code explicitly assumes the two
> are different — see the TODOs in RestServer.initializeResources:
> //
> connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java:245-246
> // TODO: we need to check if these listeners are same as 'listeners'
> // TODO: the following code assumes that they are different
> There is no cross-validation of the two configs. admin.listeners is
> validated only by anyNonDuplicateValues
> (RestServerConfig.java:148-153, checks for duplicates within itself), and
> listeners uses ListenersValidator
> (RestServerConfig.java:166, validates URL format only). Neither compares
> the two lists.
>
> What happens with overlapping config
> Example:
> listeners=http://0.0.0.0:8083
> admin.listeners=http://0.0.0.0:8083
>
> RestServer.createConnectors (RestServer.java:120-138) unconditionally
> creates a Jetty ServerConnector per main listener and a
> separate admin ServerConnector per admin listener. With the config above
> this produces two connectors bound to the identical
> host:port — a main connector named http_0.0.0.08083 (RestServer.java:180)
> and an admin connector named Admin (RestServer.java:185).
> initializeResources then binds the admin context to the @Admin connector
> via virtual host (RestServer.java:270).
>
> Because nothing enables SO_REUSEPORT, two listening sockets on the same
> host:port collide. The expected result is that the second
> bind() fails at jettyServer.start() (RestServer.java:215) with
> BindException: Address already in use, which initializeServer wraps
> as ConnectException("Unable to initialize REST server") — i.e. the worker
> fails to start, with an error message that gives no hint
> the real cause is admin.listeners overlapping listeners.
> (Note: the precise runtime manifestation — startup bind failure vs. a
> running server with broken admin endpoints — should be
> confirmed with a reproduction test as part of the fix. Code analysis points
> to a startup BindException. The earlier "404 / orphaned
> resources" framing appears inaccurate: the admin connector is created; it's
> the bind that conflicts.)
>
> Either way, a plausible misconfiguration (a user wanting admin endpoints on
> the main port may set the two equal, not realizing that
> leaving admin.listeners unset already does exactly that) yields a confusing
> failure with no actionable message.
> Proposed solutions
> - Option A: treat an overlapping/identical admin.listeners as unset
> (inherit admin resources onto the main listeners).
> - Option B (recommended): fail fast. Validate early (before any connector
> is built) that no admin.listeners entry shares a host:port
> with any listeners entry, and throw a ConfigException directing the user to
> use distinct host:port(s) or to leave admin.listeners
> unset to share the main listener.
> Option B is recommended because it is explicit, naturally covers partial
> overlap (e.g. listeners=...:8083,...:9083 with
> admin.listeners=...:8083), and avoids silently reinterpreting a non-null
> config as null. The unset/null path already provides Option
> A's intended "admin on the main listener" behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
