[
https://issues.apache.org/jira/browse/KAFKA-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Zeger updated KAFKA-20644:
--------------------------------
Description:
The Kafka Connect REST server lets operators expose admin endpoints on a
separate listener via admin.listeners. RestServer handles three intended cases:
# admin.listeners unset (null) → admin resources are served on the main
listeners.
# admin.listeners set to an empty list → admin resources are disabled.
# admin.listeners set to values distinct from listeners → admin resources are
served on a separate Jetty connector.
It does not handle a fourth case: when admin.listeners overlaps listeners
(shares a host:port). The code explicitly assumes the two are different - see
the TODOs in RestServer.initializeResources:
{code:java}
//
connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java:245-246
// TODO: we need to check if these listeners are same as 'listeners'
// TODO: the following code assumes that they are different {code}
There is no cross-validation of the two configs. admin.listeners is validated
only by anyNonDuplicateValues in RestServerConfig.java, checks for duplicates
within itself), and listeners uses ListenersValidator in RestServerConfig.java.
Neither compares the two lists.
What happens with overlapping config:
{code:java}
listeners=http://0.0.0.0:8083
admin.listeners=http://0.0.0.0:8083 {code}
RestServer.createConnectors unconditionally creates a Jetty ServerConnector per
main listener and a separate admin ServerConnector per admin listener. With the
config above this produces two connectors bound to the identical host:port - a
main connector named http_0.0.0.08083 (RestServer.java:180) and an admin
connector named Admin (RestServer.java:185). initializeResources then binds the
admin context to the @Admin connector via virtual host (RestServer.java:270).
Because nothing enables SO_REUSEPORT, two listening sockets on the same
host:port collide. The expected result is that the second bind() fails at
jettyServer.start() with BindException: Address already in use, which
initializeServer wraps as ConnectException("Unable to initialize REST server")
— i.e. the worker fails to start, with an error message that gives no hint
the real cause is admin.listeners overlapping listeners.
(Note: the precise runtime manifestation — startup bind failure vs. a running
server with broken admin endpoints — should be
confirmed with a reproduction test as part of the fix. Code analysis points
to a startup BindException. The earlier "404 / orphaned
resources" framing appears inaccurate: the admin connector is created; it's
the bind that conflicts.)
Either way, a plausible misconfiguration (a user wanting admin endpoints on
the main port may set the two equal, not realizing that
leaving admin.listeners unset already does exactly that) yields a confusing
failure with no actionable message.
Proposed solutions
- Option A: treat an overlapping/identical admin.listeners as unset (inherit
admin resources onto the main listeners).
- Option B (recommended): fail fast. Validate early (before any connector is
built) that no admin.listeners entry shares a host:port
with any listeners entry, and throw a ConfigException directing the user to
use distinct host:port(s) or to leave admin.listeners
unset to share the main listener.
Option B is recommended because it is explicit, naturally covers partial
overlap (e.g. listeners=...:8083,...:9083 with
admin.listeners=...:8083), and avoids silently reinterpreting a non-null
config as null. The unset/null path already provides Option
A's intended "admin on the main listener" behavior.
was:
The Kafka Connect REST server lets operators expose admin endpoints on a
separate listener via admin.listeners. RestServer handles
three intended cases:
1. admin.listeners unset (null) → admin resources are served on the main
listeners.
2. admin.listeners set to an empty list → admin resources are disabled.
3. admin.listeners set to values distinct from listeners → admin resources
are served on a separate Jetty connector.
It does not handle a fourth case: when admin.listeners overlaps listeners
(shares a host:port). The code explicitly assumes the two
are different — see the TODOs in RestServer.initializeResources:
//
connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java:245-246
// TODO: we need to check if these listeners are same as 'listeners'
// TODO: the following code assumes that they are different
There is no cross-validation of the two configs. admin.listeners is validated
only by anyNonDuplicateValues
(RestServerConfig.java:148-153, checks for duplicates within itself), and
listeners uses ListenersValidator
(RestServerConfig.java:166, validates URL format only). Neither compares the
two lists.
What happens with overlapping config
Example:
listeners=http://0.0.0.0:8083
admin.listeners=http://0.0.0.0:8083
RestServer.createConnectors (RestServer.java:120-138) unconditionally creates
a Jetty ServerConnector per main listener and a
separate admin ServerConnector per admin listener. With the config above this
produces two connectors bound to the identical
host:port — a main connector named http_0.0.0.08083 (RestServer.java:180) and
an admin connector named Admin (RestServer.java:185).
initializeResources then binds the admin context to the @Admin connector via
virtual host (RestServer.java:270).
Because nothing enables SO_REUSEPORT, two listening sockets on the same
host:port collide. The expected result is that the second
bind() fails at jettyServer.start() (RestServer.java:215) with BindException:
Address already in use, which initializeServer wraps
as ConnectException("Unable to initialize REST server") — i.e. the worker
fails to start, with an error message that gives no hint
the real cause is admin.listeners overlapping listeners.
(Note: the precise runtime manifestation — startup bind failure vs. a running
server with broken admin endpoints — should be
confirmed with a reproduction test as part of the fix. Code analysis points
to a startup BindException. The earlier "404 / orphaned
resources" framing appears inaccurate: the admin connector is created; it's
the bind that conflicts.)
Either way, a plausible misconfiguration (a user wanting admin endpoints on
the main port may set the two equal, not realizing that
leaving admin.listeners unset already does exactly that) yields a confusing
failure with no actionable message.
Proposed solutions
- Option A: treat an overlapping/identical admin.listeners as unset (inherit
admin resources onto the main listeners).
- Option B (recommended): fail fast. Validate early (before any connector is
built) that no admin.listeners entry shares a host:port
with any listeners entry, and throw a ConfigException directing the user to
use distinct host:port(s) or to leave admin.listeners
unset to share the main listener.
Option B is recommended because it is explicit, naturally covers partial
overlap (e.g. listeners=...:8083,...:9083 with
admin.listeners=...:8083), and avoids silently reinterpreting a non-null
config as null. The unset/null path already provides Option
A's intended "admin on the main listener" behavior.
> Connect’s RestServer assumes admin.listeners differ from listeners, silently
> orphaning admin resources when they match
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20644
> URL: https://issues.apache.org/jira/browse/KAFKA-20644
> Project: Kafka
> Issue Type: Bug
> Components: connect
> Reporter: Pavel Zeger
> Assignee: Pavel Zeger
> Priority: Major
>
> The Kafka Connect REST server lets operators expose admin endpoints on a
> separate listener via admin.listeners. RestServer handles three intended
> cases:
> # admin.listeners unset (null) → admin resources are served on the main
> listeners.
> # admin.listeners set to an empty list → admin resources are disabled.
> # admin.listeners set to values distinct from listeners → admin resources
> are served on a separate Jetty connector.
> It does not handle a fourth case: when admin.listeners overlaps listeners
> (shares a host:port). The code explicitly assumes the two are different - see
> the TODOs in RestServer.initializeResources:
> {code:java}
> //
> connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java:245-246
> // TODO: we need to check if these listeners are same as 'listeners'
> // TODO: the following code assumes that they are different {code}
> There is no cross-validation of the two configs. admin.listeners is validated
> only by anyNonDuplicateValues in RestServerConfig.java, checks for duplicates
> within itself), and listeners uses ListenersValidator in
> RestServerConfig.java. Neither compares the two lists.
>
> What happens with overlapping config:
> {code:java}
> listeners=http://0.0.0.0:8083
> admin.listeners=http://0.0.0.0:8083 {code}
> RestServer.createConnectors unconditionally creates a Jetty ServerConnector
> per main listener and a separate admin ServerConnector per admin listener.
> With the config above this produces two connectors bound to the identical
> host:port - a main connector named http_0.0.0.08083 (RestServer.java:180) and
> an admin connector named Admin (RestServer.java:185). initializeResources
> then binds the admin context to the @Admin connector via virtual host
> (RestServer.java:270).
>
> Because nothing enables SO_REUSEPORT, two listening sockets on the same
> host:port collide. The expected result is that the second bind() fails at
> jettyServer.start() with BindException: Address already in use, which
> initializeServer wraps as ConnectException("Unable to initialize REST
> server") — i.e. the worker fails to start, with an error message that gives
> no hint
> the real cause is admin.listeners overlapping listeners.
> (Note: the precise runtime manifestation — startup bind failure vs. a
> running server with broken admin endpoints — should be
> confirmed with a reproduction test as part of the fix. Code analysis points
> to a startup BindException. The earlier "404 / orphaned
> resources" framing appears inaccurate: the admin connector is created; it's
> the bind that conflicts.)
>
> Either way, a plausible misconfiguration (a user wanting admin endpoints on
> the main port may set the two equal, not realizing that
> leaving admin.listeners unset already does exactly that) yields a confusing
> failure with no actionable message.
> Proposed solutions
> - Option A: treat an overlapping/identical admin.listeners as unset
> (inherit admin resources onto the main listeners).
> - Option B (recommended): fail fast. Validate early (before any connector
> is built) that no admin.listeners entry shares a host:port
> with any listeners entry, and throw a ConfigException directing the user to
> use distinct host:port(s) or to leave admin.listeners
> unset to share the main listener.
> Option B is recommended because it is explicit, naturally covers partial
> overlap (e.g. listeners=...:8083,...:9083 with
> admin.listeners=...:8083), and avoids silently reinterpreting a non-null
> config as null. The unset/null path already provides Option
> A's intended "admin on the main listener" behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
