Lucas Brutschy created KAFKA-21486:
--------------------------------------
Summary: Streams group heartbeat fails with UNKNOWN_SERVER_ERROR
when a member sets application.server and the group's source topics are missing
Key: KAFKA-21486
URL: https://issues.apache.org/jira/browse/KAFKA-21486
Project: Kafka
Issue Type: Bug
Components: streams, group-coordinator
Affects Versions: 4.5.0
Reporter: Lucas Brutschy
If a member of a streams group sends a heartbeat with a user endpoint
({{application.server}}) while the group's configured topology has no
subtopologies, the coordinator fails the heartbeat with
{{UNKNOWN_SERVER_ERROR}}. A configured topology has no subtopologies when topic
configuration failed, for example because source topics do not exist. In that
case the broker should report the missing source topics in the response
statuses ({{MISSING_SOURCE_TOPICS}}), which it does for members that do not set
an endpoint. The client instead receives {{StreamsGroupHeartbeatRequest failed
due to unexpected error UNKNOWN_SERVER_ERROR: null}}, which it treats as fatal,
so the application stops instead of waiting for the topics to appear.
On the broker, {{EndpointToPartitionsManager.endpointToPartitions}} calls
{{streamsGroup.configuredTopology().flatMap(ConfiguredTopology::subtopologies).get()}},
and that Optional is empty:
{code}
java.util.NoSuchElementException: No value present
at
EndpointToPartitionsManager.endpointToPartitions(EndpointToPartitionsManager.java:74)
at
EndpointToPartitionsManager.maybeEndpointToPartitions(EndpointToPartitionsManager.java:62)
at StreamsGroup.getOrComputeEndpointToPartitions(StreamsGroup.java:1571)
at StreamsGroup.buildEndpointToPartitions(StreamsGroup.java:1550)
at GroupMetadataManager.streamsGroupHeartbeat(GroupMetadataManager.java:2421)
{code}
The failure also hits other members. {{buildEndpointToPartitions}} computes the
endpoint mapping for every member of the group, so one member with an endpoint
breaks the heartbeats of all of them.
*Steps to reproduce:* start a Kafka Streams application with
{{group.protocol=streams}} and {{application.server=host:port}}, using a
topology whose source topic does not exist. The {{OrdersService}} example from
the Confluent microservices examples does this before its topics are created.
The same application without {{application.server}} receives the
missing-source-topics status and keeps waiting.
*Expected behaviour:* when the configured topology has no subtopologies,
{{buildEndpointToPartitions}} should return an empty list, since there is
nothing to map yet, and the heartbeat should return the normal
missing-source-topics status.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)