Lucas Brutschy created KAFKA-21486:
--------------------------------------

             Summary: Streams group heartbeat fails with UNKNOWN_SERVER_ERROR 
when a member sets application.server and the group's source topics are missing
                 Key: KAFKA-21486
                 URL: https://issues.apache.org/jira/browse/KAFKA-21486
             Project: Kafka
          Issue Type: Bug
          Components: streams, group-coordinator
    Affects Versions: 4.5.0
            Reporter: Lucas Brutschy


If a member of a streams group sends a heartbeat with a user endpoint 
({{application.server}}) while the group's configured topology has no 
subtopologies, the coordinator fails the heartbeat with 
{{UNKNOWN_SERVER_ERROR}}. A configured topology has no subtopologies when topic 
configuration failed, for example because source topics do not exist. In that 
case the broker should report the missing source topics in the response 
statuses ({{MISSING_SOURCE_TOPICS}}), which it does for members that do not set 
an endpoint. The client instead receives {{StreamsGroupHeartbeatRequest failed 
due to unexpected error UNKNOWN_SERVER_ERROR: null}}, which it treats as fatal, 
so the application stops instead of waiting for the topics to appear.

On the broker, {{EndpointToPartitionsManager.endpointToPartitions}} calls 
{{streamsGroup.configuredTopology().flatMap(ConfiguredTopology::subtopologies).get()}},
 and that Optional is empty:

{code}
java.util.NoSuchElementException: No value present
  at 
EndpointToPartitionsManager.endpointToPartitions(EndpointToPartitionsManager.java:74)
  at 
EndpointToPartitionsManager.maybeEndpointToPartitions(EndpointToPartitionsManager.java:62)
  at StreamsGroup.getOrComputeEndpointToPartitions(StreamsGroup.java:1571)
  at StreamsGroup.buildEndpointToPartitions(StreamsGroup.java:1550)
  at GroupMetadataManager.streamsGroupHeartbeat(GroupMetadataManager.java:2421)
{code}

The failure also hits other members. {{buildEndpointToPartitions}} computes the 
endpoint mapping for every member of the group, so one member with an endpoint 
breaks the heartbeats of all of them.

*Steps to reproduce:* start a Kafka Streams application with 
{{group.protocol=streams}} and {{application.server=host:port}}, using a 
topology whose source topic does not exist. The {{OrdersService}} example from 
the Confluent microservices examples does this before its topics are created. 
The same application without {{application.server}} receives the 
missing-source-topics status and keeps waiting.

*Expected behaviour:* when the configured topology has no subtopologies, 
{{buildEndpointToPartitions}} should return an empty list, since there is 
nothing to map yet, and the heartbeat should return the normal 
missing-source-topics status.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to