TaiJuWu created KAFKA-21463:
-------------------------------

             Summary: Share group ignores share.auto.offset.reset=latest when 
the initialization of a new topic is retried
                 Key: KAFKA-21463
                 URL: https://issues.apache.org/jira/browse/KAFKA-21463
             Project: Kafka
          Issue Type: Bug
            Reporter: TaiJuWu
            Assignee: TaiJuWu


When a share group subscribes to a new topic, the group coordinator sends 
`InitializeShareGroupState` with start offset -1, so that the partition leader 
applies `share.auto.offset.reset`. If the share coordinator does not answer 
within `group.share.initialize.retry.interval.ms`, the group coordinator 
retries. The retry sends start offset 0, because the topic is already in 
`initializingTopics` and is treated as a known topic whose new partitions start 
at 0.

The retry has a higher state epoch, so the share coordinator keeps it: if it 
arrives first, the first request fails with `FENCED_STATE_EPOCH`, and if it 
arrives second, it replaces the first one. The persisted start offset ends up 
as 0, and a group with `share.auto.offset.reset=latest` reads the whole topic 
from the beginning. It is only avoided when the partition leader happens to 
initialize the share partition from the first request before the retry replaces 
it, and then the persisted state does not match what the leader delivers.

*Steps*

# 3 brokers, `__share_group_state` with 1 partition and replication factor 1, 
`group.share.initialize.retry.interval.ms=2000`.
# Create a topic on the brokers which do not host the share state partition, 
and produce 100 records.
# Shut down the broker hosting the share state partition.
# Two share consumers in a group with `share.auto.offset.reset=latest` 
subscribe to the topic and poll for 10 seconds, so the initialization is 
retried.
# Start the broker again and wait for the partition to be assigned.
# Produce records until one is delivered, so that the latest start offset is 
resolved, and then produce 10 more records.

*Expected:* none of the 100 records produced before the subscription is 
delivered, and the 10 new records are.
*Actual:* the 100 records produced before the subscription are delivered. This 
happened in 13 out of 13 runs on trunk. The broker logs `Received error while 
calling initialize state for <group> on persister, errorCode: 124`, which shows 
that two initialize requests were sent.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to