TaiJuWu created KAFKA-21463:
-------------------------------
Summary: Share group ignores share.auto.offset.reset=latest when
the initialization of a new topic is retried
Key: KAFKA-21463
URL: https://issues.apache.org/jira/browse/KAFKA-21463
Project: Kafka
Issue Type: Bug
Reporter: TaiJuWu
Assignee: TaiJuWu
When a share group subscribes to a new topic, the group coordinator sends
`InitializeShareGroupState` with start offset -1, so that the partition leader
applies `share.auto.offset.reset`. If the share coordinator does not answer
within `group.share.initialize.retry.interval.ms`, the group coordinator
retries. The retry sends start offset 0, because the topic is already in
`initializingTopics` and is treated as a known topic whose new partitions start
at 0.
The retry has a higher state epoch, so the share coordinator keeps it: if it
arrives first, the first request fails with `FENCED_STATE_EPOCH`, and if it
arrives second, it replaces the first one. The persisted start offset ends up
as 0, and a group with `share.auto.offset.reset=latest` reads the whole topic
from the beginning. It is only avoided when the partition leader happens to
initialize the share partition from the first request before the retry replaces
it, and then the persisted state does not match what the leader delivers.
*Steps*
# 3 brokers, `__share_group_state` with 1 partition and replication factor 1,
`group.share.initialize.retry.interval.ms=2000`.
# Create a topic on the brokers which do not host the share state partition,
and produce 100 records.
# Shut down the broker hosting the share state partition.
# Two share consumers in a group with `share.auto.offset.reset=latest`
subscribe to the topic and poll for 10 seconds, so the initialization is
retried.
# Start the broker again and wait for the partition to be assigned.
# Produce records until one is delivered, so that the latest start offset is
resolved, and then produce 10 more records.
*Expected:* none of the 100 records produced before the subscription is
delivered, and the 10 new records are.
*Actual:* the 100 records produced before the subscription are delivered. This
happened in 13 out of 13 runs on trunk. The broker logs `Received error while
calling initialize state for <group> on persister, errorCode: 124`, which shows
that two initialize requests were sent.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)