TengYao Chi created KAFKA-20696:
-----------------------------------
Summary: Periodic plugin.deleteTopology cleanup for
naturally-expired streams groups
Key: KAFKA-20696
URL: https://issues.apache.org/jira/browse/KAFKA-20696
Project: Kafka
Issue Type: Sub-task
Reporter: TengYao Chi
Periodic topology description cleanup task: when a plugin is configured, the
broker runs a cleanup cycle every offsets.retention.check.interval.ms to
release plugin-side state for naturally-expired streams groups before the
offset-expiration sweep tombstones them.
Cycle: fan out a read-only eligibility query across the broker's hosted
__consumer_offsets partitions (isEmpty && allOffsetsExpired &&
StoredDescriptionTopologyEpoch != -1), then for each eligible group call
plugin.deleteTopology and, on success, clear StoredDescriptionTopologyEpoch via
a metadata record conditional on the epoch the cycle observed (no-op if a
concurrent setTopology has advanced it). Plugin failures leave the field set;
the same group is retried on the next cycle. Single-flight: cycles don't
overlap.
Gate the existing offset-expiration sweep: when a plugin is configured, streams
groups must satisfy StoredDescriptionTopologyEpoch == -1 before the sweep is
allowed to tombstone them, so plugin-side state is always released ahead of the
in-memory record. Without a plugin the sweep behaves as today; operators
disabling a previously-configured plugin are responsible for plugin-side
cleanup out of band.
Lifecycle: timer starts when a plugin is configured at startup, stops on
shutdown. Cleanup interval is offsets.retention.check.interval.ms.
Metrics under kafka.server:type=group-coordinator-metrics:
streams-group-topology-description-cleanup-cycle-\{rate,count} and
streams-group-topology-description-cleanup-eligible-\{rate,count}. The
per-group plugin.deleteTopology outcome reuses the delete-\{success,error}
sensors introduced by KAFKA-20623.
Tests: cycle behaviour (single-flight, no-op without plugin, per-group
plugin.deleteTopology + conditional clear), the offset-expiration-sweep gate,
and a broker integration test where a streams group expires naturally and the
plugin sees deleteTopology before the group is tombstoned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
