[
https://issues.apache.org/jira/browse/KAFKA-20696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TengYao Chi reassigned KAFKA-20696:
-----------------------------------
Assignee: TengYao Chi
> Periodic plugin.deleteTopology cleanup for naturally-expired streams groups
> ---------------------------------------------------------------------------
>
> Key: KAFKA-20696
> URL: https://issues.apache.org/jira/browse/KAFKA-20696
> Project: Kafka
> Issue Type: Sub-task
> Reporter: TengYao Chi
> Assignee: TengYao Chi
> Priority: Major
>
> Periodic topology description cleanup task: when a plugin is configured, the
> broker runs a cleanup cycle every offsets.retention.check.interval.ms to
> release plugin-side state for naturally-expired streams groups before the
> offset-expiration sweep tombstones them.
> Cycle: fan out a read-only eligibility query across the broker's hosted
> __consumer_offsets partitions (isEmpty && allOffsetsExpired &&
> StoredDescriptionTopologyEpoch != -1), then for each eligible group call
> plugin.deleteTopology and, on success, clear StoredDescriptionTopologyEpoch
> via a metadata record conditional on the epoch the cycle observed (no-op if a
> concurrent setTopology has advanced it). Plugin failures leave the field set;
> the same group is retried on the next cycle. Single-flight: cycles don't
> overlap.
> Gate the existing offset-expiration sweep: when a plugin is configured,
> streams groups must satisfy StoredDescriptionTopologyEpoch == -1 before the
> sweep is allowed to tombstone them, so plugin-side state is always released
> ahead of the in-memory record. Without a plugin the sweep behaves as today;
> operators disabling a previously-configured plugin are responsible for
> plugin-side cleanup out of band.
> Lifecycle: timer starts when a plugin is configured at startup, stops on
> shutdown. Cleanup interval is offsets.retention.check.interval.ms.
> Metrics under kafka.server:type=group-coordinator-metrics:
> streams-group-topology-description-cleanup-cycle-\{rate,count} and
> streams-group-topology-description-cleanup-eligible-\{rate,count}. The
> per-group plugin.deleteTopology outcome reuses the delete-\{success,error}
> sensors introduced by KAFKA-20623.
> Tests: cycle behaviour (single-flight, no-op without plugin, per-group
> plugin.deleteTopology + conditional clear), the offset-expiration-sweep gate,
> and a broker integration test where a streams group expires naturally and the
> plugin sees deleteTopology before the group is tombstoned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
