[
https://issues.apache.org/jira/browse/KAFKA-20627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lucas Brutschy updated KAFKA-20627:
-----------------------------------
Description:
Please use your own judgement to extend the following list of tests that we
already have in the POC. These should be the minum:
StreamsGroupTopologyDescriptionRequestTest.scala (with plugin configured),
StreamsGroupTopologyDescriptionNoPluginRequestTest.scala (UNSUPPORTED_VERSION
behavior). TopologyDescriptionPluginIntegrationTest.java (end-to-end: push →
describe → delete). IntegrationTestHarness, GroupCoordinatorBaseRequestTest.
Add InMemoryTopologyDescriptionPlugin to
EmbeddedKafkaCluster.addDefaultBrokerPropsIfAbsent via putIfAbsent. This
enables the plugin in all streams integration tests without any per-test
changes, ensuring existing tests exercise the topology description path and
catch regressions.
TopologyDescriptionPluginIntegrationTest — 4 tests covering the core push
lifecycle on a live cluster:
- topology description reaches the plugin after join
- two-member group deduplicates: only one push happens
- permanent plugin failure ratchets failedDescriptionTopologyEpoch and stops
re-solicitation
- explicit DeleteGroups triggers plugin.deleteTopology
TopologyDescriptionPluginExpirationIntegrationTest — 2 tests covering natural
group expiry:
- group with stored topology: cleanup cycle calls plugin.deleteTopology and
clears the stored epoch before the offset-expiration sweep tombstones it
- group whose topology push permanently failed (storedEpoch == -1): tombstone
fires directly without any deleteTopology call
*CLI test*
{{To be added to DescribeStreamsGroupTest}}
({{{}tools/src/test/java/org/apache/kafka/tools/streams/DescribeStreamsGroupTest.java{}}}).
Both run against a real {{KafkaStreams}} instance backed by
{{{}EmbeddedKafkaCluster{}}}, which has {{InMemoryTopologyDescriptionPlugin}}
configured by default.
*{{testDescribeStreamsGroupWithTopologyOption}}*
Exercises the {{kafka-streams-groups.sh --describe --topology}} CLI path
end-to-end. The test invokes {{StreamsGroupCommand.execute}} with {{--describe
--topology --group <appId>}} and waits until the output contains
{{"Topologies:"}} and the source topic name. This covers the full flow: the
Streams client sends a {{StreamsGroupTopologyDescriptionUpdate}} RPC when
solicited by the heartbeat, the broker stores the description via the plugin,
and the CLI retrieves and formats it on describe.
*{{}}*
h4. Admin API test
We also need to include a test for the admin API. We could consider extending
PlaintextAdminIntegrationTest, but it doesn't have a streams instance at the
moment, but with some mocking we should be able to make it work.
was:
StreamsGroupTopologyDescriptionRequestTest.scala (with plugin configured),
StreamsGroupTopologyDescriptionNoPluginRequestTest.scala (UNSUPPORTED_VERSION
behavior). TopologyDescriptionPluginIntegrationTest.java (end-to-end: push →
describe → delete). IntegrationTestHarness, GroupCoordinatorBaseRequestTest.
Add InMemoryTopologyDescriptionPlugin to
EmbeddedKafkaCluster.addDefaultBrokerPropsIfAbsent via putIfAbsent. This
enables the plugin in all streams integration tests without any per-test
changes, ensuring existing tests exercise the topology description path and
catch regressions.
TopologyDescriptionPluginIntegrationTest — 4 tests covering the core push
lifecycle on a live cluster:
- topology description reaches the plugin after join
- two-member group deduplicates: only one push happens
- permanent plugin failure ratchets failedDescriptionTopologyEpoch and stops
re-solicitation
- explicit DeleteGroups triggers plugin.deleteTopology
TopologyDescriptionPluginExpirationIntegrationTest — 2 tests covering natural
group expiry:
- group with stored topology: cleanup cycle calls plugin.deleteTopology and
clears the stored epoch before the offset-expiration sweep tombstones it
- group whose topology push permanently failed (storedEpoch == -1): tombstone
fires directly without any deleteTopology call
> Add broker integration tests for topology description plugin
> ------------------------------------------------------------
>
> Key: KAFKA-20627
> URL: https://issues.apache.org/jira/browse/KAFKA-20627
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
> Reporter: Alieh Saeedi
> Assignee: Alieh Saeedi
> Priority: Major
> Fix For: 4.4.0
>
>
> Please use your own judgement to extend the following list of tests that we
> already have in the POC. These should be the minum:
> StreamsGroupTopologyDescriptionRequestTest.scala (with plugin configured),
> StreamsGroupTopologyDescriptionNoPluginRequestTest.scala (UNSUPPORTED_VERSION
> behavior). TopologyDescriptionPluginIntegrationTest.java (end-to-end: push →
> describe → delete). IntegrationTestHarness, GroupCoordinatorBaseRequestTest.
>
> Add InMemoryTopologyDescriptionPlugin to
> EmbeddedKafkaCluster.addDefaultBrokerPropsIfAbsent via putIfAbsent. This
> enables the plugin in all streams integration tests without any per-test
> changes, ensuring existing tests exercise the topology description path and
> catch regressions.
>
> TopologyDescriptionPluginIntegrationTest — 4 tests covering the core push
> lifecycle on a live cluster:
> - topology description reaches the plugin after join
> - two-member group deduplicates: only one push happens
> - permanent plugin failure ratchets failedDescriptionTopologyEpoch and
> stops re-solicitation
> - explicit DeleteGroups triggers plugin.deleteTopology
> TopologyDescriptionPluginExpirationIntegrationTest — 2 tests covering
> natural group expiry:
> - group with stored topology: cleanup cycle calls plugin.deleteTopology and
> clears the stored epoch before the offset-expiration sweep tombstones it
> - group whose topology push permanently failed (storedEpoch == -1):
> tombstone fires directly without any deleteTopology call
>
>
> *CLI test*
> {{To be added to DescribeStreamsGroupTest}}
> ({{{}tools/src/test/java/org/apache/kafka/tools/streams/DescribeStreamsGroupTest.java{}}}).
> Both run against a real {{KafkaStreams}} instance backed by
> {{{}EmbeddedKafkaCluster{}}}, which has {{InMemoryTopologyDescriptionPlugin}}
> configured by default.
> *{{testDescribeStreamsGroupWithTopologyOption}}*
> Exercises the {{kafka-streams-groups.sh --describe --topology}} CLI path
> end-to-end. The test invokes {{StreamsGroupCommand.execute}} with
> {{--describe --topology --group <appId>}} and waits until the output contains
> {{"Topologies:"}} and the source topic name. This covers the full flow: the
> Streams client sends a {{StreamsGroupTopologyDescriptionUpdate}} RPC when
> solicited by the heartbeat, the broker stores the description via the plugin,
> and the CLI retrieves and formats it on describe.
> *{{}}*
>
> h4. Admin API test
> We also need to include a test for the admin API. We could consider extending
> PlaintextAdminIntegrationTest, but it doesn't have a streams instance at the
> moment, but with some mocking we should be able to make it work.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
