[
https://issues.apache.org/jira/browse/KAFKA-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016167#comment-17016167
]
Nikolay Izhikov commented on KAFKA-9323:
----------------------------------------
Hello, [~ableegoldman]
Can you, please, confirm my analysis of the current source code of stream
system tests:
We have 4 python scripts with the upgrade tests:
* streams_broker_compatibility_test.py
* streams_cooperative_rebalance_upgrade_test.py
* streams_statis_memebership_test.py
* streams_upgrade_test.py
> a) only the version probing test tests "forwards compatibility" (upgrade from
> latest to future version)
streams_upgrade_test.py#test_version_probing_upgrade
> b) nothing tests version probing "backwards compatibility" (upgrade from
> older version to latest), except:
a new test that should be created.
> c) the cooperative rebalancing test actually happens to involve a version
> probing step, and so coincidentally DOES test VP (but only starting with 2.4)
stream_cooperative_rebalance_upgrade_test.py#test_upgrade_to_cooperative_rebalance
> we should have a single matrix that we update with each new version that
> specifies which upgrade path that version combination actually requires
Do we have this matrix written somewhere?
As I can understand your proposal we should refactor all script from the
mentioned files.
Is it correct?
> Refactor Streams' upgrade system tests
> ---------------------------------------
>
> Key: KAFKA-9323
> URL: https://issues.apache.org/jira/browse/KAFKA-9323
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Sophie Blee-Goldman
> Assignee: Nikolay Izhikov
> Priority: Major
>
> With the introduction of version probing in 2.0 and cooperative rebalancing
> in 2.4, the specific upgrade path depends heavily on the to & from version.
> This can be a complex operation, and we should make sure to test a realistic
> upgrade scenario across all possible combinations. The current system tests
> have gaps however, which have allowed bugs in the upgrade path to slip by
> unnoticed for several versions.
> Our current system tests include a metadata upgrade test, a version probing
> test, and a cooperative upgrade test. This has a few drawbacks:
> a) only the version probing test tests "forwards compatibility" (upgrade from
> latest to future version)
> b) nothing tests version probing "backwards compatibility" (upgrade from
> older version to latest), except:
> c) the cooperative rebalancing test actually happens to involve a version
> probing step, and so coincidentally DOES test VP (but only starting with 2.4)
> d) each test itself tries to test the upgrade across different versions,
> meaning there may be overlap and/or unnecessary tests
> e) as new versions are released, it is unclear to those not directly involved
> in these tests and/or projects whether and what needs to be updated (eg
> should this new version be added to the cooperative test? should the old
> version be aded to the metadata test?)
> We should definitely fill in the testing gap here, but how to do so is of
> course up for discussion.
> I would propose to refactor the upgrade tests, and rather than maintain
> different lists of versions to pass as input to each different test, we
> should have a single matrix that we update with each new version that
> specifies which upgrade path that version combination actually requires. We
> can then loop through each version combination and test only the actual
> upgrade path that users would actually need to follow. This way we can be
> sure we are not missing anything, as each and every possible upgrade would be
> tested.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
