Sophie Blee-Goldman commented on KAFKA-9323:

I can help you figure out the upgrade matrix though. Might need some input on 
the much much older versions, which may not be compatible for upgrades at all, 
but here are the general rules (the first matching upgrade path is the correct 
 # Cooperative upgrade: an upgrade from EAGER to COOPERATIVE rebalancing. The 
cooperative protocol was introduced in 2.4 and turned on by default, so this 
includes upgrades where from_version < 2.4 and to_version >= 2.4 
 # Version probing upgrade: an upgrade across a metadata version bump (see 
below), where from_version >= 2.0 (version probing was introduced in 2.0)
 # Metadata upgrade: an upgrade across a metadata version bump, where 
from_version < 2.0
 # Simple upgrade: normal upgrade (no change in rebalance protocol or metadata 

The Streams versions corresponding to each metadata version so far are:
 # --> 0.10.0
 # --> 0.10.1, 0.10.2, 0.11.0, 2.0, 1.1
 # --> 2.0
 # --> 2.1, 2.2, 2.3
 # --> 2.4

> Refactor Streams'  upgrade system tests
> ---------------------------------------
>                 Key: KAFKA-9323
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9323
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Sophie Blee-Goldman
>            Assignee: Nikolay Izhikov
>            Priority: Major
> With the introduction of version probing in 2.0 and cooperative rebalancing 
> in 2.4, the specific upgrade path depends heavily on the to & from version. 
> This can be a complex operation, and we should make sure to test a realistic 
> upgrade scenario across all possible combinations. The current system tests 
> have gaps however, which have allowed bugs in the upgrade path to slip by 
> unnoticed for several versions. 
> Our current system tests include a "simple upgrade downgrade" test, a 
> metadata upgrade test, a version probing test, and a cooperative upgrade 
> test. This has a few drawbacks and current issues:
> a) only the version probing test tests "forwards compatibility" (upgrade from 
> latest to future version)
> b) nothing tests version probing "backwards compatibility" (upgrade from 
> older version to latest), except:
> c) the cooperative rebalancing test actually happens to involve a version 
> probing step, and so coincidentally DOES test VP (but only starting with 2.4)
> d) each test itself tries to test the upgrade across different versions, 
> meaning there may be overlap and/or unnecessary tests. For example, currently 
> both the metadata_upgrade and cooperative_upgrade tests will test the upgrade 
> of 1.0 -> 2.4
> e) worse, a number of (to, from) pairs are not tested according to the 
> correct upgrade path at all, which has lead to easily reproducible bugs 
> slipping past for several versions.
> f) we have a test_simple_upgrade_downgrade test which does not actually do a 
> downgrade, and for some reason only tests upgrading within the range [0.10.1 
> - 1.1]
> g) as new versions are released, it is unclear to those not directly involved 
> in these tests and/or projects whether and what needs to be updated (eg 
> should this new version be added to the cooperative test? should the old 
> version be aded to the metadata test?)
> We should definitely fill in the testing gap here, but how to do so is of 
> course up for discussion.
> I would propose to refactor the upgrade tests, and rather than maintain 
> different lists of versions to pass as input to each different test, we 
> should have a single matrix that we update with each new version that 
> specifies which upgrade path that version combination actually requires. We 
> can then loop through each version combination and test only the actual 
> upgrade path that users would actually need to follow. This way we can be 
> sure we are not missing anything, as each and every possible upgrade would be 
> tested.

This message was sent by Atlassian Jira

Reply via email to