Hi everyone, A while ago we met to discuss upgrade tests as a way to catch and avoid DB migration failures at CI time. I took the meeting notes [1] and started a POC [2] and I want to share my perspective. I found myself doing more workarounds than I expected, and at first, I thought this work was a no-go for this reason, but now I'm starting to think if I had to do these workarounds because we didn't have these tests before.
I spread multiple comments on [2] to try to explain my reasoning or the why of the workarounds, please take a look at the PR and share your thoughts. Some plugins need other plugins while running tests, it adds one more challenge of keeping all versions compatible e.g. pulp-2to3-migration + pulpcore + pulp_rpm + ... Another challenge is dealing with requirements from previous versions, I had to do a hack for pinning django for example, and having additional plugins may add more complexity to it. The last challenge I noticed was about external commands or specific CI scripts for specific releases and breaking changes. I had to do some workarounds for the app label change for example and I'm not sure if I covered all the places (at least one of the tests is failing because of this). A while ago pulpcore needed an external command to be run before the migration (the artifact path change) I suspect this is the reason for the failure of another test. My personal opinion: I think it really worth having this kind of test, I started looking for testing only migration, but during the process, I became aware of dependencies, commands, and other things that may have an influence on the upgrade process. The far we go in the past, the more challenging it gets, so for starting I would only test from the last release to master. After doing this for a while it may get easier to test 2 or 3 releases ago. [1] https://hackmd.io/IwhWRn6OS9ekKJNfQrGYFw [2] https://github.com/pulp/pulp_file/pull/503 Best regards, Fabricio Aguiar Software Engineer, Pulp Project Red Hat Brazil - Latam <https://www.redhat.com/> +55 22 999000595
_______________________________________________ Pulp-dev mailing list Pulp-dev@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-dev
