Hi,
I'm writing this email because of a recurring problem with our test coverage in
combination with releases of Qt.
Many times in the past we've had the situation where a change was applied to
the CI system that temporarily removed an entire platform, for example in favor
of adding a new one in steps. I'll cite two examples right off the top of my
head, but upon request I can dig up more:
(1) During times of qtqa-testconfig we added and removed configurations
without observing what their test coverage was, and at some point near the
5.3/5.4 release ended up in a situation where our entire cross-platform network
stack was only tested partially on Linux, partly as a consequence of removing
an older Windows (where we ran tests) in favor of a newer Windows (where we
"insignifified" all network tests). We made several releases with a network
stack that was untested on Windows and macOS.
(2) Today we are in the situation where we are building Qt and running
tests on macOS 10.11 in the 5.6 branch of Qt. However the changes that added
10.11 to the CI configuration to provide build and test coverage were never
applied to the 5.7, 5.8 or the dev branch of Qt, leaving us with no tests run
on 10.11.
During a related discussion in https://codereview.qt-project.org/#/c/183219/ I
suggested that we use a release blocking bug to track the progress of changing
platforms and that way ensure that we don't release Qt before the regression of
temporarily removing build and/or test coverage is fixed.
Tony declined the suggestion with the following statement:
"We had that. We had a P0 bug to put 10.11 in to 5.7 as it was already in 5.6,
but it got show down as it was seen as definitely not being able to block a
release. And this happened again in 5.8. If everything else is fine, a missing
platform from CI was seen as not to be able to block the release."
I could not find any mention of that in the release team meeting logs regarding
such a decision.
I would like to raise the attention to this topic and request either approval
to use blocking release bugs to track these regressions or ask for suggestions
of alternate methods to prevent us from repeatedly running into the scenario of
releasing Qt with regressed build and test coverage.
Simon
_______________________________________________
Releasing mailing list
[email protected]
http://lists.qt-project.org/mailman/listinfo/releasing
