Danylo Piliaiev <danylo.pilia...@gmail.com> writes:
> I want to sum up what happened from my perspective, I think it could be
> useful to improve the process:
>
> 1) There was a regression in 20.*
> (https://gitlab.freedesktop.org/mesa/mesa/-/issues/2758)
> 2) I "fixed" the regression and broke non-iris drivers
> (https://gitlab.freedesktop.org/mesa/mesa/-/commit/d684fb37bfbc47d098158cb03c0672119a4469fe)
> 3) I "fixed" the new regression of fix 2)
> (https://gitlab.freedesktop.org/mesa/mesa/-/commit/829013d0cad0fa2513b32ae07cf8d745f6e5c62d)
> 4) After that, it appeared that due to a bug in piglit, Intel CI didn't
> run piglit tests which gave me a false sense of commit's correctness
> (https://gitlab.freedesktop.org/mesa/mesa/-/issues/2815)
> 5) I aimed to have a fix before the next minor release on 2020-04-29 by
> consulting the release calendar.
> 6) I accidentally saw 20.0.5 being released with one of the two of my
> commits.
>
> I see multiple failure points:
> 1) Me not carefully examining all code paths, because at least one that
> failed should have been obvious to test.
You missed one important failure point:
1a) untested piglit commits pushed to master which disabled the test
suite.
I was impressed by how quickly regressions were added to master in the
few days that piglit was disabled in CI. At least 4 regressions in as
many days.
> 2) CI not communicating that piglit tests were not executed. Again, I
> could have seen this, examined CI results, but did not.
This was my fault. Over 5 years ago, I change the CI harness to ignore
errors thrown by piglit when it is run on an empty test set. There are
valid CI use cases (when bisecting) where the CI will attempt to run a
small test set on older platforms that do not support any of the tests.
A more defensive implementation would check that an empty test set is
allowed only in the specific/expected use case.
> 3) After restoration of CI capabilities test what added to "expected
> failure" and this may have contributed to regression
> being missed when testing the release. I'm not sure about this part
> so correct me if I'm wrong.
You are correct. CI cannot discern the context of a text failure
automatically. Typically, test failures on the stable branch are not
release blockers. Test failures which represent regressions are written
up as gitlab issues and tracked there.
Which brings up another failure point:
3a) Bisected regressions tagged with Fixes or mesa-stable are
automatically applied to Mesa's release branch.
This failure point has burned us many times, most recently with the 20.0
regression fixed by 2120f106e0e.
Mesa currently has no mechanism for blocking a release with a gitlab
issue. This current example is tagged with "bisected" and "regression",
but the important distinction is that the bisected commit ALSO has tags
which apply it to stable releases.
Mesa's release process does not include a check of bugs that have been
written up in gitlab (https://www.mesa3d.org/releasing.html). My own
opinion is that gitlab's issues are unusable for this purpose, due to
its lack of search functionality. I have found no way to audit gitlab
issues leading up to a release.
Gitlab's issues may work well for developing on master, but they are not
as good as Bugzilla for managing releases.
> 4) I didn't know about this release and that this release was help up
> for the fix of 2758.
> 5) There were now window between announcing the scope of the release and
> release itself. Since I knew about regression
> I could have notified about it. Also there is no milestone for minor
> releases so it's problematic to link issue and release.
>
> It's a second release in a row with clear regression crept in. I believe
> that we can use this to improve the process and
> safeguard against such regressions in the future.
Does anyone have recommendations for how to use Gitlab to verify that
there are no identified-but-unfixed bugs in a pending release?
> P.S. I'm preparing and will test a final fix which will be sent soon.
>
> Danylo
>
> On 23.04.20 07:40, Dylan Baker wrote:
>> Quoting Ilia Mirkin (2020-04-22 15:47:59)
>>> On Wed, Apr 22, 2020 at 6:39 PM Danylo Piliaiev
>>> <danylo.pilia...@gmail.com> wrote:
>>>> I'm sorry for this trouble. However looking at it I think: maybe something
>>>> could be
>>>> changed about applying patches to stable to safeguard against such issues.
>>> We used to get pre-announcements a few days prior to a release which
>>> would allow developers to look things over, which would allow us to
>>> notice things like that. Not sure when this got dropped.
>>>
>>> Cheers,
>>>
>>> -ilia
>> That was dropped in favor of a live staging branch that is updated daily (at
>> least on week days).
>>
>> Dylan
>
> _______________________________________________
> mesa-dev mailing list
> mesa-dev@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/mesa-dev
_______________________________________________
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/mesa-dev
