> On 14 Oct 2020, at 08:14, Yedidyah Bar David <d...@redhat.com> wrote: > > On Tue, Oct 13, 2020 at 6:46 PM Nir Soffer <nsof...@redhat.com> wrote: >> >> On Mon, Oct 12, 2020 at 9:05 AM Yedidyah Bar David <d...@redhat.com> wrote: >>> The next run of the job (480) did finish successfully. No idea if it >>> was already fixed by a patch, or is simply a random/env issue. >> >> I think this is env issue, we run on overloaded vms with small amount of >> memory. >> I have seen such radnom failures before. > > Generally speaking, I think we must aim for zero failures due to "env > issues" - and not ignore them as such.
Exactly. We cannot ignore that any longer. > It would obviously be nice if we had more hardware in CI, no doubt. there’s never enough > But I wonder if perhaps stressing the system like we do (due to resources > scarcity) is actually a good thing - that it helps us find bugs that real > users might also run into in actually legitimate scenarios yes, it absolutely does. > - meaning, using > what we recommend in terms of hardware etc. but with a load that is higher > than what we have in CI per-run - as, admittedly, we only have minimal > _data_ there. > > So: If we decide that some code "worked as designed" and failed due to > "env issue", I still think we should fix this - either in our code, or > in CI. yes! > > For latter, I do not think it makes sense to just say "the machines are > overloaded and not have enough memory" - we must come up with concrete > details - e.g. "We need at least X MiB RAM". I’ve spent quite some time analyzing the flakes in basic suite this past half year…so allow me to say that that’s usually just an excuse for a lousy test (or functionality:) > > For current issue, if we are certain that this is due to low mem, it's > quite easy to e.g. revert this patch: > > https://gerrit.ovirt.org/110530 > > Obviously it will mean either longer queues or over-committing (higher > load). Not sure which. it’s difficult to pinpoint the reason really. If it’s happening rarely (as this one is) you’d need a statistically relevant comparison. Which takes time… About this specific sparsify test - it was me uncommenting it few months ago, after running around 100 tests over a weekend. It may have failed once (there were/are still some other flakes)…but to me considering the overall success rate being quite low at that time it sounded acceptable. If this is now happening more often then it does sound like a regression somewhere. Could be all the OST changes or tests rearrangements, but it also could be a code regression. Either way it’s supposed to be predictable. And it is, just not in this environment we use for this particular job - it’s the old one without ost-images, inside the troublesome mock, so you don’t know what it picked up, what’s the really underlying system(outside of mock) Thanks, michal > > But personally, I wouldn't do that without knowing more (e.g. following > the other thread). > > Best regards, > -- > Didi > _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/6I3RB4XJVYVN2WEZ76L2QZTHRN6CAST2/
