Thanks Giacomo, I've jumped in and added timeouts to the Nightly builds and the compiler checks.
The Nightly builds will now fail (instead of just "abort", which apparently is a silent failure) after 23 hours. Most of the time the nightly builds complete within 12ish hours (though this does vary a lot depending on how much compilation needs done). The Compiler Tests typically take 3 to 4 days. I've added a timeout at 6 days. I'm probably being a bit liberal with these timeouts, but I don't want to keep going back and extending them as we extend the tests. This doesn't stop us having checks at individual tests (i.e., we may want checks to ensure certain executions complete in a certain time). Kind regards, Bobby. -- Dr. Bobby R. Bruce Room 2235, Kemper Hall, UC Davis Davis, CA, 95616 web: https://www.bobbybruce.net Please consider making a submission to GI@ICSE '21: http://geneticimprovementofsoftware.com/gi2021icse.html On Mon, Jan 18, 2021 at 10:57 AM Jason Lowe-Power via gem5-dev < [email protected]> wrote: > Thanks, Giacomo! We may be a bit slow to respond (it's a holiday here in > the states today and Bobby is off the next couple of days). > > I don't think we want to do a global timeout since we have some tests that > take days. The compiler test takes about 3 days, IIRC. I think per-test > timeout is going to be required. > > Cheers, > Jason > > On Mon, Jan 18, 2021 at 9:43 AM Giacomo Travaglini via gem5-dev < > [email protected]> wrote: > >> Hi all, >> >> This has been fixed and it's going to be merged today. >> >> This made me realize we might want to introduce a timeout for our >> regressions. A global, Jenkins level one is easy to add and doesn't require >> any code modifications. >> >> However a per-test timeout is probably the best approach. In this way we >> can: >> >> a) Understand which regression is deadlocking from the output log >> b) Killing the failing test earlier thus saving computational costs: >> >> e.g: I will kill the test after 6 minutes if it is supposed to complete >> in a minute, rather than having a global abort set up after 24 hours. >> >> How to do this is up do debate. This falls within the typical wall-clock >> time vs cpu time issue; setting up a timeout for wall-clock time is >> straightforward with python subprocess, but cpu time is what we would >> really care about... >> Though if we make sure our timing out margin is relatively high >> (loosening the safety net) we can probably get around the intrinsic >> difference between the two metrics and just use wall-clock time. >> >> Does anybody have any thoughts on this? >> >> Giacomo >> >> > -----Original Message----- >> > From: Giacomo Travaglini >> > Sent: 18 January 2021 10:22 >> > To: gem5 Developer List <[email protected]> >> > Subject: Nightly stuck >> > >> > Hi devs, >> > >> > It seems like the following Nightly run #191 is stuck: >> > >> > https://jenkins.gem5.org/job/Nightly/191/ >> > >> > I am running those tests locally to understand if it is a tool problem >> or broken >> > long regressions are currently broken Will keep you posted >> > >> > Kind Regards >> > >> > Giacomo >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> _______________________________________________ >> gem5-dev mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s >> > _______________________________________________ > gem5-dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
_______________________________________________ gem5-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
