Hi Giacomo, Jason, Sorry for late reply. Re: tool chain - scons and python are available in the test environment AFAIK. Re: test throughput - How long does kokoro take to finish current presubmit test?
If you have much longer tests to run at periodic frequency, say once in a few days, on HEAD, kokoro also has a periodic continuous build mode. Such test will send out email report, serving as a pulse check for ToT. Presubmit jobs will not be influenced. I can look in to periodic builds if it fits for gem5 long regression use case. RT On Fri, May 3, 2019 at 6:17 AM Giacomo Travaglini < [email protected]> wrote: > Hi Jason, > > I have seen patches being under review for a long time, and IMHO adding an > extra 10-30 mins is not the real bottleneck. > I'd rather wait a little bit more but being sure I am not breaking > anything... > > about ruby protocols, let me say that we (in arm) are relatively happy > with the current setup: > > 1) Quick regressions are run on a commit base (your presubmit.sh). This > means it won't take a lot of time/computation > 2) Long regressions (ruby protocols are tested here) are run every night > on a batch of MERGED commits. > Which means at a certain point we just checkout origin/HEAD and we run > long regressions. > > This is the setup I would actually recommend: a sanity suite (quick) being > run on a commit base, and more serious tests > (long) being run periodically. If you are scared about overloading the > framework, you can always scale down our ambition > and run long regressions every two nights. > Running less frequently is better than not running at all 🙂 > > Please let me know what you think about this; other devs are welcome to > comment as well, > > Giacomo > > > ------------------------------ > *From:* Jason Lowe-Power <[email protected]> > *Sent:* 02 May 2019 22:59 > *To:* Giacomo Travaglini; Rahul Thakur > *Cc:* gem5 Developer List > *Subject:* Re: [gem5-dev] Continuous integration is live! > > Hi Giacomo, > > In tests/main.py we call scons and use the current environment defaults to > build gem5. I don't know if the kokoro infrastructure supports other > compilers. This might be something that Rahul can address. > > I'm also not sure if we can find a way to run more compilations in > parallel on Kokoro. I'm happy to refactor the test scripts to do this. > However, as it is, we are currently compiling at least 4 binaries mostly > sequentially, which is making the testing take a significant amount of > time. If we add more compilers (and more Ruby protocols), this is going to > begin to get out of hand. It would also be good to compile .fast, .opt, and > .debug, but I believe we're only compiling .opt right now. > > Cheers, > Jason > > On Thu, May 2, 2019 at 5:53 AM Giacomo Travaglini < > [email protected]> wrote: > > Hi Jason, > > I understand; Another thing I would like to ask: > > Which script is building gem5 in jenkins? Ideally it would be nice to > build with BOTH gcc and clang (so that we avoid > periodic "fix clang build" patches. I would also make the version > configurable/visible from the script so that > we can track changes in compiler support and people can compare failures > in case they managed to build > seamlessly on their local workspace > > Giacomo > ------------------------------ > *From:* Jason Lowe-Power <[email protected]> > *Sent:* 26 April 2019 17:49 > *To:* Giacomo Travaglini > *Cc:* gem5 Developer List > *Subject:* Re: [gem5-dev] Continuous integration is live! > > Hi Giacomo, > > You *do* have permission :). Anyone can modify tests/jenkins/presubmit.cfg > and presubmit.sh. In fact, if you look at the history of the presubmit.sh, > it *was* running the old tests. See > https://gem5-review.googlesource.com/c/testing/jenkins-gem5-prod/+/18028, > for instance. > > The problem is that we can't distribute most of the binaries (e.g., SPEC > binaries). We could probably upload them to a private location on the > Google Cloud and have jenkins consume them that way, but I believe that > will be more work than it's worth. > > I personally believe that putting effort into porting tests is more worth > everyone's time than trying to get the old tests to run, but that's just > my opinion. I'm happy to merge changes to run the old tests. I personally > believe we should only merge tests into the verification tester which > everyone can run locally, but I'm open to proprietary tests, especially in > the short term if we have a plan to make them not proprietary. > > Cheers, > Jason > > On Fri, Apr 26, 2019 at 9:36 AM Giacomo Travaglini < > [email protected]> wrote: > > Hi Jason, > > It's really amazing that we have a testing framework in place, thanks for > your effort! > At the moment as far as I can tell we are only running tests registered > within the new > testing library. > > I was wondering if we could temporarily enable the system to run legacy > quick regressions as well, > while waiting for porting those to the new library. I guess it is > something that shouldn't require a lot of work > > (just calling .util/regress I guess) > > I am saying this since a patch recently merged broke some syscall > emulation tests and I think it would > be beneficial for us to run the entire test suite straightaway while > porting tests manually. > I could even handle it myself if I had permission to configure the system. > > Let me know your thoughts, > > Giacomo > > ------------------------------ > *From:* gem5-dev <[email protected]> on behalf of Jason > Lowe-Power <[email protected]> > *Sent:* 16 April 2019 16:30 > *To:* gem5 Developer List; Rahul Thakur > *Subject:* [gem5-dev] Continuous integration is live! > > Hi all, > > We now have initial support for continuous integration testing! We should > all thank Google for donating the CPU time and infrastructure to run these > tests. Specifically, Rahul Thakur has been incredibly helpful for the past > two years in getting this off the ground. Thanks, Rahul and the rest of the > team at Google who has been helping us set this up! > > Now, if you submit a patch to gerrit and receive a maintainer +1, "kokoro" > will kick off a build / test of gem5. Once that is complete, you will > receive a verified +1. If it fails, you will receive a verified -1. The > logs can be viewed by anyone once the job is completed by following the > link posted by kokoro (the https://source.cloud.google.com, not the sponge > link). You can see an example on a patch I recently submitted here: > https://gem5-review.googlesource.com/c/public/gem5/+/18068. Note that the > tests take a couple of hours to run. However, I believe there is no limit > to the number of different changes that can be tested at the same time. > > Soon, we are going to enable commit gating with the verified +1 tag. I.e., > you will have to pass the continuous integration tests before you can > commit your code. > > Note that this is using the "new" testing infrastructure. You can run this > locally by running "./main.py" in the tests directory. More information > about how to run tests and add tests can be found in the TESTING.md file. > If there are any questions/issues do not hesitate to contact me or the > list. The documentation for the new infrastructure can still be improved. > Right now, we're running about 30 tests. You can find the tests that we are > running in the tests/gem5 directory. > > We are looking for volunteers to help us port more of the old tests to the > new infrastructure and to expand the coverage of our tests. I'm happy to > help anyone get started on this and point out which tests still need to be > migrated, where our biggest coverage holes are, etc. > > Cheers, > Jason > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
