My preferred way of working on this problem:
1. Get parity with what happens at http://buildbot.scons.org/#/. I can't work out from there how to read the actual commands being run in the CI job, so I'm mostly replicating what happens in the Travis-based builds. 2. Get all of scons' build processes onto cloud-based CI. Everything that is at http://scons.org/pages/download.html should come out of a CI job. 3. Get the tests running faster. This needs some profiling, but from the logs it reads like every test is run in a separate process, which feels pretty expensive to me. There's also references to 2008-era blog posts which I take with a large grain of salt. We need the tests running as a pre-requisite of course. Devising way to slice up long running tests feels like solving the wrong problem. 4. Move away from a QMTest to something more commonplace. The canonical home of QMTest seems to be https://github.com/MentorEmbedded/qmtest, last updated in 2011. I think it's time to look at something else, e.g. plain old unittest or pytest. Perhaps it's worth splitting off another thread for roadmapping? Andrew On 20 December 2017 at 14:00, Daniel Moody <[email protected]> wrote: > What about this: > > --interval-start FLOAT Percentile as float (0.0 - 1.0) of what test > to start on from the > determined list of all tests > --interval-end FLOAT Percentile as float (0.0 - 1.0) of what test > to start on from the > determined list of all tests > > interval-start default is 0 > interval-end default is 1 > > A little more flexible I think. > > > > On Wed, Dec 20, 2017 at 8:41 AM, Bill Deegan <[email protected]> > wrote: > >> Maybe add a flag directly to runtest.py to split up the tests. >> --test_mod 4 --test_index 0..3 ? >> >> Or something to that affect. >> >> On Wed, Dec 20, 2017 at 8:27 AM, Daniel Moody <[email protected]> >> wrote: >> >>> Hey Andrew, >>> >>> I also was working on this. I think in your case the builds timed out >>> due to issues with vswhere: >>> https://github.com/Microsoft/vswhere/issues/87 >>> https://github.com/Microsoft/vswhere/issues/91 >>> >>> In this case we will need to use the Visual Studio 2017 image. >>> >>> Also just to note; the issue mentioned above is also an issue with SCons >>> MSVS detection in general now for SCons 3 and newer since that is when it >>> was switched to use vswhere. >>> >>> I liked some of the things you had in your script so I took those and >>> merge them into my script. I also implemented a script that will split the >>> build up into multiple jobs like the travis script does, however I have not >>> been able to get appveyor to do multi-line scripts correctly so it is all >>> in a one liner at the moment: >>> https://github.com/dmoody256/scons/blob/AppveyorCI/.appveyor.yml >>> https://ci.appveyor.com/project/dmoody256/scons/build/1.0.53 >>> >>> Currently python 3 will fail from several tests so I have them commented >>> out. >>> >>> >>> On Wed, Dec 20, 2017 at 8:00 AM, Bill Deegan <[email protected]> >>> wrote: >>> >>>> Parallel should help. >>>> >>>> On my buildbot worker (with 2 other builds running single threaded >>>> tests) it takes 2:05. >>>> So on a reasonably modern machine, -j2 should finish in under an hour >>>> if not, try -j3? >>>> >>>> Or we can split up runs as we've done with the travis run.. >>>> >>>> -Bill >>>> >>>> On Wed, Dec 20, 2017 at 5:54 AM, Andrew Featherstone < >>>> [email protected]> wrote: >>>> >>>>> I've been trying to get AppVeyor working for Windows-based CI, but >>>>> we're hitting their 1 hour time limit (see >>>>> https://github.com/ajf58/scons/blob/appveyor/.appveyor.yml and >>>>> https://ci.appveyor.com/project/ajf58/scons. >>>>> >>>>> My next pass at this will be trying to run the unit tests in parallel >>>>> (as the Windows VM has two cores available). Either that, or we split the >>>>> job matrix a different way, e.g. run tests grouped by something else other >>>>> than Python version. >>>>> >>>>> Thoughts? >>>>> >>>>> Andrew >>>>> >>>>> On 18 December 2017 at 22:51, Bill Deegan <[email protected]> >>>>> wrote: >>>>> >>>>>> Daniel, >>>>>> >>>>>> Can we get travis to test with py2.7, 3.5, and 3.6 ? >>>>>> >>>>>> -Bill >>>>>> >>>>>> On Wed, Dec 6, 2017 at 12:03 AM, Bill Deegan < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Thanks! >>>>>>> That's pretty cool. >>>>>>> I'll try to get the coverage hooked up soon. >>>>>>> That'll also be very useful.. >>>>>>> >>>>>>> On Tue, Dec 5, 2017 at 8:27 PM, Jonathon Reinhart < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Yes, it should automatically do that. >>>>>>>> >>>>>>>> See this (merged) PR from one of my projects: >>>>>>>> https://github.com/JonathonReinhart/scuba/pull/98 >>>>>>>> >>>>>>>> Towards the bottom you'll see a "View Details" button. >>>>>>>> Clicking that will expand a box showing the results of all the >>>>>>>> "checks" that ran. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Dec 5, 2017 at 11:13 PM, Bill Deegan < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Is there a way to get travis to post the results back into the >>>>>>>>> pull request? >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Scons-dev mailing list >>>>>>>> [email protected] >>>>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Scons-dev mailing list >>>>>> [email protected] >>>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Scons-dev mailing list >>>>> [email protected] >>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Scons-dev mailing list >>>> [email protected] >>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>>> >>>> >>> >>> _______________________________________________ >>> Scons-dev mailing list >>> [email protected] >>> https://pairlist2.pair.net/mailman/listinfo/scons-dev >>> >>> >> >> _______________________________________________ >> Scons-dev mailing list >> [email protected] >> https://pairlist2.pair.net/mailman/listinfo/scons-dev >> >> > > _______________________________________________ > Scons-dev mailing list > [email protected] > https://pairlist2.pair.net/mailman/listinfo/scons-dev > >
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
