On Wed, 12 Dec 2012 12:18:56 +0000 Matthew Gretton-Dann <[email protected]> wrote:
> Paul, > > Before I start picking holes and asking questios: Thanks for doing > this it looks like you are starting on the right track! > > On Wednesday 12 Dec 2012 13:17:05 Paul Sokolovsky wrote: > > Hello, > > > > As a heads-up for CBuild/LAVA integration project, > > http://ec2-54-243-13-36.compute-1.amazonaws.com/helpers/scheduler > > is now available, which has initial(?) web frontend integration > > with LAVA. That page (currently) shows job > > "gcc-linaro-4.6-2012.09.job", submitted against queue > > "lava-panda-mock", with a link to latest build in LAVA (extra > > feature, I couldn't find a way in CBuild frontend to get to native > > build results). > > So as soon as a board starts executing a job it should no longer > appear in the 'Pending' list - just the Jobs list see: > http://cbuild.validation.linaro.org/helpers/scheduler > This is the moment the 'Taken' column gets filled in. Thanks, we'll look into managing .lock file more consistently for LAVA scheduler case. > > You know a job has finished because it appears in recent: > http://cbuild.validation.linaro.org/helpers/recent > These have a direct link to the build logs. > > You can also get to the build logs via: > http://cbuild.validation.linaro.org/helpers/buildlog Ok, we'll look how to implement those as close as possible with LAVA. One thing we don't fully understand is "build" concept as used in CBuild. For example, in Jenkins it's simple - there's "job" which is kind of class and "build" which is kind of instance of job, built against particular revision of sources and capturing build logs (so, for each job, it's possible to browse all recent builds and their results). But with CBuild, it seems to me that it tracks only latest build of a job. What happens with previous builds - are they just removed, or their build artifacts still available there in some distinct directory, just not referenced in web frontend? > > > If you want to try it yourself, it's accessible to same TCWG people > > as http://cbuild.validation.linaro.org/ , plus me, Stevan, and > > Danilo. So, if you're not in list, please drop us a line. "Spawn" > > and "Bump up" actions support integration with LAVA currently. > > We're in the process of identifying what other features of web > > frontend would require LAVA integration, hints from TCWG are > > appreciated. > > Spawn and Bump up seem a good start. > > 'Release' would be useful as well - which actually means: > 'Respawn' (i.e. do it again) Ok, will look into that. > Another one which would be useful (and which we currently do manually > with "killall make" on toolchain64.lab) is: Kill all current jobs. > This is useful in release week when we want to clear the way for the > release builds. Well, that likely will need to go as separate feature request for LAVA first, I'm not sure if that supported. > > Now a bit about architecture. We're targetting to support both > > native CBuild and LAVA schedulers. This is per-queue, i.e. one > > queue can use native scheduler and another LAVA. LAVA is detected > > by the presence of LAVA job definition template file in a scheduler > > queue dir: > > http://bazaar.launchpad.net/~linaro-infrastructure/cbuild/cbuild-scheduler_c > > build-lava/files/head:/queue/lava-panda-mock/ lava-test-shell > > script with actual commands to run for build is also served by > > CBuild frontend based on a template in that dir. > > > > Currently, sandbox defines 3 test queue for LAVA integration: > > > > lava-panda - build something on panda (tested only with gcc, needs > > more work) > > > > lava-panda-mock - doesn't really compiles stuff, just downloads > > snapshot of previous gcc build. This allows for quicker (~15min) > > turnaround during testing, if you test snadbox, please use this > > queue. > > > > lava-qemu-mininal - for local development with even shorter > > turnaround, won't work on sandbox. > > > > > > If you have any feedback on the architecture above, please share it. > > Otherwise, we're proceeding to support it in cbuild-tools. > > So I do have some Lava issues - which are probably mostly due to Lava > terms not being what I would expect them to mean: > > The following Link is to the 'test results bundle' > https://validation.linaro.org/lava- > server/dashboard/streams/anonymous/cbuild/bundles/9c87b6a32ef38e06d081f12f115c08e79ce89f8b/3df9cc54-60ec-484b-91bf- > cc4d34e3a494/ Thanks for looking into that. I wanted add few sentences about LAVA result bundle browsing, but skipped that to not make mail too long, so I'd appreciate your exploring it. > > This shows four test cases - which reflect the historic build steps > shown under 'Details' in the historic Scheduler (gcc-configure, > gcc-build, gcc-install, cbuild-global) > > So currently these seem to reflect stages in the build process, > rather than actual test results. Yes. Using LAVA as a backend for this project was my concern from the very beginning. Fortunately, architecturely and feature-wise LAVA fits well, but there's another "organizational/representative" aspect of it - all the LAVA is build around "test" paradigm, which includes terminology, application wording, documentation, etc. At the CBuild/LAVA initiation session at the last Connect, I raised concern that it's both confusing and "looking hacky" and asked if we should discuss ways to deal with that. My concern wasn't shared at that time though. Well, I'm still sure it's valid, so thanks for sharing your experiences, hopefully this will catch attention from the other stakeholders. From my side, I submitted https://bugs.launchpad.net/lava-dashboard/+bug/1089384 as one small starting step to make it less confusing. > Eventually you will run the > 'gcc-testsuite' build stage which actually does the regression > testing (all 100K+ of tests) and I don't expect that to succeed > usually (in terms of make check returning 0), although it will > succeed in running the testsuite. > > What is the intention here? Are we going to have Lava be the > high-level builder, and so its 'tests' are each of the build steps, > and then the testsuite analysis is done elsewhere? This is the current setup > (see > http://cbuild.validation.linaro.org/helpers/testcompare/gcc- > linaro-4.6-2012.09/logs/x86_64-precise-cbuild367-oort3-x86_64r1/gcc- > testsuite.txt - except this is currently returning an Internal Server > Error). Or is the intention for Lava to handle all of the > gcc-testsuite results individually? My intention was to explore what result representation LAVA allows, and actually to have experience with that. And it's pretty flexible, so both of options you mentioned are possible (recording each case individually will of course require more work - either add commands to record each testcase results to existing makefiles/scripts, or to write parser for test output, or both). Which exact to use is up to the stakeholders, and but that I mean not only TCWG, but also Validation, QA, Release Managers, Platform in general, which may be interested in detailed and standardly represented test details for toolchain. I see Infrastructure's role as ensuring the various options are possible, and providing focused documentation/examples how to achieve that. So, that's definitely something to discuss, but likely in 13.01, once basic integration is complete. > > Thanks, > > Matt > -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog _______________________________________________ linaro-validation mailing list [email protected] http://lists.linaro.org/mailman/listinfo/linaro-validation
