+ Yang Wang Yang and me have been discussing about this Integration work (bugzilla [1]) and trying to breakdown the tasks needed.
Nico: Yang and me talk about will be if Yocto Project can get a token to access staging LAVA instance in order to test the integration. [2] [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13016 [2] https://staging.validation.linaro.org/ On Fri, 9 Nov 2018 at 14:59, Anibal Limon <[email protected]> wrote: > > > On Thu, 8 Nov 2018 at 20:49, Chan, Aaron Chun Yew < > [email protected]> wrote: > >> Hi Anibal/RP, >> >> > In order to do a distributed Boards testing the Yocto Autobuilder >> > needs to publish in some place accessible the artifacts (image, >> > kernel, etc) to flash/boot the board and the test suite expected to >> > execute. >> >> [Reply] That is correct, since Linaro have this in place to use >> https://archive.validation.linaro.org/directories/ and I have look into >> this as well, we can leverage on this >> but I am up for any suggestion you might have. So the idea >> here is that we have a placeholder to store the publish artifacts remotely >> and deploy using >> native curl command with token access. Then based on your >> LAVA job definitions we can instruct LAVA to source the images via https. >> Having said that, the deploy stage in LAVA must have some >> capabilities to read a token file in the LAVA job defintion and pick up the >> binaries from public repo (git LFS). >> >> In order for Board Distributed Tests to happen, there are 2 >> items in my wish lists >> >> 1. Public hosting of binary repository - with access >> control >> > > For publish the artifacts (Rootfs, Kernel image, Test suite), if there is > a public build a token isn't needed like targeting some boards already > commercialized and can be published anywhere like in > http://downloads.yoctoproject.org. > > >> 2. Ease Handshaking between two(2) different systems CI >> (e.g. Jenkins/Autobuilder) with LAVA >> a. Exchange build property (metadata) - includes >> hardware info, system info >> > > You can add meta-data to a LAVA test definition. > > b. Test reporting results >> > > For notify job results LAVA test definition support the notify block in > test jobs or you can poll the API using for both needs a LAVA token. > > > >> >> > I created a simple LAVA test definition that allows run testimage >> > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a >> > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to >> > get target and server IP addresses. >> >> [Reply] Although the lava test shell have these capabilities to use >> lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the >> way we retrieve the Ipv4 >> address is reading the logs from LAVA thru XML-RPC and grep >> the pattern matching string which contains IP even before the HW get >> initialize entirely then parse >> IP back to the Yocto Autobuilder. >> >> >> http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs > > > Yes, that's my idea the Yocto Autobuilder dosen't need to know about > particular network configuration in tha LAVA server for execute the job, in > this way the Yocto Autobuilder can communicate with LAVA server to retrieve > the testing job results, and in a case that needs to debug the board LAVA > support hacking sessions to allow connect to the board outside the LAB. > > https://validation.linaro.org/static/docs/v2/hacking-session.html > > >> >> >> > Some of the tasks, I identified, (if is accepted) >> > >> > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior , >> > move the EXTRA_PLAIN_CMDS to a class. >> > - Poky/OE: Review/fix test-export or provide other mechanism to export >> > the test suite. > - Poky/OE: Review/fix test-export or provide other >> mechanism to export >> > the test suite. >> >> [Reply] I would like to understand further what is the implementation >> here and how it addresses the problems that we have today. I believe in the >> past, Tim has tried >> to enable testexport and transfer the testexport into the >> DUT but it was not very successful and we found breakage. >> > > Agree, The testexport functionality is on not usage so there are some bugs > on it. > Yang commented me that He is using testexport but I agree that isn't not has full functionality as testimage, so in any case a mechanism to use the test suite + artifacts is needed, could be making a copy of full build environment (bitbake + layers + config) and compress in order be able to execute inside LAVA. > > >> >> > - Yocto-aubuilder-helper: Create a better approach to re-use LAVA job >> > templates across boards. >> >> [Reply] I couldn’t be more supportive on this having a common LAVA job >> template across boards but I would like to stress this, we don’t exactly >> know how >> community will define their own LAVA job definition, >> therefore what I had in mind as per today is to create a placeholde where >> LAVA job templates >> can be define and other boards/community can reuse the same >> template if it fits their use cases. In general the templates we have today >> are >> created to fit into Yocto Project use cases. >> > > Agree, I not mean a single template but a manner to add easily new LAVA > templates for boards in Yocto Autobuilder, this involves some base LAVA job > templates and a set of scripts to > generate the final template, like you are doing. For example there are > different ways to deploy a board but the login process is the same for > core-image's (login as root wo passwd). > > Cheers, > Anibal > > >> >> Lastly there are some works I've done on provisiong QEMU on LAVA >> sourceing from Yocto Project public releases, I am looking at where we can >> upstream this >> https://github.com/lab-github/yoctoproject-lava-test-shell >> >> Thanks! >> >> Cheers, >> Aaron Chan >> Open Source Technology Center Intel >> >> -----Original Message----- >> From: [email protected] [mailto: >> [email protected]] >> Sent: Thursday, November 8, 2018 6:45 AM >> To: Anibal Limon <[email protected]>; [email protected] >> Cc: Nicolas Dechesne <[email protected]>; Chan, Aaron Chun Yew >> <[email protected]> >> Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration >> >> Hi Anibal, >> >> On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote: >> > We know the need to execute OE testimage over real HW not only QEMU, >> > >> > I'm aware that currently there is an implementation on the Yocto >> > Autobuilder Helper , this initial implementation looks pretty well >> > separating parts for template generation [1] and the script to send >> > jobs to LAVA [2]. >> > >> > There are some limitations. >> > >> > - Requires that the boards are accessible trough SSH (same network?) >> > by the Autobuilder, so no distributed LAB testing. >> > - LAVA doesn't know about test results because the execution is >> > injected via SSH. >> > >> > In order to do a distributed Boards testing the Yocto Autobuilder >> > needs to publish in some place accessible the artifacts (image, >> > kernel, etc) to flash/boot the board and the test suite expected to >> > execute. >> > >> > Currently there is a functionality called testexport (not too >> > used/maintained) that allows you to export the test suite. >> >> I continue to have mixed feelings about testexport. It adds complexity >> but I'm not sure its actually worth it. >> >> An alternative would be to specify a set of commit hashes for the >> configuration under test (poky or oe-core+bitbake and any other layers), >> then have LAVA obtain those pieces and run the tests directly. >> >> Its worth considering that we already now have two difference pieces of >> code trying to package up the build system/layers, eSDK and testexport. >> Ideally if we had some kind of standardised layer setup/configuration >> approach we'd then just have a config file to share, then the tools could >> recreate the environment and allow the tests to be run there without >> testexport. Layer-setup is itself a harder subject but for example the >> layer setup code in autobuilder-helper could easily be reused as things >> stand today... >> >> In fact the more I think about it, the more I think we may want to do it >> that way... >> >> > I created a simple LAVA test definition that allows run testimage >> > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a >> > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to >> > get target and server IP addresses. >> > >> > In this way the LAVA server handles all the testing and finally the >> > Yocto Autobuilder can get/poll an event to know what was the actual >> > result of the job and the job could be send to different LAVA LAB's. >> >> That does sound useful and is likely a way we could end up doing this. >> Its probably worth highlighting that we now have a way of summarising the >> result of the test in the form of the json file the tests all generate. >> Sharing that back to the Yocto autobuilder would give us the test results >> we need. >> >> > Some of the tasks, I identified, (if is accepted) >> > >> > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior , >> > move the EXTRA_PLAIN_CMDS to a class. >> > - Yocto-aubuilder-helper: Create a better approach to re-use LAVA job >> > templates across boards. >> > - Poky/OE: Review/fix test-export or provide other mechanism to export >> > the test suite. >> >> I think some of these are also independent of each other and good things >> to work on regardless... >> >> I would like to hear feedback from those at Intel using LAVA who >> submitted the existing code. >> >> Cheers, >> >> Richard >> >>
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#47397): https://lists.yoctoproject.org/g/yocto/message/47397 Mute This Topic: https://lists.yoctoproject.org/mt/61337961/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
