On Mon, 25 Nov 2019 at 10:23, Nicolas Dechesne <[email protected]> wrote:
> hey, > > > On Mon, Nov 25, 2019 at 2:49 AM Chan, Aaron Chun Yew > <[email protected]> wrote: > > > > Hi Anibal. > > > > > > > > Hope that all is well with you and good to hear from someone from the > community. > > > > > > > > We are maintaining our own LAVA server/dispatcher and only the > administrator can create a user account for you. > > > > With the access, each user can create their own authentication tokens. > Steps are: > > > > > > > > API (tab) à Authentication Tokens à New (+) à Enter the Description of > new token à Copy the token > > > > > > > > You can define the server URL and token into the yoctoabb/config.py like > this: > > > > > > > > “lava-server”: { > > > > “server”: “https://staging.validation.linaro.org/”, > > > > “token”: <New Generated User Token> > > > > }, > > > > “artifactorial”: { > > > > “server”: “ > https://archive.validation.linaro.org/artifacts/team/qa/2019/11/24/12/28/ > ”, > > > > “token”: <New Generated User Token> > > > > } > > > > > > > > To summarize on what had just mentioned about a year ago, the concept is > that if your hardware is located on a remote site, > > > > we need to have user access to the LAVA server in order to schedule > task/jobs and to retrieve the IP addr on the device > > > > which allow host to connect to the device on the network where both > servers can communicate with each other. > > Can you please summarize/explain how the device under test is being > used during the testing? The devices we have in our LAVA lab instance > are generally not accessible from the outside world with SSH. I think > I remember that SSH access was required to run the tests, but I am not > sure about the details. > Aaron is explaining the current implementation in the Yocto Autobuilder it relays of being in the same network/access with LAVA and the tests are driven only by OE testimage and right it requires to have SSH access details. > > The way we (Linaro) run tests on our devices in our LAVA lab (that > includes all kind of tests, such as kernel only, but also YP/OE based, > Android, ... ) is that the device under test is controlled via the > serial console, not SSH. The test is driven from a LAVA test case > definition. Do you think we can modify the YP ABB to behave in a > similar way? > We can run OE ptest on the OE images that we built, here is an example > of LAVA test case that shows ptest was run: > https://validation.linaro.org/results/1890697/0_linux-ptest > > To run ptest, we are using the following LAVA test definition (snippet): > > https://github.com/Linaro/test-definitions/blob/master/automated/linux/ptest/ptest.yaml > > which in turns use this test execution script: > > https://github.com/Linaro/test-definitions/blob/master/automated/linux/ptest/ptest.py > > LAVA was essentially designed to be used with a test definition, and I > was hoping we would find a way to integrate and link YP ABB 'output' > with LAVA in this way. > > What do you think? > > > > > > > > > We had also done publishing the artifacts into Artifactorial (similar to > https://archive.validation.linaro.org/directories/), > > > > Artifactorial uses curl command to upload/download artifacts store on > the remote server, we can definitely integrate > > > > a python script using pycurl. > > > > However the setback on Artifactorial is that it creates a timestamp > based on /<path>/<year>/<month>/<day>/<hour>/<minute>/<seconds> > > > > which can be tricky at times which automation may require to handle as > we do not want to pick up the wrong image and flash into the > > > > hardware (e.g. beaglebone). > > > > > > > > On another approach you can bring up your own LAVA server thru Docker - > https://hub.docker.com/r/lavasoftware/lava-server > > > > We had also explore this option in the past and its working for us. This > way you can have access control and miniture board farm which you > > > > can run tests on, if we do not have the hardware which you require you > may still need to have access to staging linaro LAVA server. > > > > > > > > Lastly, you may consider to have access to LAVA dispatcher on Linaro > end, as “board_info.json” is generated on the hardware booted on Yocto > > > > will contain board information which maybe helpful in the future. The > dispatcher holds the rootfs of the image were local results/data are stored. > > > > For do_testimage the results are already handle by the bitbake framework > and does not require the effort to retrieve the results. > > > > > > > > Cheers, > > > > Aaron > > > > Open Source Technology Center Intel > > > > > > > > From: Anibal Limon <[email protected]> > > Sent: Sunday, November 24, 2019 2:40 AM > > To: Chan, Aaron Chun Yew <[email protected]> > > Cc: Richard Purdie <[email protected]>; > [email protected]; Nicolas Dechesne <[email protected]>; > Orling, Timothy T <[email protected]>; Sangal, Apoorv < > [email protected]> > > Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration > > > > > > > > + Yang Wang > > > > > > > > Yang and me have been discussing about this Integration work (bugzilla > [1]) and trying to breakdown the tasks needed. > > > > > > > > Nico: Yang and me talk about will be if Yocto Project can get a token to > access staging LAVA instance in order to test the integration. [2] > > > > > > > > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13016 > > > > [2] https://staging.validation.linaro.org/ > > > > > > > > On Fri, 9 Nov 2018 at 14:59, Anibal Limon <[email protected]> > wrote: > > > > > > > > On Thu, 8 Nov 2018 at 20:49, Chan, Aaron Chun Yew < > [email protected]> wrote: > > > > Hi Anibal/RP, > > > > > In order to do a distributed Boards testing the Yocto Autobuilder > > > needs to publish in some place accessible the artifacts (image, > > > kernel, etc) to flash/boot the board and the test suite expected to > > > execute. > > > > [Reply] That is correct, since Linaro have this in place to use > https://archive.validation.linaro.org/directories/ and I have look into > this as well, we can leverage on this > > but I am up for any suggestion you might have. So the idea > here is that we have a placeholder to store the publish artifacts remotely > and deploy using > > native curl command with token access. Then based on your > LAVA job definitions we can instruct LAVA to source the images via https. > > Having said that, the deploy stage in LAVA must have some > capabilities to read a token file in the LAVA job defintion and pick up the > binaries from public repo (git LFS). > > > > In order for Board Distributed Tests to happen, there are > 2 items in my wish lists > > > > 1. Public hosting of binary repository - with access > control > > > > > > > > For publish the artifacts (Rootfs, Kernel image, Test suite), if there > is a public build a token isn't needed like targeting some boards already > commercialized and can be published anywhere like in > http://downloads.yoctoproject.org. > > > > > > > > 2. Ease Handshaking between two(2) different systems CI > (e.g. Jenkins/Autobuilder) with LAVA > > a. Exchange build property (metadata) - includes > hardware info, system info > > > > > > > > You can add meta-data to a LAVA test definition. > > > > > > > > b. Test reporting results > > > > > > > > For notify job results LAVA test definition support the notify block in > test jobs or you can poll the API using for both needs a LAVA token. > > > > > > > > > > > > > > > I created a simple LAVA test definition that allows run testimage > > > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a > > > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to > > > get target and server IP addresses. > > > > [Reply] Although the lava test shell have these capabilities to use > lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the > way we retrieve the Ipv4 > > address is reading the logs from LAVA thru XML-RPC and > grep the pattern matching string which contains IP even before the HW get > initialize entirely then parse > > IP back to the Yocto Autobuilder. > > > > > http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs > > > > > > > > Yes, that's my idea the Yocto Autobuilder dosen't need to know about > particular network configuration in tha LAVA server for execute the job, in > this way the Yocto Autobuilder can communicate with LAVA server to retrieve > the testing job results, and in a case that needs to debug the board LAVA > support hacking sessions to allow connect to the board outside the LAB. > > > > > > > > https://validation.linaro.org/static/docs/v2/hacking-session.html > > > > > > > > > > > > > Some of the tasks, I identified, (if is accepted) > > > > > > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior , > > > move the EXTRA_PLAIN_CMDS to a class. > > > - Poky/OE: Review/fix test-export or provide other mechanism to export > > > the test suite. > - Poky/OE: Review/fix test-export or provide other > mechanism to export > > > the test suite. > > > > [Reply] I would like to understand further what is the implementation > here and how it addresses the problems that we have today. I believe in the > past, Tim has tried > > to enable testexport and transfer the testexport into the > DUT but it was not very successful and we found breakage. > > > > > > > > Agree, The testexport functionality is on not usage so there are some > bugs on it. > > > > > > > > Yang commented me that He is using testexport but I agree that isn't not > has full functionality as testimage, so in any case a mechanism to use the > test suite + artifacts is needed, could > > > > be making a copy of full build environment (bitbake + layers + config) > and compress in order be able to execute inside LAVA. > > > > > > > > > > > > > > > - Yocto-aubuilder-helper: Create a better approach to re-use LAVA job > > > templates across boards. > > > > [Reply] I couldn’t be more supportive on this having a common LAVA job > template across boards but I would like to stress this, we don’t exactly > know how > > community will define their own LAVA job definition, > therefore what I had in mind as per today is to create a placeholde where > LAVA job templates > > can be define and other boards/community can reuse the > same template if it fits their use cases. In general the templates we have > today are > > created to fit into Yocto Project use cases. > > > > > > > > Agree, I not mean a single template but a manner to add easily new LAVA > templates for boards in Yocto Autobuilder, this involves some base LAVA job > templates and a set of scripts to > > > > generate the final template, like you are doing. For example there are > different ways to deploy a board but the login process is the same for > core-image's (login as root wo passwd). > > > > > > > > Cheers, > > > > Anibal > > > > > > > > > > Lastly there are some works I've done on provisiong QEMU on LAVA > sourceing from Yocto Project public releases, I am looking at where we can > upstream this > > https://github.com/lab-github/yoctoproject-lava-test-shell > > > > Thanks! > > > > Cheers, > > Aaron Chan > > Open Source Technology Center Intel > > > > -----Original Message----- > > From: [email protected] [mailto: > [email protected]] > > Sent: Thursday, November 8, 2018 6:45 AM > > To: Anibal Limon <[email protected]>; [email protected] > > Cc: Nicolas Dechesne <[email protected]>; Chan, Aaron Chun > Yew <[email protected]> > > Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration > > > > Hi Anibal, > > > > On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote: > > > We know the need to execute OE testimage over real HW not only QEMU, > > > > > > I'm aware that currently there is an implementation on the Yocto > > > Autobuilder Helper , this initial implementation looks pretty well > > > separating parts for template generation [1] and the script to send > > > jobs to LAVA [2]. > > > > > > There are some limitations. > > > > > > - Requires that the boards are accessible trough SSH (same network?) > > > by the Autobuilder, so no distributed LAB testing. > > > - LAVA doesn't know about test results because the execution is > > > injected via SSH. > > > > > > In order to do a distributed Boards testing the Yocto Autobuilder > > > needs to publish in some place accessible the artifacts (image, > > > kernel, etc) to flash/boot the board and the test suite expected to > > > execute. > > > > > > Currently there is a functionality called testexport (not too > > > used/maintained) that allows you to export the test suite. > > > > I continue to have mixed feelings about testexport. It adds complexity > but I'm not sure its actually worth it. > > > > An alternative would be to specify a set of commit hashes for the > configuration under test (poky or oe-core+bitbake and any other layers), > then have LAVA obtain those pieces and run the tests directly. > > > > Its worth considering that we already now have two difference pieces of > code trying to package up the build system/layers, eSDK and testexport. > > Ideally if we had some kind of standardised layer setup/configuration > approach we'd then just have a config file to share, then the tools could > recreate the environment and allow the tests to be run there without > testexport. Layer-setup is itself a harder subject but for example the > layer setup code in autobuilder-helper could easily be reused as things > stand today... > > > > In fact the more I think about it, the more I think we may want to do it > that way... > > > > > I created a simple LAVA test definition that allows run testimage > > > (oe-test runtime) in my own LAVA LAB, is very simplistic only has a > > > regex to parse results and uses lava-target-ip and lava-echo-ipv4 to > > > get target and server IP addresses. > > > > > > In this way the LAVA server handles all the testing and finally the > > > Yocto Autobuilder can get/poll an event to know what was the actual > > > result of the job and the job could be send to different LAVA LAB's. > > > > That does sound useful and is likely a way we could end up doing this. > > Its probably worth highlighting that we now have a way of summarising > the result of the test in the form of the json file the tests all generate. > Sharing that back to the Yocto autobuilder would give us the test results > we need. > > > > > Some of the tasks, I identified, (if is accepted) > > > > > > - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior , > > > move the EXTRA_PLAIN_CMDS to a class. > > > - Yocto-aubuilder-helper: Create a better approach to re-use LAVA job > > > templates across boards. > > > - Poky/OE: Review/fix test-export or provide other mechanism to export > > > the test suite. > > > > I think some of these are also independent of each other and good things > to work on regardless... > > > > I would like to hear feedback from those at Intel using LAVA who > submitted the existing code. > > > > Cheers, > > > > Richard >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#47417): https://lists.yoctoproject.org/g/yocto/message/47417 Mute This Topic: https://lists.yoctoproject.org/mt/61337961/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
