Hi Anibal.
Hope that all is well with you and good to hear from someone from the community.
We are maintaining our own LAVA server/dispatcher and only the administrator
can create a user account for you.
With the access, each user can create their own authentication tokens. Steps
are:
API (tab) --> Authentication Tokens --> New (+) --> Enter the Description of
new token --> Copy the token
You can define the server URL and token into the yoctoabb/config.py like this:
“lava-server”: {
“server”: “https://staging.validation.linaro.org/”,
“token”: <New Generated User Token>
},
“artifactorial”: {
“server”:
“https://archive.validation.linaro.org/artifacts/team/qa/2019/11/24/12/28/”,
“token”: <New Generated User Token>
}
To summarize on what had just mentioned about a year ago, the concept is that
if your hardware is located on a remote site,
we need to have user access to the LAVA server in order to schedule task/jobs
and to retrieve the IP addr on the device
which allow host to connect to the device on the network where both servers can
communicate with each other.
We had also done publishing the artifacts into Artifactorial (similar to
https://archive.validation.linaro.org/directories/),
Artifactorial uses curl command to upload/download artifacts store on the
remote server, we can definitely integrate
a python script using pycurl.
However the setback on Artifactorial is that it creates a timestamp based on
/<path>/<year>/<month>/<day>/<hour>/<minute>/<seconds>
which can be tricky at times which automation may require to handle as we do
not want to pick up the wrong image and flash into the
hardware (e.g. beaglebone).
On another approach you can bring up your own LAVA server thru Docker -
https://hub.docker.com/r/lavasoftware/lava-server
We had also explore this option in the past and its working for us. This way
you can have access control and miniture board farm which you
can run tests on, if we do not have the hardware which you require you may
still need to have access to staging linaro LAVA server.
Lastly, you may consider to have access to LAVA dispatcher on Linaro end, as
“board_info.json” is generated on the hardware booted on Yocto
will contain board information which maybe helpful in the future. The
dispatcher holds the rootfs of the image were local results/data are stored.
For do_testimage the results are already handle by the bitbake framework and
does not require the effort to retrieve the results.
Cheers,
Aaron
Open Source Technology Center Intel
From: Anibal Limon <[email protected]>
Sent: Sunday, November 24, 2019 2:40 AM
To: Chan, Aaron Chun Yew <[email protected]>
Cc: Richard Purdie <[email protected]>;
[email protected]; Nicolas Dechesne <[email protected]>; Orling,
Timothy T <[email protected]>; Sangal, Apoorv <[email protected]>
Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
+ Yang Wang
Yang and me have been discussing about this Integration work (bugzilla [1]) and
trying to breakdown the tasks needed.
Nico: Yang and me talk about will be if Yocto Project can get a token to access
staging LAVA instance in order to test the integration. [2]
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13016
[2] https://staging.validation.linaro.org/
On Fri, 9 Nov 2018 at 14:59, Anibal Limon
<[email protected]<mailto:[email protected]>> wrote:
On Thu, 8 Nov 2018 at 20:49, Chan, Aaron Chun Yew
<[email protected]<mailto:[email protected]>> wrote:
Hi Anibal/RP,
> In order to do a distributed Boards testing the Yocto Autobuilder
> needs to publish in some place accessible the artifacts (image,
> kernel, etc) to flash/boot the board and the test suite expected to
> execute.
[Reply] That is correct, since Linaro have this in place to use
https://archive.validation.linaro.org/directories/ and I have look into this as
well, we can leverage on this
but I am up for any suggestion you might have. So the idea here
is that we have a placeholder to store the publish artifacts remotely and
deploy using
native curl command with token access. Then based on your LAVA
job definitions we can instruct LAVA to source the images via https.
Having said that, the deploy stage in LAVA must have some
capabilities to read a token file in the LAVA job defintion and pick up the
binaries from public repo (git LFS).
In order for Board Distributed Tests to happen, there are 2 items
in my wish lists
1. Public hosting of binary repository - with access control
For publish the artifacts (Rootfs, Kernel image, Test suite), if there is a
public build a token isn't needed like targeting some boards already
commercialized and can be published anywhere like in
http://downloads.yoctoproject.org.
2. Ease Handshaking between two(2) different systems CI (e.g.
Jenkins/Autobuilder) with LAVA
a. Exchange build property (metadata) - includes hardware
info, system info
You can add meta-data to a LAVA test definition.
b. Test reporting results
For notify job results LAVA test definition support the notify block in test
jobs or you can poll the API using for both needs a LAVA token.
> I created a simple LAVA test definition that allows run testimage
> (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> get target and server IP addresses.
[Reply] Although the lava test shell have these capabilities to use
lava-target-ip or/and lava-echo-ipv4 this only works within LAVA scope, the way
we retrieve the Ipv4
address is reading the logs from LAVA thru XML-RPC and grep the
pattern matching string which contains IP even before the HW get initialize
entirely then parse
IP back to the Yocto Autobuilder.
http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/lava/trigger-lava-jobs
Yes, that's my idea the Yocto Autobuilder dosen't need to know about particular
network configuration in tha LAVA server for execute the job, in this way the
Yocto Autobuilder can communicate with LAVA server to retrieve the testing job
results, and in a case that needs to debug the board LAVA support hacking
sessions to allow connect to the board outside the LAB.
https://validation.linaro.org/static/docs/v2/hacking-session.html
> Some of the tasks, I identified, (if is accepted)
>
> - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> move the EXTRA_PLAIN_CMDS to a class.
> - Poky/OE: Review/fix test-export or provide other mechanism to export
> the test suite. > - Poky/OE: Review/fix test-export or provide other
> mechanism to export
> the test suite.
[Reply] I would like to understand further what is the implementation here and
how it addresses the problems that we have today. I believe in the past, Tim
has tried
to enable testexport and transfer the testexport into the DUT but
it was not very successful and we found breakage.
Agree, The testexport functionality is on not usage so there are some bugs on
it.
Yang commented me that He is using testexport but I agree that isn't not has
full functionality as testimage, so in any case a mechanism to use the test
suite + artifacts is needed, could
be making a copy of full build environment (bitbake + layers + config) and
compress in order be able to execute inside LAVA.
> - Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> templates across boards.
[Reply] I couldn’t be more supportive on this having a common LAVA job template
across boards but I would like to stress this, we don’t exactly know how
community will define their own LAVA job definition, therefore
what I had in mind as per today is to create a placeholde where LAVA job
templates
can be define and other boards/community can reuse the same
template if it fits their use cases. In general the templates we have today are
created to fit into Yocto Project use cases.
Agree, I not mean a single template but a manner to add easily new LAVA
templates for boards in Yocto Autobuilder, this involves some base LAVA job
templates and a set of scripts to
generate the final template, like you are doing. For example there are
different ways to deploy a board but the login process is the same for
core-image's (login as root wo passwd).
Cheers,
Anibal
Lastly there are some works I've done on provisiong QEMU on LAVA sourceing
from Yocto Project public releases, I am looking at where we can upstream this
https://github.com/lab-github/yoctoproject-lava-test-shell
Thanks!
Cheers,
Aaron Chan
Open Source Technology Center Intel
-----Original Message-----
From:
[email protected]<mailto:[email protected]>
[mailto:[email protected]<mailto:[email protected]>]
Sent: Thursday, November 8, 2018 6:45 AM
To: Anibal Limon <[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Cc: Nicolas Dechesne
<[email protected]<mailto:[email protected]>>; Chan, Aaron
Chun Yew <[email protected]<mailto:[email protected]>>
Subject: Re: [RFC] Yocto Autobuilder and LAVA Integration
Hi Anibal,
On Wed, 2018-11-07 at 16:25 -0600, Anibal Limon wrote:
> We know the need to execute OE testimage over real HW not only QEMU,
>
> I'm aware that currently there is an implementation on the Yocto
> Autobuilder Helper , this initial implementation looks pretty well
> separating parts for template generation [1] and the script to send
> jobs to LAVA [2].
>
> There are some limitations.
>
> - Requires that the boards are accessible trough SSH (same network?)
> by the Autobuilder, so no distributed LAB testing.
> - LAVA doesn't know about test results because the execution is
> injected via SSH.
>
> In order to do a distributed Boards testing the Yocto Autobuilder
> needs to publish in some place accessible the artifacts (image,
> kernel, etc) to flash/boot the board and the test suite expected to
> execute.
>
> Currently there is a functionality called testexport (not too
> used/maintained) that allows you to export the test suite.
I continue to have mixed feelings about testexport. It adds complexity but I'm
not sure its actually worth it.
An alternative would be to specify a set of commit hashes for the configuration
under test (poky or oe-core+bitbake and any other layers), then have LAVA
obtain those pieces and run the tests directly.
Its worth considering that we already now have two difference pieces of code
trying to package up the build system/layers, eSDK and testexport.
Ideally if we had some kind of standardised layer setup/configuration approach
we'd then just have a config file to share, then the tools could recreate the
environment and allow the tests to be run there without testexport. Layer-setup
is itself a harder subject but for example the layer setup code in
autobuilder-helper could easily be reused as things stand today...
In fact the more I think about it, the more I think we may want to do it that
way...
> I created a simple LAVA test definition that allows run testimage
> (oe-test runtime) in my own LAVA LAB, is very simplistic only has a
> regex to parse results and uses lava-target-ip and lava-echo-ipv4 to
> get target and server IP addresses.
>
> In this way the LAVA server handles all the testing and finally the
> Yocto Autobuilder can get/poll an event to know what was the actual
> result of the job and the job could be send to different LAVA LAB's.
That does sound useful and is likely a way we could end up doing this.
Its probably worth highlighting that we now have a way of summarising the
result of the test in the form of the json file the tests all generate. Sharing
that back to the Yocto autobuilder would give us the test results we need.
> Some of the tasks, I identified, (if is accepted)
>
> - Yocto-aubuilder-helper: Implement/adapt to cover this new behavior ,
> move the EXTRA_PLAIN_CMDS to a class.
> - Yocto-aubuilder-helper: Create a better approach to re-use LAVA job
> templates across boards.
> - Poky/OE: Review/fix test-export or provide other mechanism to export
> the test suite.
I think some of these are also independent of each other and good things to
work on regardless...
I would like to hear feedback from those at Intel using LAVA who submitted the
existing code.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#47401): https://lists.yoctoproject.org/g/yocto/message/47401
Mute This Topic: https://lists.yoctoproject.org/mt/61337961/21656
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
