Re: [U-Boot] [PATCH v2 8/8] travis-ci: Add test.py for various qemu platforms

[Tom Rini]

On Mon, Oct 24, 2016 at 04:21:36PM -0600, Stephen Warren wrote:
[snip]
> BTW, does/can Travis save the test-log.html from the test.py
> invocations? That would make debugging any failures a little easier.

I'm far from a travis "expert".  But the real nice thing about it is
that you can hook up and test via github.  It took maybe 2 minutes to
get my u-boot clone going in travis-ci so I could test things out, and
that's something I want to emphasize in -rc3 is how easy it is to get a
lot more coverage with minimal effort.

So since a script clause is just a bit of bash commands basically, some
sort of quick wrapper or maybe just pass in -s to test.py since another
nice thing is that the web view will collapse output from each step?

-- 
Tom


signature.asc
Description: Digital signature
___
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Re: [U-Boot] [PATCH v2 8/8] travis-ci: Add test.py for various qemu platforms

[Stephen Warren]

On 10/22/2016 06:56 AM, Tom Rini wrote:

On Fri, Oct 21, 2016 at 12:24:07PM -0600, Stephen Warren wrote:

On 10/20/2016 01:40 PM, Tom Rini wrote:

- Add a PPA for a more recent qemu (required for PowerPC to work)
- Add tests to run test.py for various QEMU platforms.  This relies on
 swarren's uboot-test-hooks repository to provide the abstractions.



+ # Clone uboot-test-hooks
+ - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git 
/tmp/uboot-test-hooks


Does it make sense to rely on my copy of the repo being around all
the time? I have no intention of removing it, but when I created it
I'd assumed others would create their own clones for their own
infra-structure. I wonder if all the dependencies for U-Boot's
testing should be hosted on U-Boot infra-structure?


A good point.  The way I've been thinking of it is that we also need a
repo with as many and varied configs as possible, which has become yours
:).  That said, we could move a copy of this over to the u-boot account
on github if you'd like.


I think it's fine for now.


That all said, this should be fine; just something to ponder.


+ - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname`


It may be useful to add the following too:

- ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname`

That way, if we ever need per-board test configuration files, the
path to access them will already be available.


A good point.  I think we could get the network tests going, maybe with
a little re-work to dynamically hash the file from the host side?  But
that's an aside.


Yes, assuming qemu networking works, I expect that would be possible. 
Yes, I think it'd be pretty easy to dynamically create a test download 
file and calculate the CRC from the per-board Python config files.



+- env:
+- CROSS_COMPILE="/usr/bin/arm-linux-gnueabihf-"
+  script:
+- ./test/py/test.py --bd vexpress_ca15_tc2 --id qemu --build;
+  ./test/py/test.py --bd vexpress_ca9x4 --id qemu --build;
+  ./test/py/test.py --bd integratorcp_cm926ejs --id qemu --build;


Does it make sense to make that 3 separate scripts? I assume that
would (a) allow the individual status of the 3 test.py invocations
to be seen separately (b) might allow/cause travis-ci to run them in
parallel; I'm not sure if this Travis config file is a sequential
list of commands, or a list of potentially parallel actions.


So, the way travis-ci works is that you can (and we do) split tasks up
to run in parallel.  The hunk above is a series to run in sequence on
the same host. The downside to moving this to parallel is the amount of
duplication you need in each stanza.


The duplication doesn't seem like a terribly big deal given it will 
allow finer-grained status on the main per-build page.


> We could fix that (at least in the

case of non-buildman-fetched toolchains) if test.py used buildman to
build, as that will determine CROSS_COMPILE correctly from the default
.buildman file we create earlier on.  Thanks!


I'll look into whether that's possible.

BTW, does/can Travis save the test-log.html from the test.py 
invocations? That would make debugging any failures a little easier.

___
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Re: [U-Boot] [PATCH v2 8/8] travis-ci: Add test.py for various qemu platforms

[Tom Rini]

On Thu, Oct 20, 2016 at 03:40:20PM -0400, Tom Rini wrote:

> - Add a PPA for a more recent qemu (required for PowerPC to work)
> - Add tests to run test.py for various QEMU platforms.  This relies on
>   swarren's uboot-test-hooks repository to provide the abstractions.
> 
> Signed-off-by: Tom Rini 

After adding in Stephen's suggested edit, applied to u-boot/master,
thanks!

-- 
Tom


signature.asc
Description: Digital signature
___
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Re: [U-Boot] [PATCH v2 8/8] travis-ci: Add test.py for various qemu platforms

[Tom Rini]

On Fri, Oct 21, 2016 at 12:24:07PM -0600, Stephen Warren wrote:
> On 10/20/2016 01:40 PM, Tom Rini wrote:
> >- Add a PPA for a more recent qemu (required for PowerPC to work)
> >- Add tests to run test.py for various QEMU platforms.  This relies on
> >  swarren's uboot-test-hooks repository to provide the abstractions.
> 
> Overall, the series,
> Acked-by: Stephen Warren 
> 
> A few comments on this particular patch below:
> 
> >diff --git a/.travis.yml b/.travis.yml
> 
> > addons:
> >   apt:
> >+sources:
> >+- sourceline: 'ppa:gns3/qemu'
> 
> Have you co-ordinated with "qns3" to ensure that PPA is always going
> to be available, and host binaries for Ubuntuy 16.04 (some PPAs
> eventually disappear as the owner moves on to other things, or
> change the set of distro releases they support at the owner's whim).

I don't like this part either.  I couldn't find an "official" PPA to
provide a newer qemu nor does it appear to be in the official backports
channel either.  My current plan is that if something happens to the PPA
I'll have to host one myself.

> >+ # Clone uboot-test-hooks
> >+ - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git 
> >/tmp/uboot-test-hooks
> 
> Does it make sense to rely on my copy of the repo being around all
> the time? I have no intention of removing it, but when I created it
> I'd assumed others would create their own clones for their own
> infra-structure. I wonder if all the dependencies for U-Boot's
> testing should be hosted on U-Boot infra-structure?

A good point.  The way I've been thinking of it is that we also need a
repo with as many and varied configs as possible, which has become yours
:).  That said, we could move a copy of this over to the u-boot account
on github if you'd like.

> That all said, this should be fine; just something to ponder.
> 
> >+ - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname`
> 
> It may be useful to add the following too:
> 
> - ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname`
> 
> That way, if we ever need per-board test configuration files, the
> path to access them will already be available.

A good point.  I think we could get the network tests going, maybe with
a little re-work to dynamically hash the file from the host side?  But
that's an aside.

> >+- env:
> >+- CROSS_COMPILE="/usr/bin/arm-linux-gnueabihf-"
> >+  script:
> >+- ./test/py/test.py --bd vexpress_ca15_tc2 --id qemu --build;
> >+  ./test/py/test.py --bd vexpress_ca9x4 --id qemu --build;
> >+  ./test/py/test.py --bd integratorcp_cm926ejs --id qemu --build;
> 
> Does it make sense to make that 3 separate scripts? I assume that
> would (a) allow the individual status of the 3 test.py invocations
> to be seen separately (b) might allow/cause travis-ci to run them in
> parallel; I'm not sure if this Travis config file is a sequential
> list of commands, or a list of potentially parallel actions.

So, the way travis-ci works is that you can (and we do) split tasks up
to run in parallel.  The hunk above is a series to run in sequence on
the same host. The downside to moving this to parallel is the amount of
duplication you need in each stanza.  We could fix that (at least in the
case of non-buildman-fetched toolchains) if test.py used buildman to
build, as that will determine CROSS_COMPILE correctly from the default
.buildman file we create earlier on.  Thanks!

-- 
Tom


signature.asc
Description: Digital signature
___
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

Re: [U-Boot] [PATCH v2 8/8] travis-ci: Add test.py for various qemu platforms

[Stephen Warren]

On 10/20/2016 01:40 PM, Tom Rini wrote:

- Add a PPA for a more recent qemu (required for PowerPC to work)
- Add tests to run test.py for various QEMU platforms.  This relies on
  swarren's uboot-test-hooks repository to provide the abstractions.


Overall, the series,
Acked-by: Stephen Warren 

A few comments on this particular patch below:


diff --git a/.travis.yml b/.travis.yml



 addons:
   apt:
+sources:
+- sourceline: 'ppa:gns3/qemu'


Have you co-ordinated with "qns3" to ensure that PPA is always going to 
be available, and host binaries for Ubuntuy 16.04 (some PPAs eventually 
disappear as the owner moves on to other things, or change the set of 
distro releases they support at the owner's whim).



+ # Clone uboot-test-hooks
+ - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git 
/tmp/uboot-test-hooks


Does it make sense to rely on my copy of the repo being around all the 
time? I have no intention of removing it, but when I created it I'd 
assumed others would create their own clones for their own 
infra-structure. I wonder if all the dependencies for U-Boot's testing 
should be hosted on U-Boot infra-structure?


That all said, this should be fine; just something to ponder.


+ - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname`


It may be useful to add the following too:

- ln -s travis-ci /tmp/uboot-test-hooks/py/`hostname`

That way, if we ever need per-board test configuration files, the path 
to access them will already be available.



+- env:
+- CROSS_COMPILE="/usr/bin/arm-linux-gnueabihf-"
+  script:
+- ./test/py/test.py --bd vexpress_ca15_tc2 --id qemu --build;
+  ./test/py/test.py --bd vexpress_ca9x4 --id qemu --build;
+  ./test/py/test.py --bd integratorcp_cm926ejs --id qemu --build;


Does it make sense to make that 3 separate scripts? I assume that would 
(a) allow the individual status of the 3 test.py invocations to be seen 
separately (b) might allow/cause travis-ci to run them in parallel; I'm 
not sure if this Travis config file is a sequential list of commands, or 
a list of potentially parallel actions.


___
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

[U-Boot] [PATCH v2 8/8] travis-ci: Add test.py for various qemu platforms

[Tom Rini]

- Add a PPA for a more recent qemu (required for PowerPC to work)
- Add tests to run test.py for various QEMU platforms.  This relies on
  swarren's uboot-test-hooks repository to provide the abstractions.

Signed-off-by: Tom Rini 
---
Ooops, Stephen has merged my request to add the conf files now, so
really reference his repository and not mine.

 .travis.yml | 35 ++-
 1 file changed, 34 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 25b0023d631b..1aff5d607d7e 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -10,6 +10,8 @@ language: c
 
 addons:
   apt:
+sources:
+- sourceline: 'ppa:gns3/qemu'
 packages:
 - cppcheck
 - sloccount
@@ -19,6 +21,10 @@ addons:
 - libsdl1.2-dev
 - python
 - python-virtualenv
+- qemu-system-arm
+- qemu-system-mips
+- qemu-system-ppc
+- qemu-system-x86
 - gcc-powerpc-linux-gnu
 - gcc-arm-linux-gnueabihf
 - iasl
@@ -27,6 +33,9 @@ install:
  # install latest device tree compiler
  - git clone --depth=1 git://git.kernel.org/pub/scm/utils/dtc/dtc.git /tmp/dtc
  - make -j4 -C /tmp/dtc
+ # Clone uboot-test-hooks
+ - git clone --depth=1 git://github.com/swarren/uboot-test-hooks.git 
/tmp/uboot-test-hooks
+ - ln -s travis-ci /tmp/uboot-test-hooks/bin/`hostname`
  # prepare buildman environment
  - export BUILDMAN_ROOT="root:"
  - echo -e "[toolchain]\n${BUILDMAN_ROOT} /usr" > ~/.buildman
@@ -38,7 +47,7 @@ install:
 
 env:
   global:
-- PATH=/tmp/dtc:$PATH
+- PATH=/tmp/dtc:/tmp/uboot-test-hooks/bin:$PATH
 - BUILD_DIR=build
 - HOSTCC="cc"
 - HOSTCXX="c++"
@@ -167,5 +176,29 @@ matrix:
 # test/py
 - script:
 - ./test/py/test.py --bd sandbox --build
+- env:
+- CROSS_COMPILE="/usr/bin/arm-linux-gnueabihf-"
+  script:
+- ./test/py/test.py --bd vexpress_ca15_tc2 --id qemu --build;
+  ./test/py/test.py --bd vexpress_ca9x4 --id qemu --build;
+  ./test/py/test.py --bd integratorcp_cm926ejs --id qemu --build;
+- env:
+- TOOLCHAIN="mips"
+  
CROSS_COMPILE="${HOME}/.buildman-toolchains/gcc-4.9.0-nolibc/mips-linux/bin/mips-linux-"
+  script:
+- ./test/py/test.py --bd qemu_mips --build -k 'not sleep';
+  ./test/py/test.py --bd qemu_mipsel --build -k 'not sleep';
+  ./test/py/test.py --bd qemu_mips64 --build -k 'not sleep';
+  ./test/py/test.py --bd qemu_mips64el --build -k 'not sleep';
+- env:
+- CROSS_COMPILE="/usr/bin/powerpc-linux-gnu-"
+  script:
+- ./test/py/test.py --bd qemu-ppce500 --build -k 'not sleep'
+- env:
+- TOOLCHAIN="x86_64"
+  BUILD_ROM=yes
+  
CROSS_COMPILE="${HOME}/.buildman-toolchains/gcc-4.9.0-nolibc/x86_64-linux/bin/x86_64-linux-"
+  script:
+- ./test/py/test.py --bd qemu-x86 --build -k 'not sleep'
 
 # TODO make it perfect ;-r
-- 
1.9.1

___
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot