Re: RFC: kernelCI media subsystem pilot (Test results for gtucker/kernelci-media - gtucker-kernelci-media-001-6-g1b2c6e5844d8)

2018-10-19 Thread Ezequiel Garcia
On Fri, 2018-10-19 at 09:06 +0200, Hans Verkuil wrote:
> On 10/18/2018 09:23 PM, Ezequiel Garcia wrote:
> > Hi everyone,
> > 
> > In Collabora, and as part of our kernelci work, we are doing
> > research on kernel functional testing with kernelci.
> > 
> > For those new to kernelci, see 
> > https://github.com/kernelci/kernelci-doc/wiki/KernelCI 
> > and https://kernelci.org/.
> > 
> > The goal is to lay down the infrastructure required to make
> > automated test coverage an integral part of our feature
> > and bugfix development process.
> > 
> > So, as a first attempt, we've decided to extend kernelci test
> > v4l2 plan support, leading the way to extending
> > other subsystems' test plans.
> > 
> > Currently, kernelci looks for a list of branches every hour and
> > see if anything changed. For any branch that has changed, it triggers
> > builds, boots, tests and reports for each branch that had some changes
> > since last time it ran.
> > 
> > For this pilot, we've decided to target just a few devices:
> > qemu with vivid, rk3399-gru-kevin and rk3288-veyron-jaq
> > with uvc.
> 
> It's running v4l2-compliance, right?
> 

Exactly.

> Looking at the test cases, they appear in the reverse order that 
> v4l2-compliance
> performs them, that's a bit odd.
> 

That's something to check in the parser. I'm sure it's fixable if you
find it annoying.

> And if we include uvc in the testing, then I need to prioritize the work I
> started for uvc to remove the last FAILs.
> 

Well, we can run anything. We decided to go for uvc and vivid, just too pick
two popular examples, but there are really no restrictions.

Thanks,
Ezequiel


Re: RFC: kernelCI media subsystem pilot (Test results for gtucker/kernelci-media - gtucker-kernelci-media-001-6-g1b2c6e5844d8)

2018-10-19 Thread Hans Verkuil
On 10/18/2018 09:23 PM, Ezequiel Garcia wrote:
> Hi everyone,
> 
> In Collabora, and as part of our kernelci work, we are doing
> research on kernel functional testing with kernelci.
> 
> For those new to kernelci, see 
> https://github.com/kernelci/kernelci-doc/wiki/KernelCI 
> and https://kernelci.org/.
> 
> The goal is to lay down the infrastructure required to make
> automated test coverage an integral part of our feature
> and bugfix development process.
> 
> So, as a first attempt, we've decided to extend kernelci test
> v4l2 plan support, leading the way to extending
> other subsystems' test plans.
> 
> Currently, kernelci looks for a list of branches every hour and
> see if anything changed. For any branch that has changed, it triggers
> builds, boots, tests and reports for each branch that had some changes
> since last time it ran.
> 
> For this pilot, we've decided to target just a few devices:
> qemu with vivid, rk3399-gru-kevin and rk3288-veyron-jaq
> with uvc.

It's running v4l2-compliance, right?

Looking at the test cases, they appear in the reverse order that v4l2-compliance
performs them, that's a bit odd.

And if we include uvc in the testing, then I need to prioritize the work I
started for uvc to remove the last FAILs.

Regards,

Hans

> 
> We'd like to get some early feedback on this, so we are sending
> an example of how a kernelci report would look like, to trigger
> some discussion around
> the direction this should take.
> 
> Thanks!
> 
> ===
> Test results for:
>   Tree:gtucker
>   Branch:  kernelci-media
>   Kernel:  gtucker-kernelci-media-002-2-gaa27eb0392c7
>   URL: https://gitlab.collabora.com/gtucker/linux.git
>   Commit:  aa27eb0392c70adec713e911a9b5267a1d853624
>   Test plans: v4l2
> 
> Summary
> ---
> 3 test groups results
> 
> 1  | v4l2   | rk3399-gru-kevin   | arm64 |  49 total:  17 PASS   4 
> FAIL  28 SKIP
> 2  | v4l2   | rk3288-veyron-jaq  | arm   |  49 total:  17 PASS   4 
> FAIL  28 SKIP
> 3  | v4l2   | qemu   | arm64 | 168 total: 102 PASS   0 
> FAIL  66 SKIP
> 
> 
> Tests
> -
> 
> 1  | v4l2   | rk3399-gru-kevin   | arm64 |  49 total:  17 PASS   4 
> FAIL  28 SKIP
> 
>   Config:  defconfig
>   Lab Name:lab-collabora-dev
>   Date:2018-10-18 18:48:52.426000
>   TXT log: 
> http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig/lab-collabora-dev
> /v4l2-rk3399-gru-kevin.txt
>   HTML log:
> http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig/lab-collabora-dev
> /v4l2-rk3399-gru-kevin.html
>   Rootfs:  
> http://staging-storage.kernelci.org/images/rootfs/debian/stretchv4l2/20181017.1//arm64/rootfs.cpio.gz
>   Test Git:git://linuxtv.org/v4l-utils.git
>   Test Commit: e889b0d56757b9a74eb8c4c8cc2ebd5b18e228b2
> 
> 
>   Test cases:
> 
> * MMAP: FAIL
> * blocking-wait: PASS
> * read/write: SKIP
> * read/write: SKIP
> * read/write: SKIP
> * VIDIOC_EXPBUF: PASS
> * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS
> * VIDIOC_TRY_DECODER_CMD: SKIP
> * VIDIOC_G_ENC_INDEX: SKIP
> * VIDIOC_TRY_ENCODER_CMD: SKIP
> * Scaling: SKIP
> * Composing: SKIP
> * Cropping: SKIP
> * VIDIOC_G_SLICED_VBI_CAP: SKIP
> * VIDIOC_S_FMT: PASS
> * VIDIOC_TRY_FMT: PASS
> * VIDIOC_G_FMT: PASS
> * VIDIOC_G_FBUF: SKIP
> * VIDIOC_G/S_PARM: FAIL
> * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS
> * VIDIOC_G/S_JPEGCOMP: SKIP
> * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS
> * VIDIOC_G/S/TRY_EXT_CTRLS: FAIL
> * VIDIOC_G/S_CTRL: PASS
> * VIDIOC_QUERYCTRL: PASS
> * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL
> * VIDIOC_G/S_EDID: SKIP
> * VIDIOC_DV_TIMINGS_CAP: SKIP
> * VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: SKIP
> * VIDIOC_ENUM/G/S/QUERY_STD: SKIP
> * VIDIOC_G/S_AUDOUT: SKIP
> * VIDIOC_G/S/ENUMOUTPUT: SKIP
> * VIDIOC_ENUMAUDOUT: SKIP
> * VIDIOC_G/S_FREQUENCY: SKIP
> * VIDIOC_G/S_MODULATOR: SKIP
> * VIDIOC_G/S_AUDIO: SKIP
> * VIDIOC_G/S/ENUMINPUT: PASS
> * VIDIOC_ENUMAUDIO: SKIP
> * VIDIOC_S_HW_FREQ_SEEK: SKIP
> * VIDIOC_G/S_FREQUENCY: SKIP
> * VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: SKIP
> * VIDIOC_LOG_STATUS: SKIP
> * VIDIOC_DBG_G/S_REGISTER: SKIP
> * for-unlimited-opens: PASS
> * VIDIOC_G/S_PRIORITY: PASS
> * VIDIOC_QUERYCAP: PASS
> * second-/dev/video0-open: PASS
> * VIDIOC_QUERYCAP: PASS
> * MC-information-see-Media-Driver-Info-above: PASS
> 
> 
> 
> 2  | v4l2   | rk3288-veyron-jaq  | arm   |  49 total:  17 PASS   4 
> FAIL  28 SKIP
> 
>   Config:  multi_v7_defconfig
>   Lab Name:lab-collabora-dev
>   Date:2018-10-18 17:00:41.724000
>   TXT log: 
> http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm/multi_v7_defconfig/lab-collab
> ora-de

RFC: kernelCI media subsystem pilot (Test results for gtucker/kernelci-media - gtucker-kernelci-media-001-6-g1b2c6e5844d8)

2018-10-18 Thread Ezequiel Garcia
Hi everyone,

In Collabora, and as part of our kernelci work, we are doing
research on kernel functional testing with kernelci.

For those new to kernelci, see 
https://github.com/kernelci/kernelci-doc/wiki/KernelCI 
and https://kernelci.org/.

The goal is to lay down the infrastructure required to make
automated test coverage an integral part of our feature
and bugfix development process.

So, as a first attempt, we've decided to extend kernelci test
v4l2 plan support, leading the way to extending
other subsystems' test plans.

Currently, kernelci looks for a list of branches every hour and
see if anything changed. For any branch that has changed, it triggers
builds, boots, tests and reports for each branch that had some changes
since last time it ran.

For this pilot, we've decided to target just a few devices:
qemu with vivid, rk3399-gru-kevin and rk3288-veyron-jaq
with uvc.

We'd like to get some early feedback on this, so we are sending
an example of how a kernelci report would look like, to trigger
some discussion around
the direction this should take.

Thanks!

===
Test results for:
  Tree:gtucker
  Branch:  kernelci-media
  Kernel:  gtucker-kernelci-media-002-2-gaa27eb0392c7
  URL: https://gitlab.collabora.com/gtucker/linux.git
  Commit:  aa27eb0392c70adec713e911a9b5267a1d853624
  Test plans: v4l2

Summary
---
3 test groups results

1  | v4l2   | rk3399-gru-kevin   | arm64 |  49 total:  17 PASS   4 FAIL 
 28 SKIP
2  | v4l2   | rk3288-veyron-jaq  | arm   |  49 total:  17 PASS   4 FAIL 
 28 SKIP
3  | v4l2   | qemu   | arm64 | 168 total: 102 PASS   0 FAIL 
 66 SKIP


Tests
-

1  | v4l2   | rk3399-gru-kevin   | arm64 |  49 total:  17 PASS   4 FAIL 
 28 SKIP

  Config:  defconfig
  Lab Name:lab-collabora-dev
  Date:2018-10-18 18:48:52.426000
  TXT log: 
http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig/lab-collabora-dev
/v4l2-rk3399-gru-kevin.txt
  HTML log:
http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig/lab-collabora-dev
/v4l2-rk3399-gru-kevin.html
  Rootfs:  
http://staging-storage.kernelci.org/images/rootfs/debian/stretchv4l2/20181017.1//arm64/rootfs.cpio.gz
  Test Git:git://linuxtv.org/v4l-utils.git
  Test Commit: e889b0d56757b9a74eb8c4c8cc2ebd5b18e228b2


  Test cases:

* MMAP: FAIL
* blocking-wait: PASS
* read/write: SKIP
* read/write: SKIP
* read/write: SKIP
* VIDIOC_EXPBUF: PASS
* VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS
* VIDIOC_TRY_DECODER_CMD: SKIP
* VIDIOC_G_ENC_INDEX: SKIP
* VIDIOC_TRY_ENCODER_CMD: SKIP
* Scaling: SKIP
* Composing: SKIP
* Cropping: SKIP
* VIDIOC_G_SLICED_VBI_CAP: SKIP
* VIDIOC_S_FMT: PASS
* VIDIOC_TRY_FMT: PASS
* VIDIOC_G_FMT: PASS
* VIDIOC_G_FBUF: SKIP
* VIDIOC_G/S_PARM: FAIL
* VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS
* VIDIOC_G/S_JPEGCOMP: SKIP
* VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS
* VIDIOC_G/S/TRY_EXT_CTRLS: FAIL
* VIDIOC_G/S_CTRL: PASS
* VIDIOC_QUERYCTRL: PASS
* VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL
* VIDIOC_G/S_EDID: SKIP
* VIDIOC_DV_TIMINGS_CAP: SKIP
* VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: SKIP
* VIDIOC_ENUM/G/S/QUERY_STD: SKIP
* VIDIOC_G/S_AUDOUT: SKIP
* VIDIOC_G/S/ENUMOUTPUT: SKIP
* VIDIOC_ENUMAUDOUT: SKIP
* VIDIOC_G/S_FREQUENCY: SKIP
* VIDIOC_G/S_MODULATOR: SKIP
* VIDIOC_G/S_AUDIO: SKIP
* VIDIOC_G/S/ENUMINPUT: PASS
* VIDIOC_ENUMAUDIO: SKIP
* VIDIOC_S_HW_FREQ_SEEK: SKIP
* VIDIOC_G/S_FREQUENCY: SKIP
* VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: SKIP
* VIDIOC_LOG_STATUS: SKIP
* VIDIOC_DBG_G/S_REGISTER: SKIP
* for-unlimited-opens: PASS
* VIDIOC_G/S_PRIORITY: PASS
* VIDIOC_QUERYCAP: PASS
* second-/dev/video0-open: PASS
* VIDIOC_QUERYCAP: PASS
* MC-information-see-Media-Driver-Info-above: PASS



2  | v4l2   | rk3288-veyron-jaq  | arm   |  49 total:  17 PASS   4 FAIL 
 28 SKIP

  Config:  multi_v7_defconfig
  Lab Name:lab-collabora-dev
  Date:2018-10-18 17:00:41.724000
  TXT log: 
http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm/multi_v7_defconfig/lab-collab
ora-dev/v4l2-rk3288-veyron-jaq.txt
  HTML log:
http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm/multi_v7_defconfig/lab-collab
ora-dev/v4l2-rk3288-veyron-jaq.html
  Rootfs:  
http://staging-storage.kernelci.org/images/rootfs/debian/stretchv4l2/20181017.1//armhf/rootfs.cpio.gz
  Test Git:git://linuxtv.org/v4l-utils.git
  Test Commit: e889b0d56757b9a74eb8c4c8cc2ebd5b18e228b2


  Test cases:

* MMAP: FAIL
* blocking-wait: PASS
* read/write: SKIP
* read/write: SKIP
* read/write: SKIP
* VIDIOC_EXPBUF: PASS
* VIDIOC_REQBUFS/CREAT