Bug#870128: libgpuarray: broken autopkgtests

[Rebecca N. Palmer]

Control: tags -1 fixed-upstream

Upstream have a different patch for skipping CUDA-only tests on OpenCL 
(which I haven't tried myself): 
https://github.com/Theano/libgpuarray/commit/f036aef3a425560161de362f390d238f4e7c1721


There are also two real issues in this set of test failures/errors:

- extract_tril/extract_triu - cast across address spaces - fixed 
upstream by 
https://github.com/Theano/libgpuarray/commit/99545026f00e8697bfa5b8fc326f1a808bbe659c


- test_elemwise_f16 add/iadd - wrong rounding mode (half precision is 
low enough for 1-in-last-digit to be out of tolerance) - fixed upstream 
by 
https://github.com/Theano/libgpuarray/commit/5761af18431576e38fa45ff05a0712db84689166


All of these are included in upstream 0.7.6, and given #901487, I 
suggest upgrading rather than cherry-picking.


(This won't stop the tests being D-Uninstallable on non-pocl 
architectures, but given that they wouldn't be usefully runnable there 
anyway (mesa-opencl-icd is available on more architectures but requires 
an AMD/ATI GPU, so won't work on a typical server), they should probably 
stay such.)

Bug#870128: libgpuarray: broken autopkgtests

[Steve Langasek]

Control: tags -1 patch
Control: user ubuntu-de...@lists.ubuntu.com
Control: usertags -1 ubuntu-patch

On Mon, Aug 28, 2017 at 11:37:05PM -0700, Steve Langasek wrote:
> Hi Ghislain,
> 
> On Sun, Jul 30, 2017 at 01:16:14PM +0100, Ghislain Vaillant wrote:
> 
> > Indeed, which allows running the upstream test suite on our CPU-based
> > builders. This is better than the default autopkgtest-pkg-python, which
> > does nothing but testing the import of the Python module.
> 
> > Your assessment is wrong. pocl is a **CPU** implementation of OpenCL,
> > so the tests should run in principle. The issue is that the test suite
> > does not skip tests for which a specific compute must be satisfied by
> > the environment, such as CUDA and float16. The rest of the tests runs
> > fine.
> 
> Ah, thanks for the clarification.
> 
> > The old basic does nothing. For such package, testing whether the
> > import is successful is hardly testing at all.
> 
> Fair enough.
> 
> >> I am filing this bug because failing autopkgtests are considered blockers
> >> for inclusion of a package in the Ubuntu release.  Newer versions of
> >> libgpuarray might not be included in Ubuntu releases until this is
> >> resolved.
> 
> > I believe this is wrong. How can we provide upstream with meaningful
> > logs without enabling the tests in the first place? I can keep the old
> > autopkgtest-pkg-python to satisfy Ubuntu's policy, but what good would
> > it make?
> 
> Well, you can generate whatever logs you want, the issue is when
> known-bad tests are treated as a *failure* of the autopkgtest.  This
> prevents the autopkgtests from being used for continuous integration,
> because there is no signal to say when the test results are good.
> 
> If the testing framework supports marking these tests as XFAIL as part of
> the autopkgtest, or otherwise skipping them when run on unsuitable hardware,
> I think that would be preferable.
> 
> Attached is a patch that appears to achieve this here.

And here is a more complete patch that should let the tests pass on all
architectures, not just on amd64.

Thanks,
-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
diff -Nru libgpuarray-0.6.9/debian/patches/series 
libgpuarray-0.6.9/debian/patches/series
--- libgpuarray-0.6.9/debian/patches/series 1969-12-31 16:00:00.0 
-0800
+++ libgpuarray-0.6.9/debian/patches/series 2017-08-28 23:19:34.0 
-0700
@@ -0,0 +1 @@
+skip-cuda-tests
diff -Nru libgpuarray-0.6.9/debian/patches/skip-cuda-tests 
libgpuarray-0.6.9/debian/patches/skip-cuda-tests
--- libgpuarray-0.6.9/debian/patches/skip-cuda-tests1969-12-31 
16:00:00.0 -0800
+++ libgpuarray-0.6.9/debian/patches/skip-cuda-tests2017-08-29 
17:04:37.0 -0700
@@ -0,0 +1,158 @@
+Description: skip those tests that we know will fail on opencl
+ Some of the upstream tests require CUDA in order to run.  Skip these tests
+ instead of treating them as errors when called in a non-CUDA environment.
+Author: Steve Langasek 
+Bug-Debian: https://bugs.debian.org/870128
+
+Index: libgpuarray-0.6.9/pygpu/tests/test_elemwise.py
+===
+--- libgpuarray-0.6.9.orig/pygpu/tests/test_elemwise.py
 libgpuarray-0.6.9/pygpu/tests/test_elemwise.py
+@@ -2,6 +2,7 @@
+ import numpy
+ from mako.template import Template
+ 
++from nose.plugins.skip import SkipTest
+ from unittest import TestCase
+ from pygpu import gpuarray, ndgpuarray as elemary
+ from pygpu.dtypes import dtype_to_ctype, get_common_dtype
+@@ -43,7 +44,10 @@
+ c, g = gen_gpuarray((50,), dtype, ctx=context, cls=elemary)
+ 
+ out_c = op(c)
+-out_g = op(g)
++try:
++out_g = op(g)
++except gpuarray.GpuArrayException:
++raise SkipTest("skipping test on this hardware")
+ 
+ assert out_c.shape == out_g.shape
+ assert out_c.dtype == out_g.dtype
+@@ -123,7 +127,10 @@
+   cls=elemary)
+ 
+ out_c = op(ac, bc)
+-out_g = op(ag, bg)
++try:
++out_g = op(ag, bg)
++except gpuarray.GpuArrayException:
++raise SkipTest("skipping test on this hardware")
+ 
+ assert out_c.shape == out_g.shape
+ assert out_c.dtype == out_g.dtype
+@@ -147,7 +154,10 @@
+ # TODO: currently, we use old Numpy semantic and tolerate more case.
+ # So we can't test that we raise the same error
+ return
+-out_g = op(ag, bg)
++try:
++out_g = op(ag, bg)
++except gpuarray.GpuArrayException:
++raise SkipTest("skipping test on this hardware")
+ 
+ assert out_g is ag
+ assert numpy.allclose(out_c, numpy.asarray(out_g), atol=1e-6)
+Index: libgpuarray-0.6.9/pygpu/tests/test_gpu_ndarray.py

Bug#870128: libgpuarray: broken autopkgtests

[Steve Langasek]

Hi Ghislain,

On Sun, Jul 30, 2017 at 01:16:14PM +0100, Ghislain Vaillant wrote:

> Indeed, which allows running the upstream test suite on our CPU-based
> builders. This is better than the default autopkgtest-pkg-python, which
> does nothing but testing the import of the Python module.

> Your assessment is wrong. pocl is a **CPU** implementation of OpenCL,
> so the tests should run in principle. The issue is that the test suite
> does not skip tests for which a specific compute must be satisfied by
> the environment, such as CUDA and float16. The rest of the tests runs
> fine.

Ah, thanks for the clarification.

> The old basic does nothing. For such package, testing whether the
> import is successful is hardly testing at all.

Fair enough.

>> I am filing this bug because failing autopkgtests are considered blockers
>> for inclusion of a package in the Ubuntu release.  Newer versions of
>> libgpuarray might not be included in Ubuntu releases until this is
>> resolved.

> I believe this is wrong. How can we provide upstream with meaningful
> logs without enabling the tests in the first place? I can keep the old
> autopkgtest-pkg-python to satisfy Ubuntu's policy, but what good would
> it make?

Well, you can generate whatever logs you want, the issue is when
known-bad tests are treated as a *failure* of the autopkgtest.  This
prevents the autopkgtests from being used for continuous integration,
because there is no signal to say when the test results are good.

If the testing framework supports marking these tests as XFAIL as part of
the autopkgtest, or otherwise skipping them when run on unsuitable hardware,
I think that would be preferable.

Attached is a patch that appears to achieve this here.

Cheers,
-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
diff -Nru libgpuarray-0.6.9/debian/changelog libgpuarray-0.6.9/debian/changelog
--- libgpuarray-0.6.9/debian/changelog  2017-08-05 09:29:35.0 -0700
+++ libgpuarray-0.6.9/debian/changelog  2017-08-28 23:30:28.0 -0700
@@ -1,3 +1,10 @@
+libgpuarray (0.6.9-2ubuntu1) artful; urgency=medium
+
+  * debian/patches/skip-cuda-tests: skip those tests that we know will
+fail on opencl.  Closes: #870128.
+
+ -- Steve Langasek   Mon, 28 Aug 2017 23:30:28 -0700
+
 libgpuarray (0.6.9-2build1) artful; urgency=medium
 
   * No-change rebuild to build to drop python3.5.
diff -Nru libgpuarray-0.6.9/debian/patches/series 
libgpuarray-0.6.9/debian/patches/series
--- libgpuarray-0.6.9/debian/patches/series 1969-12-31 16:00:00.0 
-0800
+++ libgpuarray-0.6.9/debian/patches/series 2017-08-28 23:19:34.0 
-0700
@@ -0,0 +1 @@
+skip-cuda-tests
diff -Nru libgpuarray-0.6.9/debian/patches/skip-cuda-tests 
libgpuarray-0.6.9/debian/patches/skip-cuda-tests
--- libgpuarray-0.6.9/debian/patches/skip-cuda-tests1969-12-31 
16:00:00.0 -0800
+++ libgpuarray-0.6.9/debian/patches/skip-cuda-tests2017-08-28 
23:30:28.0 -0700
@@ -0,0 +1,90 @@
+Description: skip those tests that we know will fail on opencl
+ Some of the upstream tests require CUDA in order to run.  Skip these tests
+ instead of treating them as errors when called in a non-CUDA environment.
+Author: Steve Langasek 
+Bug-Debian: https://bugs.debian.org/870128
+
+Index: libgpuarray-0.6.9/pygpu/tests/test_elemwise.py
+===
+--- libgpuarray-0.6.9.orig/pygpu/tests/test_elemwise.py
 libgpuarray-0.6.9/pygpu/tests/test_elemwise.py
+@@ -2,6 +2,7 @@
+ import numpy
+ from mako.template import Template
+ 
++from nose.plugins.skip import SkipTest
+ from unittest import TestCase
+ from pygpu import gpuarray, ndgpuarray as elemary
+ from pygpu.dtypes import dtype_to_ctype, get_common_dtype
+@@ -43,7 +44,10 @@
+ c, g = gen_gpuarray((50,), dtype, ctx=context, cls=elemary)
+ 
+ out_c = op(c)
+-out_g = op(g)
++try:
++out_g = op(g)
++except gpuarray.GpuArrayException:
++raise SkipTest("skipping test on this hardware")
+ 
+ assert out_c.shape == out_g.shape
+ assert out_c.dtype == out_g.dtype
+@@ -123,7 +127,10 @@
+   cls=elemary)
+ 
+ out_c = op(ac, bc)
+-out_g = op(ag, bg)
++try:
++out_g = op(ag, bg)
++except gpuarray.GpuArrayException:
++raise SkipTest("skipping test on this hardware")
+ 
+ assert out_c.shape == out_g.shape
+ assert out_c.dtype == out_g.dtype
+@@ -147,7 +154,10 @@
+ # TODO: currently, we use old Numpy semantic and tolerate more case.
+ # So we can't test that we raise the same error
+ return
+-out_g = op(ag, bg)
++try:
++out_g = op(ag, bg)
++except 

Bug#870128: libgpuarray: broken autopkgtests

[Ghislain Vaillant]

On Sun, 2017-07-30 at 17:35 +0200, Graham Inggs wrote:
> Can you test your package with mesa-opencl-icd instead of pocl-
> opencl-icd?

mesa-opencl-icd is for devices compatible with amdgpu.

> This will allow the test suite to run on more architectures than only
> amd64 and i386.

Not really, unless all Debian builders are spec'd with semi-recent AMD
graphics or APUs. I suspect this is not the case.

I am afraid the only solution for testing OpenCL packages in a
hardware-neutral way is pocl.

Ghis

Bug#870128: libgpuarray: broken autopkgtests

[Graham Inggs]

Hi Ghislain

Can you test your package with mesa-opencl-icd instead of pocl-opencl-icd?
This will allow the test suite to run on more architectures than only
amd64 and i386.

Regards
Graham

Bug#870128: libgpuarray: broken autopkgtests

[Ghislain Vaillant]

On Sat, 29 Jul 2017 23:33:21 -0700 Steve Langasek
 wrote:
>
> Your latest upload of libgpuarray has replaced the previous python
> autopkgtest with one based on pocl.

Indeed, which allows running the upstream test suite on our CPU-based
builders. This is better than the default autopkgtest-pkg-python, which
does nothing but testing the import of the Python module.

> Unfortunately, this means that the autopkgtests now consistently fail
> on ci.debian.net, because they can no longer be run on a system without
> a gpu, e.g.:

Your assessment is wrong. pocl is a **CPU** implementation of OpenCL,
so the tests should run in principle. The issue is that the test suite
does not skip tests for which a specific compute must be satisfied by
the environment, such as CUDA and float16. The rest of the tests runs
fine.

I filed the following issues upstream as a result:

https://github.com/Theano/libgpuarray/issues/491
https://github.com/Theano/libgpuarray/issues/492

> ==
> ERROR: pygpu.tests.test_gpu_ndarray.test_asfortranarray((),
'float32', True, True, 1, 'f')
> ---
---
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in
runTest
> self.test(*self.arg)
>   File "/usr/lib/python2.7/dist-packages/pygpu/tests/support.py",
line 44, in f
> func(*args, **kwargs)
>   File "/usr/lib/python2.7/dist-
packages/pygpu/tests/test_gpu_ndarray.py", line 196, in asfortranarray
> assert b.gpudata == gpu.gpudata
>   File "pygpu/gpuarray.pyx", line 2260, in
pygpu.gpuarray.GpuArray.gpudata.__get__ (pygpu/gpuarray.c:28840)
> TypeError: This is for CUDA arrays.
> 
> https://ci.debian.net/packages/libg/libgpuarray/unstable/amd64/
> 
> I would suggest that it's more useful to have a basic autopkgtest
that
> passes on generic hardware than to have a deep autopkgtest which will
never
> succeed there.

The old basic does nothing. For such package, testing whether the
import is successful is hardly testing at all.

> I am filing this bug because failing autopkgtests are considered
blockers
> for inclusion of a package in the Ubuntu release.  Newer versions of
> libgpuarray might not be included in Ubuntu releases until this is
resolved.

I believe this is wrong. How can we provide upstream with meaningful
logs without enabling the tests in the first place? I can keep the old
autopkgtest-pkg-python to satisfy Ubuntu's policy, but what good would
it make?

Ghis

Bug#870128: libgpuarray: broken autopkgtests

[Steve Langasek]

Source: libgpuarray
Version: 0.6.9-2
Severity: important
User: ubuntu-de...@lists.ubuntu.com
Usertags: origin-ubuntu artful autopkgtest

Hi Ghislain,

Your latest upload of libgpuarray has replaced the previous python
autopkgtest with one based on pocl.  Unfortunately, this means that the
autopkgtests now consistently fail on ci.debian.net, because they can no
longer be run on a system without a gpu, e.g.:

==
ERROR: pygpu.tests.test_gpu_ndarray.test_asfortranarray((), 'float32', True, 
True, 1, 'f')
--
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/pygpu/tests/support.py", line 44, in f
func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pygpu/tests/test_gpu_ndarray.py", line 
196, in asfortranarray
assert b.gpudata == gpu.gpudata
  File "pygpu/gpuarray.pyx", line 2260, in 
pygpu.gpuarray.GpuArray.gpudata.__get__ (pygpu/gpuarray.c:28840)
TypeError: This is for CUDA arrays.

https://ci.debian.net/packages/libg/libgpuarray/unstable/amd64/

I would suggest that it's more useful to have a basic autopkgtest that
passes on generic hardware than to have a deep autopkgtest which will never
succeed there.

I am filing this bug because failing autopkgtests are considered blockers
for inclusion of a package in the Ubuntu release.  Newer versions of
libgpuarray might not be included in Ubuntu releases until this is resolved.

Thanks,
-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature