Control: tags -1 + patch
Gilles Filippini a écrit le 06/04/2020 à 14:23 :
> Drew Parsons a écrit le 06/04/2020 à 05:08 :
>> On 2020-04-06 09:56, Drew Parsons wrote:
>>> On 2020-04-06 01:48, Gilles Filippini wrote:
>>>> Drew Parsons a écrit le 05/04/2020 à 18:57 :
>>>>>
>>>>> Another option is to create an environment variable to force h5py to
>>>>> load the mpi version even when run in a serial environment without
>>>>> mpirun. Easy enough to set up, though I'm interested to see if "mpirun
>>>>> -n 1 dh_auto_build" or a variation of that is viable. Maybe
>>>>> %:
>>>>> mpirun -n 1 dh $@ --with python3 --buildsystem=pybuild
>>>>
>>>> This, way the test cases run against python3.7 is OK, but it fails
>>>> against python3.8 with:
>>>>
>>>> I: pybuild base:217: cd
>>>> /build/bitshuffle-z2ZvpN/bitshuffle-0.3.5/.pybuild/cpython3_3.8_bitshuffle/build;
>>>>
>>>> python3.8 -m unittest discover -v
>>>> [pinibrem15:43725] OPAL ERROR: Unreachable in file ext3x_client.c at
>>>> line 112
>>>> *** An error occurred in MPI_Init_thread
>>>> *** on a NULL communicator
>>>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>>>> *** and potentially your MPI job)
>>>> [pinibrem15:43725] Local abort before MPI_INIT completed completed
>>>> successfully, but am not able to aggregate error messages, and not able
>>>> to guarantee that all other processes were killed!
>>>> E: pybuild pybuild:352: test: plugin distutils failed with: exit code=1:
>>>> cd
>>>> /build/bitshuffle-z2ZvpN/bitshuffle-0.3.5/.pybuild/cpython3_3.8_bitshuffle/build;
>>>>
>>>> python3.8 -m unittest discover -v
>>>> dh_auto_test: error: pybuild --test -i python{version} -p "3.7 3.8"
>>>> returned exit code 13
>>>>
>>>> But the HDF5 error is no more present with python3.7. So it seems a good
>>>> point.
>>>
>>>
>>> Strange again. I would have expected the same behaviour in python3.8
>>> and python3.7, whether successful or unsuccessful.
>>
>>
>> Putting dh into mpirun seems to be interfering with process spawning.
>> Once MPI is initialised (for the python3.7 test) it's not reinitialised
>> for the python3.8 and so it's in a bad state for the test. Something
>> like that.
>>
>> It's only in the tests where h5py is invoked that we get the problems.
>> This variant works, applying mpirun separately for each test run:
>>
>> override_dh_auto_test:
>> set -e; \
>> for py in `py3versions -s -v`; do \
>> mpirun -n 1 pybuild --test -i python{version} -p $$py; \
>> done
>>
>> (could use mpirun -n $(NPROC) for real mpi testing).
>
> Yes, it works! \o/
>
>> Do we want to use this as a solution? Or would you prefer an environment
>> variable that h5py can check to allow mpi invocation on a serial process?
>
> I let this decision up to you. Whatever you choose it deserve a bit fat
> note in README.Debian.
>
>> Note that this means bitshuffle as built now is expressly tied in with
>> hdf5-mpi and h5py-mpi (this seems intentional by debian/rules and
>> debian/control, though the Build-Depends must be updated to
>> python3-h5py-mpi). It's a separate question whether it's desirable to
>> also support a hdf5-serial build of bitshuffle. Likewise we need to
>> think about what we want to happen when bitshuffle is invoked in a
>> serial process.
>
> I'll let that to the bitshuffle maintainer. I'll propose a patch to fix
> the current FTBFS, sticking on the mpi flavour to be conservative vs
> bitshuffle's previous builds.Here it is. _g.
diff -Nru bitshuffle-0.3.5/debian/changelog bitshuffle-0.3.5/debian/changelog
--- bitshuffle-0.3.5/debian/changelog 2019-12-01 19:03:38.000000000 +0100
+++ bitshuffle-0.3.5/debian/changelog 2020-04-06 14:47:10.000000000 +0200
@@ -1,3 +1,21 @@
+bitshuffle (0.3.5-3.1) UNRELEASED; urgency=medium
+
+ * Non-maintainer upload.
+
+ [ Drew Parsons <dpars...@debian.org>, Gilles Filippini <p...@debian.org> ]
+ * Closes: #955456
+ - fix-deprecated.patch: fix test_h5filter.py and test_h5plugin.py
+ to open files with flag 'w' when required
+ - Build-Depends: python3-h5py-mpi to force using the mpi flavour
+ of h5py
+ - override_dh_auto_test:
+ - Run the tests via mpirun so that h5py knows it has to invoke its
+ mpi implementation
+ - Launch the tests for each python version separatly to permit MPI
+ initialization at each run
+
+ -- Gilles Filippini <p...@debian.org> Mon, 06 Apr 2020 14:47:10 +0200
+
bitshuffle (0.3.5-3) unstable; urgency=medium
* don't use -march=native when building the package
diff -Nru bitshuffle-0.3.5/debian/control bitshuffle-0.3.5/debian/control
--- bitshuffle-0.3.5/debian/control 2019-12-01 19:03:38.000000000 +0100
+++ bitshuffle-0.3.5/debian/control 2020-04-06 14:46:51.000000000 +0200
@@ -11,7 +11,7 @@
# , libopenmpi-dev
, openmpi-bin
, python3-setuptools
- , python3-h5py
+ , python3-h5py-mpi
, quilt
, cmake
, pkg-config
diff -Nru bitshuffle-0.3.5/debian/patches/fix-deprecated.patch
bitshuffle-0.3.5/debian/patches/fix-deprecated.patch
--- bitshuffle-0.3.5/debian/patches/fix-deprecated.patch 1970-01-01
01:00:00.000000000 +0100
+++ bitshuffle-0.3.5/debian/patches/fix-deprecated.patch 2020-04-06
14:46:44.000000000 +0200
@@ -0,0 +1,53 @@
+Index: bitshuffle-0.3.5/bitshuffle/tests/test_h5filter.py
+===================================================================
+--- bitshuffle-0.3.5.orig/bitshuffle/tests/test_h5filter.py
++++ bitshuffle-0.3.5/bitshuffle/tests/test_h5filter.py
+@@ -23,7 +23,7 @@ class TestFilter(unittest.TestCase):
+ dtype = np.int64
+ data = np.arange(shape[0])
+ fname = "tmp_test_filters.h5"
+- f = h5py.File(fname)
++ f = h5py.File(fname, 'w')
+ h5.create_dataset(f, b"range", shape, dtype, chunks,
+ filter_pipeline=(32008, 32000),
+ filter_flags=(h5z.FLAG_MANDATORY, h5z.FLAG_MANDATORY),
+@@ -43,7 +43,7 @@ class TestFilter(unittest.TestCase):
+ dtype = np.int64
+ data = np.arange(shape[0])
+ fname = "tmp_test_filters.h5"
+- f = h5py.File(fname)
++ f = h5py.File(fname, 'w')
+ h5.create_dataset(f, b"range", shape, dtype, chunks,
+ filter_pipeline=(32008, 32000),
+ filter_flags=(h5z.FLAG_MANDATORY, h5z.FLAG_MANDATORY),
+@@ -65,7 +65,7 @@ class TestFilter(unittest.TestCase):
+ dtype = np.int64
+ data = np.arange(shape[0])
+ fname = "tmp_test_filters.h5"
+- f = h5py.File(fname)
++ f = h5py.File(fname, 'w')
+ h5.create_dataset(f, b"range", shape, dtype, chunks,
+ filter_pipeline=(32008,),
+ filter_flags=(h5z.FLAG_MANDATORY,),
+Index: bitshuffle-0.3.5/bitshuffle/tests/test_h5plugin.py
+===================================================================
+--- bitshuffle-0.3.5.orig/bitshuffle/tests/test_h5plugin.py
++++ bitshuffle-0.3.5/bitshuffle/tests/test_h5plugin.py
+@@ -34,7 +34,7 @@ class TestFilterPlugins(unittest.TestCas
+ dtype = np.int64
+ data = np.arange(shape[0])
+ fname = "tmp_test_filters.h5"
+- f = h5py.File(fname)
++ f = h5py.File(fname, 'w')
+ tid = h5t.py_create(dtype, logical=1)
+ sid = h5s.create_simple(shape, shape)
+ #
+@@ -72,7 +72,7 @@ class TestFilterPlugins(unittest.TestCas
+ # return
+ # # Does not appear to be supported by h5py.
+ # fname = "tmp_test_h5py_hl.h5"
+- # f = h5py.File(fname)
++ # f = h5py.File(fname, 'w')
+ # f.create_dataset("range", np.arange(1024, dtype=np.int64),
+ # compression=32008)
+
diff -Nru bitshuffle-0.3.5/debian/patches/series
bitshuffle-0.3.5/debian/patches/series
--- bitshuffle-0.3.5/debian/patches/series 2019-12-01 19:03:38.000000000
+0100
+++ bitshuffle-0.3.5/debian/patches/series 2020-04-06 14:46:44.000000000
+0200
@@ -1,2 +1,3 @@
hdf5-old-api.patch
change-build-flags.patch
+fix-deprecated.patch
diff -Nru bitshuffle-0.3.5/debian/rules bitshuffle-0.3.5/debian/rules
--- bitshuffle-0.3.5/debian/rules 2019-08-20 19:29:38.000000000 +0200
+++ bitshuffle-0.3.5/debian/rules 2020-04-06 14:46:58.000000000 +0200
@@ -24,5 +24,11 @@
%:
dh $@ --with python3 --buildsystem=pybuild
+override_dh_auto_test:
+ set -e; \
+ for py in `py3versions -s -v`; do \
+ mpirun -n 1 pybuild --test -i python{version} -p $$py; \
+ done
+
#override_dh_auto_clean:
# dh_auto_clean
signature.asc
Description: OpenPGP digital signature
