On 2019-08-14 18:05, Steffen Möller wrote:
On 13.08.19 06:01, Drew Parsons wrote:

To reiterate, having h5py-mpi available will be transparent to a user
interacting with hdf as a serial library. It doesn't break serial use,
it just provides the capability to also run multicpu jobs.


This sounds like an omission not to feature, then. Please go for it.


h5py 2.9.0-3 will migrate to testing in a day or two, we can proceed with the mpi then.


How do autotests work for MPI?
We simply configure the test script to invoke the same tests using
mpirun.

I am somewhat uncertain that Debian needs to be the instance testing
this. But given all the hick-ups that are possibly introduced by
parallelization - would be good to test it. And Debian should then take
some pride in it and announce that.

Once we've got mpi activated in h5py, we can check whether the parallelisation does in fact improve your own workflow. Even on a laptop or desktop, most come with at least 4 cpus these days. Even mobile phones. Do you deal with GB-size hdf5 datasets, data for which access time is noticeable? Ideally your data handling will speed up according to the number of cpus added.

I don't think switching on mpi in h5py is itself such a big deal. But if we can demonstrate that it measurably improves performance for a real workflow, then that is worth crowing about.

Does Debian have any mechanisms to indicate that a software can run in
parallel? I am thinking about all the automation that now controls
workflows - like toil and/or cwl - or the testing of reverse
dependencies on some buildd. These can check for the presence for a
binary but don't immediately know if they should start it with mpirun.

No specific mechanism, since normally we known if the program is intended to be mpi enabled or not.

But at the level of the package, we can look at dependencies, e.g.
  apt-cache depends python3-scipy | grep mpi
  apt-cache depends python3-dolfin | grep mpi

At the level of a given library or executable, objdump can be helpful, e.g.
  objdump -p /usr/lib/x86_64-linux-gnu/libsuperlu.so | grep mpi
  objdump -p /usr/lib/x86_64-linux-gnu/libsuperlu_dist.so | grep mpi

For autopkgtest, it's our own tests so we already know if the program is compiled with mpi or not. It wouldn't really make sense for the scripts in debian/tests to check whether the program being tested was compiled with mpi.

Drew

Reply via email to