So far, Kwant does not contain any support for parallelism. However, it works OK with libraries for task parallelism, e.g. concurrent.futures. (See the Kwant paper for an example). Task parallelism means that several independent tasks (e.g. related to different leads or different energies), are processed in parallel. Support for task parallelism could be improved (Kwant systems could be made pickleable, for example), but this does not diminish the validity of the following.
Where Kwant is lacking utterly is data parallelism. Data parallel programs process (different parts) of the same data synchronously. Different cores work on solving different parts of the same linear system, for example.
The inability to harness data parallelism is a severe handicap for Kwant. As you all know, one does not always have more independent tasks than available cores. This is especially true for comparatively short-running exploratory workloads. A solution exists: direct linear solvers seem to scale quite well up to a hundred or so cores [1].
Our direct linear solver of choice, MUMPS, supports data parallelism through MPI. The same seems to be true for all other serious options that I could find, notably Pardiso. (MUMPS, however, seems to be the only library that is both efficient and free.) As such, it seems to me that we should embrace MPI in Kwant. I believe that MPI support can be added to Kwant in a way that at the same time
• keeps MPI completely optional for Kwant users, • and respects best practices for MPI-enabled libraries [2].The idea is to add an optional "comm" (for communicator) parameter to operations that can be performed in a data-parallel way, like kwant.smatrix. The default value of this parameter would be "None" to indicate non-parallel execution (just like now).
For example, turning a sequential Kwant script into a data-parallel one would have to
• add "from mpi4py import MPI" in the preamble,• substitute kwant.smatrix(…) -> kwant.smatrix(…, comm=MPI.COMM_WORLD), • execute the script using "mpiexec", or a queuing system, or using jupyter in an appropriate way (it’s possible)
I do not see a way to completely hide MPI from the user, nor do I think that this is desirable (it would make it impossible to use Kwant together with other libraries that use MPI). However, the above seems to me quite acceptable, especially because only users who want to go data-parallel would be affected.
Some people claim that MPI has already seen its best days [3], however I do not think that this analysis is relevant to Kwant: I do not see how to base Kwant on something like Spark, because there are no direct linear solvers for that framework. MPI will certainly continue to be the standard HPC framework for many years (the body of existing code alone is enough to justify this). If/when a viable alternative emerges in 5-10 years, we should be able to adapt quite easily, since Kwant will most likely continue to depend only weakly on it (by using libraries like MUMPS).
[1] http://mumps.enseeiht.fr/doc/poster_mumps.pdf [2] https://htor.inf.ethz.ch/publications/index.php?pub=135 [3] http://www.dursi.ca/hpc-is-dying-and-mpi-is-killing-it/
smime.p7s
Description: S/MIME cryptographic signature
