[Kwant-devel] Retirement of kwant-devel
Dear subscribers of kwant-devel and of kwant-discuss, Due to the very limited usage of the kwant-devel mailing list (most development discussions happen in the issues and merge requests of our gitlab instance [1]), we have decided to retire it. Still, a mailing list is a useful forum also for development, especially when more general topics are being discussed. That's why, starting from now, we recommend [2] the use the kwant-discuss list also for development discussions. This will have the welcome side-effect of giving development discussions a wider audience. Note that kwant-discuss was never meant to be a pure Q list. A number of subscribers to kwant-devel are not members of kwant-discuss. Would you please consider subscribing? The average volume is rather low: a couple of messages per week. In order to subscribe, simply send an arbitrary message to kwant-discuss-j...@kwant-project.org (the subject and body are ignored). (Those who are subscribed to both lists have received this message twice. My apologies for that.) As far as kwant-devel is concerned, you have nothing to do. Posting is technically still allowed for the time being. We will eventually disable it, but the list and its archives will continue to linger. Cheers, Christoph [1] https://gitlab.kwant-project.org/ [2] https://kwant-project.org/community smime.p7s Description: S/MIME cryptographic signature
Re: [Kwant-devel] Important missing information in the kwant doc
alexandre.berna...@u-psud.fr a écrit : > I noticed, if I am not mistaken, that the attribute > "kwant.system.System.sites" is not present in the documentation, and > this attribute is very usefull to get the correspondance index <---> > position. A while ago, we added kwant.builder.FiniteSystem and kwant.builder.InifiniteSystem to the documentation in order do document this. But I notice that kwant.builder.Builder.finalized [1] still refers to kwant.system.FiniteSystem etc. Perhaps we should also add a reference from kwant.system to the systems in kwant.builder, because in practice it's them who are used most often. What do you think, Alexandre? Christoph [1] https://kwant-project.org/doc/1/reference/generated/kwant.builder.Builder#kwant.builder.Builder.finalized smime.p7s Description: S/MIME cryptographic signature
[Kwant-devel] (Composite) systems
Overview In the past, we discussed a new format for Kwant systems [1] that will allow more general symmetries. Joe and Daniel have implemented a prototype [2] that supports general symmetries. Since general symmetries turn out to be tricky, the current consensus is to concentrate on translational symmetries at a first time. More general symmetries can then be added later. This staged approach does not have to complicate the final format since it seems anyway a good idea to have separate layers for translational and other spatial symmetries. This way the non-translational symmetries can remain hidden from solvers who are not interested in them. There is another important aspect of the "new system" effort: we need a way to couple systems of different symmetry into one composite system. Such composite systems exist in today's Kwant but are not a separate concept from finite systems (this is easy to overcome). However, they are limited to attaching 1d leads to 0d scattering regions. Overcoming this limitation is tricky and I'd like to start a discussion on how to best do it. [1] https://www.mail-archive.com/kwant-devel@kwant-project.org/msg00063.html [2] https://gitlab.kwant-project.org/kwant/system2 Idea 1: unique sites Our experience has shown that it is useful to associate high-level sites with low-level systems. Users who build systems in terms of such high-level sites expect to find them again in the low-level systems. One attractive way of defining composite systems is to give additional importance to high-level sites: to require that they have a unique identity *across* the multiple systems that constitute a composite system. Observe that this is not the case in current Kwant: it is possible to attach a lead to the center of a ring even if both share the same lattice. The sites where the lead "hovers" over a sector of the ring exist twice as physical degrees of freedom. This "unique site" approach allows to elegantly specify complex composite systems. For example, to embed a 0d island in a 2d bulk it is enough to describe both constituents separately and to specify that sites From the lower-symmetry subsystem "overwrite" sites from the higher-symmetry subsystem. The "unique site" approach has the problem of not defining any canonical dense numbering of the present degrees of freedom. However any solver that performs useful computations for a composite system will have to establish such a numbering. Since the different parts of the composite system are connected using sites the usage of hash-tables to map sites to integer site indices (that necessarily appear at some point) seems inevitable. An additional problem is that the coupling between different parts is implicit and must be computed. For example a composite system that consists of a 0d central region with an attached 1d lead would require a precalculation that is equivalent to today's Builder.attach_lead() function. This preprocessing is also tricky: consider the simple configuration of a central region with two parallel leads (both belonging to the same lattice). Nothing remarkable happens if one lead is rotated slightly *away* from the other, but rotating it *towards* the other suddenly changes the very topology of the composite system! This simple example shows that correctly interpreting a composite system that uses the "unique site" approach is rather tricky. If this preprocessing can be done in a general way, it should obviously be cached but then shouldn't the cached results *become* the low-level composite system description instead? Idea 2: Systems as numbering schemes for degrees of freedom === The following is an attempt to reconcile various ideas that came up recently (e.g. putting all operators on equal footing, reunifying symmetries of lattices with system symmetries) with a low-level format that is easy and efficient to use by solvers. I propose to redefine "system" within Kwant to mean a numbering scheme for degrees of freedom together with associated symmetries. This corresponds to what we understand today as system minus the Hamiltonian operator. An example of a very basic system is a single site (numbered 0) with two orthogonal translational symmetries. Note that this can be interpreted as a lattice! Thanks to the numbering, operators that act on a system can be defined easily. An operator is a mapping from group elements of the system's symmetry to (sparse) matrices of some form. We can introduce systems that correspond to a parent system but with a reduced symmetry. For example, a 1d wire could be derived from the above 2d lattice by picking 3 sites of the lattice (e.g. at the coordinates (0, 0), (0, 1), (0, 2)), numbering them starting with 0, and specifying the remaining symmetry (which must be compatible with that of the parent system). Note that the choice of the sites fixes the unit cell of the
Re: [Kwant-devel] Kwant for Julia
Hi George, I’m adding a bit to Joe’s reply. If you are motivated to make a Kwant version for Julia, I guess the best is to start by simply wrapping Kwant’s Python interface. George Datseris wrote: However, doing an implementation in Julia would probably have big benefits in terms of speed for the part of the code that does not call low-level language. Kwant does currently have some performance bottlenecks. The worst is system construction which is slow (as designed), but since it’s O(N) in time (and solvers are typically some higher power of N) it doesn’t matter much for most applications. However, since there are some applications that do suffer from the slowness of builder, we are working on vectorizing it (along with evaluation of the system Hamiltonian). If we decide that some parts of Kwant need to be rewritten for speed, we will most likely stick with our current approach of either using Cython or C/C++. smime.p7s Description: S/MIME cryptographic signature
Re: [Kwant-devel] kwant | setup.py: remove from __future__ import (!73)
This is a comment to the discussion at [1] that I’d like to bring to broader attention. Anton Akhmerov wrote: BTW please don't use the filename in the commit message (filenames are accessible by other means). I agree that listing the filenames that get modified in the title of a commit is bad practice. But it can be very useful to mention the "subsystem", and that's what I mean when prepending "setup.py" to the commit message. In this particular case, I don't think that mentioning "setup.py" is necessary, because this commit was supposed to be purely cosmetic. But consider a commit message like setup.py: move initialization code into functions This one does not seem complete without mentioning the subsystem. [1] https://gitlab.kwant-project.org/kwant/kwant/merge_requests/73
Re: [Kwant-devel] Symmetries
Thanks, Anton and Dániel, for your insightful postings. I think that, ultimately, there is not so much difference between the reduced and expanded approaches. The work that needs to be done to expand the reduced system is equivalent to the work to check that the expanded system indeed obeys the declared symmetries. (Or do you see any problem with the statement, Dániel?) The one advantage of the reduced approach that I can see, and that could tip the scale, is that it works better with Kwant's value functions, i.e. functions that return bits of the Hamiltonian dynamically when given some optional parameters. I think it’s more robust to specify that these functions will be only ever evaluated on the FD than to say that the functions will be evaluated on the UC, but their results must always respect the declared symmetries. In the latter case if some unitary transformations are involved, we might be forced to even do almost_equal checks up to some epsilon. I’m not yet sure how to best handle symmetries that put constraints on the value of a single site or hopping. We still want value functions to return a matrix in that case I guess, so we would have to do the necessary consistency checks on that matrix. But probably that’s not different from requiring that onsite Hamiltonians are Hermitian, and that we already do. Do you agree with the following? Even if we go the reduced way, we still want to have integer numberings of all the sites and hoppings in the expanded cell, so that • solvers that don’t need point symmetries, don’t have to deal with them, • we can store quantities (e.g. observables) for the expanded unit cell. So in that sense we can forget about the point symmetries when designing the API for expanded low-systems. All we need to make sure is that the API for expanded low-level systems can be implemented efficiently for reduced low-level systems as well. smime.p7s Description: S/MIME cryptographic signature
[Kwant-devel] Low-level system proposal
Dear all, I’d like to start a thread on new low-level systems. As usual, everybody interested is invited to join. Let me list and summarize the state of discussion with regard to new low-level systems. (1) Previous discussions on the topic were summarized and concertized by Joseph in a wiki document [1]. In a nutshell, it proposes to store tight-binding systems as COO-format matrices. The H matrix is split up into a number of “terms” where each term is homogeneous with regard to the number of orbitals per site and the corresponding spatial symmetry group element. Note that the proposed format is “dumb”: there’s no way to query whether a particular site or hopping belongs to the system other than by searching. We thought that this is a good thing, since the things that we currently do with systems (=feeding them to kwant.solvers) do not need any further information. (2) Recently there has been a discussion [2] [3] about low-level systems and point group symmetries. The discussion is still ongoing, but we may record as a result that it may not be a good idea (as I thought previously) to store only the smallest possible fundamental domain of a system, as opposed to the (often larger) translational unit cell. In any case, there needs to be a way for system consumers (e.g. solvers) who do not care about point group symmetries to ignore these. So it looks to me that we can (contrary to what I believed before) simply ignore point group symmetries in a first design of low level systems. (3) Even more recently Joseph raised the issue of indexing computed observables (like charge or current) with high-level site and hopping objects as opposed to integer labels [4]. I think that this is an important point, however as I explain in [4] I do believe that the right way to do this is to teach low-level systems about sites and hoppings as opposed to making only finalized builders (which are only one specific realization of low-level systems) smarter. This matters even if the only low-level systems to be used were finalized builders: we have the ambition for low-level systems to be efficient (comparable to C-code and not Python-code). [1] https://gitlab.kwant-project.org/kwant/kwant/wikis/Low-Level-System-Format [2] https://www.mail-archive.com/kwant-devel@kwant-project.org/msg00054.html [3] https://www.mail-archive.com/kwant-devel@kwant-project.org/msg00057.html [4] https://gitlab.kwant-project.org/kwant/kwant/issues/65 Note that the point (3) is not compatible with the “dumb” format of (1), since with the “dumb” format there’s no way (other than searching) to even know whether some site or hopping is present in the system. We would like to have site/hopping-indexable arrays [4] soon, but we don’t want to introduce something now that will be inconsistent with something else that we’re working on. I think the best way forward is to finalize a new low-level format as soon as possible. We don’t even have to ship it yet (but hopefully we can), but it would be great to have a prototype implementation. Let me list the requirements as I see them currently: (a) Multiple translational symmetries. (It looks like more general symmetries can be added seamlessly later.) (b) Vectorized evaluation: it must be possible to evaluate Hamiltonians without an O(N) cost in Python function calls. (c) Time efficiency: A C-implemented Kwant solver should be able to work with a C-implemented Kwant system without any significant Python-related overhead. It should be possible to plug both together on Python level. (d) Smartness: high-level sites and site families are a concept that is known to low-level systems. It must be possible to check (at O(1) cost) whether a system contains some site/hopping and what the corresponding value is. This should also work in a vectorized way. (e) Storage efficiency: it shouldn’t be necessary to store the graph (think of large regular graphs). That’s why we need a low-level system *API* rather than a format. But since most often the graph will be stored, and the API should not stand in the way, I propose to think of a format, and then design an API that would be mostly transparent for that format but also could be implemented for large regular graphs without any storage. (Note that the CSR sparse matrix format can be reformulated as an API in a transparent way.) The proposal [1] addresses (a), (b) & (c) but doesn’t address (d) & (e). I think that by extending it a bit and transforming into an API, we can address all the requirements. Here it goes: First some definitions (mostly taken from [1]): • Sites are enumerated by integers. • The sequence of all sites consists of subsequent consecutive sub-sequences of *site ranges*. A site range shares some common features (like number of orbitals per site). • We deal only with n-dimensional translational symmetries. A symmetry group element is
Re: [Kwant-devel] Ideas about Kwant Symmetry API
Espen Flage-Larsen wrote: I would just like to point you in the direction of spglib (https://atztogo.github.io/spglib/). It is one of the fastest ways to for instance generate irreducible meshes including mapping tables. It works on integers wherever possible. Rotational matrices etc. also comes out. Also has a Python interface. Hi Espen! I had a look at spglib when you first told me about it, but back then I didn’t see how it could be useful for us. This was because I was thinking that we would be storing only the fundamental domain (of all the symmetries, not just translations), and then create the translational unit cell from that. (Just like translational symmetries work currently in Kwant.) If we go the other way (as Dániel seems think it’s preferable) and store the translational unit cell, then spglib could suddenly be useful for detecting the spatial symmetries that are present. Dániel (and anyone else interested), could you have a look at [1]? This seems to cover all we need for 3d (and probably also 2d) spatial symmetries. If we decide to go this way, we could actually keep the builder interface pretty much unchanged, since already today it’s possible to create a builder with more than one translational symmetry and that’s all that is needed. We could then use spglib to detect any spatial symmetries (there are some subtle points here though with regard to the values and how they transform and value functions). This would mean, however, that translational symmetries are treated quite differently from all the other symmetries. Perhaps that’s OK. One type of problem that I can see with this approach is that someone is trying to build a system with a given symmetry, but due to a mistake of the user the desired symmetry is not detected. [1] https://atztogo.github.io/spglib/python-spglib.html smime.p7s Description: S/MIME cryptographic signature
Re: [Kwant-devel] Ideas about Kwant Symmetry API
Dániel Varjas wrote: I'll try to address all of your questions. I discuss some more issues in my reply to your old email on the topic. Thanks! These cover almost all possible symmetries of a time-independent system of free particles I can think of, but I'm not aware of any theorem about this. One other thing we could think of are continuous on-site symmetries, like SU(2) spin rotations, these could be implemented using the group generators. Continuous symmetries like this are always on-site and can be decoupled from any spatial symmetry (i.e. every symmetry operation is a product of an on-site continuous symmetry and a spatial symmetry). OK, since continuous symmetries can be decoupled, let's forget about them for now. Or do you see any danger (in the sense of taking wrong design decisions that may block us later) of doing so? If you think of storing the matrices in the basis of the primitive lattice vectors (such that all rotations are invertible integer matrices) sounds like a good idea. On the other hand, it is not very intuitive, we'll have to provide an interface to transform at a higher level. We should figure out whether we can treat the internal parts of the symmetries in a symbolic way as well. It seems somewhat more complicated, think of a pi/6 spin rotation for example. These usually still have closed forms in terms of exponential/trigonometric functions, maybe we could use sympy to treat them exactly? Hmm, I’d prefer not to depend on sympy for low-level Kwant data structures. But isn’t it possible to represent any finite group by matrices? (These matrices could be generated using sympy, if that’s the easiest way.) Yes, we could generate the space groups ourselves. This code system looks a bit over the top for me, it's just an obfuscated way of storing 3*3 matrices of trits (1,0,-1) in a single integer. If we want to automate more fancy representation theory (like representations of nonsymmorphic space groups at high symmetry points of the BZ) it could be necessary to consult online databases as generating these tables seems rather cumbersome. First we probably just should implement generating representations and characters, which is quite straightforward, and leave it to the user to consult the appropriate database to find out what it corresponds to. I agree that this is the way to go. In the first iteration we should actually concentrate only on the low-level format and not on how it is constructed. But at later stages (still before releasing it) it’s important to verify that there are no gotchas in the low-level format that cause problems when using it in the ways we want. ### Storing the lattice (…) I do not comment on this for now, since it seems that we should rather finish the other thread first. smime.p7s Description: S/MIME cryptographic signature
Re: [Kwant-devel] Symmetries
Dániel Varjas wrote: (...) Considering all these complications, I'm not sure that storing hoppings in the way you propose is feasible. As I understand, you would specify the sites for the hopping by saying it connects site i in the fundamental domain to site j in the image of the fundamental domain under some SG symmetry g. I don't see why this representation would simplify calculations and it's quite hard to keep track of it for humans, compared to just specifying everything in the translational unit cell. Thanks for this very clear analysis. Perhaps you are right and storing the complete translational unit cell is the way to go. But then, isn't the difficulty of verifying that the provided translational unit cell indeed satisfies the declared symmetries comparable to the construction of the translational unit cell from the fundamental domain and said symmetries? In the end whether we chose to store only the fundamental domain or the full translational unit cell probably doesn't make a difference in terms of computing efficiency. (It could matter if, say, rotational symmetries can also apply to bigger structures, for example a large quantum dot.) So we can choose whater is easier to work with. I'm still not sure, however, *what* is easier. What do you think? What does matter for sure, however, is which user interface is presented to the user who builds systems with symmetries. The idea to only store the fundamental domain came naturally to me because this is what we’re doing in Kwant builders currently. (Only that there are no other symmetries but translational, hence fundamental domain = translational unit cell.) When creating a system with symmetry in current Kwant, the user first specifies the symmetry and then performs actions on the system (=adding and removing sites/hoppings). Kwant makes sure that these actions are done in a way that preserves the declared symmetries, e.g. adding a single site adds all of its images as well. In practice this is implemented by storing only the fundamental domain. I assumed that this way of working also makes sense with symmetries other than translational. (Of course there would be a way to plot the automatically generated unit cell.) But I’m actually not sure. What do you think? Perhaps a good example would be making a diamond lattice. Is it easier to declare the full symmetries and specify a single atom, or to declare the symmetries and the the full unit cell? The former seems cleaner, but also more abstract. The latter is perhaps easier to imagine, but my first reaction is that it’s the inferior approach. However, if you can argue that systems with symmetries should be _stored_ as complete translational unit cells, this will give us a good reason not to bother with the minimalist approach during construction as well. If we consider not only spatial but also internal symmetries this becomes even more complicated, but perhaps we can first try to come to a conclusion disregarding that. smime.p7s Description: S/MIME cryptographic signature
Re: [Kwant-devel] Ideas about Kwant Symmetry API
Hi Dániel, It’s good to have someone around with experience in crystallography! What I find the most interesting about what you wrote is your enumeration of possible internal symmetries. This is exactly where we were not so sure about things. Generalized low-level systems have been on our agenda for an eternity. There is a post from the begin of this year that summarizes our plans so far: https://www.mail-archive.com/kwant-devel@kwant-project.org/msg00042.html. I would be grateful if you could read it and comment on it. (Note that you can reply to that message by simply going to the above page and pressing the “reply via email” button at its very bottom. Be sure to CC your reply also to kwant-devel@kwant-project.) In the following I comment on specific points of your message: Dániel Varjas wrote: ### Storing symmetry elements A generic symmetry element g should have the following properties: 1. Spatial part Correct me if I’m wrong but it seems to me that a good representation of the spatial part would be rational matrices like the ones returned by http://www.cryst.ehu.es/cgi-bin/plane/programs/nph-plane_getgen?gnum=4=plane=table for the plane group “pg”. Is this what you propose? 2. Internal/gauge part Is there a theorem that states that the combination of a unitary matrix with optional complex conjugation and optional antisymmetry is all that one needs to cover “all” (however defined) symmetries? ### Generating the group All these discrete groups are finitely generarated. We can split off the pure lattice translation part, and generate the finite point group part as a quotient. This works in a brute force way (generate products up to a certain length), or we can think of something smarter. Storing the operators in the above linear representation form has the advantage of taking products is quite straightforward. We may need to worry about numerical accuracy when deciding equivalence of operators in this representation, but it shouldn't be a big problem. I wonder if it’s not better to store the group elements not as real-space transformation matrices (that are subject to numerical error), but rather as “skewed” matrices filled with rational numbers that can be processed exactly. It seems to me that the real-space transformation matrices can be recovered from the exact ones by means of a coordinate transformation. If we want to implement the full list of space groups we could fetch crystallographic tables from http://www.cryst.ehu.es/ for example, there is plain-text interface to basic data, like the group elements (only the real spce part) for all space groups. Isn’t there any public library for doing this? It’s all maths after all, so it should not be necessary to consult online databases. I found a paper that outlines methods to generate such data. (http://www.intechopen.com/books/recent-advances-in-crystallography/unique-and-effective-characterization-of-space-groups-in-computer-applications) The attached Python script is based on the code on page 21 (or 264) of the above paper. It generates what seem to be all the matrices that are needed to generate all the point groups in space. The script demonstrates that cubing a matrix that corresponds to 1/3-rotation results in a unity matrix. ### Storing the lattice 1. Lattice sites should be grouped in symmetry related families. Can you elaborate on what you mean? Do you mean “families” in the sense of Kwant’s site families? Or rather as an abstract concept? When you speak of sites here, do you mean the sites that belong to the fundamental domain (or translational unit cell) of the symmetry, or rather the sites that make up a complete Kwant system? 2. Could be nice to also allow grouping of orbitals of the sites according to symmetry representations. Grouping in which sense? 3. There is an ambiguity in the basis because of different choices of the angular momentum quantization basis on various sites, so it would be useful to allow attaching a basis (SU(2) or O(3) matrix) to each site, to keep track of the various quantization axes. This can be useful when dealing with situations like the anisotropic atoms in 2/1 if the axis is different on every site, this happens on lattices of interest like pyrochlores. Can you provide a simple but complete that illustrates the issue? # http://www.intechopen.com/books/recent-advances-in-crystallography/unique-and-effective-characterization-of-space-groups-in-computer-applications import numpy as np def mat(code): ret = np.empty((3, 3), int) for k in range(2, -1, -1): for j in range(2, -1, -1): ret[k, j] = (code % 3) - 1 code = code // 3 return ret codes = [16484, 3200, 3360, 7898, 12270, 7410, 14459, 7817, 10816, 3198, 16482, 16322, 11784, 7412, 12272, 5223, 11865, 8866] for id, code in enumerate(codes): print id, code print mat(code) print
[Kwant-devel] Pip expert needed to test build configuration and pip
In the development version of Kwant (what will be released as 1.3) the build configuration via the file “build.conf” became more powerful [1]. It is now also possible to provide an alternative path for “build.conf” via the option “--configfile=PATH” to “setup.py”. This closed an issue in the bugtracker [2]. Before this change, it was simply not possible to configure the build when installing Kwant through pip, since there was no way to provide a “build.conf” in pip’s build directory [3]. This was bad, since Kwant’s “setup.py” detects MUMPS automatically only in some cases. Now it should be possible to do something like this: pip install --global-option="--configfile=/tmp/build.conf" kwant I’m looking for someone who knows pip better than me and would volunteer to check the following (with Kwant from git of course): • Does the above pip command actually work? • Can/should a relative path for “build.conf” be used instead instead of the absolute one? • Are there things that one has to be aware of, e.g. pip’s caching? If Kwant is built a second time (in the same or in a different venv) with a different build.conf, will this work as intended? To make me perfectly happy, the honorable volunteer could propose how to modify [3] once Kwant 1.3 has been released. Thanks! [1] https://kwant-project.org/doc/dev/pre/whatsnew/1.3#improved-build-configuration [2] https://gitlab.kwant-project.org/kwant/kwant/issues/32 [3] https://kwant-project.org/install#automatic-installation-using-pip smime.p7s Description: S/MIME cryptographic signature
Re: [Kwant-devel] Parallel solvers
This is an update on my reflections about parallelizing Kwant. Please feel invited to comment and join the discussion. So far, Kwant does not contain any support for parallelism. However, it works OK with libraries for task parallelism, e.g. concurrent.futures. (See the Kwant paper for an example). Task parallelism means that several independent tasks (e.g. related to different leads or different energies), are processed in parallel. Support for task parallelism could be improved (Kwant systems could be made pickleable, for example), but this does not diminish the validity of the following. Where Kwant is lacking utterly is data parallelism. Data parallel programs process (different parts) of the same data synchronously. Different cores work on solving different parts of the same linear system, for example. The inability to harness data parallelism is a severe handicap for Kwant. As you all know, one does not always have more independent tasks than available cores. This is especially true for comparatively short-running exploratory workloads. A solution exists: direct linear solvers seem to scale quite well up to a hundred or so cores [1]. Our direct linear solver of choice, MUMPS, supports data parallelism through MPI. The same seems to be true for all other serious options that I could find, notably Pardiso. (MUMPS, however, seems to be the only library that is both efficient and free.) As such, it seems to me that we should embrace MPI in Kwant. I believe that MPI support can be added to Kwant in a way that at the same time • keeps MPI completely optional for Kwant users, • and respects best practices for MPI-enabled libraries [2]. The idea is to add an optional "comm" (for communicator) parameter to operations that can be performed in a data-parallel way, like kwant.smatrix. The default value of this parameter would be "None" to indicate non-parallel execution (just like now). For example, turning a sequential Kwant script into a data-parallel one would have to • add "from mpi4py import MPI" in the preamble, • substitute kwant.smatrix(…) -> kwant.smatrix(…, comm=MPI.COMM_WORLD), • execute the script using "mpiexec", or a queuing system, or using jupyter in an appropriate way (it’s possible) I do not see a way to completely hide MPI from the user, nor do I think that this is desirable (it would make it impossible to use Kwant together with other libraries that use MPI). However, the above seems to me quite acceptable, especially because only users who want to go data-parallel would be affected. Some people claim that MPI has already seen its best days [3], however I do not think that this analysis is relevant to Kwant: I do not see how to base Kwant on something like Spark, because there are no direct linear solvers for that framework. MPI will certainly continue to be the standard HPC framework for many years (the body of existing code alone is enough to justify this). If/when a viable alternative emerges in 5-10 years, we should be able to adapt quite easily, since Kwant will most likely continue to depend only weakly on it (by using libraries like MUMPS). [1] http://mumps.enseeiht.fr/doc/poster_mumps.pdf [2] https://htor.inf.ethz.ch/publications/index.php?pub=135 [3] http://www.dursi.ca/hpc-is-dying-and-mpi-is-killing-it/ smime.p7s Description: S/MIME cryptographic signature
