Gus,
Please see my answers inline.
> On Jan 16, 2015, at 14:24 , Gus Correa <g...@ldeo.columbia.edu> wrote:
>
> Hi George
>
> It is still not clear to me how to deal with strides in
> MPI_Create_type_subarray.
> The function/subroutine interface doesn’t mention strides at all.
That’s indeed a little tricky. However, the trick here is that when you try to
understand the subarray type you should think recursively about the datatypes
involved in the operation.
> It is a pity that there is little literature (books) about MPI,
> and the existing books are lagging behind the new MPI developments and
> standards (MPI-2, MPI-3).
> My most reliable sources so far were the "MPI - The complete reference"
> books, vol-1 (2nd ed.) and vol-2 (which presumably covers MPI-2).
> However, they do not even mention MPI_Create_type_subarray,
> which is part of the MPI-2 standard.
Let me do a wild guess: you two guys must be the firsts to ask questions about
it …
> I found it in the MPI-2 standard on the web, but it is not clear to me
> how to achieve the same effect of strides that are available in
> MPI_Type_vector.
> MPI_Create_type_subarray is in section 4.1.3.
> The OMPI MPI_Create_type_subarray man page says there is an example in
> section 9.9.2 of the MPI-2 standard.
> However, there is no section 9.9.2.
> Chapter 9 is about something else ("The info object"), not derived types.
> No good example of MPI_Create_type_subarray in section 4.1.3 either,
> which is written in the typical terse and hermetic style in which
> standards are.
No comments on subjective topics … ;) You just blew my day away, I was totally
under the impression that the MPI standard reads like a children’s bedtime
story book !!!
> So, how can one handle strides in MPI_Create_type_subarray?
> Would one have to first create several MPI_Type_vector for the various
> dimensions, then use them as "oldtype" in MPI_Create_type_subarray?
> That sounds awkward, because there is only one “oldtype" in
> MPI_Create_type_subarray, not one for each dimension.
Exactly. Take a look at how we handle the subarray in Open MPI, more precisely
at the file ompi/datatype/ompi_datatype_create_subarray.c. My comment from few
days ago referred exactly to this code, where the subarray is basically
described in terms of vector (well maybe vector as I was lazy to check the
LB/UB).
As I said above think recursively. You start with the old type, then build
another try on a dimension, then you use this to expose the second dimensions
and so on. For each dimension your basic type is not the user provided old
type, but the type you built so far.
- size_array[i] is the number of elements in big data in the dimension i
- subsize_array[i] is the of element you will include in your datatype in the
dimension i
- start_array[i] is how many elements you will skip in the dimension i before
you start including them in your datatype. start[i] + subside[i] must be
smaller or equal to size[i]
> Is there any simple example of how to achieve stride effect with
> MPI_Create_type_subarray in a multi-dimensional array?
Not as far as I know. But now that people expressed interest in this topic,
maybe someone will write a blog or something about.
> BTW, when are you gentlemen going to write an updated version of the
> “MPI - The Complete Reference"? :)
Maybe after the release of MPI 4.0 would be a good target … A lot of new and
exciting technologies will hopefully be going in by then, writing a new book
might be worth the effort.
George.
>
> Thank you,
> Gus Correa
> (Hijacking Diego Avesani's thread, apologies to Diego.)
> (Also, I know this question is not about Open MPI, but about MPI in general.
> But the lack of examples may warrant asking the question here.)
>
>
> On 01/16/2015 01:39 AM, George Bosilca wrote:
>> The subarray creation is an multi-dimension extension of the vector type.
> You can see it as a vector of vector of vector and so on, one vector per
> dimension.
> The stride array is used to declare on each dimension what is the relative
> displacement
> (in number of elements) from the beginning of the dimension array.
>>
>> It is important to use regular type creation when you can take advantage of
>> such
> regularity instead of resorting to use of struct or h*. This insure better
> packing/unpacking performance, as well as possible future support for
> one-sided
> communications.
>>
>> George.
>>
>>
>>
>>> On Jan 15, 2015, at 19:31, Gus Correa <g...@ldeo.columbia.edu> wrote:
>>>
>>> I never used MPI_Type_create_subarray, only MPI_Type_Vector.
>>> What I like about MPI_Type_Vector is that you can define a stride,
>>> hence you can address any regular pattern in memory.
>>> However, it envisages the array layout in memory as a big 1-D array,
>>> with a linear index progressing in either Fortran or C order.
>>>
>>> Somebody correct me please if I am wrong, but at first sight
>>> MPI_Type_Vector sounds more flexible to me than MPI_Type_create_subarray,
>>> exactly because the latter doesn't have strides.
>>>
>>> The downside is that you need to do some index arithmetic to figure
>>> the right strides, etc, to match the corresponding
>>> Fortran90 array sections.
>>>
>>> There are good examples in the "MPI - The complete reference" books I
>>> suggested to you before (actually in vol 1).
>>>
>>> Online I could find the two man pages (good information, but no example):
>>>
>>> http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_vector.3.php
>>> http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_create_subarray.3.php
>>>
>>> There is a very simple 2D example of MPI_Type_vector using strides here:
>>>
>>> https://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types
>>>
>>> and a similar one here:
>>>
>>> http://static.msi.umn.edu/tutorial/scicomp/general/MPI/content6.html
>>>
>>> Gus Correa
>>>
>>>> On 01/15/2015 06:53 PM, Diego Avesani wrote:
>>>> dear George, dear Gus, dear all,
>>>> Could you please tell me where I can find a good example?
>>>> I am sorry but I can not understand the 3D array.
>>>>
>>>>
>>>> Really Thanks
>>>>
>>>> Diego
>>>>
>>>>
>>>> On 15 January 2015 at 20:13, George Bosilca <bosi...@icl.utk.edu
>>>> <mailto:bosi...@icl.utk.edu>> wrote:
>>>>
>>>>
>>>>> On Jan 15, 2015, at 06:02 , Diego Avesani <diego.aves...@gmail.com
>>>>> <mailto:diego.aves...@gmail.com>> wrote:
>>>>>
>>>>> Dear Gus, Dear all,
>>>>> Thanks a lot.
>>>>> MPI_Type_Struct works well for the first part of my problem, so I
>>>>> am very happy to be able to use it.
>>>>>
>>>>> Regarding MPI_TYPE_VECTOR.
>>>>>
>>>>> I have studied it and for simple case it is clear to me what id
>>>>> does (at least I believe). Foe example if I have a matrix define as:
>>>>> REAL, ALLOCATABLE (AA(:,:))
>>>>> ALLOCATE AA(100,5)
>>>>>
>>>>> I could send part of it defining
>>>>>
>>>>> CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,/MY_NEW_TYPE/)
>>>>>
>>>>> after that I can send part of it with
>>>>>
>>>>> CALL MPI_SEND( AA(1:/10/,:), /10/, /MY_NEW_TYPE/, 1, 0,
>>>>> MPI_COMM_WORLD );
>>>>>
>>>>> Have I understood correctly?
>>>>>
>>>>> What I can do in case of three dimensional array? for example
>>>>> AA(:,:,:), I am looking to MPI_TYPE_CREATE_SUBARRAY.
>>>>> Is that the correct way?
>>>>>
>>>>> Thanks again
>>>>
>>>> Indeed, using the subarray is the right approach independent on the
>>>> number of dimensions of the data (you can use it instead of
>>>> MPI_TYPE_VECTOR as well).
>>>>
>>>> George.
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Diego
>>>>>
>>>>>
>>>>> On 13 January 2015 at 19:04, Gus Correa <g...@ldeo.columbia.edu
>>>>> <mailto:g...@ldeo.columbia.edu>> wrote:
>>>>>
>>>>> Hi Diego
>>>>> I guess MPI_Type_Vector is the natural way to send and receive
>>>>> Fortran90 array sections (e.g. your QQMLS(:,50:100,:)).
>>>>> I used that before and it works just fine.
>>>>> I think that is pretty standard MPI programming style.
>>>>> I guess MPI_Type_Struct tries to emulate Fortran90 and C
>>>>> structures
>>>>> (as you did in your previous code, with all the surprises
>>>>> regarding alignment, etc), not array sections.
>>>>> Also, MPI type vector should be more easy going (and probably
>>>>> more efficient) than MPI type struct, with less memory
>>>>> alignment problems.
>>>>> I hope this helps,
>>>>> Gus Correa
>>>>>
>>>>> PS - These books have a quite complete description and several
>>>>> examples
>>>>> of all MPI objects and functions, including MPI types (native
>>>>> and user defined):
>>>>> http://mitpress.mit.edu/books/__mpi-complete-reference-0
>>>>> <http://mitpress.mit.edu/books/mpi-complete-reference-0>
>>>>> http://mitpress.mit.edu/books/__mpi-complete-reference-1
>>>>> <http://mitpress.mit.edu/books/mpi-complete-reference-1>
>>>>>
>>>>> [They cover MPI 1 and 2. I guess there is a new/upcoming book
>>>>> with MPI 3, but for what you're doing 1 and 2 are more than
>>>>> enough.]
>>>>>
>>>>>
>>>>> On 01/13/2015 09:22 AM, Diego Avesani wrote:
>>>>>
>>>>> Dear all,
>>>>>
>>>>> I had some wonderful talking about MPI_type_create_struct adn
>>>>> isend\irecv with
>>>>> Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is
>>>>> more clear and my
>>>>> program works.
>>>>>
>>>>> Now I have another question. In may program I have matrix:
>>>>>
>>>>> /QQMLS(:,:,:) /that is allocate as
>>>>>
>>>>> /ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of
>>>>> particles
>>>>>
>>>>> QQMLS is double precision.
>>>>>
>>>>> I would like to sent form a CPU to another part of it, for
>>>>> example,
>>>>> sending QQMLS(:,50:100,:). I mean sending the QQMLS of the
>>>>> particles
>>>>> between 50 to 100.
>>>>> I suppose that i could use MPI_Type_vector but I am not
>>>>> sure. The
>>>>> particle that I want to sent could be from 25 to 50 ecc..
>>>>> ecc..so
>>>>> blocklength changes everytime.
>>>>>
>>>>> Do I have to use MPI_type_create_struct?
>>>>> Do I have correctly understood MPI_Type_vector?
>>>>>
>>>>> Thanks a lot
>>>>>
>>>>>
>>>>> Diego
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>> Subscription:
>>>>> http://www.open-mpi.org/__mailman/listinfo.cgi/users
>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>>>>> Link to this post:
>>>>>
>>>>> http://www.open-mpi.org/__community/lists/users/2015/01/__26171.php
>>>>>
>>>>> <http://www.open-mpi.org/community/lists/users/2015/01/26171.php>
>>>>>
>>>>>
>>>>> _________________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>> Subscription:
>>>>> http://www.open-mpi.org/__mailman/listinfo.cgi/users
>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/__community/lists/users/2015/01/__26172.php
>>>>> <http://www.open-mpi.org/community/lists/users/2015/01/26172.php>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26184.php
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2015/01/26192.php
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2015/01/26193.php
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2015/01/26194.php
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2015/01/26195.php
>>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/01/26206.php
