Actually, you are not the first to encounter the problem with
'MPI_Type_indexed' for very large datatypes. I also run with a 1.6
release, and solved the problem by switching to
'MPI_Type_Create_Hindexed' for the datatype. The critical difference is
that the displacements for 'MPI_type_indexed' are type integer, i.e. 32
bit values, while for 'MPI_Type_Create_Hindexed' the displacements are
in bytes, but with type 'MPI_Address_Kind', i.e. normally 64 bit, and
therefore of effectively infinite size. Otherwise the 2 'types' can be
used identically.
T. Rosmond
On Thu, 2015-03-05 at 12:31 -0500, George Bosilca wrote:
> Bogdan,
>
>
> As far as I can tell your code is correct, and the problem is coming
> from Open MPI. More specifically, I used alloca in the optimization
> stage in MPI_Type_commit, and as your arrays of length were too large,
> alloca failed and lead to a segfault. I fixed in the trunk (3c489ea),
> and this will get into our next release.
>
>
> Unfortunately there is no fix for the 1.6 that I can think of.
> Apparently, you are really the first that run into such kind of
> problems, so guess you are the first creating gigantic datatypes.
>
>
> Thanks for the bug report,
> George.
>
>
>
> On Thu, Mar 5, 2015 at 9:09 AM, Bogdan Sataric
> <bogdan.sata...@gmail.com> wrote:
> I've been having problems with my 3D matrix transpose program.
> I'm using MPI_Type_indexed in order to allign specific blocks
> that I want to send and receive across one or multiple nodes
> of a cluster. Up to few days ago I was able to run my program
> without any errors. However several test cases on the cluster
> in last few days exposed segmentation fault when I try to form
> indexed type on some specific matrix configurations.
>
>
>
> The code that forms indexed type is as follows:
>
>
> #include <stdio.h>
> #include <stdlib.h>
> #include <mpi.h>
>
>
> int main(int argc, char **argv) {
>
>
> int Nx = 800;
> int Ny = 640;
> int Nz = 480;
> int gsize;
> int i, j;
>
>
> MPI_Init(&argc, &argv);
> MPI_Comm_size(MPI_COMM_WORLD, &gsize);
>
>
> printf("GSIZE: %d\n", gsize);
>
>
> MPI_Datatype double_complex_type;
> MPI_Datatype block_send_complex_type;
>
>
> int * send_displ = (int *) malloc(Nx * Ny/gsize *
> sizeof(int));
> int * send_blocklen = (int *) malloc(Nx * Ny/gsize *
> sizeof(int));
>
>
> MPI_Type_contiguous(2, MPI_DOUBLE, &double_complex_type);
> MPI_Type_commit(&double_complex_type);
>
>
> for (i = Ny/gsize - 1; i >= 0 ; i--) {
> for (j = 0; j < Nx; j++) {
> send_displ[(Ny/gsize - 1 - i) * Nx + j] = i * Nz +
> j * Ny * Nz;
> send_blocklen[(Ny/gsize - 1 - i) * Nx + j] = Nz;
>
> }
> }
>
>
> MPI_Type_indexed(Nx * Ny/gsize, send_blocklen, send_displ,
> double_complex_type, &block_send_complex_type);
> MPI_Type_commit(&block_send_complex_type);
>
>
> free(send_displ);
> free(send_blocklen);
>
>
> MPI_Finalize();
> }
>
>
> Values of the Nx, Ny and Nz respectively are 800, 640 and 480.
> Value of gsize for this test was 1 (simulation of MPI program
> on 1 node). The node has 32GB of RAM and no other memory has
> been allocated (only this code has been run).
>
>
> In code basically I'm creating double_complex_type to
> represent complex number (2 contiguous MPI_DOUBLE) values. The
> whole matrix has 800 * 640 * 480 of these values and I'm
> trying to catch these values in the indexed type. One indexed
> type block length is the whole Nz "rod" while ordering of
> these "rods" in displacements array is given by the formula i
> * Nz + j * Ny * Nz. Basically displacements start from top
> row, and left column of the 3D matrix. Then I gradually sweep
> to the right sight of that top row, then go to one row below
> sweep to the right side and so on until the bottom row.
>
>
> The strange thing is that this formula and algorithm WORK if I
> put MPI_DOUBLE type instead of derived complex type (1 instead
> of 2 in MPI_TYPE_CONTIGIOUS). Also this formula WORKS if I put
> 1 for Nz dimension instead of 480. However if I change Nz to
> even 2 I get segmentation fault error in the MPI_Type_commit
> call.
>
>
> I checked all of the displacements and they seem fine. There
> is no overlapping of displacements or going under 0 or over
> extent of the formed indexed type. Also the size of the
> datatype is below 4GB (which is I believe limit of MPI
> datatypes (since MPI_Type_size function returns int * ). Also
> I believe amount of memory is not an issue as even if I put Nz
> to be 2, I get the same segmentation fault error, and the node
> has 32GB of RAM just for this test.
>
>
> What bothers me is that most of other indexed type
> configurations (with plain MPI_DOUBLE type elements), or
> complex type with smaller matrix (say 400 * 640 * 480) WORK
> without segmentation fault. Also If I commit the indexed type
> with MPI_DOUBLE type elements even larger matrices work (say
> 960 x 800 x 640) which have exactly the same type size as 800
> x 640 x 480 complex indexed type (just under 4GB)! So
> basically the type size is not an issue here, but somehow
> either number of blocks, size of particular blocks, or size of
> block elements create problems. I'm not sure weather there is
> problem in implementation of OpenMPI or something in my code
> is wrong...
>
>
> I would greatly appreciate any help as I've been stuck on this
> problem for days now and nothing in MPI documentation and the
> examples I found on the internet is giving me a clue where the
> error might be.
>
>
> At the end I would like to say that code has been compiled
> with Open-MPI version 1.6.5.
>
>
> Thank you,
>
>
> Bogdan Sataric
> ----
>
>
> Bogdan Sataric
>
>
>
> email: bogdan.sata...@gmail.com
> phone: +381 21-485-2441
>
>
> Teaching & Research Assistant
> Chair for Applied Computer Science
> Faculty of Technical Sciences, Novi Sad, Serbia
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription:
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/03/26430.php
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/03/26431.php
