Takahiro,
Nice catch, I really wonder how this one survived for soo long. I pushed a
patch in r28535 addressing this issue. It is not the best solution, but it
provide an easy way to address the issue.
A little bit of history. A datatype is composed by (let's keep it short) 2
component, a high-level description containing among others the size and the
name of the datatype and a low level description (the desc_t part) containing
the basic predefined elements in the datatype. As most of the predefined
datatypes defined in the MPI layer are synonyms to some basic predefined
datatypes (such as the equivalent POSIX types MPI_INT32_T), the design of the
datatype allowed for the sharing of the desc_t part between datatypes. This
approach allows us to have similar datatypes (MPI_INT and MPI_INT32_T) with
different names but with the same backend internal description. However, when
we split the datatype engine in two, we duplicate this common description (in
OPAL and OMPI). The OMPI desc_t was pointing to OPAL desc_t for almost
everything … except the datatypes that were not defined by OPAL such as the
Fortran one. This turned the management of the common desc_t into a nightmare …
with the effect you noticed few days ago. Too bad for the optimization part. I
now duplicate the desc_t between the two layers, and all OMPI datatypes have
now their own desc_t.
Thanks for finding and analyzing so deeply this issue.
George.
On May 16, 2013, at 12:04 , KAWASHIMA Takahiro <rivis.kawash...@nifty.com>
wrote:
> Hi,
>
> I'm reading the datatype code in Open MPI trunk and have a question.
> A bit long.
>
> See the following program.
>
> ----------------------------------------------------------------
> #include <stdio.h>
> #include <mpi.h>
>
> struct opal_datatype_t;
> extern int opal_init(int *pargc, char ***pargv);
> extern int opal_finalize(void);
> extern void opal_datatype_dump(struct opal_datatype_t *type);
> extern struct opal_datatype_t opal_datatype_int8;
>
> int main(int argc, char **argv)
> {
> opal_init(NULL, NULL);
> opal_datatype_dump(&opal_datatype_int8);
> MPI_Init(NULL, NULL);
> opal_datatype_dump(&opal_datatype_int8);
> MPI_Finalize();
> opal_finalize();
> return 0;
> }
> ----------------------------------------------------------------
>
> All variables/functions declared as 'extern' are defined in OPAL.
> opal_datatype_dump() function outputs internal data of a datatype.
> I expect the same output on two opal_datatype_dump() calls.
> But when I run it on an x86_64 machine, I get the following output.
>
> ----------------------------------------------------------------
> ompi-trunk/opal-datatype-dump && ompiexec -n 1 ompi-trunk/opal-datatype-dump
> [ppc.rivis.jp:27886] Datatype 0x600c60[OPAL_INT8] size 8 align 8 id 7 length
> 1 used 1
> true_lb 0 true_ub 8 (true_extent 8) lb 0 ub 8 (extent 8)
> nbElems 1 loops 0 flags 136 (commited contiguous )-cC---P-DB-[---][---]
> contain OPAL_INT8
> --C---P-D--[---][---] OPAL_INT8 count 1 disp 0x0 (0) extent 8 (size 8)
> No optimized description
>
> [ppc.rivis.jp:27886] Datatype 0x600c60[OPAL_INT8] size 8 align 8 id 7 length
> 1 used 1
> true_lb 0 true_ub 8 (true_extent 8) lb 0 ub 8 (extent 8)
> nbElems 1 loops 0 flags 136 (commited contiguous )-cC---P-DB-[---][---]
> contain OPAL_INT8
> --C---P-D--[---][---] count 1 disp 0x0 (0) extent 8 (size
> 8971008)
> No optimized description
> ----------------------------------------------------------------
>
> The former output is what I expected. But the latter one is not
> identical to the former one and its content datatype has no name
> and a very large size.
>
> This line is output in opal_datatype_dump_data_desc() function in
> opal/datatype/opal_datatype_dump.c file. It refers
> opal_datatype_basicDatatypes[pDesc->elem.common.type]->name and
> opal_datatype_basicDatatypes[pDesc->elem.common.type]->size for
> the content datatype.
>
> In this case, pDesc->elem.common.type is
> opal_datatype_int8.desc.desc[0].elem.common.type and is initialized to 7
> in opal_datatype_init() function in opal/datatype/opal_datatype_module.c
> file, which is called during opal_init() function.
> opal_datatype_int8.desc.desc points &opal_datatype_predefined_elem_desc[7*2].
>
> But if we call MPI_Init() function, the value is overwritten.
> ompi_datatype_init() function in ompi/datatype/ompi_datatype_module.c
> file, which is called during MPI_Init() function, has similar
> procedure to initialize OMPI datatypes.
>
> On initializing ompi_mpi_aint in it, ompi_mpi_aint.dt.super.desc.desc
> points &opal_datatype_predefined_elem_desc[7*2], which is also pointed
> by opal_datatype_int8, because ompi_mpi_aint is defined by
> OMPI_DATATYPE_INIT_PREDEFINED_BASIC_TYPE macro and it uses
> OPAL_DATATYPE_INITIALIZER_INT8 macro. So
> opal_datatype_int8.desc.desc[0].elem.common.type is overwritten
> to 37.
>
> Therefore in the second opal_datatype_dump() function call in my
> program, opal_datatype_basicDatatypes[37] is accessed.
> But the array length of opal_datatype_basicDatatypes is 25.
>
> Summarize:
>
> static initializer:
> opal_datatype_predefined_elem_desc[25] = {{0, ...}, ...};
> opal_datatype_int8.desc.desc = &opal_datatype_predefined_elem_desc[7*2];
> ompi_mpi_aint.dt.super.desc.desc =
> &opal_datatype_predefined_elem_desc[7*2];
>
> opal_init:
> opal_datatype_int8.desc.desc.elem.common.type = 7;
>
> MPI_Init:
> ompi_mpi_aint.dt.super.desc.desc.elem.common.type = 37;
>
> opal_datatype_dump:
> access to opal_datatype_predefined_elem_desc[37]
>
> While opal_datatype_dump() function might not be called from
> user's programs, breaking opal_datatype_predefined_elem_desc
> array in ompi_datatype_init() function is not good.
>
> Though the above is described for opal_datatype_int8 and ompi_mpi_aint,
> the same thing happens to other datatypes.
>
> Though I tried to fix this problem, I could not figure out the
> correct solution.
>
> - The first loop in ompi_datatype_init() function should be removed?
> But OMPI Fortran datatypes should be initialized in it?
>
> - All OMPI datatypes should point ompi_datatype_predefined_elem_desc
> array? But having same 'type' value in OPAL datatypes and OMPI
> datatypes is allowed?
>
> Regards,
> KAWASHIMA Takahiro
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
