Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Jed Brown 
Satish Balay  writes:

>> typedef struct { char dummy; } F90Array1d;
>> typedef struct { char dummy; } F90Array2d;
>> typedef struct { char dummy; } F90Array3d;
>> 
>> These are all distinct types.  
>
> I'm not sure if they need to be distinct. Perhaps they are collapsible
> into a single type. [I suspect its a design we used to keep it in sync
> with our old f90 interface code. We had both versions for a while]

If our current code does not mix them, then the types should be distinct
so that they aren't accidentally mixed.

> For now - I've updated the commit to use the above typedef.

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Satish Balay 
On Tue, 15 May 2018, Jed Brown wrote:

> Satish Balay  writes:
> 
> > On Tue, 15 May 2018, Jed Brown wrote:
> >
> >> > Your notation requires the following change
> >> >
> >> > PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec 
> >> > *v,F90Array1d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> >> > PETSC_EXTERN PetscErrorCode 
> >> > F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d* 
> >> > PETSC_F90_2PTR_PROTO_NOVAR);
> >> >
> >> > to:
> >> >
> >> > PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec 
> >> > *v,F90Array1d a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> >> > PETSC_EXTERN PetscErrorCode 
> >> > F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d 
> >> > PETSC_F90_2PTR_PROTO_NOVAR);
> >> 
> >> Yes.
> >> 
> >> > [this doesn't look right.] 
> >> 
> >> I don't know why not, 
> >
> > output argument without a explicit pointer in definition is weird for c - 
> > and can result in future bugs [when this code is changed]
> 
> The pointer is inside a typedef for all the basic PETSc types.

Sure - but we have: 'MatCreate(MPI_Comm,Mat*)' and not 
'MatCreate(MPI_Comm,Mat)' - which the above prototype would suggest.

> 
> >> but I don't care about this narrow bit of
> >> compiler-enforced type safety enough to push for the more intrusive fix.
> >
> > I'm not sure I understand what extra type safety is obtained by 
> > 'F90Array1d*' vs 'F90Array1d' usage
> >
> > Or does 'typedef struct { void *ptr; } F90Array3d;' give the extra safety? 
> > I could change this part of typedef.
> 
> Yeah, that's an alternative way to get a strong typedef, and you could
> keep the other code unmodified.
> 
> typedef struct { char dummy; } F90Array1d;
> typedef struct { char dummy; } F90Array2d;
> typedef struct { char dummy; } F90Array3d;
> 
> These are all distinct types.  

I'm not sure if they need to be distinct. Perhaps they are collapsible
into a single type. [I suspect its a design we used to keep it in sync
with our old f90 interface code. We had both versions for a while]

For now - I've updated the commit to use the above typedef.

Satish

> Better to use char for the dummy because
> we don't have reason to know that the pointer is word aligned, but the
> standard says that any data pointer can be cast to a char*.
> 
> >> 
> >> > So I guess 'pointer to address (void*)' is the correct notation - as
> >> > we pass these back to fortran calling routine wrt
> >> > dmdavecgetarrayf901_() and an output argument for F90Array1dCreate().
> >> 
> >> It's actually just the pointer that we have.  The thing it points to is
> >> of some type that is not known statically in F90Array1dCreate/Destroy.
> >
> > Yes. we get something opaque - and we pass it down to fortran utility 
> > routine or up to the calling fortran routine - unmodified.
> >
> > My current patch is at balay/fix-F90Array__Destroy/maint
> >
> > Satish
>

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Jed Brown 
Satish Balay  writes:

> On Tue, 15 May 2018, Jed Brown wrote:
>
>> > Your notation requires the following change
>> >
>> > PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec 
>> > *v,F90Array1d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
>> > PETSC_EXTERN PetscErrorCode 
>> > F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d* 
>> > PETSC_F90_2PTR_PROTO_NOVAR);
>> >
>> > to:
>> >
>> > PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec 
>> > *v,F90Array1d a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
>> > PETSC_EXTERN PetscErrorCode 
>> > F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d 
>> > PETSC_F90_2PTR_PROTO_NOVAR);
>> 
>> Yes.
>> 
>> > [this doesn't look right.] 
>> 
>> I don't know why not, 
>
> output argument without a explicit pointer in definition is weird for c - and 
> can result in future bugs [when this code is changed]

The pointer is inside a typedef for all the basic PETSc types.

>> but I don't care about this narrow bit of
>> compiler-enforced type safety enough to push for the more intrusive fix.
>
> I'm not sure I understand what extra type safety is obtained by 'F90Array1d*' 
> vs 'F90Array1d' usage
>
> Or does 'typedef struct { void *ptr; } F90Array3d;' give the extra safety? I 
> could change this part of typedef.

Yeah, that's an alternative way to get a strong typedef, and you could
keep the other code unmodified.

typedef struct { char dummy; } F90Array1d;
typedef struct { char dummy; } F90Array2d;
typedef struct { char dummy; } F90Array3d;

These are all distinct types.  Better to use char for the dummy because
we don't have reason to know that the pointer is word aligned, but the
standard says that any data pointer can be cast to a char*.

>> 
>> > So I guess 'pointer to address (void*)' is the correct notation - as
>> > we pass these back to fortran calling routine wrt
>> > dmdavecgetarrayf901_() and an output argument for F90Array1dCreate().
>> 
>> It's actually just the pointer that we have.  The thing it points to is
>> of some type that is not known statically in F90Array1dCreate/Destroy.
>
> Yes. we get something opaque - and we pass it down to fortran utility routine 
> or up to the calling fortran routine - unmodified.
>
> My current patch is at balay/fix-F90Array__Destroy/maint
>
> Satish

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Satish Balay 
On Tue, 15 May 2018, Jed Brown wrote:

> > Your notation requires the following change
> >
> > PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec 
> > *v,F90Array1d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> > PETSC_EXTERN PetscErrorCode 
> > F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d* 
> > PETSC_F90_2PTR_PROTO_NOVAR);
> >
> > to:
> >
> > PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec 
> > *v,F90Array1d a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> > PETSC_EXTERN PetscErrorCode 
> > F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d 
> > PETSC_F90_2PTR_PROTO_NOVAR);
> 
> Yes.
> 
> > [this doesn't look right.] 
> 
> I don't know why not, 

output argument without a explicit pointer in definition is weird for c - and 
can result in future bugs [when this code is changed]

> but I don't care about this narrow bit of
> compiler-enforced type safety enough to push for the more intrusive fix.

I'm not sure I understand what extra type safety is obtained by 'F90Array1d*' 
vs 'F90Array1d' usage

Or does 'typedef struct { void *ptr; } F90Array3d;' give the extra safety? I 
could change this part of typedef.
 
> 
> > So I guess 'pointer to address (void*)' is the correct notation - as
> > we pass these back to fortran calling routine wrt
> > dmdavecgetarrayf901_() and an output argument for F90Array1dCreate().
> 
> It's actually just the pointer that we have.  The thing it points to is
> of some type that is not known statically in F90Array1dCreate/Destroy.

Yes. we get something opaque - and we pass it down to fortran utility routine 
or up to the calling fortran routine - unmodified.

My current patch is at balay/fix-F90Array__Destroy/maint

Satish

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Jed Brown 
Satish Balay  writes:

> On Tue, 15 May 2018, Jed Brown wrote:
>
>> Satish Balay  writes:
>> 
>> > On Tue, 15 May 2018, Jed Brown wrote:
>> >
>> >> Wow, how did this ever work with other compilers?
>> >> 
>> >> To ensure everything gets fixed, I see we have
>> >> 
>> >>   #define F90Array3d void
>> >> 
>> >> and then arguments are defined as
>> >> 
>> >>   F90Array3d *ptr
>> >> 
>> >> We could make this type-safe by
>> >> 
>> >>   typedef struct { void *ptr; } F90Array3d;
>> >
>> > The following appears to be sufficient. [void* more appropriate than void 
>> > - as these are pointers anyway?]
>> 
>> They're declared as pointers, so it's "pointer to void" versus "pointer
>> to void*".
>
> Your notation requires the following change
>
> PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec *v,F90Array1d 
> *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> PETSC_EXTERN PetscErrorCode 
> F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d* 
> PETSC_F90_2PTR_PROTO_NOVAR);
>
> to:
>
> PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec *v,F90Array1d 
> a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> PETSC_EXTERN PetscErrorCode 
> F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d 
> PETSC_F90_2PTR_PROTO_NOVAR);

Yes.

> [this doesn't look right.] 

I don't know why not, but I don't care about this narrow bit of
compiler-enforced type safety enough to push for the more intrusive fix.

> So I guess 'pointer to address (void*)' is the correct notation - as
> we pass these back to fortran calling routine wrt
> dmdavecgetarrayf901_() and an output argument for F90Array1dCreate().

It's actually just the pointer that we have.  The thing it points to is
of some type that is not known statically in F90Array1dCreate/Destroy.

> Satish
>
>> But the point of my suggestion was to get stronger type
>> checking, including between arrays of different dimensions.
>> 
>> > -#define F90Array1d void
>> > -#define F90Array2d void
>> > -#define F90Array3d void
>> > -#define F90Array4d void
>> > +typedef void* F90Array1d;
>> > +typedef void* F90Array2d;
>> > +typedef void* F90Array3d;
>> > +typedef void* F90Array4d;
>> >
>> >> 
>> >> and use this for arguments:
>> >> 
>> >>   F90Array3d a
>> >
>> > Don't need this change..
>> >
>> > Attaching my current patch..
>> >
>> > Satish
>> > diff --git a/include/petsc/private/f90impl.h 
>> > b/include/petsc/private/f90impl.h
>> > index a35efb76bd..4f26c8ffea 100644
>> > --- a/include/petsc/private/f90impl.h
>> > +++ b/include/petsc/private/f90impl.h
>> > @@ -15,11 +15,10 @@
>> >  #endif
>> >  
>> >  #if defined(PETSC_USING_F90)
>> > -
>> > -#define F90Array1d void
>> > -#define F90Array2d void
>> > -#define F90Array3d void
>> > -#define F90Array4d void
>> > +typedef void* F90Array1d;
>> > +typedef void* F90Array2d;
>> > +typedef void* F90Array3d;
>> > +typedef void* F90Array4d;
>> >  
>> >  PETSC_EXTERN PetscErrorCode 
>> > F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d* 
>> > PETSC_F90_2PTR_PROTO_NOVAR);
>> >  PETSC_EXTERN PetscErrorCode 
>> > F90Array1dAccess(F90Array1d*,MPI_Datatype,void** 
>> > PETSC_F90_2PTR_PROTO_NOVAR);
>> > diff --git a/src/dm/impls/composite/f90-custom/zfddaf90.c 
>> > b/src/dm/impls/composite/f90-custom/zfddaf90.c
>> > index abc0a27a01..2633c9e86a 100644
>> > --- a/src/dm/impls/composite/f90-custom/zfddaf90.c
>> > +++ b/src/dm/impls/composite/f90-custom/zfddaf90.c
>> > @@ -24,8 +24,8 @@ PETSC_EXTERN void PETSC_STDCALL 
>> > dmcompositegetaccessvpvp_(DM *dm,Vec *v,Vec *v1,
>> >  PETSC_EXTERN void PETSC_STDCALL dmcompositerestoreaccessvpvp_(DM *dm,Vec 
>> > *v,Vec *v1,F90Array1d *p1,Vec *v2,F90Array1d *p2,PetscErrorCode *ierr 
>> > PETSC_F90_2PTR_PROTO(ptrd1) PETSC_F90_2PTR_PROTO(ptrd2))
>> >  {
>> >*ierr = DMCompositeRestoreAccess(*dm,*v,v1,0,v2,0);
>> > -  *ierr = F90Array1dDestroy(&p1,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd1));
>> > -  *ierr = F90Array1dDestroy(&p2,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd2));
>> > +  *ierr = F90Array1dDestroy(p1,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd1));
>> > +  *ierr = F90Array1dDestroy(p2,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd2));
>> >  }
>> >  
>> >  PETSC_EXTERN void PETSC_STDCALL dmcompositegetentriesarray_(DM *dm, DM 
>> > *dmarray, PetscErrorCode *ierr)
>> > diff --git a/src/dm/impls/da/f90-custom/zda1f90.c 
>> > b/src/dm/impls/da/f90-custom/zda1f90.c
>> > index 082027725f..41cc58534f 100644
>> > --- a/src/dm/impls/da/f90-custom/zda1f90.c
>> > +++ b/src/dm/impls/da/f90-custom/zda1f90.c
>> > @@ -74,7 +74,7 @@ PETSC_EXTERN void PETSC_STDCALL 
>> > dmdavecrestorearrayf901_(DM *da,Vec *v,F90Array1
>> >PetscScalar *fa;
>> >*ierr = F90Array1dAccess(a,MPIU_SCALAR,(void**)&fa 
>> > PETSC_F90_2PTR_PARAM(ptrd));
>> >*ierr = VecRestoreArray(*v,&fa);if (*ierr) return;
>> > -  *ierr = F90Array1dDestroy(&a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
>> > +  *ierr = F90Array1dDestroy(a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
>> >  }
>> >  
>> >  PETSC_EXTERN

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Satish Balay 
On Tue, 15 May 2018, Jed Brown wrote:

> Satish Balay  writes:
> 
> > On Tue, 15 May 2018, Jed Brown wrote:
> >
> >> Wow, how did this ever work with other compilers?
> >> 
> >> To ensure everything gets fixed, I see we have
> >> 
> >>   #define F90Array3d void
> >> 
> >> and then arguments are defined as
> >> 
> >>   F90Array3d *ptr
> >> 
> >> We could make this type-safe by
> >> 
> >>   typedef struct { void *ptr; } F90Array3d;
> >
> > The following appears to be sufficient. [void* more appropriate than void - 
> > as these are pointers anyway?]
> 
> They're declared as pointers, so it's "pointer to void" versus "pointer
> to void*".

Your notation requires the following change

PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec *v,F90Array1d 
*a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
PETSC_EXTERN PetscErrorCode 
F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d* 
PETSC_F90_2PTR_PROTO_NOVAR);

to:

PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf901_(DM *da,Vec *v,F90Array1d 
a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
PETSC_EXTERN PetscErrorCode 
F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d 
PETSC_F90_2PTR_PROTO_NOVAR);

[this doesn't look right.] So I guess 'pointer to address (void*)' is
the correct notation - as we pass these back to fortran calling
routine wrt dmdavecgetarrayf901_() and an output argument for
F90Array1dCreate().

Satish

> But the point of my suggestion was to get stronger type
> checking, including between arrays of different dimensions.
> 
> > -#define F90Array1d void
> > -#define F90Array2d void
> > -#define F90Array3d void
> > -#define F90Array4d void
> > +typedef void* F90Array1d;
> > +typedef void* F90Array2d;
> > +typedef void* F90Array3d;
> > +typedef void* F90Array4d;
> >
> >> 
> >> and use this for arguments:
> >> 
> >>   F90Array3d a
> >
> > Don't need this change..
> >
> > Attaching my current patch..
> >
> > Satish
> > diff --git a/include/petsc/private/f90impl.h 
> > b/include/petsc/private/f90impl.h
> > index a35efb76bd..4f26c8ffea 100644
> > --- a/include/petsc/private/f90impl.h
> > +++ b/include/petsc/private/f90impl.h
> > @@ -15,11 +15,10 @@
> >  #endif
> >  
> >  #if defined(PETSC_USING_F90)
> > -
> > -#define F90Array1d void
> > -#define F90Array2d void
> > -#define F90Array3d void
> > -#define F90Array4d void
> > +typedef void* F90Array1d;
> > +typedef void* F90Array2d;
> > +typedef void* F90Array3d;
> > +typedef void* F90Array4d;
> >  
> >  PETSC_EXTERN PetscErrorCode 
> > F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d* 
> > PETSC_F90_2PTR_PROTO_NOVAR);
> >  PETSC_EXTERN PetscErrorCode 
> > F90Array1dAccess(F90Array1d*,MPI_Datatype,void** 
> > PETSC_F90_2PTR_PROTO_NOVAR);
> > diff --git a/src/dm/impls/composite/f90-custom/zfddaf90.c 
> > b/src/dm/impls/composite/f90-custom/zfddaf90.c
> > index abc0a27a01..2633c9e86a 100644
> > --- a/src/dm/impls/composite/f90-custom/zfddaf90.c
> > +++ b/src/dm/impls/composite/f90-custom/zfddaf90.c
> > @@ -24,8 +24,8 @@ PETSC_EXTERN void PETSC_STDCALL 
> > dmcompositegetaccessvpvp_(DM *dm,Vec *v,Vec *v1,
> >  PETSC_EXTERN void PETSC_STDCALL dmcompositerestoreaccessvpvp_(DM *dm,Vec 
> > *v,Vec *v1,F90Array1d *p1,Vec *v2,F90Array1d *p2,PetscErrorCode *ierr 
> > PETSC_F90_2PTR_PROTO(ptrd1) PETSC_F90_2PTR_PROTO(ptrd2))
> >  {
> >*ierr = DMCompositeRestoreAccess(*dm,*v,v1,0,v2,0);
> > -  *ierr = F90Array1dDestroy(&p1,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd1));
> > -  *ierr = F90Array1dDestroy(&p2,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd2));
> > +  *ierr = F90Array1dDestroy(p1,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd1));
> > +  *ierr = F90Array1dDestroy(p2,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd2));
> >  }
> >  
> >  PETSC_EXTERN void PETSC_STDCALL dmcompositegetentriesarray_(DM *dm, DM 
> > *dmarray, PetscErrorCode *ierr)
> > diff --git a/src/dm/impls/da/f90-custom/zda1f90.c 
> > b/src/dm/impls/da/f90-custom/zda1f90.c
> > index 082027725f..41cc58534f 100644
> > --- a/src/dm/impls/da/f90-custom/zda1f90.c
> > +++ b/src/dm/impls/da/f90-custom/zda1f90.c
> > @@ -74,7 +74,7 @@ PETSC_EXTERN void PETSC_STDCALL 
> > dmdavecrestorearrayf901_(DM *da,Vec *v,F90Array1
> >PetscScalar *fa;
> >*ierr = F90Array1dAccess(a,MPIU_SCALAR,(void**)&fa 
> > PETSC_F90_2PTR_PARAM(ptrd));
> >*ierr = VecRestoreArray(*v,&fa);if (*ierr) return;
> > -  *ierr = F90Array1dDestroy(&a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
> > +  *ierr = F90Array1dDestroy(a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
> >  }
> >  
> >  PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf902_(DM *da,Vec 
> > *v,F90Array2d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> > @@ -113,7 +113,7 @@ PETSC_EXTERN void PETSC_STDCALL 
> > dmdavecrestorearrayf902_(DM *da,Vec *v,F90Array2
> >PetscScalar *fa;
> >*ierr = F90Array2dAccess(a,MPIU_SCALAR,(void**)&fa 
> > PETSC_F90_2PTR_PARAM(ptrd));
> >*ierr = VecRestoreArray(*v,&fa);if (*ierr) return;
> > -  *ierr = F90Array2dDestroy(&a,MPIU_SCALAR PETSC_

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Jed Brown 
Satish Balay  writes:

> On Tue, 15 May 2018, Jed Brown wrote:
>
>> Wow, how did this ever work with other compilers?
>> 
>> To ensure everything gets fixed, I see we have
>> 
>>   #define F90Array3d void
>> 
>> and then arguments are defined as
>> 
>>   F90Array3d *ptr
>> 
>> We could make this type-safe by
>> 
>>   typedef struct { void *ptr; } F90Array3d;
>
> The following appears to be sufficient. [void* more appropriate than void - 
> as these are pointers anyway?]

They're declared as pointers, so it's "pointer to void" versus "pointer
to void*".  But the point of my suggestion was to get stronger type
checking, including between arrays of different dimensions.

> -#define F90Array1d void
> -#define F90Array2d void
> -#define F90Array3d void
> -#define F90Array4d void
> +typedef void* F90Array1d;
> +typedef void* F90Array2d;
> +typedef void* F90Array3d;
> +typedef void* F90Array4d;
>
>> 
>> and use this for arguments:
>> 
>>   F90Array3d a
>
> Don't need this change..
>
> Attaching my current patch..
>
> Satish
> diff --git a/include/petsc/private/f90impl.h b/include/petsc/private/f90impl.h
> index a35efb76bd..4f26c8ffea 100644
> --- a/include/petsc/private/f90impl.h
> +++ b/include/petsc/private/f90impl.h
> @@ -15,11 +15,10 @@
>  #endif
>  
>  #if defined(PETSC_USING_F90)
> -
> -#define F90Array1d void
> -#define F90Array2d void
> -#define F90Array3d void
> -#define F90Array4d void
> +typedef void* F90Array1d;
> +typedef void* F90Array2d;
> +typedef void* F90Array3d;
> +typedef void* F90Array4d;
>  
>  PETSC_EXTERN PetscErrorCode 
> F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d* 
> PETSC_F90_2PTR_PROTO_NOVAR);
>  PETSC_EXTERN PetscErrorCode F90Array1dAccess(F90Array1d*,MPI_Datatype,void** 
> PETSC_F90_2PTR_PROTO_NOVAR);
> diff --git a/src/dm/impls/composite/f90-custom/zfddaf90.c 
> b/src/dm/impls/composite/f90-custom/zfddaf90.c
> index abc0a27a01..2633c9e86a 100644
> --- a/src/dm/impls/composite/f90-custom/zfddaf90.c
> +++ b/src/dm/impls/composite/f90-custom/zfddaf90.c
> @@ -24,8 +24,8 @@ PETSC_EXTERN void PETSC_STDCALL 
> dmcompositegetaccessvpvp_(DM *dm,Vec *v,Vec *v1,
>  PETSC_EXTERN void PETSC_STDCALL dmcompositerestoreaccessvpvp_(DM *dm,Vec 
> *v,Vec *v1,F90Array1d *p1,Vec *v2,F90Array1d *p2,PetscErrorCode *ierr 
> PETSC_F90_2PTR_PROTO(ptrd1) PETSC_F90_2PTR_PROTO(ptrd2))
>  {
>*ierr = DMCompositeRestoreAccess(*dm,*v,v1,0,v2,0);
> -  *ierr = F90Array1dDestroy(&p1,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd1));
> -  *ierr = F90Array1dDestroy(&p2,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd2));
> +  *ierr = F90Array1dDestroy(p1,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd1));
> +  *ierr = F90Array1dDestroy(p2,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd2));
>  }
>  
>  PETSC_EXTERN void PETSC_STDCALL dmcompositegetentriesarray_(DM *dm, DM 
> *dmarray, PetscErrorCode *ierr)
> diff --git a/src/dm/impls/da/f90-custom/zda1f90.c 
> b/src/dm/impls/da/f90-custom/zda1f90.c
> index 082027725f..41cc58534f 100644
> --- a/src/dm/impls/da/f90-custom/zda1f90.c
> +++ b/src/dm/impls/da/f90-custom/zda1f90.c
> @@ -74,7 +74,7 @@ PETSC_EXTERN void PETSC_STDCALL dmdavecrestorearrayf901_(DM 
> *da,Vec *v,F90Array1
>PetscScalar *fa;
>*ierr = F90Array1dAccess(a,MPIU_SCALAR,(void**)&fa 
> PETSC_F90_2PTR_PARAM(ptrd));
>*ierr = VecRestoreArray(*v,&fa);if (*ierr) return;
> -  *ierr = F90Array1dDestroy(&a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
> +  *ierr = F90Array1dDestroy(a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
>  }
>  
>  PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf902_(DM *da,Vec 
> *v,F90Array2d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> @@ -113,7 +113,7 @@ PETSC_EXTERN void PETSC_STDCALL 
> dmdavecrestorearrayf902_(DM *da,Vec *v,F90Array2
>PetscScalar *fa;
>*ierr = F90Array2dAccess(a,MPIU_SCALAR,(void**)&fa 
> PETSC_F90_2PTR_PARAM(ptrd));
>*ierr = VecRestoreArray(*v,&fa);if (*ierr) return;
> -  *ierr = F90Array2dDestroy(&a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
> +  *ierr = F90Array2dDestroy(a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
>  }
>  
>  PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf903_(DM *da,Vec 
> *v,F90Array3d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> @@ -154,7 +154,7 @@ PETSC_EXTERN void PETSC_STDCALL 
> dmdavecrestorearrayf903_(DM *da,Vec *v,F90Array3
>PetscScalar *fa;
>*ierr = F90Array3dAccess(a,MPIU_SCALAR,(void**)&fa 
> PETSC_F90_2PTR_PARAM(ptrd));
>*ierr = VecRestoreArray(*v,&fa);if (*ierr) return;
> -  *ierr = F90Array3dDestroy(&a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
> +  *ierr = F90Array3dDestroy(a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
>  }
>  
>  PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf904_(DM *da,Vec 
> *v,F90Array4d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
> @@ -190,7 +190,7 @@ PETSC_EXTERN void PETSC_STDCALL 
> dmdavecrestorearrayf904_(DM *da,Vec *v,F90Array4
>*/
>*ierr = F90Array4dAccess(a,MPIU_SCALAR,(void**)&fa 
> PETSC_F90_2PTR_PARAM(ptrd));
>*ierr = VecRestoreArray(*v,&fa);if (*ierr) re

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Satish Balay 
On Tue, 15 May 2018, Jed Brown wrote:

> Wow, how did this ever work with other compilers?
> 
> To ensure everything gets fixed, I see we have
> 
>   #define F90Array3d void
> 
> and then arguments are defined as
> 
>   F90Array3d *ptr
> 
> We could make this type-safe by
> 
>   typedef struct { void *ptr; } F90Array3d;

The following appears to be sufficient. [void* more appropriate than void - as 
these are pointers anyway?]

-#define F90Array1d void
-#define F90Array2d void
-#define F90Array3d void
-#define F90Array4d void
+typedef void* F90Array1d;
+typedef void* F90Array2d;
+typedef void* F90Array3d;
+typedef void* F90Array4d;

> 
> and use this for arguments:
> 
>   F90Array3d a

Don't need this change..

Attaching my current patch..

Satishdiff --git a/include/petsc/private/f90impl.h b/include/petsc/private/f90impl.h
index a35efb76bd..4f26c8ffea 100644
--- a/include/petsc/private/f90impl.h
+++ b/include/petsc/private/f90impl.h
@@ -15,11 +15,10 @@
 #endif
 
 #if defined(PETSC_USING_F90)
-
-#define F90Array1d void
-#define F90Array2d void
-#define F90Array3d void
-#define F90Array4d void
+typedef void* F90Array1d;
+typedef void* F90Array2d;
+typedef void* F90Array3d;
+typedef void* F90Array4d;
 
 PETSC_EXTERN PetscErrorCode 
F90Array1dCreate(void*,MPI_Datatype,PetscInt,PetscInt,F90Array1d* 
PETSC_F90_2PTR_PROTO_NOVAR);
 PETSC_EXTERN PetscErrorCode F90Array1dAccess(F90Array1d*,MPI_Datatype,void** 
PETSC_F90_2PTR_PROTO_NOVAR);
diff --git a/src/dm/impls/composite/f90-custom/zfddaf90.c 
b/src/dm/impls/composite/f90-custom/zfddaf90.c
index abc0a27a01..2633c9e86a 100644
--- a/src/dm/impls/composite/f90-custom/zfddaf90.c
+++ b/src/dm/impls/composite/f90-custom/zfddaf90.c
@@ -24,8 +24,8 @@ PETSC_EXTERN void PETSC_STDCALL dmcompositegetaccessvpvp_(DM 
*dm,Vec *v,Vec *v1,
 PETSC_EXTERN void PETSC_STDCALL dmcompositerestoreaccessvpvp_(DM *dm,Vec 
*v,Vec *v1,F90Array1d *p1,Vec *v2,F90Array1d *p2,PetscErrorCode *ierr 
PETSC_F90_2PTR_PROTO(ptrd1) PETSC_F90_2PTR_PROTO(ptrd2))
 {
   *ierr = DMCompositeRestoreAccess(*dm,*v,v1,0,v2,0);
-  *ierr = F90Array1dDestroy(&p1,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd1));
-  *ierr = F90Array1dDestroy(&p2,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd2));
+  *ierr = F90Array1dDestroy(p1,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd1));
+  *ierr = F90Array1dDestroy(p2,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd2));
 }
 
 PETSC_EXTERN void PETSC_STDCALL dmcompositegetentriesarray_(DM *dm, DM 
*dmarray, PetscErrorCode *ierr)
diff --git a/src/dm/impls/da/f90-custom/zda1f90.c 
b/src/dm/impls/da/f90-custom/zda1f90.c
index 082027725f..41cc58534f 100644
--- a/src/dm/impls/da/f90-custom/zda1f90.c
+++ b/src/dm/impls/da/f90-custom/zda1f90.c
@@ -74,7 +74,7 @@ PETSC_EXTERN void PETSC_STDCALL dmdavecrestorearrayf901_(DM 
*da,Vec *v,F90Array1
   PetscScalar *fa;
   *ierr = F90Array1dAccess(a,MPIU_SCALAR,(void**)&fa 
PETSC_F90_2PTR_PARAM(ptrd));
   *ierr = VecRestoreArray(*v,&fa);if (*ierr) return;
-  *ierr = F90Array1dDestroy(&a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
+  *ierr = F90Array1dDestroy(a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
 }
 
 PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf902_(DM *da,Vec *v,F90Array2d 
*a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
@@ -113,7 +113,7 @@ PETSC_EXTERN void PETSC_STDCALL dmdavecrestorearrayf902_(DM 
*da,Vec *v,F90Array2
   PetscScalar *fa;
   *ierr = F90Array2dAccess(a,MPIU_SCALAR,(void**)&fa 
PETSC_F90_2PTR_PARAM(ptrd));
   *ierr = VecRestoreArray(*v,&fa);if (*ierr) return;
-  *ierr = F90Array2dDestroy(&a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
+  *ierr = F90Array2dDestroy(a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
 }
 
 PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf903_(DM *da,Vec *v,F90Array3d 
*a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
@@ -154,7 +154,7 @@ PETSC_EXTERN void PETSC_STDCALL dmdavecrestorearrayf903_(DM 
*da,Vec *v,F90Array3
   PetscScalar *fa;
   *ierr = F90Array3dAccess(a,MPIU_SCALAR,(void**)&fa 
PETSC_F90_2PTR_PARAM(ptrd));
   *ierr = VecRestoreArray(*v,&fa);if (*ierr) return;
-  *ierr = F90Array3dDestroy(&a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
+  *ierr = F90Array3dDestroy(a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
 }
 
 PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayf904_(DM *da,Vec *v,F90Array4d 
*a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
@@ -190,7 +190,7 @@ PETSC_EXTERN void PETSC_STDCALL dmdavecrestorearrayf904_(DM 
*da,Vec *v,F90Array4
   */
   *ierr = F90Array4dAccess(a,MPIU_SCALAR,(void**)&fa 
PETSC_F90_2PTR_PARAM(ptrd));
   *ierr = VecRestoreArray(*v,&fa);if (*ierr) return;
-  *ierr = F90Array4dDestroy(&a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
+  *ierr = F90Array4dDestroy(a,MPIU_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
 }
 
 PETSC_EXTERN void PETSC_STDCALL dmdavecgetarrayreadf901_(DM *da,Vec 
*v,F90Array1d *a,PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(ptrd))
@@ -223,7 +223,7 @@ PETSC_EXTERN void PETSC_STDCALL 
dmdavecrestorearrayreadf901_(DM *da,Vec *v,F90Ar
   const PetscScalar *fa;
   *ierr = F90Array1dAccess(a,MPIU_S

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Satish Balay 
Looks like this issue got introduced with:

http://bitbucket.org/petsc/petsc/commits/fcfd50eb5fb

Satish

On Tue, 15 May 2018, Jed Brown wrote:

> Wow, how did this ever work with other compilers?
> 
> To ensure everything gets fixed, I see we have
> 
>   #define F90Array3d void
> 
> and then arguments are defined as
> 
>   F90Array3d *ptr
> 
> We could make this type-safe by
> 
>   typedef struct { void *ptr; } F90Array3d;
> 
> and use this for arguments:
> 
>   F90Array3d a
> 
> 
> 
> Randall Mackie  writes:
> 
> >> On Apr 10, 2018, at 5:40 PM, Satish Balay  wrote:
> >> 
> >> On Tue, 10 Apr 2018, Jeff Hammond wrote:
> >> 
> >>> Can you try on a non-KNL host?  It's a bug either way but I want to
> >>> determine if KNL host is the issue.
> >> 
> >> Breaks on 'E5-2695 v4' aswell (bebop.lcrc) with '-axcore-avx2' and 'icc 
> >> (ICC) 18.0.1 20171018'
> >> 
> >>> Based only what I see below, Randy doesn't seem to be reporting a
> >>> KNL-specific issue.  Is that incorrect?
> >> 
> >> Hardware details weren't mentioned in this thread.
> >> 
> >>> Again, there is clearly a bug here, but it helps to localize the problem 
> >>> as
> >>> much as possible.
> >> 
> >>> On Thu, 5 Apr 2018, Randall Mackie wrote:
> >> 
> >> so I assume this is an Intel bug, but before we submit a bug
> >> report I wanted to see if anyone else had similar experiences?
> >> 
> >> Randy,
> >> 
> >> I'll leave this to you to file a report with Intel.
> >> 
> >> Thanks,
> >> Satish
> >
> >
> > Hi Satish,
> >
> > As requested we filed a report with Intel, and this is their response:
> >
> >
> >  
> > From: Intel Customer Support  > > 
> > Sent: 14 May 2018 19:56
> > Subject: Intel Developer Products Support - Update to Service 
> > Request#:03369230
> >  
> >
> >  
> >
> > Hello,
> > An update was made to service request on May 14, 2018:
> >  
> > Thank you for the additional information. Our engineering team investigated 
> > this case. Please see the following resolution:
> >
> >  
> >
> > There is a bug in customer’s code:
> >
> >  
> >
> > At line 157 of “/src/dm/impls/da/f90-custom/zda1f90.c”
> >
> >  
> >
> > *ierr = F90Array3dDestroy(&a,PETSC_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
> >
> >  
> >
> > Here “&a” should be “a” because “a” was passed from call at line 79 of 
> > “test.F90” as the 3rd arguments 
> > DMDAVecRestoreArrayF90(da1,vec1,ptr_v1,ierr). In fortran the array 
> > descriptor of assumed-shape array will be passed by address so when it is 
> > passed to another C function it shouldn’t be taken address again. There are 
> > other places in the code having the same error. After removing “&” the code 
> > can be built and run without error.
> >
> >  
> >
> > In addition, this issue was discussed 4 years ago 
> > https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-April/021232.html 
> > 
> >  without noticing the bug in the code, more likely a user coding error.
> >
> > I am going to close out this case as not a compiler defect. You can reopen 
> > by posting a reply if you have any problem with the above resolution.
> >
> > Sign in 
> > 
> >  to view and update your request or to get additional information. You can 
> > also reply to this email with questions or comments.
> >  
> > Regards,
> >  
> > Devorah
> > Intel Developer Products Support
> >  
> >
> > Intel will use your personal information solely for the purpose it was 
> > collected. We will not use your personal information for a different 
> > purpose without first asking your permission. In order to fulfill the 
> > purpose, we may need to share your personal information within Intel 
> > Corporation, Intel subsidiaries worldwide, or with authorized third parties.
> > Privacy  – Cookies 
> > 
> > Intel may contact you in order to obtain your feedback on the quality of 
> > the support you received. We give you many choices regarding our use of 
> > your personal information for quality assurance and marketing purposes. You 
> > may update and request access to your contact details and communication 
> > preferences by using one of the following methods: visit the specific 
> > product or service website; use the Contact Us 
> >  
> > form; or send a letter to the postal address below.
> > Intel Corporation; Mailstop RNB4-145; 2200 Mission College Blvd.; Santa 
> > Clara, CA 95054 USA
> > ® Intel Corporation – Legal Information 
> >  – 
> > www.intel.com 
> > Intel is a registered trademark of Intel Corporation or its subsidiaries in 
> > the United States and other countries.
> > *Other names and brands may be claimed as the prop

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Jed Brown 
Wow, how did this ever work with other compilers?

To ensure everything gets fixed, I see we have

  #define F90Array3d void

and then arguments are defined as

  F90Array3d *ptr

We could make this type-safe by

  typedef struct { void *ptr; } F90Array3d;

and use this for arguments:

  F90Array3d a



Randall Mackie  writes:

>> On Apr 10, 2018, at 5:40 PM, Satish Balay  wrote:
>> 
>> On Tue, 10 Apr 2018, Jeff Hammond wrote:
>> 
>>> Can you try on a non-KNL host?  It's a bug either way but I want to
>>> determine if KNL host is the issue.
>> 
>> Breaks on 'E5-2695 v4' aswell (bebop.lcrc) with '-axcore-avx2' and 'icc 
>> (ICC) 18.0.1 20171018'
>> 
>>> Based only what I see below, Randy doesn't seem to be reporting a
>>> KNL-specific issue.  Is that incorrect?
>> 
>> Hardware details weren't mentioned in this thread.
>> 
>>> Again, there is clearly a bug here, but it helps to localize the problem as
>>> much as possible.
>> 
>>> On Thu, 5 Apr 2018, Randall Mackie wrote:
>> 
>> so I assume this is an Intel bug, but before we submit a bug
>> report I wanted to see if anyone else had similar experiences?
>> 
>> Randy,
>> 
>> I'll leave this to you to file a report with Intel.
>> 
>> Thanks,
>> Satish
>
>
> Hi Satish,
>
> As requested we filed a report with Intel, and this is their response:
>
>
>  
> From: Intel Customer Support  > 
> Sent: 14 May 2018 19:56
> Subject: Intel Developer Products Support - Update to Service 
> Request#:03369230
>  
>
>  
>
> Hello,
> An update was made to service request on May 14, 2018:
>  
> Thank you for the additional information. Our engineering team investigated 
> this case. Please see the following resolution:
>
>  
>
> There is a bug in customer’s code:
>
>  
>
> At line 157 of “/src/dm/impls/da/f90-custom/zda1f90.c”
>
>  
>
> *ierr = F90Array3dDestroy(&a,PETSC_SCALAR PETSC_F90_2PTR_PARAM(ptrd));
>
>  
>
> Here “&a” should be “a” because “a” was passed from call at line 79 of 
> “test.F90” as the 3rd arguments DMDAVecRestoreArrayF90(da1,vec1,ptr_v1,ierr). 
> In fortran the array descriptor of assumed-shape array will be passed by 
> address so when it is passed to another C function it shouldn’t be taken 
> address again. There are other places in the code having the same error. 
> After removing “&” the code can be built and run without error.
>
>  
>
> In addition, this issue was discussed 4 years ago 
> https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-April/021232.html 
>  
> without noticing the bug in the code, more likely a user coding error.
>
> I am going to close out this case as not a compiler defect. You can reopen by 
> posting a reply if you have any problem with the above resolution.
>
> Sign in 
> 
>  to view and update your request or to get additional information. You can 
> also reply to this email with questions or comments.
>  
> Regards,
>  
> Devorah
> Intel Developer Products Support
>  
>
> Intel will use your personal information solely for the purpose it was 
> collected. We will not use your personal information for a different purpose 
> without first asking your permission. In order to fulfill the purpose, we may 
> need to share your personal information within Intel Corporation, Intel 
> subsidiaries worldwide, or with authorized third parties.
> Privacy  – Cookies 
> 
> Intel may contact you in order to obtain your feedback on the quality of the 
> support you received. We give you many choices regarding our use of your 
> personal information for quality assurance and marketing purposes. You may 
> update and request access to your contact details and communication 
> preferences by using one of the following methods: visit the specific product 
> or service website; use the Contact Us 
>  
> form; or send a letter to the postal address below.
> Intel Corporation; Mailstop RNB4-145; 2200 Mission College Blvd.; Santa 
> Clara, CA 95054 USA
> ® Intel Corporation – Legal Information 
>  – 
> www.intel.com 
> Intel is a registered trademark of Intel Corporation or its subsidiaries in 
> the United States and other countries.
> *Other names and brands may be claimed as the property of others.
> *PLEASE DO NOT DELETE*
> Thread ID: ref:_00DU0YT3c._5000PhEKLa:ref
> You must include this text in any reply to this email. Thank you.
> *PLEASE DO NOT DELETE*

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-05-15 Thread Randall Mackie 

> On Apr 10, 2018, at 5:40 PM, Satish Balay  wrote:
> 
> On Tue, 10 Apr 2018, Jeff Hammond wrote:
> 
>> Can you try on a non-KNL host?  It's a bug either way but I want to
>> determine if KNL host is the issue.
> 
> Breaks on 'E5-2695 v4' aswell (bebop.lcrc) with '-axcore-avx2' and 'icc (ICC) 
> 18.0.1 20171018'
> 
>> Based only what I see below, Randy doesn't seem to be reporting a
>> KNL-specific issue.  Is that incorrect?
> 
> Hardware details weren't mentioned in this thread.
> 
>> Again, there is clearly a bug here, but it helps to localize the problem as
>> much as possible.
> 
>> On Thu, 5 Apr 2018, Randall Mackie wrote:
> 
> so I assume this is an Intel bug, but before we submit a bug
> report I wanted to see if anyone else had similar experiences?
> 
> Randy,
> 
> I'll leave this to you to file a report with Intel.
> 
> Thanks,
> Satish


Hi Satish,

As requested we filed a report with Intel, and this is their response:


 
From: Intel Customer Support mailto:supportrepl...@intel.com>> 
Sent: 14 May 2018 19:56
Subject: Intel Developer Products Support - Update to Service Request#:03369230
 

 

Hello,
An update was made to service request on May 14, 2018:
 
Thank you for the additional information. Our engineering team investigated 
this case. Please see the following resolution:

 

There is a bug in customer’s code:

 

At line 157 of “/src/dm/impls/da/f90-custom/zda1f90.c”

 

*ierr = F90Array3dDestroy(&a,PETSC_SCALAR PETSC_F90_2PTR_PARAM(ptrd));

 

Here “&a” should be “a” because “a” was passed from call at line 79 of 
“test.F90” as the 3rd arguments DMDAVecRestoreArrayF90(da1,vec1,ptr_v1,ierr). 
In fortran the array descriptor of assumed-shape array will be passed by 
address so when it is passed to another C function it shouldn’t be taken 
address again. There are other places in the code having the same error. After 
removing “&” the code can be built and run without error.

 

In addition, this issue was discussed 4 years ago 
https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2014-April/021232.html 
 
without noticing the bug in the code, more likely a user coding error.

I am going to close out this case as not a compiler defect. You can reopen by 
posting a reply if you have any problem with the above resolution.

Sign in 

 to view and update your request or to get additional information. You can also 
reply to this email with questions or comments.
 
Regards,
 
Devorah
Intel Developer Products Support
 

Intel will use your personal information solely for the purpose it was 
collected. We will not use your personal information for a different purpose 
without first asking your permission. In order to fulfill the purpose, we may 
need to share your personal information within Intel Corporation, Intel 
subsidiaries worldwide, or with authorized third parties.
Privacy  – Cookies 
Intel may contact you in order to obtain your feedback on the quality of the 
support you received. We give you many choices regarding our use of your 
personal information for quality assurance and marketing purposes. You may 
update and request access to your contact details and communication preferences 
by using one of the following methods: visit the specific product or service 
website; use the Contact Us 
 
form; or send a letter to the postal address below.
Intel Corporation; Mailstop RNB4-145; 2200 Mission College Blvd.; Santa Clara, 
CA 95054 USA
® Intel Corporation – Legal Information 
 – 
www.intel.com 
Intel is a registered trademark of Intel Corporation or its subsidiaries in the 
United States and other countries.
*Other names and brands may be claimed as the property of others.
*PLEASE DO NOT DELETE*
Thread ID: ref:_00DU0YT3c._5000PhEKLa:ref
You must include this text in any reply to this email. Thank you.
*PLEASE DO NOT DELETE*

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-12 Thread Randall Mackie 

> On Apr 12, 2018, at 3:47 PM, Satish Balay  wrote:
> 
> On Thu, 12 Apr 2018, Victor Eijkhout wrote:
> 
>> 
>> 
>> On Apr 12, 2018, at 3:30 PM, Satish Balay 
>> mailto:ba...@mcs.anl.gov>> wrote:
>> 
>> I can reproudce this issue with 'icc (ICC) 18.0.2 20180210'. Is there a 
>> newer version?
>> 
>> Probably not. But I can’t figure out Intel version numbers. Here’s what I 
>> find:
>> 
>> compilers_and_libraries_2018.2.199/
> 
> bash-4.2$ which icc
> /homes/intel/18u2/compilers_and_libraries_2018.2.199/linux/bin/intel64/icc
> 
> Ok - its the same version.
> 
> Satish



Thanks Satish, we can also confirm the bug still exists in update 2 of the 
Intel compiler, and we have updated our bug report with Intel to indicate this.

Randy M.

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-12 Thread Satish Balay 
On Thu, 12 Apr 2018, Victor Eijkhout wrote:

> 
> 
> On Apr 12, 2018, at 3:30 PM, Satish Balay 
> mailto:ba...@mcs.anl.gov>> wrote:
> 
> I can reproudce this issue with 'icc (ICC) 18.0.2 20180210'. Is there a newer 
> version?
> 
> Probably not. But I can’t figure out Intel version numbers. Here’s what I 
> find:
> 
> compilers_and_libraries_2018.2.199/

bash-4.2$ which icc
/homes/intel/18u2/compilers_and_libraries_2018.2.199/linux/bin/intel64/icc

Ok - its the same version.

Satish

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-12 Thread Victor Eijkhout 


On Apr 12, 2018, at 3:30 PM, Satish Balay 
mailto:ba...@mcs.anl.gov>> wrote:

I can reproudce this issue with 'icc (ICC) 18.0.2 20180210'. Is there a newer 
version?

Probably not. But I can’t figure out Intel version numbers. Here’s what I find:

compilers_and_libraries_2018.2.199/

Victor.

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-12 Thread Satish Balay 
On Thu, 12 Apr 2018, Victor Eijkhout wrote:

> 
> 
> > On Apr 11, 2018, at 9:12 AM, Randall Mackie  wrote:
> > 
> > Hi Satish,
> > 
> > Thanks to you and Jeff for taking a look at this. We will submit a bug 
> > report to Intel.
> 
> Intel has released 18 update 2 which seems to fix several of these problems, 
> though I have not seen this exact problem with earlier versions.

I can reproudce this issue with 'icc (ICC) 18.0.2 20180210'. Is there a newer 
version?

Satish

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-11 Thread Victor Eijkhout 


> On Apr 11, 2018, at 9:12 AM, Randall Mackie  wrote:
> 
> Hi Satish,
> 
> Thanks to you and Jeff for taking a look at this. We will submit a bug report 
> to Intel.

Intel has released 18 update 2 which seems to fix several of these problems, 
though I have not seen this exact problem with earlier versions.

Victor.

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-11 Thread Randall Mackie 
Thanks Satish, we’ve submitted a bug report with Intel and are following up 
with them.

Randy


> On Apr 12, 2018, at 12:08 AM, Satish Balay  wrote:
> 
> I see this error with "-xAVX" build aswell.
> 
> Satish
> 
> On Wed, 11 Apr 2018, Zhang, Hong wrote:
> 
>> Performance wise, I would suggest to use "-xAVX" instead of "-axcore-avx2". 
>> Based on our experience with running PETSc on a variety of Xeon processors 
>> (including KNL), using AVX2 yields comparable and sometimes worse 
>> performance than using AVX. But if your machine supports AVX-512, it is 
>> definitely beneficial to use AVX-512.
>> 
>> Hong (Mr.)
>> 
>>> On Apr 5, 2018, at 10:03 AM, Randall Mackie  wrote:
>>> 
>>> Dear PETSc users,
>>> 
>>> I’m curious if anyone else experiences problems using DMDAVecGetArrayF90 in 
>>> conjunction with Intel compilers?
>>> We have had many problems (typically 11 SEGV segmentation violations) when 
>>> PETSc is compiled in optimize mode (with various combinations of options).
>>> These same codes run valgrind clean with gfortran, so I assume this is an 
>>> Intel bug, but before we submit a bug report I wanted to see if anyone else 
>>> had similar experiences?
>>> We have basically gone back and replaced our calls to DMDAVecGetArrayF90 
>>> with calls to VecGetArrayF90 and pass those pointers into a “local” 
>>> subroutine that works fine.
>>> 
>>> In case anyone is curious, the attached test code shows this behavior when 
>>> PETSc is compiled with the following options:
>>> 
>>> ./configure \
>>> --with-clean=1 \
>>> --with-debugging=0 \
>>> --with-fortran=1 \
>>> --with-64-bit-indices \
>>> --download-mpich=../mpich-3.3a2.tar.gz \
>>> --with-blas-lapack-dir=/opt/intel/mkl \
>>> --with-cc=icc \
>>> --with-fc=ifort \
>>> --with-cxx=icc \
>>> --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
>>> --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
>>> --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2’ \
>>> 
>>> 
>>> 
>>> Thanks, Randy M.
>>> 
>>> 
>> 
>>

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-11 Thread Satish Balay 
I see this error with "-xAVX" build aswell.

Satish

On Wed, 11 Apr 2018, Zhang, Hong wrote:

> Performance wise, I would suggest to use "-xAVX" instead of "-axcore-avx2". 
> Based on our experience with running PETSc on a variety of Xeon processors 
> (including KNL), using AVX2 yields comparable and sometimes worse performance 
> than using AVX. But if your machine supports AVX-512, it is definitely 
> beneficial to use AVX-512.
> 
> Hong (Mr.)
> 
> > On Apr 5, 2018, at 10:03 AM, Randall Mackie  wrote:
> > 
> > Dear PETSc users,
> > 
> > I’m curious if anyone else experiences problems using DMDAVecGetArrayF90 in 
> > conjunction with Intel compilers?
> > We have had many problems (typically 11 SEGV segmentation violations) when 
> > PETSc is compiled in optimize mode (with various combinations of options).
> > These same codes run valgrind clean with gfortran, so I assume this is an 
> > Intel bug, but before we submit a bug report I wanted to see if anyone else 
> > had similar experiences?
> > We have basically gone back and replaced our calls to DMDAVecGetArrayF90 
> > with calls to VecGetArrayF90 and pass those pointers into a “local” 
> > subroutine that works fine.
> > 
> > In case anyone is curious, the attached test code shows this behavior when 
> > PETSc is compiled with the following options:
> > 
> > ./configure \
> >  --with-clean=1 \
> >  --with-debugging=0 \
> >  --with-fortran=1 \
> >  --with-64-bit-indices \
> >  --download-mpich=../mpich-3.3a2.tar.gz \
> >  --with-blas-lapack-dir=/opt/intel/mkl \
> >  --with-cc=icc \
> >  --with-fc=ifort \
> >  --with-cxx=icc \
> >  --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
> >  --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
> >  --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2’ \
> > 
> > 
> > 
> > Thanks, Randy M.
> > 
> > 
> 
>

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-11 Thread Zhang, Hong 
Performance wise, I would suggest to use "-xAVX" instead of "-axcore-avx2". 
Based on our experience with running PETSc on a variety of Xeon processors 
(including KNL), using AVX2 yields comparable and sometimes worse performance 
than using AVX. But if your machine supports AVX-512, it is definitely 
beneficial to use AVX-512.

Hong (Mr.)

> On Apr 5, 2018, at 10:03 AM, Randall Mackie  wrote:
> 
> Dear PETSc users,
> 
> I’m curious if anyone else experiences problems using DMDAVecGetArrayF90 in 
> conjunction with Intel compilers?
> We have had many problems (typically 11 SEGV segmentation violations) when 
> PETSc is compiled in optimize mode (with various combinations of options).
> These same codes run valgrind clean with gfortran, so I assume this is an 
> Intel bug, but before we submit a bug report I wanted to see if anyone else 
> had similar experiences?
> We have basically gone back and replaced our calls to DMDAVecGetArrayF90 with 
> calls to VecGetArrayF90 and pass those pointers into a “local” subroutine 
> that works fine.
> 
> In case anyone is curious, the attached test code shows this behavior when 
> PETSc is compiled with the following options:
> 
> ./configure \
>  --with-clean=1 \
>  --with-debugging=0 \
>  --with-fortran=1 \
>  --with-64-bit-indices \
>  --download-mpich=../mpich-3.3a2.tar.gz \
>  --with-blas-lapack-dir=/opt/intel/mkl \
>  --with-cc=icc \
>  --with-fc=ifort \
>  --with-cxx=icc \
>  --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
>  --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
>  --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2’ \
> 
> 
> 
> Thanks, Randy M.
> 
>

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-11 Thread Randall Mackie 
Hi Satish,

Thanks to you and Jeff for taking a look at this. We will submit a bug report 
to Intel.

Randy


> On Apr 11, 2018, at 2:40 AM, Satish Balay  wrote:
> 
> On Tue, 10 Apr 2018, Jeff Hammond wrote:
> 
>> Can you try on a non-KNL host?  It's a bug either way but I want to
>> determine if KNL host is the issue.
> 
> Breaks on 'E5-2695 v4' aswell (bebop.lcrc) with '-axcore-avx2' and 'icc (ICC) 
> 18.0.1 20171018'
> 
>> Based only what I see below, Randy doesn't seem to be reporting a
>> KNL-specific issue.  Is that incorrect?
> 
> Hardware details weren't mentioned in this thread.
> 
>> Again, there is clearly a bug here, but it helps to localize the problem as
>> much as possible.
> 
>> On Thu, 5 Apr 2018, Randall Mackie wrote:
> 
> so I assume this is an Intel bug, but before we submit a bug
> report I wanted to see if anyone else had similar experiences?
> 
> Randy,
> 
> I'll leave this to you to file a report with Intel.
> 
> Thanks,
> Satish

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-10 Thread Satish Balay 
On Tue, 10 Apr 2018, Jeff Hammond wrote:

> Can you try on a non-KNL host?  It's a bug either way but I want to
> determine if KNL host is the issue.

Breaks on 'E5-2695 v4' aswell (bebop.lcrc) with '-axcore-avx2' and 'icc (ICC) 
18.0.1 20171018'

> Based only what I see below, Randy doesn't seem to be reporting a
> KNL-specific issue.  Is that incorrect?

Hardware details weren't mentioned in this thread.

> Again, there is clearly a bug here, but it helps to localize the problem as
> much as possible.

> > > > > On Thu, 5 Apr 2018, Randall Mackie wrote:

> > > > so I assume this is an Intel bug, but before we submit a bug
> > > > report I wanted to see if anyone else had similar experiences?

Randy,

I'll leave this to you to file a report with Intel.

Thanks,
Satish

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-10 Thread Jeff Hammond 
On Tue, Apr 10, 2018 at 4:39 PM, Satish Balay  wrote:

> On Tue, 10 Apr 2018, Jeff Hammond wrote:
>
> > This should generate an SSE2 binary:
> >
> > 'COPTFLAGS=-g',
> > 'FOPTFLAGS=-g',
> >
> > This should generate a KNL binary:
> >
> > 'COPTFLAGS=-g -xMIC-AVX512 -O3',
> > 'FOPTFLAGS=-g -xMIC-AVX512 -O3',
> >
> > This should generate a SSE2 binary that also supports CORE-AVX2 dispatch.
> >
> > '--COPTFLAGS=-g -axcore-avx2',
> > '--FOPTFLAGS=-g -axcore-avx2',
> >
> > I don't see a good reason for the third option to fail.  Please report
> this
> > bug to Intel.
> >
> > You might also verify that this works:
> >
> > '--COPTFLAGS=-g -xCORE-AVX2',
> > '--FOPTFLAGS=-g -xCORE-AVX2',
>
> This fails the same way as  -axcore-avx2
>
>
Can you try on a non-KNL host?  It's a bug either way but I want to
determine if KNL host is the issue.


> >
> > In general, one should avoid compiling for SSE on KNL, because SSE-AVX
> > transition penalties need to be avoided (google should find the details).
> > Are you trying to generate a single binary that is portable to ancient
> > Core/Xeon and KNL?
>
> My usage here is to reproduce this issue reported by Randy - assumed the
> knl box we have is the easiest way..
>
>
Based only what I see below, Randy doesn't seem to be reporting a
KNL-specific issue.  Is that incorrect?

I strongly recommend generating KNL-specific binaries for KNL, in which
case, the original issue should be investigated on non-KNL systems.

Again, there is clearly a bug here, but it helps to localize the problem as
much as possible.

Jeff


> Satish
>
> >  I recommend that you use AVX (Sandy Bridge) -
> > preferably AVX2 (Haswell) - as your oldest ISA target when generating a
> > portable binary that includes KNL support.
> >
> > Jeff
> >
> > On Tue, Apr 10, 2018 at 2:23 PM, Satish Balay  wrote:
> >
> > > I tried a few builds with:
> > >
> > > '--with-64-bit-indices=1',
> > > '--with-memalign=64',
> > > '--with-blaslapack-dir=/home/intel/18/compilers_and_
> > > libraries_2018.0.128/linux/mkl',
> > > '--with-cc=icc',
> > > '--with-fc=ifort',
> > > '--with-cxx=0',
> > > '--with-debugging=0',
> > > '--with-mpi=0',
> > >
> > > And then changed the OPTFLAGS:
> > >
> > > 1.  'basic -g' - works fine
> > >
> > > 'COPTFLAGS=-g',
> > > 'FOPTFLAGS=-g',
> > >
> > > 2. 'avx512' - works fine
> > >
> > > 'COPTFLAGS=-g -xMIC-AVX512 -O3',
> > > 'FOPTFLAGS=-g -xMIC-AVX512 -O3',
> > >
> > > 3. 'avx2' - breaks.
> > >
> > > '--COPTFLAGS=-g -axcore-avx2',
> > > '--FOPTFLAGS=-g -axcore-avx2',
> > >
> > > with a breakpoint at dmdavecrestorearrayf903_() in gdb - I see - the
> > > stack is fine during the first call to dmdavecrestorearrayf903_() -
> > > but is corrupted when it goes to the second call to
> > > dmdavecrestorearrayf903_() i.e ierr=0x7fffb4a0 changes to
> > > ierr=0x0]
> > >
> > > >>
> > >
> > > Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 ,
> > > v=0x6030c0 , a=0x401abd ,
> > > ierr=0x7fffb4a0) at /home/petsc/petsc.barry-test/
> > > src/dm/impls/da/f90-custom/zda1f90.c:153
> > > 153 {
> > > (gdb) where
> > > #0  dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030c0
> > > , a=0x401abd , ierr=0x7fffb4a0)
> > > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/
> > > zda1f90.c:153
> > > #1  0x00401abd in test () at ex1f.F90:80
> > > #2  0x004011ae in main ()
> > > #3  0x7fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6
> > > #4  0x004010b9 in _start ()
> > > (gdb) c
> > > Continuing.
> > >
> > > Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 ,
> > > v=0x6030b8 , a=0x401ada , ierr=0x0)
> > > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/
> > > zda1f90.c:153
> > > 153 {
> > > (gdb) where
> > > #0  dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030b8
> > > , a=0x401ada , ierr=0x0)
> > > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/
> > > zda1f90.c:153
> > > #1  0x00401ada in test () at ex1f.F90:81
> > > #2  0x004011ae in main ()
> > > #3  0x7fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6
> > > #4  0x004010b9 in _start ()
> > > (gdb)
> > >
> > > >
> > >
> > > Its not clear to me why this happens. [and why it would work with
> > > -xMIC-AVX512 but breaks with -axcore-avx2].
> > >
> > > Perhaps Richard, Jeff have better insight on this.
> > >
> > > BTW: The above run is with:
> > >
> > > bash-4.2$ icc --version
> > > icc (ICC) 18.0.0 20170811
> > >
> > > Satish
> > >
> > > On Mon, 9 Apr 2018, Satish Balay wrote:
> > >
> > > > I'm able to reproduce this problem on knl box [with the attached test
> > > code]. But it goes away if I rebuild without the option
> > > --with-64-bit-indices.
> > > >
> > > > Will have to check further..
> > > >
> > > > Satish
> > > >
> > > >
> > > > On Thu, 5 Apr 2018, Randall Mackie wrote:
> > > >
> > > > > Dear PETSc users,
> > > > >
> > > > > I’m

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-10 Thread Satish Balay 
On Tue, 10 Apr 2018, Jeff Hammond wrote:

> This should generate an SSE2 binary:
> 
> 'COPTFLAGS=-g',
> 'FOPTFLAGS=-g',
> 
> This should generate a KNL binary:
> 
> 'COPTFLAGS=-g -xMIC-AVX512 -O3',
> 'FOPTFLAGS=-g -xMIC-AVX512 -O3',
> 
> This should generate a SSE2 binary that also supports CORE-AVX2 dispatch.
> 
> '--COPTFLAGS=-g -axcore-avx2',
> '--FOPTFLAGS=-g -axcore-avx2',
> 
> I don't see a good reason for the third option to fail.  Please report this
> bug to Intel.
> 
> You might also verify that this works:
> 
> '--COPTFLAGS=-g -xCORE-AVX2',
> '--FOPTFLAGS=-g -xCORE-AVX2',

This fails the same way as  -axcore-avx2

> 
> In general, one should avoid compiling for SSE on KNL, because SSE-AVX
> transition penalties need to be avoided (google should find the details).
> Are you trying to generate a single binary that is portable to ancient
> Core/Xeon and KNL?

My usage here is to reproduce this issue reported by Randy - assumed the knl 
box we have is the easiest way..

Satish

>  I recommend that you use AVX (Sandy Bridge) -
> preferably AVX2 (Haswell) - as your oldest ISA target when generating a
> portable binary that includes KNL support.
> 
> Jeff
> 
> On Tue, Apr 10, 2018 at 2:23 PM, Satish Balay  wrote:
> 
> > I tried a few builds with:
> >
> > '--with-64-bit-indices=1',
> > '--with-memalign=64',
> > '--with-blaslapack-dir=/home/intel/18/compilers_and_
> > libraries_2018.0.128/linux/mkl',
> > '--with-cc=icc',
> > '--with-fc=ifort',
> > '--with-cxx=0',
> > '--with-debugging=0',
> > '--with-mpi=0',
> >
> > And then changed the OPTFLAGS:
> >
> > 1.  'basic -g' - works fine
> >
> > 'COPTFLAGS=-g',
> > 'FOPTFLAGS=-g',
> >
> > 2. 'avx512' - works fine
> >
> > 'COPTFLAGS=-g -xMIC-AVX512 -O3',
> > 'FOPTFLAGS=-g -xMIC-AVX512 -O3',
> >
> > 3. 'avx2' - breaks.
> >
> > '--COPTFLAGS=-g -axcore-avx2',
> > '--FOPTFLAGS=-g -axcore-avx2',
> >
> > with a breakpoint at dmdavecrestorearrayf903_() in gdb - I see - the
> > stack is fine during the first call to dmdavecrestorearrayf903_() -
> > but is corrupted when it goes to the second call to
> > dmdavecrestorearrayf903_() i.e ierr=0x7fffb4a0 changes to
> > ierr=0x0]
> >
> > >>
> >
> > Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 ,
> > v=0x6030c0 , a=0x401abd ,
> > ierr=0x7fffb4a0) at /home/petsc/petsc.barry-test/
> > src/dm/impls/da/f90-custom/zda1f90.c:153
> > 153 {
> > (gdb) where
> > #0  dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030c0
> > , a=0x401abd , ierr=0x7fffb4a0)
> > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/
> > zda1f90.c:153
> > #1  0x00401abd in test () at ex1f.F90:80
> > #2  0x004011ae in main ()
> > #3  0x7fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6
> > #4  0x004010b9 in _start ()
> > (gdb) c
> > Continuing.
> >
> > Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 ,
> > v=0x6030b8 , a=0x401ada , ierr=0x0)
> > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/
> > zda1f90.c:153
> > 153 {
> > (gdb) where
> > #0  dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030b8
> > , a=0x401ada , ierr=0x0)
> > at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/
> > zda1f90.c:153
> > #1  0x00401ada in test () at ex1f.F90:81
> > #2  0x004011ae in main ()
> > #3  0x7fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6
> > #4  0x004010b9 in _start ()
> > (gdb)
> >
> > >
> >
> > Its not clear to me why this happens. [and why it would work with
> > -xMIC-AVX512 but breaks with -axcore-avx2].
> >
> > Perhaps Richard, Jeff have better insight on this.
> >
> > BTW: The above run is with:
> >
> > bash-4.2$ icc --version
> > icc (ICC) 18.0.0 20170811
> >
> > Satish
> >
> > On Mon, 9 Apr 2018, Satish Balay wrote:
> >
> > > I'm able to reproduce this problem on knl box [with the attached test
> > code]. But it goes away if I rebuild without the option
> > --with-64-bit-indices.
> > >
> > > Will have to check further..
> > >
> > > Satish
> > >
> > >
> > > On Thu, 5 Apr 2018, Randall Mackie wrote:
> > >
> > > > Dear PETSc users,
> > > >
> > > > I’m curious if anyone else experiences problems using
> > DMDAVecGetArrayF90 in conjunction with Intel compilers?
> > > > We have had many problems (typically 11 SEGV segmentation violations)
> > when PETSc is compiled in optimize mode (with various combinations of
> > options).
> > > > These same codes run valgrind clean with gfortran, so I assume this is
> > an Intel bug, but before we submit a bug report I wanted to see if anyone
> > else had similar experiences?
> > > > We have basically gone back and replaced our calls to
> > DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers
> > into a “local” subroutine that works fine.
> > > >
> > > > In case anyone is curious, the attached test code shows this behavior
> > when PETSc is compiled with the following optio

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-10 Thread Jeff Hammond 
This should generate an SSE2 binary:

'COPTFLAGS=-g',
'FOPTFLAGS=-g',

This should generate a KNL binary:

'COPTFLAGS=-g -xMIC-AVX512 -O3',
'FOPTFLAGS=-g -xMIC-AVX512 -O3',

This should generate a SSE2 binary that also supports CORE-AVX2 dispatch.

'--COPTFLAGS=-g -axcore-avx2',
'--FOPTFLAGS=-g -axcore-avx2',

I don't see a good reason for the third option to fail.  Please report this
bug to Intel.

You might also verify that this works:

'--COPTFLAGS=-g -xCORE-AVX2',
'--FOPTFLAGS=-g -xCORE-AVX2',

In general, one should avoid compiling for SSE on KNL, because SSE-AVX
transition penalties need to be avoided (google should find the details).
Are you trying to generate a single binary that is portable to ancient
Core/Xeon and KNL?  I recommend that you use AVX (Sandy Bridge) -
preferably AVX2 (Haswell) - as your oldest ISA target when generating a
portable binary that includes KNL support.

Jeff

On Tue, Apr 10, 2018 at 2:23 PM, Satish Balay  wrote:

> I tried a few builds with:
>
> '--with-64-bit-indices=1',
> '--with-memalign=64',
> '--with-blaslapack-dir=/home/intel/18/compilers_and_
> libraries_2018.0.128/linux/mkl',
> '--with-cc=icc',
> '--with-fc=ifort',
> '--with-cxx=0',
> '--with-debugging=0',
> '--with-mpi=0',
>
> And then changed the OPTFLAGS:
>
> 1.  'basic -g' - works fine
>
> 'COPTFLAGS=-g',
> 'FOPTFLAGS=-g',
>
> 2. 'avx512' - works fine
>
> 'COPTFLAGS=-g -xMIC-AVX512 -O3',
> 'FOPTFLAGS=-g -xMIC-AVX512 -O3',
>
> 3. 'avx2' - breaks.
>
> '--COPTFLAGS=-g -axcore-avx2',
> '--FOPTFLAGS=-g -axcore-avx2',
>
> with a breakpoint at dmdavecrestorearrayf903_() in gdb - I see - the
> stack is fine during the first call to dmdavecrestorearrayf903_() -
> but is corrupted when it goes to the second call to
> dmdavecrestorearrayf903_() i.e ierr=0x7fffb4a0 changes to
> ierr=0x0]
>
> >>
>
> Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 ,
> v=0x6030c0 , a=0x401abd ,
> ierr=0x7fffb4a0) at /home/petsc/petsc.barry-test/
> src/dm/impls/da/f90-custom/zda1f90.c:153
> 153 {
> (gdb) where
> #0  dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030c0
> , a=0x401abd , ierr=0x7fffb4a0)
> at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/
> zda1f90.c:153
> #1  0x00401abd in test () at ex1f.F90:80
> #2  0x004011ae in main ()
> #3  0x7fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6
> #4  0x004010b9 in _start ()
> (gdb) c
> Continuing.
>
> Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 ,
> v=0x6030b8 , a=0x401ada , ierr=0x0)
> at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/
> zda1f90.c:153
> 153 {
> (gdb) where
> #0  dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030b8
> , a=0x401ada , ierr=0x0)
> at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/
> zda1f90.c:153
> #1  0x00401ada in test () at ex1f.F90:81
> #2  0x004011ae in main ()
> #3  0x7fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6
> #4  0x004010b9 in _start ()
> (gdb)
>
> >
>
> Its not clear to me why this happens. [and why it would work with
> -xMIC-AVX512 but breaks with -axcore-avx2].
>
> Perhaps Richard, Jeff have better insight on this.
>
> BTW: The above run is with:
>
> bash-4.2$ icc --version
> icc (ICC) 18.0.0 20170811
>
> Satish
>
> On Mon, 9 Apr 2018, Satish Balay wrote:
>
> > I'm able to reproduce this problem on knl box [with the attached test
> code]. But it goes away if I rebuild without the option
> --with-64-bit-indices.
> >
> > Will have to check further..
> >
> > Satish
> >
> >
> > On Thu, 5 Apr 2018, Randall Mackie wrote:
> >
> > > Dear PETSc users,
> > >
> > > I’m curious if anyone else experiences problems using
> DMDAVecGetArrayF90 in conjunction with Intel compilers?
> > > We have had many problems (typically 11 SEGV segmentation violations)
> when PETSc is compiled in optimize mode (with various combinations of
> options).
> > > These same codes run valgrind clean with gfortran, so I assume this is
> an Intel bug, but before we submit a bug report I wanted to see if anyone
> else had similar experiences?
> > > We have basically gone back and replaced our calls to
> DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers
> into a “local” subroutine that works fine.
> > >
> > > In case anyone is curious, the attached test code shows this behavior
> when PETSc is compiled with the following options:
> > >
> > > ./configure \
> > >   --with-clean=1 \
> > >   --with-debugging=0 \
> > >   --with-fortran=1 \
> > >   --with-64-bit-indices \
> > >   --download-mpich=../mpich-3.3a2.tar.gz \
> > >   --with-blas-lapack-dir=/opt/intel/mkl \
> > >   --with-cc=icc \
> > >   --with-fc=ifort \
> > >   --with-cxx=icc \
> > >   --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
> > >   --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
> > >   --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2’ \
> > >
> > >
> > >
> > > Thanks, Randy

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-10 Thread Satish Balay 
I tried a few builds with:

'--with-64-bit-indices=1',
'--with-memalign=64',

'--with-blaslapack-dir=/home/intel/18/compilers_and_libraries_2018.0.128/linux/mkl',
'--with-cc=icc',
'--with-fc=ifort',
'--with-cxx=0',
'--with-debugging=0',
'--with-mpi=0',

And then changed the OPTFLAGS:

1.  'basic -g' - works fine

'COPTFLAGS=-g',
'FOPTFLAGS=-g',

2. 'avx512' - works fine

'COPTFLAGS=-g -xMIC-AVX512 -O3',
'FOPTFLAGS=-g -xMIC-AVX512 -O3',

3. 'avx2' - breaks.

'--COPTFLAGS=-g -axcore-avx2',
'--FOPTFLAGS=-g -axcore-avx2',
 
with a breakpoint at dmdavecrestorearrayf903_() in gdb - I see - the
stack is fine during the first call to dmdavecrestorearrayf903_() -
but is corrupted when it goes to the second call to
dmdavecrestorearrayf903_() i.e ierr=0x7fffb4a0 changes to
ierr=0x0]

>>

Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030c0 
, a=0x401abd , 
ierr=0x7fffb4a0) at 
/home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/zda1f90.c:153
153 {
(gdb) where
#0  dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030c0 
, a=0x401abd , ierr=0x7fffb4a0)
at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/zda1f90.c:153
#1  0x00401abd in test () at ex1f.F90:80
#2  0x004011ae in main ()
#3  0x7fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6
#4  0x004010b9 in _start ()
(gdb) c
Continuing.

Breakpoint 1, dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030b8 
, a=0x401ada , ierr=0x0)
at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/zda1f90.c:153
153 {
(gdb) where
#0  dmdavecrestorearrayf903_ (da=0x603098 , v=0x6030b8 
, a=0x401ada , ierr=0x0)
at /home/petsc/petsc.barry-test/src/dm/impls/da/f90-custom/zda1f90.c:153
#1  0x00401ada in test () at ex1f.F90:81
#2  0x004011ae in main ()
#3  0x7fffef1c3c05 in __libc_start_main () from /lib64/libc.so.6
#4  0x004010b9 in _start ()
(gdb) 

>

Its not clear to me why this happens. [and why it would work with -xMIC-AVX512 
but breaks with -axcore-avx2].

Perhaps Richard, Jeff have better insight on this.

BTW: The above run is with:

bash-4.2$ icc --version
icc (ICC) 18.0.0 20170811

Satish

On Mon, 9 Apr 2018, Satish Balay wrote:

> I'm able to reproduce this problem on knl box [with the attached test code]. 
> But it goes away if I rebuild without the option --with-64-bit-indices.
> 
> Will have to check further..
> 
> Satish
> 
> 
> On Thu, 5 Apr 2018, Randall Mackie wrote:
> 
> > Dear PETSc users,
> > 
> > I’m curious if anyone else experiences problems using DMDAVecGetArrayF90 in 
> > conjunction with Intel compilers?
> > We have had many problems (typically 11 SEGV segmentation violations) when 
> > PETSc is compiled in optimize mode (with various combinations of options).
> > These same codes run valgrind clean with gfortran, so I assume this is an 
> > Intel bug, but before we submit a bug report I wanted to see if anyone else 
> > had similar experiences?
> > We have basically gone back and replaced our calls to DMDAVecGetArrayF90 
> > with calls to VecGetArrayF90 and pass those pointers into a “local” 
> > subroutine that works fine.
> > 
> > In case anyone is curious, the attached test code shows this behavior when 
> > PETSc is compiled with the following options:
> > 
> > ./configure \
> >   --with-clean=1 \
> >   --with-debugging=0 \
> >   --with-fortran=1 \
> >   --with-64-bit-indices \
> >   --download-mpich=../mpich-3.3a2.tar.gz \
> >   --with-blas-lapack-dir=/opt/intel/mkl \
> >   --with-cc=icc \
> >   --with-fc=ifort \
> >   --with-cxx=icc \
> >   --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
> >   --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
> >   --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2’ \
> > 
> > 
> > 
> > Thanks, Randy M.
> > 
> > 
>

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-09 Thread Satish Balay 
I'm able to reproduce this problem on knl box [with the attached test code]. 
But it goes away if I rebuild without the option --with-64-bit-indices.

Will have to check further..

Satish


On Thu, 5 Apr 2018, Randall Mackie wrote:

> Dear PETSc users,
> 
> I’m curious if anyone else experiences problems using DMDAVecGetArrayF90 in 
> conjunction with Intel compilers?
> We have had many problems (typically 11 SEGV segmentation violations) when 
> PETSc is compiled in optimize mode (with various combinations of options).
> These same codes run valgrind clean with gfortran, so I assume this is an 
> Intel bug, but before we submit a bug report I wanted to see if anyone else 
> had similar experiences?
> We have basically gone back and replaced our calls to DMDAVecGetArrayF90 with 
> calls to VecGetArrayF90 and pass those pointers into a “local” subroutine 
> that works fine.
> 
> In case anyone is curious, the attached test code shows this behavior when 
> PETSc is compiled with the following options:
> 
> ./configure \
>   --with-clean=1 \
>   --with-debugging=0 \
>   --with-fortran=1 \
>   --with-64-bit-indices \
>   --download-mpich=../mpich-3.3a2.tar.gz \
>   --with-blas-lapack-dir=/opt/intel/mkl \
>   --with-cc=icc \
>   --with-fc=ifort \
>   --with-cxx=icc \
>   --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
>   --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
>   --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2’ \
> 
> 
> 
> Thanks, Randy M.
> 
>

Re: [petsc-users] Problems with DMDAVecGetArrayF90 + Intel

2018-04-05 Thread Fabian.Jakub 
Hi,

Not sure if this is related to your current issues but
I once had issues with this particular function with IBM XLF compilers
but never got to a solution because...

a) VecGetArrayF90 also worked for me
b) and the IBM Machine got switched off

Anyway, I did not have any issues since then... not with intel nor with
gfortran compilers.

The helpdesk back then created a petsc bug report but I think the issue
was never resolved...

For the sake of completeness, I am attaching the messages we had then,
maybe the debugging efforts are related to your problems?

Good Luck,

Fabian


On 04/05/2018 05:03 PM, Randall Mackie wrote:
> Dear PETSc users,
>
> I’m curious if anyone else experiences problems using
DMDAVecGetArrayF90 in conjunction with Intel compilers?
> We have had many problems (typically 11 SEGV segmentation violations)
when PETSc is compiled in optimize mode (with various combinations of
options).
> These same codes run valgrind clean with gfortran, so I assume this is
an Intel bug, but before we submit a bug report I wanted to see if
anyone else had similar experiences?
> We have basically gone back and replaced our calls to
DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers
into a “local” subroutine that works fine.
>
> In case anyone is curious, the attached test code shows this behavior
when PETSc is compiled with the following options:
>
> ./configure \
>   --with-clean=1 \
>   --with-debugging=0 \
>   --with-fortran=1 \
>   --with-64-bit-indices \
>   --download-mpich=../mpich-3.3a2.tar.gz \
>   --with-blas-lapack-dir=/opt/intel/mkl \
>   --with-cc=icc \
>   --with-fc=ifort \
>   --with-cxx=icc \
>   --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
>   --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \
>   --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2’ \
>
>
>
> Thanks, Randy M.
>
--- Begin Message ---
Hello,
My userid is b380246.

I would like to run my Fortran model, which depends on a recent version
of PETSC, http://www.mcs.anl.gov/petsc/.
While it compiles fine, it crashes at a certain point.
My knowledge of the code is however very limited and I am at the same
time not familiar with the Blizzard machine.

I hoped that, before I send this as a bug request to the Petsc People,
you might be able to have a look at compile flags if I am doing
something silly or maybe if you could even reproduce the error.

In the following, I try to elaborate on the steps I used to compile PETSC:

# load modules... while I tried also the default ones, I currently have:

Currently Loaded Modulefiles:

IBM/xlf14.1.0.8

IBM/xlC12.1.0.9

IBM/xlc12.1.0.9

GCC/gcc-4.5.1

NAG/5.1.340

NCAR/ncarg5.1.0 

 
 

IGES/grads2.0.a6

 
 

UNITE/1.0   

 
 

./default   

 
 

NETCDF/4.2.1.1


# Get current petsc version I also tried stable version 3.5.2... and
git branch master and next here however check out git branch
stable/maint:

/sw/aix61/git-1.7.4.1/bin/git clone
ssh://g...@bitbucket.org/petsc/petsc.git petsc -b maint


#export PETSC_DIR to point to the just created git directory

export PETSC_DIR=$(pwd)/petsc




# The following is a script configure petsc to compile on blizzard:

export PETSC_DIR=$(pwd)

export PETSC_ARCH=debug


CONFIGURE_SCRIPT="conftest-$PETSC_ARCH"

RECONFIGURE_SCRIPT="reconfigure-$PETSC_ARCH.py"

rm $RECONFIGURE_SCRIPT $CONFIGURE_SCRIPT


make allclean

./configure \

  --with-make-np=8 \

  --with-batch=1 \

  --with-mpi-shared=0 \

  --with-cc=mpcc_r \

  --with-cxx=mpCC_r \

  --with-fc="mpxlf_r"\

  --with-debugging=1 \

  --known-mpi-shared-libraries=0  \

  --with-shared-libraries=0   \

  --with-fortran  \

  --with-fortran-interfaces   \

  --download-sowing-configure-options='CFLAGS=-maix64 CXXFLAGS=-maix64
LDFLAGS=-maix64 CPPFLAGS=-maix64' \

  --with-c2html=0 \

  --with-cmake=$(which cmake) \

  --with-blas-lapack-dir=/sw/aix53/lapack-3.2.0/ \

  COPTFLAGS='-g -O0 ' \

  FOPTFLAGS='-qextname -qwarn64  -O0 -qfullpath -C -g -qextchk -qnosave  ' \

  --LIBS="-L/sw/aix53/lapack-3.2.0/lib -llapack -lessl -lblas"


  if [ ! -e $CONFIGURE_SCRIPT ] ; then

echo 'Configure failed in creating conftest script'

exit

  fi


cat > petsc_conf-$PETSC_ARCH.ll << EOF

#!/client/bin/ksh

# 1 node, ST mode (Single Threading)

# 1 MPI Processes

#---

#

# @ shell =