Re: [Mpi-forum] Giving up on C11 _Generic

[Jeff Squyres (jsquyres) via mpi-forum]

Some points:

1. C11 _Generic support in an MPI implementation has to be optional, at least 
for now, because not all compilers support C11 _Generic (just like the F08 
bindings are still optional, and just like aspects of the C++ bindings were 
optional back in the '90s when C++ compilers still hadn't implemented all of 
the required features).

2. Given that an MPI implementation and/or compiler may or may not support 
MPI_Count-enabled C11 _Generic-style polymorphism for functions like MPI_Send, 
and given the defined behavior of the C/C++ language in terms of casting a 
large int down into a small int, it is quite possible that a valid MPI-4 
enabled application (i.e., using MPI_Count params with >2^32 values for 
functions like MPI_Send) with a valid MPI implementation can get truncated at 
run time.   **That is bad** (Dan's email exemplified this point)

--> Jeff Hammond has pointed out that there are some compiler flags (outside of 
-Wall) that can emit such warnings.
--> Jim Dinan has pointed out that implementations can add additional sizeof 
checks via macros. 

Fair points.  However, my response to both is: this is just making it hard for 
the average user to utilize the functionality correctly.  Sure, we can say 
"it's the users' fault", but the fact is that the language polymorphism 
mechanism itself is pretty weak and downright confusing (as a Forum, we've been 
discussing this for ***months*** -- how will average users/programmers to sort 
this stuff out?).

Let me ask a simple question: how will users to write portable MPI programs in 
C with large count values?

Answer: they will explicitly call MPI_Send_x(), and not rely on C11 _Generic.

Which then raises the question: what's the point of using C11 _Generic?  Its 
main functionality can lead to [potentially] silent run-time errors, and/or 
require additional error checking in every single code path by the 
implementation.  That just seems like bad design, IMNSHO.  That's why the WG 
decided to bring this to the Forum list (especially given the compressed 
timeframe for MPI-4).



> On Aug 8, 2019, at 7:37 AM, Jim Dinan via mpi-forum 
>  wrote:
> 
> In fairness, unsafe conversions are a feature of the C language, I assume 
> since very early on.  Assignment gets a pass to do horribly unsafe 
> conversions.  Unfortunately, when calling a function (with a prototype), the 
> argument expressions are converted as if by assignment, which is the 
> underlying cause for the truncation sadness.
> 
>  ~Jim.
> 
> On Thu, Aug 8, 2019 at 9:45 AM Jeff Hammond via mpi-forum 
>  wrote:
> That you use code that does unsafe conversions really has nothing to do with 
> the business of the MPI Forum. 
> 
> Again, if you think C89 is the best C, then use and teach that. No one here 
> is trying to make you use C11.
> 
> Jeff
> 
> > On Aug 8, 2019, at 5:56 AM, N.M. Maclaren via mpi-forum 
> >  wrote:
> > 
> >> On Aug 7 2019, Jeff Hammond via mpi-forum wrote:
> >> 
> >> "silently truncated at run time" is trivially addressed with -Wconversion
> >> or -Wshorten-64-to-32.  The example program below is addressed by this.
> > 
> > Unfortunately, no.  While I have no trouble using such options on MY
> > code, I have frequently found them unusable on imported packages, because
> > of the flood of non-errors they generate.  For example, the following
> > code is both common and reasonable, and includes narrowing conversions:
> > 
> >   int i;
> >   i = sizeof(double);
> >   float x;
> >   x = 1.0;
> ___
> mpi-forum mailing list
> mpi-forum@lists.mpi-forum.org
> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum
> ___
> mpi-forum mailing list
> mpi-forum@lists.mpi-forum.org
> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum


-- 
Jeff Squyres
jsquy...@cisco.com

___
mpi-forum mailing list
mpi-forum@lists.mpi-forum.org
https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

Re: [Mpi-forum] Giving up on C11 _Generic

[N.M. Maclaren via mpi-forum]

On Aug 8 2019, Jeff Hammond wrote:

That you use code that does unsafe conversions really has nothing to do 
with the business of the MPI Forum.


Again, if you think C89 is the best C, then use and teach that. No one 
here is trying to make you use C11.


Please don't be infantile.  I am thankfully retired, but spent 43 years
supporting scientific computing, including as our organisation's C expert,
and 15 years on SC22WG14.  As I have said (twice now, if I recall) "This
is not about MY code."

Jim Dinan has pointed out that unsafe conversions are a feature of the
C language, and I can confirm that they have been basic to the language
since K C and still are, in both C and C++, and are relied on in the
vast majority of the many dozens of large C codes I have ported, worked
on and advised people with.

He actually understated the problem, as assignment is not the only factor
that makes this such a foul issue.


Regards,
Nick Maclaren.

___
mpi-forum mailing list
mpi-forum@lists.mpi-forum.org
https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

Re: [Mpi-forum] Giving up on C11 _Generic

[Jim Dinan via mpi-forum]

In fairness, unsafe conversions are a feature of the C language, I assume
since very early on.  Assignment gets a pass to do horribly unsafe
conversions.  Unfortunately, when calling a function (with a prototype),
the argument expressions are converted as if by assignment, which is the
underlying cause for the truncation sadness.

 ~Jim.

On Thu, Aug 8, 2019 at 9:45 AM Jeff Hammond via mpi-forum <
mpi-forum@lists.mpi-forum.org> wrote:

> That you use code that does unsafe conversions really has nothing to do
> with the business of the MPI Forum.
>
> Again, if you think C89 is the best C, then use and teach that. No one
> here is trying to make you use C11.
>
> Jeff
>
> > On Aug 8, 2019, at 5:56 AM, N.M. Maclaren via mpi-forum <
> mpi-forum@lists.mpi-forum.org> wrote:
> >
> >> On Aug 7 2019, Jeff Hammond via mpi-forum wrote:
> >>
> >> "silently truncated at run time" is trivially addressed with
> -Wconversion
> >> or -Wshorten-64-to-32.  The example program below is addressed by this.
> >
> > Unfortunately, no.  While I have no trouble using such options on MY
> > code, I have frequently found them unusable on imported packages, because
> > of the flood of non-errors they generate.  For example, the following
> > code is both common and reasonable, and includes narrowing conversions:
> >
> >   int i;
> >   i = sizeof(double);
> >   float x;
> >   x = 1.0;
> ___
> mpi-forum mailing list
> mpi-forum@lists.mpi-forum.org
> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum
>
___
mpi-forum mailing list
mpi-forum@lists.mpi-forum.org
https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

Re: [Mpi-forum] Giving up on C11 _Generic

[Jeff Hammond via mpi-forum]

That you use code that does unsafe conversions really has nothing to do with 
the business of the MPI Forum. 

Again, if you think C89 is the best C, then use and teach that. No one here is 
trying to make you use C11.

Jeff

> On Aug 8, 2019, at 5:56 AM, N.M. Maclaren via mpi-forum 
>  wrote:
> 
>> On Aug 7 2019, Jeff Hammond via mpi-forum wrote:
>> 
>> "silently truncated at run time" is trivially addressed with -Wconversion
>> or -Wshorten-64-to-32.  The example program below is addressed by this.
> 
> Unfortunately, no.  While I have no trouble using such options on MY
> code, I have frequently found them unusable on imported packages, because
> of the flood of non-errors they generate.  For example, the following
> code is both common and reasonable, and includes narrowing conversions:
> 
>   int i;
>   i = sizeof(double);
>   float x;
>   x = 1.0;
___
mpi-forum mailing list
mpi-forum@lists.mpi-forum.org
https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

Re: [Mpi-forum] Giving up on C11 _Generic

[N.M. Maclaren via mpi-forum]

On Aug 7 2019, Jeff Squyres (jsquyres) via mpi-forum wrote:


After much debate, the BigCount WG has decided that C11 _Generic just 
isn't worth it. That's no reason to penalize Fortran, though.


That is certainly the simplest solution.  There may be others, but
see below.

At a wild guess, this is probably trappable by implementations (at some
effort, and needing deep knowledge of the standards and issues) for MPI
being called directly from conforming C compilers, but not from C++ ones
using their C interface.  And that is a very serious gotcha.


On Aug 7 2019, Anthony Skjellum via mpi-forum wrote:


Conclusion: You can't use non-compliant MPI-4 with C11 compilers that are
non-compliant with C11, except maybe with a special flag???
Is there more that I am missing?


It's a LOT more complicated than that, as I tried to explain earlier.

Even when I was most active in this area, it would have taken me many
hours, probably days, of investigation (including on systems to which I
had no access), studying at least the C99, C11, C++ and MPI standards in
great detail, and working out flow-charts, to be fairly sure of the
situation.  This area is a gibbering nightmare, and is a major reason
the UK voted "no" to C99.


On Aug 7 2019, Jeff Hammond via mpi-forum wrote:


"silently truncated at run time" is trivially addressed with -Wconversion
or -Wshorten-64-to-32.  The example program below is addressed by this.


Unfortunately, no.  While I have no trouble using such options on MY
code, I have frequently found them unusable on imported packages, because
of the flood of non-errors they generate.  For example, the following
code is both common and reasonable, and includes narrowing conversions:

   int i;
   i = sizeof(double);
   float x;
   x = 1.0;


On Aug 8 2019, Gropp, William D via mpi-forum wrote:

That -Wall is a lie is also true for gcc. Yes, this is a extremely poor 
design. If you want to provide -Wmostcommon then do that (and maybe 
abbreviate to -Wmost).


Agreed, but it's a digression here!  I believe that the gcc people agree,
too, but changing it now is just too evil.


Regards,
Nick Maclaren.

___
mpi-forum mailing list
mpi-forum@lists.mpi-forum.org
https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

Re: [Mpi-forum] Giving up on C11 _Generic

[Gropp, William D via mpi-forum]

That -Wall is a lie is also true for gcc. Yes, this is a extremely poor design. 
If you want to provide -Wmostcommon then do that (and maybe abbreviate to 
-Wmost).

Bill

William Gropp
Director and Chief Scientist, NCSA
Thomas M. Siebel Chair in Computer Science
University of Illinois Urbana-Champaign






On Aug 8, 2019, at 6:42 AM, HOLMES Daniel via mpi-forum 
mailto:mpi-forum@lists.mpi-forum.org>> wrote:

After a brief search that I never knew was necessary, I find that “all” is 
smaller than “everything” in clang world.
http://clang.llvm.org/docs/UsersManual.html#enabling-all-diagnostics

So, my question “why isn’t -Wconversion implied by -Wall?” might have an 
astonishing answer.

___
mpi-forum mailing list
mpi-forum@lists.mpi-forum.org
https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

Re: [Mpi-forum] Giving up on C11 _Generic

[HOLMES Daniel via mpi-forum]

Hi JeffH,

After a brief search that I never knew was necessary, I find that “all” is 
smaller than “everything” in clang world.
http://clang.llvm.org/docs/UsersManual.html#enabling-all-diagnostics

So, my question “why isn’t -Wconversion implied by -Wall?” might have an 
astonishing answer.

The resulting workflow is then:
1) notice the symptoms of a bug in the output/effect of the code
2) re-compile with -Weverything -Wno-c++98-compat -Wno-c++-compat
3) re-compile with -Weverything -Wno-c++98-compat -Wno-c++-compat 
-Wno-
4) re-compile with -Weverything -Wno-c++98-compat -Wno-c++-compat 
-Wno- -Wno-
5) address all remaining warnings by changing code
Experience and repeated exposure would permit combining steps 2-4.

This is documented, once you know to go look for it. However, it was not 
obvious to me that looking beyond “-Wall” was needed.

Whilst I agree wholeheartedly that “incorrect programs are incorrect”, I think 
designers of interfaces/standards have some responsibility to select a design 
that makes it easy to discover incorrect usage over one that makes it hard. 
Ideally, a design should be selected that makes it hard to form incorrect usage 
rather than one that makes misuse easy.

Cheers,
Dan.
—
Dr Daniel Holmes PhD
Architect (HPC Research)
d.hol...@epcc.ed.ac.uk
Phone: +44 (0) 131 651 3465
Mobile: +44 (0) 7940 524 088
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT
—
The University of Edinburgh is a charitable body, registered in Scotland, with 
registration number SC005336.
—

On 7 Aug 2019, at 21:14, Jeff Hammond via mpi-forum 
mailto:mpi-forum@lists.mpi-forum.org>> wrote:

I don't care that much about C11 _Generic, which is why I have always focused 
on a C99 solution to the large-count problem, but I disagree with your reasons 
for abandoning it.

"silently truncated at run time" is trivially addressed with -Wconversion or 
-Wshorten-64-to-32.  The example program below is addressed by this.

$ clang -Wshorten-64-to-32 -c truncation.c
truncation.c:10:9: warning: implicit conversion loses integer precision: 'long 
long' to 'int' [-Wshorten-64-to-32]
foo(i);
~~~ ^
1 warning generated.

$ gcc-9 -Wconversion -c truncation.c
truncation.c: In function 'main':
truncation.c:10:9: warning: conversion from 'long long int' to 'int' may change 
value [-Wconversion]
   10 | foo(i);
  | ^

$ icc -Wconversion -c truncation.c
truncation.c(10): warning #1682: implicit conversion of a 64-bit integral type 
to a smaller integral type (potential portability problem)
  foo(i);
  ^

In any case, incorrect programs are incorrect.  It is a 
quality-of-implementation issue whether C compilers and MPI libraries detect 
incorrect usage.  We have never designed MPI around people who can't follow 
directions and we should not start now.

Jeff

On Wed, Aug 7, 2019 at 6:59 AM Jeff Squyres (jsquyres) via mpi-forum 
mailto:mpi-forum@lists.mpi-forum.org>> wrote:
SHORT VERSION
=

Due to the possibility of silently introducing errors into user applications, 
the BigCount WG no longer thinks that C11 _Generic is a good idea.  We are 
therefore dropping that from our proposal.  The new proposal will therefore 
essentially just be the addition of a bunch of MPI_Count-enabled "_x" functions 
in C, combined with the addition of a bunch of polymorphic MPI_Count-enabled 
interfaces in Fortran.

MORE DETAIL
===

Joseph Schuchart raised a very important point in a recent mailing thread: the 
following C/C++ code does not raise a compiler warning:

-
#include 

static void foo(int j) {
printf("foo(j) = %d\n", j);
}

int main(int argc, char *argv[]) {
/* 8589934592LL == 2^33 */
long long i = 8589934592LL + 11;
foo(i);
return 0;
}
-

If you compile and run this program on a commodity x86-64 platform, a) you 
won't get a warning from the compiler, and b) you'll see "11" printed out.  I 
tried with gcc 9 and clang 8 -- both with the C and C++ compilers.  I even 
tried with "-Wall -pedantic".  No warnings.

This is because casting from a larger int type to a smaller int type is 
perfectly valid C/C++.

Because of this, there is a possibility that we could be silently introducing 
errors into user applications.  Consider:

1. An application upgrades its "count" parameters to type MPI_Count for all 
calls to MPI_Send.
   --> Recall that "MPI_Count" already exists in MPI-3.1, and is likely of type 
(long long) on commodity x86-64 platforms
2. The application then uses values in that "count" parameter that are greater 
than 2^32.

If the user's MPI implementation and compiler both support C11 _Generic, 
everything is great.

But if either the MPI implementation or the compiler do not support C11 
_Generic, ***the "count" value will be silently truncated at run time***.

This seems like a very bad idea, from a design standpoint.

We have therefore come full circle: we are back to adding a bunch 

Re: [Mpi-forum] Giving up on C11 _Generic

[HOLMES Daniel via mpi-forum]

Hi Jim,

Are we sure the function pointer “worked" in your example? It silently 
truncated the large value, because the function pointer bypasses the 
redirection macro.

Cheers,
Dan.
—
Dr Daniel Holmes PhD
Architect (HPC Research)
d.hol...@epcc.ed.ac.uk
Phone: +44 (0) 131 651 3465
Mobile: +44 (0) 7940 524 088
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT
—
The University of Edinburgh is a charitable body, registered in Scotland, with 
registration number SC005336.
—

On 7 Aug 2019, at 22:00, Jim Dinan via mpi-forum 
mailto:mpi-forum@lists.mpi-forum.org>> wrote:

Even simpler than this, we could just forward all calls to the MPI_Count 
interface (see below).  The int count argument should type convert to MPI_Count 
without issue.  Note that it still needs to be a function-like macro so that 
function pointers work.

Don't give up yet!  :D

 ~Jim.

#include 



typedef int MPI_Datatype;
typedef int MPI_Comm;

int MPI_Send(const void* buf, int count, MPI_Datatype datatype, int dest,
 int tag, MPI_Comm comm)
{
printf("MPI_Send(count = %d)\n", count);
return 0;
}

int MPI_Send_x(const void* buf, long long count, MPI_Datatype datatype, int 
dest,
   int tag, MPI_Comm comm)
{
printf("MPI_Send_x(count = %lld)\n", count);
return 0;
}

#define MPI_Send(buf, count, datatype, dest, tag, comm) MPI_Send_x(buf, count, 
datatype, dest, tag, comm)

int main(int argc, char *argv[]) {
/* 8589934592LL == 2^33 */
long long i = 8589934592LL + 11;
int ret;
int (*snd_ptr)(const void*, int, MPI_Datatype, int, int, MPI_Comm) = 
_Send;
ret = MPI_Send(NULL, i, 0, 0, 0, 0);
ret = MPI_Send(NULL, 5, 0, 0, 0, 0);
ret = (*snd_ptr)(NULL, i, 0, 0, 0, 0);
ret = (*snd_ptr)(NULL, 5, 0, 0, 0, 0);
return 0;

}

MPI_Send_x(count = 8589934603)
MPI_Send_x(count = 5)
MPI_Send(count = 11)

MPI_Send(count = 5)
___
mpi-forum mailing list
mpi-forum@lists.mpi-forum.org
https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

___
mpi-forum mailing list
mpi-forum@lists.mpi-forum.org
https://lists.mpi-forum.org/mailman/listinfo/mpi-forum