On Jun 16, 2005, at 2:58 PM, Ben Allan wrote:
Ah -- I thought that that would be a different issue (I presume you're
speaking of the compile/lib flags command, like gnome-config et
al.)...? Are you saying that the compile/lib flags should be
accessible from ompi_info in a fine-grained fashion as well? (they're
not right now -- only "compile flags" and "link flags")
As much info we can get, in as flexible a format as possible,
is the best thing I can imagine. And of course some attention
is needed so that from release to release, information output
is consistent with the prior releases, i.e. the labels don't
change gratuitously. What's called "c compiler" in one release
shouldn't be called ISO-C-compiler in the next.
New labels can be added, of course.
That's the intent.
That would obviate using the frameworks like gnome-config (which can
read arbitrary *Conf.sh files), or ... er... I swear there was another
one, but I can't seem to find it at the moment. I'm not saying that
this is necessarily a Bad Thing; it's just something else that would
need to be implemented.
Actually, i'm perfectly happy with a
ompiConf.sh provided there's an open-mpi-config that will
tell me where it sits (and maybe even query it for me).
The only reason to have something like ompiConf.sh is to use the
frameworks that already exist (like the gnome-conf thingy). I was only
tossing that out as an example -- I didn't know if you were looking to
use a standardized tool or didn't really care where the info came from
as long as there was a defined interface to it.
It sounds like the latter.
Did you look at the output when you run with the -parsable flag? (see
my other mail about this)
I tried that just now and it doesn't look different. Will check your
other mail (which apparently i haven't reached yet in the mail reader).
I take that back, apparently -parseable isn't recognized but -parsable
is.
Hah! Since there is bound to be confusion over this, I'll add a
synonym for both -parsable and -parseable (to-MAY-to, to-MAH-to).
A :-separated output results. tolerable. :)
Kudos to you all, actually. I haven't seen anything this useful
from the mpich team yet. {now matt can correct me...}
I have a strong sysadmin and build system background. I feel your pain.
I'm still not sure that you're getting what you want, though. Note
that there's two sets of flags provided by "ompi_info -all" -- the
flags that Open MPI was built with and the flags that are added by the
wrapper compilers. Are you just extracting the wrapper compiler
flags?
Are they sufficient?
It is useful to know both.
Also note that the wrapper compilers will report their flags to you as
well:
mpicc -showme
mpicc -showme:compile
mpicc -showme:link
As usual, mpicc --help doesn't show showme as an option.
The *only* flag that mpicc (and friends) recognizes is --showme.
*Everything* else is passed to the underlying compiler. We didn't want
to take the chance, for example, that --help was actually a valid flag
for the underlying compiler.
This kinda hamstrings the ability to add features into wrapper
compilers, but we can figure out something safe if we need to.
Finally, is there a reason you can't just use the wrapper compilers
themselves? They can even be layered with other compilers if
necessary. Unless there's a technical reason that you can't, I would
strongly advise using the wrapper compilers -- we wrote them for
exactly this purpose.
I love it when compiler wrappers work.
But in the context of
multi-language builds,
cranky c++ and fortran compilers competing for who gets to link
either the executable or construct the shared library,
mis-installations by sysadmins,
portability to horrors like AIX, etc, all wrappers are
taken with a grain of salt.
I guess I still don't understand why -- all the reasons you cited above
are going to be problematic regardless of whether you're using a
wrapper compiler or not. All that mpicc (and friends) do is add the
proper -I, -L, -l, and other flags (like -pthread). That's it. Which
you use to link your application / create your shared library is still
up to you (e.g., mpicc/c++/f77/f90), for example.
I remember that MPI wrapper compilers in the mid- and late-90's were
pretty crappy. But I think they're all gotten pretty reasonable of
late (I could be wrong here, though...?).
More specifically: if all that you're doing is extracting the flags
from the wrapper compilers and inserting them in your own build
process, are you really doing anything different than what the wrapper
compiler is doing?
Here's a snipit from OMPI's wrapper compiler documentation (something
similar will someday make into a man page, similar to LAM's):
/// At most, we end up with a command line in the following form:
///
/// [compiler(*)] [Xppflags(*)] [Xflags(*)] [user args] [ldflags(*)]
[libs(*)]
///
/// Items with (*) can be overridden at run time with environment
variables.
///
/// X can be replaced with C, CXX, F, and FC for C, C++, Fortran 77,
/// and Fortran 90, respectively.
I would suggest that if this is not suitable for your usage that we
morph it until it is. The whole point of the wrapper compilers is for
exactly this kind of purpose, so if they're not actually useful, then
we should change them so that they are.
Let me know if the wrappers work for you (ditto for LAM/MPI; the
wrappers in OMPI are basically the same -- but slightly expanded -- as
the wrappers from LAM/MPI). I have only seen one situation (extremely
recently) where a LAM user *couldn't* use the wrapper compilers, but
they wanted to intercept MPI calls in a fairly non-standard way, so we
judged that an acceptable failure for the wrappers (i.e., the user was
satisfied with --showme:foo).
My users expect to combine c,c++,fortran,python,java(!),and
fortran-variant-x
all in the same executable on a diversity of platforms.
And when it doesn't work, they don't go to you, they
tell me "hey, make it work, my mpi isn't broken -- it
runs my vanilla C code all the time." The real issue is,
of course, the utter insanity of history that is the linker.
The workaround always involves reverse-engineering the
compiler wrappers and assembling the link line details
explicitly. Far better that this kind of insanity be
testable and the work-arounds picked out by my configure
scripts than all the users coming back to me for
individual attention.
Are you saying that your configure re-orders the flags that you're
getting back from MPI installations?
Please, extract the full path name to the compilers your
wrappers are going to invoke and put them in ompi_info.
Actually, it is whatever was given to configure. In this case, only
"gcc" was given to configure. For example, if you:
with
./configure CC=/path/to/gcc
You'll see:
C compiler: /path/to/gcc
And that's also the name that mpicc will fork/exec to compile C
applications. I'd prefer to leave it this way for the following
reasons:
- this is the way that I've seen most Autoconf-enabled build systems
work
- if people want to use absolute names for compilers, they can
- those who don't want absolute names aren't forced to (there's many
an
installation out there that only has the C bindings and don't give a
whit about C++ or Fortran)
Those are all fine reasons, and i'm not saying (since for example)
compiler:c:command:gcc
should be changed to read
compiler:c:command:/some/godawful/experimental/gcc
No, i want whatever is there to keep going just as is so that
no existing build processes break.
I'm just requesting that be added:
compiler:which_c:command:/usr/local/bin/gcc
compiler:which_f77:command:/opt/intel/whatever
compiler:which_cxx:command:/opt/intel/whatever
where which_$X indicates the full path to the
critter X that was built with.
Ah, ok. That's easy enough to do (too late for beta, I'll commit this
on the trunk tonight -- we try not to make configure.ac changes during
the work day; it keeps peer developer frustration down ;-) ).
Slight rename, though:
-----
shell$ ./ompi_info --parsable -c | grep compiler: | egrep
':command:|:absolute:'
compiler:c:command:gcc
compiler:c:absolute:/usr/i686-pc-linux-gnu/gcc-bin/3.3.5-20050130/gcc
compiler:cxx:command:g++
compiler:cxx:absolute:/usr/i686-pc-linux-gnu/gcc-bin/3.3.5-20050130/g++
compiler:f77:command:g77
compiler:f77:absolute:/usr/i686-pc-linux-gnu/gcc-bin/3.3.5-20050130/g77
compiler:f90:command:none
compiler:f90:absolute:none
-----
That's two minor changes:
1. Making the second field stay the name for easy grouped grepping
(e.g., grepping on "compiler:c:" gets all info about the C compiler);
make the 3rd field be different.
2. Change it from "which" to "absolute", because "which" reflects a
command that not everyone will necessarily be familiar with. Indeed,
we're really presenting the absolute filename of the compiler as it was
located in the $PATH.
Acceptable?
In some fantasy future I might then be able to
submit a patch to the c++/fortran compiler wrapper so that
if the wrapper is used in an environment where
the compiler invoked is not the same as the
compiler open-mpi libraries were built with I could
issue a compiler warning. This would save a lot
of headaches to a lot of naive users.
Not a bad idea.
I'd also want to make it possible for systems to *not* give that
warning, or perhaps give aliases that mean "don't warn if I'm using
this compiler", or somesuch. You can easily imagine scenarios where
this could present false warnings. For example: a sysadmin installs a
compiler at location X, but later moves (reinstalls) it to location Y.
If we didn't provide a way to disable such warnings (and/or provide
aliases), either the sysadmin has to reinstall OMPI (which makes the
sysadmin unhappy), or users would always get false warnings about this
situation (and software really shouldn't warn about indecisive
sysadmins ;-) ).
More work than I have time for today, unfortunately. I'll see if I can
sneak in such functionality while waiting for long compiles, etc. No
promises about timeframe...
The autoconf process, if ompi_info is to be believed,
checks a lot of machine specific and compiler specific
things to do with alignment, size, type existence, etc
and records these assumptions. Throwing a random
compiler (or worse, the primitive size-changing fortran switches)
into a compiler wrapper's path is just asking for trouble.
We need good config info to diagnose these kinds of user idiocy
efficiently.
Yep -- that's exactly the purpose of ompi_info (we've been requesting
the full output of laminfo along with bug reports for the past few
years, and it's been incredibly helpful). Your point about adding the
absolute compiler path is a good step in the right direction for this
functionality.
Any other suggestions?
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
