On Mon, 25 Mar 2019, Mills, Richard Tran via petsc-dev wrote:
> Folks,
>
> I've spent a while looking at the BuildSystem code, and I think this is going
> to take me more time than I have available right now to figure it out on my
> own. Someone more familiar with BuildSystem needs to give me some hints --
> soon, if possible, as I really think that building with non-GCC compilers and
> CUDA should be supported in the upcoming release.
Well I think the check is good to have - but it shouldn't prevent non-GCC
compiler support.
i.e we should remove the gcc check first - and then improve the other
infrastructure.
>
> What I want to do is to add a test inside cuda.py that checks to see if
> something like
>
> nvcc --compiler-option=<compiler options PETSc has identified> <CUDAFLAGS>
> hello.c
>
> will return successfully.
Configure is too convoluted for me as-well. Perhaps cuda.py is not the correct
place for this check.
gmakefile.test:PETSC_COMPILE.cu = $(call quiet,CUDAC) -c $(CUDAC_FLAGS)
--compiler-options="$(PCC_FLAGS) $(CXXFLAGS) $(CCPPFLAGS)"
PCC_FLAGS is composed at the very end of configure run - and can have
MPI_INCLUDE [for mpi.h] - and perhaps others?
config/PETSc/Configure.py:
self.addMakeMacro('PCC_FLAGS',self.setCompilers.getCompilerFlags())
But perhaps mpi.h is not critical for this test - so its ok to do this test in
cuda.py.
>
> What I wasn't sure about was how to get at the values for a bunch of the
> above variables within the cuda.py code.
I guess we'll have to see how configure currently writes these values..
But first check what goes into "$(PCC_FLAGS) $(CXXFLAGS) $(CCPPFLAGS)"
PCC_FLAGS is set by configure in petscvariables
CXXFLAGS is something the use can specify at command line.
CCPPFLAGS is constructed from values set by configure in
lib/petsc/conf/variables
PETSC_CCPPFLAGS = ${PETSC_CC_INCLUDES} ${PETSCFLAGS} ${CPP_FLAGS}
${CPPFLAGS}
CCPPFLAGS = ${PETSC_CCPPFLAGS}
CPPFLAGS is set by user. PETSCFLAGS looks unused. Perhaps PETSC_CC_INCLUDES can
be ignored. So its just PCC_FLAGS and CPP_FLAGS
balay@sb /home/balay/petsc (master=)
$ git grep PCC_FLAGS config
config/PETSc/Configure.py:
self.addMakeMacro('PCC_FLAGS',self.setCompilers.getCompilerFlags())
config/gmakegen.py: fd.write('c_flags = %(PETSC_CC_INCLUDES)s %(PCC_FLAGS)s
%(CCPPFLAGS)s\n' % conf)
balay@sb /home/balay/petsc (master=)
$ git grep CPP_FLAGS config
config/PETSc/Configure.py:
self.addMakeMacro('CPP_FLAGS',self.setCompilers.CPPFLAGS)
Does this help?
Satish
> After deciding I couldn't really follow everything that is happening in the
> code buy just looking at it, I used the 'pdb' python debugger to stick a
> breakpoint in the configureLibrary() method in cuda.py so I could poke around.
>
> **** Aside: Looking at contents of configure objects? ****
> I had hoped I could look at everything that is stashed in the different
> objects by doing things like
>
> (Pdb) p dir(self.compilers)
>
> But this doesn't actually list everything in there. There is no 'CUDAC'
> attribute listed, for instance, but it is there for me to print:
>
> (Pdb) p self.compilers.CUDAC
> 'nvcc'
>
> Is there a good way for me to actually see all the attributes in something
> like the self.compilers object? Sorry, my Python skills are very rusty --
> haven't written much Python in about a decade.
> **** End aside ****
>
> It appears that what I need to construct my command line is then available in
>
> self.compilers.CUDAC -- The invocation for the CUDA compiler
> self.compilers.CXXFLAGS -- The flags passed to the C++ compiler (our "host")
> self.compilers.CUDAFLAGS -- The flags like "-ccbin pgc++" being passed to
> nvcc or whatever CUDAC is
>
> I could use these to construct a command that I then pass to the command
> shell, and maybe I should just do this, but this doesn't seem to follow the
> BuildSystem paradigm. It seems like I should be able to run this test by
> doing something like
>
> self.pushLanguage('CUDA')
> self.checkCompile(cuda_test)
>
> which is, in fact, invoked in checkCUDAVersion(). But the command put
> together by checkCompile() does not include "--compiler-option=<compiler
> options PETSc has identified>". Should I be modifying the code the code
> somewhere so that this argument goes into the compiler invocation constructed
> in self.checkCompile? If so, where should I be doing this?
>
> --Richard
>
>
>
> On 3/22/19 10:24 PM, Mills, Richard Tran wrote:
>
>
> On 3/22/19 3:28 PM, Mills, Richard Tran wrote:
> On 3/22/19 12:13 PM, Balay, Satish wrote:
>
> Is there currently an existing check like this somewhere? Or will things just
> fail when running 'make' right now?
>
>
>
> Most likely no. Its probably best to attempt the error case - and
> figure-out how to add a check.
>
> I gave things a try and verified that there is no check for this anywhere in
> configure -- things just fail at 'make' time. I think that all we need is a
> test that will try to compile any simple, valid C program using "nvcc
> --compiler-options=<compiler options PETSc has identified> <CUDAFLAGS>". If
> the test fails, it should report something like "Compiler flags do not work
> with CUDA compiler; perhaps you need to provide to use -ccbin in CUDAFLAGS to
> specify the intended host compiler".
>
> I'm not sure where this test should go. Does it make sense for this to go in
> cuda.py with the other checks like checkNVCCDoubleAlign()? If so, how do I
> get at the values of <compiler options PETSc has identified> and <CUDAFLAGS>?
> I'm not sure what modules I need to import from BuildSystem...
> OK, answering part of my own question here: Re-familiarizing myself with how
> the configure packages work, and then looking through the makefiles, I see
> that the argument to "--compiler-options" is filled in by the makefile
> variables
>
> ${PCC_FLAGS} ${CFLAGS} ${CCPPFLAGS}
>
> and it appears that this partly maps to self.compilers.CFLAGS in BuildSystem.
> But so far I've not managed to employ the right combination of find and grep
> to figure out where PCC_FLAGS and CCPPFLAGS come from.
>
> --Richard
>
> --Richard
>
> Satish
>
> On Fri, 22 Mar 2019, Mills, Richard Tran via petsc-dev wrote:
>
>
>
> On 3/18/19 7:29 PM, Balay, Satish wrote:
>
> On Tue, 19 Mar 2019, Mills, Richard Tran via petsc-dev wrote:
>
>
>
> Colleagues,
>
> It took me a while to get PETSc to build at all with anything on Summit other
> than the GNU compilers, but, once this was accomplished, editing out the
> isGNU() test and then passing something like
>
> '--with-cuda=1',
> '--with-cudac=nvcc -ccbin pgc++',
>
>
>
> Does the following also work?
>
> --with-cuda=1 --with-cudac=nvcc CUDAFLAGS='-ccbin pgc++'
>
> Yes, using CUDAFLAGS as above also works, and that does seem to be a better
> way to do things.
>
> After experimenting with a lot of different builds on Summit, and doing more
> reading about how CUDA compilation works on different platforms, I'm now
> thinking that perhaps configure.py should *avoid* doing anything clever to
> try figure out what the value of "-ccbin" should be. For one, this is not
> anything that NVIDIA's toolchain does for the user in the first place: If you
> want to use nvcc with a host compiler that isn't whatever NVIDIA considers
> the default (g++ on Linux, clang on Mac OS, MSVC on Windows), NVIDIA expects
> you to provide the appropriate '-ccbin' argument. Second, nvcc isn't the only
> CUDA compiler that a user might want to use: some people use Clang directly
> to compile CUDA code. Third, which host compilers are supported appears to be
> platform independent; for example, GCC is the default/preferred host compiler
> on Linux, but isn't even supported on Mac OS! Figuring out what is supported
> is very convoluted, and I think that trying to get configure to determine
> this may be more trouble than it is worth. I think we should instead let the
> user try whatever, and print out a helpful message how they "may need to
> specify host compiler to nvcc with -ccbin" if the CUDA compiler doesn't seem
> to work. Also, I'll put something about this in the CUDA configure examples.
> Any objections?
>
>
>
>
> Sometimes we have extra options in configure for specific features for
> ex: --with-pic --with-visibility etc.
>
> But that gets messy. On cuda side - we've have --with-cuda-arch and at
> some point elimiated it [so CUDAFLAGS is now the interface for this
> flag]. We could add --with-cuda-internal-compiler option to petsc
> configure - but it will again have similar drawbacks. I personally
> think most users will gravitate towards specifying such option via
> CUDAFLAGS
>
>
>
>
> to configure works fine. So, I should make a change to the BuildSystem
> cuda.py along these lines. I'm wondering exactly how I should make this work.
> I could just remove the check,
>
>
>
> sure
>
>
>
> but I think that maybe the better thing to do is to check isGNU(), then if
> the compiler is *not* GNU, configure should add the appropriate '-ccbin'
> argument to "--with-cudac", unless the user has specified '-ccbin' in their
> '--with-cudac' already. Or do we need to get this fancy?
>
>
>
> The check should be: do --compiler-options= constructed by PETSc configure
> work with CUDAC
>
> Is there currently an existing check like this somewhere? Or will things just
> fail when running 'make' right now?
>
>
>
> [or perhaps we should - just trim the --compiler-options to only -I flags?]
>
> I think we should avoid explict check for a compiler type [i.e isGNU() check]
> as much as possible.
>
>
>
>
> CUDA is only supposed to work with certain compilers, but there doesn't seem
> to be a correct official list (for instance, it supposedly won't work with
> the IBM XL compilers, but they certainly *are* actually supported on Summit).
> Heck, the latest GCC suite won't even work right now. Since what compilers
> are supported seems to be in flux, I suggest we just let the user try
> anything and then let things fail if it doesn't work.
>
>
>
> I suspec the list is dependent on the install [for ex: linux vs Windows vs
> somthing else?] and version of cuda [for ex: each version of cuda supports
> only specific versions of gcc]
>
> Yes, you are correct about this, as I detailed above.
>
>
>
> Satish
>
>
>
>
> --Richard
>
> On 3/12/19 8:45 PM, Smith, Barry F. wrote:
>
>
> Richard,
>
> You need to remove the isGNU() test and then experiment with getting the
> Nvidia tools to use the compiler you want it to use.
>
> No one has made a serious effort to use any other compilers but Gnu (at
> least not publicly).
>
> Barry
>
>
>
>
>
> On Mar 12, 2019, at 10:40 PM, Mills, Richard Tran via petsc-dev
> <[email protected]><mailto:[email protected]><mailto:[email protected]><mailto:[email protected]><mailto:[email protected]><mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>
> wrote:
>
> Fellow PETSc developers,
>
> If I try to configure PETSc with CUDA support on the ORNL Summit system using
> non-GNU compilers, I run into an error due to the following code in
> packages/cuda.py:
>
> def configureTypes(self):
> import config.setCompilers
> if not config.setCompilers.Configure.isGNU(self.setCompilers.CC,
> self.log):
> raise RuntimeError('Must use GNU compilers with CUDA')
> ...
>
> Is this just because this code predates support for other host compilers with
> nvcc, or is there perhaps some more subtle reason that I, with my
> inexperience using CUDA, don't know about? I'm guessing that I just need to
> add support for using '-ccbin' appropriately to set the location of the
> non-GNU host compiler, but maybe there is something that I'm missing. I poked
> around in the petsc-dev mailing list archives and can find a few old threads
> on using non-GNU compilers, but I'm not sure what conclusions were reached.
>
> Best regards,
> Richard
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
