https://gitlab.com/petsc/petsc/-/merge_requests/6805
Add hints to discussion of Spack installs for optimization to docs. (!6805) ·
Merge requests · PETSc / petsc · GitLab
gitlab.com
> On Aug 8, 2023, at 4:55 PM, Liu Wei AWE via petsc-dev <petsc-dev@mcs.anl.gov>
> wrote:
>
>> I see 'spack install cflags=-O3' is working - but not spack install
>> cflags='-O3 -g' [something to debug]
>
> I got the same issue with 0.19/0.20, likely to be a bug, I will raise the
> issue with Todd after evaluating the dev branch.
>
>> The other issue is mapping cflags from spack to CFLAGS/COPTFLAGS - in some
>> cases petsc build requires some default CLFAGS to work - and overriding them
>> from spack might cause issues. [again this part is not properly handed]..
>
> mfem team implemented the option to pass spack cflags as COPTFLAGS
> https://github.com/spack/spack/blame/develop/var/spack/repos/builtin/packages/petsc/package.py#L380
>
>
>>> The question is should optimisation be amended at spack recipe stage or
>>> should PETSc configure enforce some compiler optimisation (if
>>> -with-debugging=0 is set) ?
>
>> This sounds reasonable, however there are pitfalls to automatically enabling
>> optimization.
>
>> The naive approach would be to check if `--with-debugging=0` is set, and if
>> so, append `-O3 -march=native -mtune=native` (assuming those work with the
>> compiler). This might work well enough for GCC/Clang (and probably MSVC, but
>> who knows), but not all compilers play nice. For example:
>
>> - Intel compilers (before the clang transition) automatically (and
>> silently!) enable the equivalent of `-ffast-math` on any optimization level
>> greater than base.
>> - PGI compilers have had some long running bugs when optimization has been
>> enabled. For example, there has been a bug with
>> __attribute__((visibility("hidden"))) (conversely, "default" if using
>> `-fvisibility=hidden`) variables, where their order in the source file (i.e.
>> is variable A literally defined before or after variable B) causes their
>> definition to be lost at link-time leading to "undefined reference to Foo"
>> errors.
>
>> All this is to say, configure now has to keep track of, and add special
>> rules for all these compilers (and versions). Since there is no clear answer
>> we have found it's easier to punt responsibility to the user.
>
> With spack, build and deployment is decentralized, whilst traditionally we
> can apply optimization + appropriate dependency packages (e.g. MKL/libsci)
> for the specific HPC platform and compilers; changing to spack means
> individual software teams builds their own package chain, with their own
> internal PETSc which by default likely to be reliant on the package recipe /
> configure, and they are certainly not going to examine the dependency stack
> in detail until spack build performs poorly to the non-spack version.
>
>
>> The space of "best" choices for optimization flags for PETSc is huge; each
>> OS, each compiler, and each particular piece of hardware may have different
>> "best" choices, in addition, the previous choice may not even allow
>> compiling with any small change in any of these things. Finally particular
>> choices may not be compatible with choices may be other packages being
>> linked with PETSc. Hence PETSc's configure punts on trying to select a good
>> choice (and just uses -O). Surely this is an issue with most portable open
>> source packages not being able to select "good" optimization flags in a
>> portable way?
>
> We did conducted some limited evaluation a while back, performance difference
> from "-g -O" -> "-g -O3 -march=native" was noticeable (although that was
> Tealeaf + older PETSc 3.10 / gcc 9.3).
>
>> Does the Spack system contain any infrastructure to help solve this common
>> problem? That ensures all the packages it builds have good compatible
>> optimization flags for the particular system?
> I don't know which takes precedence, optimization enforced by cmake within
> the package, or package recipes, or via spack compiler spec.
>
>> Perhaps the configure message you mention printed by PETSc is inappropriate
>> for Spack builds of PETSc and we should print something different to help
>> guide the user to the correct Spack solution to the issue?
>
> If the consensus is setting optimization via spack (instead of configure),
> possibly also give a more detailed breakdown of "spack petsc install" on
> PETSc install page (maybe even a separate page)?
>
> e.g.
> # gnu optimized version
> spack install petsc %gcc@10 cflags='-g -O3 -march=native -mtune=native'
> fflags='-g -O3 -march=native -mtune=native' cxxflags='-g -O3 -march=native
> -mtune=native'
>
> # petsc without default spack extras
> spack install petsc ~superlu-dist ~metis ~hypre ~hdf5
>
>
> Regards
> Wei Liu
>
>
>
>
> The information in this email and in any attachment(s) is commercial in
> confidence. If you are not the named addressee(s) or if you receive this
> email in error then any distribution, copying or use of this communication or
> the information in it is strictly prohibited. Please notify us immediately by
> email at admin.internet(at)awe.co.uk, and then delete this message from your
> computer. While attachments are virus checked, AWE plc does not accept any
> liability in respect of any virus which is not detected. AWE Plc Registered
> in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7
> 4PR
