Sure - once any change works locally [for gcc and xlf]
When I try - I get a bunch of errors.. [yet to digest them.]
Satish
On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:
> > I'm not sure what would happen if these 'use' statements are removed [whats
> > required and what can be removed?]
> >
> > The relevant code that adds this is in
> > lib/petsc/bin/maint/generatefortranstubs.py
> >
> > fd.write(' use petsc'+mansec+'def\n')
>
> I suppose we can run it through CI, see if it breaks?
>
> Best regards,
>
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
>
> > On Mar 3, 2021, at 12:49, Satish Balay <[email protected]> wrote:
> >
> > On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:
> >
> >> Hello All,
> >>
> >> I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago
> >> that would crash the compiler when compiling petsc fortran interfaces. The
> >> TL;DR of it is that the xl compiler creates a function dictionary for
> >> every function imported in fortran modules, and since petsc fortran
> >> interfaces seem to import entire packages writ-large this exceeds the
> >> number of dictionary entries (2**21):
> >>
> >>> The reason for the Internal Compiler Error is because we can't grow an
> >>> interal dictionary anymore (ie we hit a 2**21 limit).
> >>> The file contains many module procedures and interfaces that use the same
> >>> helper module. As a result, we are importing the dictionary entries for
> >>> that module repeatedly reaching
> >>> the limit.
> >>>
> >>> Can you please give the following source code workaround a try?
> >>> Since there is already "use petscvecdefdummy" at the module scope, one
> >>> workaround might be to remove the unnecessary "use petscvecdefdummy" in
> >>> vecnotequal and vecequals
> >>> and all similar procedures.
> >>>
> >>> For example, the test case has:
> >>> module petscvecdef
> >>> use petscvecdefdummy
> >>> ...
> >>> function vecnotequal(A,B)
> >>> use petscvecdefdummy
> >>> logical vecnotequal
> >>> type(tVec), intent(in) :: A,B
> >>> vecnotequal = (A%v .ne. B%v)
> >>> end function
> >>> function vecequals(A,B)
> >>> use petscvecdefdummy
> >>> logical vecequals
> >>> type(tVec), intent(in) :: A,B
> >>> vecequals = (A%v .eq. B%v)
> >>> end function
> >>> ...
> >>> end module
> >>> Another workaround would be to put the procedure definitions from this
> >>> large module into several submodules. Each submodule would be able to
> >>> accommodate a dictionary with 2**21 entries.
> >>>
> >>>
> >>> Please let us know if one of the above workarounds resolve the issue.
> >>
> >>
> >> The proposed fix from IBM would be to pull “use moduleXXX” out of
> >> subroutines or to have our auto-fortran interfaces detect which symbols to
> >> include from the respective modules and only include those in the
> >> subroutines. I’m not familiar at all with how the interfaces are generated
> >> so I don’t even know if this is possible.
> >
> > I'm not sure what would happen if these 'use' statements are removed [whats
> > required and what can be removed?]
> >
> > The relevant code that adds this is in
> > lib/petsc/bin/maint/generatefortranstubs.py
> >
> > fd.write(' use petsc'+mansec+'def\n')
> >
> > Satish
> >
> >>> IBM provided the following additional explanation and example. Can the
> >>> process used to generate these routines and functions determine the
> >>> specific symbols required and then use the only keyword or import
> >>> statement to include them?
> >>>
> >>> When factoring out use statements out of module procedures, you can just
> >>> delete them. But you can't completely remove them from interface blocks.
> >>> Instead, you can limit them either by using use <module>, only: <symbol>
> >>> or import <symbol> . if the hundreds of use statements in the program are
> >>> factored out / limited in this way, that should reduce the dictionary
> >>> size sufficiently for the program to compile.
> >>>
> >>> For example
> >>> Interface
> >>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >>> use petscvecdef
> >>> real(kind=selected_real_kind(10)), pointer :: array(:)
> >>> integer(kind=selected_int_kind(5)) ierr
> >>> type(tVec) v
> >>> End Subroutine
> >>> End Interface
> >>>
> >>> imports all symbols from petscvecdef into the dictionary even though we
> >>> only need tVec . So we can either:
> >>>
> >>> Interface
> >>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >>> use petscvecdef, only: tVec
> >>> implicit none
> >>> real(kind=selected_real_kind(10)), pointer :: array(:)
> >>> integer(kind=selected_int_kind(5)) ierr
> >>> type(tVec) v
> >>> End Subroutine
> >>> End Interface
> >>>
> >>> or if use petscvecdef is used in the outer scope, we can:
> >>> Interface
> >>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >>> import tVec
> >>> implicit none
> >>> real(kind=selected_real_kind(10)), pointer :: array(:)
> >>> integer(kind=selected_int_kind(5)) ierr
> >>> type(tVec) v
> >>> End Subroutine
> >>> End Interface
> >>> (The two methods (use, only vs import) are equivalent in terms of impact
> >>> to the dictionary.)
> >>>
> >>
> >> Is this compiler ~feature~ something that we intend to work around?
> >> Thoughts?
> >>
> >> Best regards,
> >>
> >> Jacob Faibussowitsch
> >> (Jacob Fai - booss - oh - vitch)
> >> Cell: (312) 694-3391
> >>
> >>> Begin forwarded message:
> >>>
> >>> From: "Roy Musselman" <[email protected]>
> >>> Subject: Re: Case TS005062693 - XLF: ICE in xlfentry compiling a module
> >>> with 358 subroutines
> >>> Date: March 3, 2021 at 08:23:17 CST
> >>> To: Jacob Faibussowitsch <[email protected]>
> >>> Cc: "Gyllenhaal, John C." <[email protected]>
> >>>
> >>> Hi Jacob,
> >>> I tried the first suggestion and commented out the use statements called
> >>> within the functions. However, I hit the following error complaining
> >>> about specific symbol dependencies provided by the library.
> >>>
> >>> .../src/vec/f90-mod/petscvecmod.F90", line 107.37: 1514-084 (S)
> >>> Identifier a is being declared with type name tvec which has not been
> >>> defined in a derived type definition.
> >>>
> >>> IBM provided the following additional explanation and example. Can the
> >>> process used to generate these routines and functions determine the
> >>> specific symbols required and then use the only keyword or import
> >>> statement to include them?
> >>>
> >>> When factoring out use statements out of module procedures, you can just
> >>> delete them. But you can't completely remove them from interface blocks.
> >>> Instead, you can limit them either by using use <module>, only: <symbol>
> >>> or import <symbol> . if the hundreds of use statements in the program are
> >>> factored out / limited in this way, that should reduce the dictionary
> >>> size sufficiently for the program to compile.
> >>>
> >>> For example
> >>> Interface
> >>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >>> use petscvecdef
> >>> real(kind=selected_real_kind(10)), pointer :: array(:)
> >>> integer(kind=selected_int_kind(5)) ierr
> >>> type(tVec) v
> >>> End Subroutine
> >>> End Interface
> >>>
> >>> imports all symbols from petscvecdef into the dictionary even though we
> >>> only need tVec . So we can either:
> >>>
> >>> Interface
> >>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >>> use petscvecdef, only: tVec
> >>> implicit none
> >>> real(kind=selected_real_kind(10)), pointer :: array(:)
> >>> integer(kind=selected_int_kind(5)) ierr
> >>> type(tVec) v
> >>> End Subroutine
> >>> End Interface
> >>>
> >>> or if use petscvecdef is used in the outer scope, we can:
> >>> Interface
> >>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >>> import tVec
> >>> implicit none
> >>> real(kind=selected_real_kind(10)), pointer :: array(:)
> >>> integer(kind=selected_int_kind(5)) ierr
> >>> type(tVec) v
> >>> End Subroutine
> >>> End Interface
> >>> (The two methods (use, only vs import) are equivalent in terms of impact
> >>> to the dictionary.)
> >>>
> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>> Roy Musselman
> >>> IBM HPC Application Analyst at Lawrence Livermore National Lab
> >>> email: [email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>
> >>> LLNL office: 925-422-6033
> >>> Cell: 507-358-8895, Home: 507-281-9565
> >>>
> >>> Roy Musselman---02/24/2021 07:08:45 PM---Hi Jacob, I opened the ticket
> >>> with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket
> >>>
> >>> From: Roy Musselman/Rochester/Contr/IBM
> >>> To: Jacob Faibussowitsch <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> Cc: "Gyllenhaal, John C." <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> Date: 02/24/2021 07:08 PM
> >>> Subject: Re: [EXTERNAL] Case TS005062693 - XLF: ICE in xlfentry
> >>> compiling a module with 358 subroutines
> >>>
> >>>
> >>>
> >>> Hi Jacob,
> >>> I opened the ticket with IBM: case TS005062693 and and the local LLNL
> >>> Sierra Jira Ticket at
> >>> https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues
> >>>
> >>> <https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues><https://urldefense.com/v3/__https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues__;!!DZ3fjg!vDUpTg4q6jg1lQwt37jm9Uzc7MqGrEdrg0wpKgGq9P5JoR3jKrqncOAKyni2BEUYOxQ$
> >>>
> >>> <https://urldefense.com/v3/__https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues__;!!DZ3fjg!vDUpTg4q6jg1lQwt37jm9Uzc7MqGrEdrg0wpKgGq9P5JoR3jKrqncOAKyni2BEUYOxQ$>>
> >>>
> >>> Today IBM provided the response below. I don't know when I'll have time
> >>> to try it on the reproducer I gave IBM. Perhaps early next week. Can you
> >>> review this and see if it helps?
> >>>
> >>> The reason for the Internal Compiler Error is because we can't grow an
> >>> interal dictionary anymore (ie we hit a 2**21 limit).
> >>> The file contains many module procedures and interfaces that use the same
> >>> helper module. As a result, we are importing the dictionary entries for
> >>> that module repeatedly reaching
> >>> the limit.
> >>>
> >>> Can you please give the following source code workaround a try?
> >>> Since there is already "use petscvecdefdummy" at the module scope, one
> >>> workaround might be to remove the unnecessary "use petscvecdefdummy" in
> >>> vecnotequal and vecequals
> >>> and all similar procedures.
> >>>
> >>> For example, the test case has:
> >>> module petscvecdef
> >>> use petscvecdefdummy
> >>> ...
> >>> function vecnotequal(A,B)
> >>> use petscvecdefdummy
> >>> logical vecnotequal
> >>> type(tVec), intent(in) :: A,B
> >>> vecnotequal = (A%v .ne. B%v)
> >>> end function
> >>> function vecequals(A,B)
> >>> use petscvecdefdummy
> >>> logical vecequals
> >>> type(tVec), intent(in) :: A,B
> >>> vecequals = (A%v .eq. B%v)
> >>> end function
> >>> ...
> >>> end module
> >>> Another workaround would be to put the procedure definitions from this
> >>> large module into several submodules. Each submodule would be able to
> >>> accommodate a dictionary with 2**21 entries.
> >>>
> >>>
> >>> Please let us know if one of the above workarounds resolve the issue.
> >>>
> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>> Roy Musselman
> >>> IBM HPC Application Analyst at Lawrence Livermore National Lab
> >>> email: [email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>
> >>> LLNL office: 925-422-6033
> >>> Cell: 507-358-8895, Home: 507-281-9565
> >>>
> >>>
> >>> Roy Musselman---02/21/2021 09:42:55 PM---Hi Jacob, After some more
> >>> experimentation, I think I may have found what is triggering the ICE. It
> >>>
> >>> From: Roy Musselman/Rochester/Contr/IBM
> >>> To: Jacob Faibussowitsch <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> Cc: "Gyllenhaal, John C." <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> Date: 02/21/2021 09:42 PM
> >>> Subject: Re: [EXTERNAL] Re: xlf90_r Internal Compiler Error
> >>>
> >>>
> >>> Hi Jacob,
> >>>
> >>> After some more experimentation, I think I may have found what is
> >>> triggering the ICE. It doesn't appear to be related to the subroutine
> >>> name length. I think the compiler may be hitting an internal limit of the
> >>> number of subroutines within a module. There are 358 subroutines
> >>> contained in the expanded petscmatmod.F90. Removing 4 subroutines will
> >>> allow the compile to complete successfully, so the limit must be 354
> >>> subroutines. Is it possible for you to bust up petscmatmod into multiple
> >>> modules? I'll package up the reproducer and pass it on to the compiler
> >>> development team.
> >>>
> >>> I've asked for user feedback a couple years ago, when the IBM Power9
> >>> CORAL-1 Sierra systems were deployed, but received minimal responses. DOE
> >>> is now working with Cray (aka HPE) developing the environment for the
> >>> CORAL-2 system (El Capitan). I'll pass your request to the LLNL person I
> >>> know that is dealing with math libraries for CORAL-2.
> >>>
> >>> We use the spack tool to download and build petsc and its specified
> >>> dependencies. I switched between the PETSC versions by changing the
> >>> PETSCDIR variable in the script I shared with you. I've attached a tar
> >>> ball containing the scripts used to build PETSc via spack.
> >>>
> >>> [attachment "bld-petsc-spack.tgz" deleted by Roy
> >>> Musselman/Rochester/Contr/IBM]
> >>>
> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>> Roy Musselman
> >>> IBM HPC Application Analyst at Lawrence Livermore National Lab
> >>> email: [email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>
> >>> LLNL office: 925-422-6033
> >>> Cell: 507-358-8895, Home: 507-281-9565
> >>>
> >>>
> >>> Jacob Faibussowitsch ---02/21/2021 12:24:11 PM---Hi Roy, > I'm not sure
> >>> which projects at LLNL are using PETSc or if they chose to build their
> >>> own ve
> >>>
> >>> From: Jacob Faibussowitsch <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> To: Roy Musselman <[email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>>
> >>> Cc: "Gyllenhaal, John C." <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> Date: 02/21/2021 12:24 PM
> >>> Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error
> >>>
> >>>
> >>>
> >>> Hi Roy, I'm not sure which projects at LLNL are using PETSc or if they
> >>> chose to build their own version. Entirely unrelated to our problem, but
> >>> is it possible to find this out? It would be great if yes, but also
> >>> completely fine if not. PETSc
> >>> Hi Roy,
> >>> I'm not sure which projects at LLNL are using PETSc or if they chose to
> >>> build their own version.
> >>> Entirely unrelated to our problem, but is it possible to find this out?
> >>> It would be great if yes, but also completely fine if not. PETSc is
> >>> potentially undergoing a rather transformative rewrite over the next few
> >>> years and we’d like to gather current usage data to get a better idea of
> >>> where PETSc fits into our users workflows. But we aren’t sure how to
> >>> gather this data (we don’t particularly want to scrape and silently send
> >>> it off without users consent/knowledge) absent user questionnaires and
> >>> HPC usage statistics.
> >>> If you are interested, I can share with you the spack recipes I use to
> >>> build petsc with hdf5, hypre, and suplerlu-dist.
> >>> Yes that would be quite useful. I can let it percolate through our dev
> >>> channels for any other recommendations etc.
> >>> 3.14.0 and 3.14.1
> >>>
> >>> "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90",
> >>> line 9.13: 1514-219 (S) Unable to access module symbol file for module
> >>> petscisdefdummy. Check path and file permissions of file. Use association
> >>> not done for this module.
> >>> 1501-511 Compilation failed for file petscvecmod.F90.
> >>> How exactly did you switch between versions? PETSc has 2 types of fortran
> >>> bindings, “ftn-custom” and “ftn-auto” (technically 3 including the F90
> >>> files, but those simply call either of the two preceding ones), a copy of
> >>> which you will find in every src directory. As the names imply ftn-auto
> >>> is auto generated while ftn-custom is hand-written.
> >>>
> >>> This also means that the ftn-auto files are __not__ tracked by git, so a
> >>> simple git checkout [new-tag] may not properly dispose of the old
> >>> auto-generated files (very rare, but IIRC we made a major enough change
> >>> to the fortran bindings within the last year to warrant having to "make
> >>> deletefortranstubs" before rebuilding).
> >>> Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch
> >>> of other warning messages, but it still encounters the ICE. So, I'm
> >>> uncertain if the subroutine name length is the root of the problem.
> >>> Our current compiler flag selection philosophy is to require a minimum
> >>> but choose the maximum available reasonable flag for the compiler (I.e.
> >>> we require C99, but very often you will find that your code is compiled
> >>> with C11 or C17 if they are available). It is therefore odd that
> >>> configure did not use the same methodology for fortran compilers. I will
> >>> relay this on our side.
> >>> Is it possible for you to use subroutines that are less than 32
> >>> characters and see if that works four you? Have you used other fortran 90
> >>> compilers and do any of them complain of this?
> >>> Of all of the small quirks fortran has this is probably the most esoteric
> >>> one I’ve come across… I’ve attached a list of all the F90 compilers, and
> >>> their flags which we use in CI/CD (all of which is run multiple times
> >>> daily and __must__ pass). I got them all via grep, so there may be some
> >>> duplicates here or there. As for using shorter names, this is also
> >>> something we can look at, but since none of the other compilers have had
> >>> issues with this I’m not sure this is the change to make.
> >>> Are there any unusual or questionable language constructs used in any of
> >>> the functions mentioned above that may possibly challenge the compiler?
> >>> Not that I am aware of, but again I will ask around our dev channels and
> >>> see if anything comes to mind.
> >>>
> >>>
> >>> Best regards,
> >>>
> >>> Jacob Faibussowitsch
> >>> (Jacob Fai - booss - oh - vitch)
> >>> Cell: (312) 694-3391[attachment "compilerList" deleted by Roy
> >>> Musselman/Rochester/Contr/IBM]
> >>> On Feb 20, 2021, at 22:05, Roy Musselman <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>> wrote:
> >>> Hi Jacob,
> >>> Thanks for letting me know that you are a PETSc developer and that you
> >>> are testing it on the LLNL lassen system. I've used the spack build tool
> >>> to build and deploy a few versions on the systems. I'm not sure which
> >>> projects at LLNL are using PETSc or if they chose to build their own
> >>> version. I did however provide a single precision version upon request
> >>> that was integrated with MVAPICH2-MPI instead of the IBM-provided
> >>> Spectrum-MPI. Here's what's available on the systems today.
> >>>
> >>>> ml avail petsc
> >>> -----------------------------------------------------
> >>> /usr/tcetmp/modulefiles/Core
> >>> -----------------------------------------------------
> >>> petsc/default petsc/3.10.2 petsc/3.11.3 petsc/3.13.0 (D)
> >>> petsc/3.13.1-mvapich2-2020.01.09-xl-2020.03.18.single
> >>>
> >>> If you are interested, I can share with you the spack recipes I use to
> >>> build petsc with hdf5, hypre, and suplerlu-dist.
> >>>
> >>> After several attempts I was able to reproduce the Internal Compiler
> >>> Errro (ICE) that you are seeing using version 3.14.4. I've whittled it
> >>> down to the petscmatmod.F90 file and it's specific dependencies.
> >>> The following script is what I'm using. Note that in the 2nd set of
> >>> compiles, the -E option is used to expand all included source files and
> >>> headers and encapsulating it into a single large source file. This can be
> >>> used to help isolate the source of the problem.
> >>>
> >>> #!/bin/bash
> >>>
> >>> PETSCDIR="../roymuss/spack-stage-petsc-3.14.4-eh5arny7l3cqjlltlfpjp6f4jofbnmz6/spack-src"
> >>>
> >>> OPTIONS=" -qmoddir=moddir -I$PETSCDIR/arch-linux-c-opt/include
> >>> -I$PETSCDIR/include"
> >>> mkdir -p moddir
> >>>
> >>> set -x
> >>>
> >>> # Compile original source files including dependencies
> >>> if [ 0 = 1 ]; then
> >>> mpif90 -c -g $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o
> >>> petscsysmod.o
> >>> mpif90 -c -g $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o
> >>> petscvecmod.o
> >>> mpif90 -c -g $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o
> >>> petscmatmod.o
> >>> fi
> >>>
> >>> # Use -E option to expand source into full source files
> >>> if [ 0 = 1 ]; then
> >>> mpif90 -c -g -E $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o
> >>> full_petscsysmod.F90
> >>> mpif90 -c -g -E $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o
> >>> full_petscvecmod.F90
> >>> mpif90 -c -g -E $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o
> >>> full_petscmatmod.F90
> >>> fi
> >>>
> >>> # Compile from full source files
> >>> if [ 1 = 1 ]; then
> >>> mpif90 -c -g -Imoddir -qmoddir=moddir full_petscsysmod.F90 -o
> >>> full_petscsysmod.o
> >>> mpif90 -c -g -Imoddir -qmoddir=moddir full_petscvecmod.F90 -o
> >>> full_petscvecmod.o
> >>> mpif90 -V -c -g -Imoddir -qmoddir=moddir full_petscmatmod.F90 -o
> >>> full_petscmatmod.o
> >>> fi
> >>>
> >>> <eof>
> >>>
> >>> Petsc 3.13.6 it the most recent version that did not fail. I tried all
> >>> subsequent versions and got the folowing results:
> >>>
> >>> 3.14.0 and 3.14.1
> >>>
> >>> "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90",
> >>> line 9.13: 1514-219 (S) Unable to access module symbol file for module
> >>> petscisdefdummy. Check path and file permissions of file. Use association
> >>> not done for this module.
> >>> 1501-511 Compilation failed for file petscvecmod.F90.
> >>>
> >>> 3.14.2, 3.14.3, and 3.14.4
> >>>
> >>> . . .
> >>> ** matnullspaceequals === End of Compilation 8 ===
> >>> *** Error in
> >>> `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free():
> >>> invalid pointer: 0x0000200001740018 ***
> >>>
> >>> Examining the tail end of petscmatmod.F90
> >>>
> >>>
> >>> 80 function matnullspaceequals(A,B)
> >>> 81 use petscmatdefdummy
> >>> 82 logical matnullspaceequals
> >>> 83 type(tMatNullSpace), intent(in) :: A,B
> >>> 84 matnullspaceequals = (A%v .eq. B%v)
> >>> 85 end function
> >>> 86
> >>> 87 #if defined(_WIN32) && defined(PETSC_USE_SHARED_LIBRARIES)
> >>> 88 !DEC$ ATTRIBUTES DLLEXPORT::matnotequal
> >>> 89 !DEC$ ATTRIBUTES DLLEXPORT::matequals
> >>> 90 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringnotequal
> >>> 91 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringequals
> >>> 92 !DEC$ ATTRIBUTES DLLEXPORT::matnullspacenotequal
> >>> 93 !DEC$ ATTRIBUTES DLLEXPORT::matnullspaceequals
> >>> 94 #endif
> >>> 95 module petscmat
> >>> 96 use petscmatdef
> >>> 97 use petscvec
> >>> 98 #include <../src/mat/f90-mod/petscmat.h90>
> >>> 99 interface
> >>> 100 #include <../src/mat/f90-mod/ftn-auto-interfaces/petscmat.h90>
> >>> 101 end interface
> >>> 102 end module
> >>> 103
> >>>
> >>> Compiling the matnullspaceequals function was successful just before
> >>> hitting the error. The error goes away when removing either or both of
> >>> the #include lines 98 and 100. Both #include statements are required to
> >>> produce the error. The 3.13.6 and 3.14.4 version of the file identified
> >>> in the first #include at line 98 are identical. The file identified in
> >>> line 100 is different between 3.13.6 and 3.14.4.
> >>> Just looking at the list of subroutines contained within each version,
> >>> the following are the differences.
> >>>
> >>> Old subroutines available in 3.13.6 but removed from 4.14.4
> >>> subroutine MatFreeIntermediateDataStructures(a,z)
> >>>
> >>> New subroutines available in 4.14.4 but not contained in 3.13.6
> >>> subroutine MatDenseReplaceArray(a,b,z)
> >>> subroutine MatIsShell(a,b,z)
> >>> subroutine MatRARtMultEqual(a,b,c,d,e,z)
> >>> subroutine MatScaLAPACKGetBlockSizes(a,b,c,z)
> >>> subroutine MatScaLAPACKSetBlockSizes(a,b,c,z)
> >>> subroutine MatSeqAIJCUSPARSESetGenerateTranspose(a,b,z)
> >>> subroutine MatSeqAIJSetTotalPreallocation(a,b,z)
> >>> subroutine MatSetLayouts(a,b,c,z)
> >>>
> >>> Methodically removing the new subroutines did not provide a consistent
> >>> result. But I did notice the extra long subroutine name
> >>> MatSeqAIJCUSPARSESetGenerateTranspose had 37 characters.
> >>> A little research found: In Fortran 90/95 the maximum length was 31
> >>> characters, in Fortran 2003 it is now 63 characters. I found the
> >>> following subroutines with greater than 31 characters
> >>>
> >>> subroutine MatCreateMPIMatConcatenateSeqMat
> >>> subroutine MatFactorFactorizeSchurComplement
> >>> subroutine MatMPIAdjCreateNonemptySubcommMat
> >>> subroutine MatSeqAIJCUSPARSESetGenerateTranspose
> >>> subroutine MatMPIAIJSetUseScalableIncreaseOverlap
> >>> subroutine MatFactorSolveSchurComplementTranspose
> >>>
> >>> I individually ifdef'd them out of the source file and was able to
> >>> compile the files successfully without encountering the ICE.
> >>>
> >>> I'm not exactly sure what the maximum subroutine name length that the XLF
> >>> compiler allows, but if it is only 31, it would be useful if the compiler
> >>> detected this and issue a message instead of the ICE.
> >>> Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch
> >>> of other warning messages, but it still encounters the ICE. So, I'm
> >>> uncertain if the subroutine name length is the root of the problem.
> >>>
> >>> Is it possible for you to use subroutines that are less than 32
> >>> characters and see if that works four you? Have you used other fortran 90
> >>> compilers and do any of them complain of this?
> >>> Are there any unusual or questionable language constructs used in any of
> >>> the functions mentioned above that may possibly challenge the compiler?
> >>>
> >>> I'll package this up and send it to the IBM XL compiler development team
> >>> for their examination and comment.
> >>>
> >>> Best Regards,
> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>> Roy Musselman
> >>> IBM HPC Application Analyst at Lawrence Livermore National Lab
> >>> email: [email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>
> >>> LLNL office: 925-422-6033
> >>> Cell: 507-358-8895, Home: 507-281-9565
> >>>
> >>> <graycol.gif>Jacob Faibussowitsch ---02/18/2021 02:17:05 PM---> The most
> >>> recently built version available on the CORAL systems is 3.13.0. (ml load
> >>> petsc/3.13.0) W
> >>>
> >>> From: Jacob Faibussowitsch <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> To: Roy Musselman <[email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>>
> >>> Cc: "Gyllenhaal, John C." <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> Date: 02/18/2021 02:17 PM
> >>> Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> The most recently built version available on the CORAL systems...
> >>> This Message Is From an External Sender
> >>> This message came from outside your organization.
> >>> The most recently built version available on the CORAL systems is 3.13.0.
> >>> (ml load petsc/3.13.0) Will that work for you?
> >>> I am building petsc from source as part of development work on petsc
> >>> itself so modules are unfortunately not useful here.
> >>> The files you sent me do not contain all the dependencies (other mod
> >>> files) required to reproduce the error.
> >>> I'll attempt to build version 3.14.4 from scratch and recreate the
> >>> failing symptom you are observing.
> >>> Yes, petsc uses an automated system to generate the fortran files from C
> >>> which goes about 20 rabbit holes deeper than I was willing to dig. Let me
> >>> know if you run into trouble configuring and building petsc, I can point
> >>> you in the right direction. I’ve attached a “reconfigure” script with
> >>> this email, it contains all of the arguments I used to configure petsc
> >>> successfully on Lassen. If you place it into your $PETSC_DIR (i.e. the
> >>> folder titled “petsc” and that contains a “configure” file) and run:
> >>>
> >>> $ python3 ./reconfigure-arch-linux-c-debug.py
> >>>
> >>> It should work. If not, you will have to
> >>>
> >>> $ ./configure —all-the-args —in-the-reconfigure —file
> >>>
> >>> Best regards,
> >>>
> >>> Jacob Faibussowitsch
> >>> (Jacob Fai - booss - oh - vitch)
> >>> Cell: (312) 694-3391[attachment "reconfigure-arch-linux-c-debug.py"
> >>> deleted by Roy Musselman/Rochester/Contr/IBM]
> >>> On Feb 18, 2021, at 15:07, Roy Musselman <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>> wrote:
> >>> Hi Jacob,
> >>>
> >>> The source file appears to come from the PETSc 3.14.4 library. The most
> >>> recently built version available on the CORAL systems is 3.13.0. (ml load
> >>> petsc/3.13.0) Will that work for you?
> >>> The files you sent me do not contain all the dependencies (other mod
> >>> files) required to reproduce the error.
> >>> I'll attempt to build version 3.14.4 from scratch and recreate the
> >>> failing symptom you are observing.
> >>>
> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>> Roy Musselman
> >>> IBM HPC Application Analyst at Lawrence Livermore National Lab
> >>> email: [email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>
> >>> LLNL office: 925-422-6033
> >>> Cell: 507-358-8895, Home: 507-281-9565
> >>>
> >>> <graycol.gif>Roy Musselman---02/18/2021 11:18:20 AM---I'll take a look.
> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Roy Musselman
> >>>
> >>> From: Roy Musselman/Rochester/Contr/IBM
> >>> To: LC Hotline <[email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>>
> >>> Cc: "Gyllenhaal, John C." <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> Date: 02/18/2021 11:18 AM
> >>> Subject: Re: [EXTERNAL] FW: xlf90_r Internal Compiler Error
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> I'll take a look.
> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>> Roy Musselman
> >>> IBM HPC Application Analyst at Lawrence Livermore National Lab
> >>> email: [email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>
> >>> LLNL office: 925-422-6033
> >>> Cell: 507-358-8895, Home: 507-281-9565
> >>>
> >>>
> >>> <graycol.gif>LC Hotline ---02/18/2021 11:03:55 AM---Hi John, Roy, Can you
> >>> help this user with the problem that he is seeing when he tries to build
> >>> with
> >>>
> >>> From: LC Hotline <[email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>>
> >>> To: "Gyllenhaal, John C." <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>, Roy Musselman <[email protected]
> >>> <mailto:[email protected]><mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> Date: 02/18/2021 11:03 AM
> >>> Subject: [EXTERNAL] FW: xlf90_r Internal Compiler Error
> >>>
> >>>
> >>>
> >>> Hi John, Roy, Can you help this user with the problem that he is...
> >>> This Message Is From an External Sender
> >>> This message came from outside your organization.
> >>> Hi John, Roy,
> >>>
> >>> Can you help this user with the problem that he is seeing when he tries
> >>> to build with xlf90 on Lassen?
> >>>
> >>> Thanks,
> >>> Ryan
> >>> --
> >>> LC Hotline
> >>>
> >>> From: Jacob Faibussowitsch <[email protected]
> >>> <mailto:[email protected]> <mailto:[email protected]
> >>> <mailto:[email protected]>>>
> >>> Date: Wednesday, February 17, 2021 at 5:27 PM
> >>> To: LC Hotline <[email protected] <mailto:[email protected]>
> >>> <mailto:[email protected] <mailto:[email protected]>>>
> >>> Subject: xlf90_r Internal Compiler Error
> >>>
> >>> Hello LC Support,
> >>>
> >>> While compiling my application on Lassen I seem have run afoul of the
> >>> xlf90 mpi compiler wrapper with the following error:
> >>>
> >>> *** Error in
> >>> `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free():
> >>> invalid pointer: 0x0000200001740018 ***
> >>>
> >>> I’m fairly certain this isn’t my fault as this is code that compiles
> >>> regularly on extensive CI/CD under various other compilers and machines,
> >>> but you can never rule it out. I have included a verbose full log of my
> >>> make run (which includes a comprehensive rundown of the environment) as
> >>> well as a separate file containing the error message and stack trace from
> >>> the compiler. Additionally I have also included the file which I believe
> >>> is causing the error. Let me know if there is anything else I should send.
> >>>
> >>> P.S. My list of loaded modules:
> >>>
> >>> Currently Loaded Modules:
> >>> 1) StdEnv (S) 4) cuda/11.1.1 7) valgrind/3.16.1
> >>> 2) clang/ibm-11.0.0 5) python/3.8.2 8) lapack/3.9.0-xl-2020.11.12
> >>> 3) spectrum-mpi/rolling-release 6) cmake/3.18.0 9) hip/3.0.0
> >>>
> >>> Best regards,
> >>>
> >>> Jacob Faibussowitsch
> >>> (Jacob Fai - booss - oh - vitch)
> >>> Cell: (312) 694-3391[attachment "errorReport.zip" deleted by Roy
> >>> Musselman/Rochester/Contr/IBM]
>
>
