Re: [petsc-dev] -with-kokkos-cuda-arch=AMPERE80 nonsense

[Matthew Knepley]

On Wed, Apr 7, 2021 at 5:47 PM Scott Kruger  wrote:

> On 2021-04-06 14:44, Matthew Knepley did write:
> > > > Does spack have some magic for this we could use?
> > > >
> > >
> > > spack developed the archspec repo to abstract all of these issues:
> > > https://github.com/archspec/archspec
> >
> >
> > I do not love it. Besides the actual code (you can always complain about
> > code),
> > they do not really do any tests. They go look in a few places that data
> > should be.
> > We can do the same thing in probably 10x less code. It would be great to
> > actually
> > test the hardware to verify.
> >
>
> My impression is that the current project is languishing because they
> are focusing on the spack side right now.   But if this is the project
> that is the ECP-annointed solution, then it has the best chance of
> succeeding through sheer resources.
>

Maybe this will end up eventually being good. However, in my lifetime at
DOE, funded
projects are usually the best indicator of what not to do.

   Matt


> The thing I like the best is that having a stand-alone project to handle
> these issues is a real forehead-slapper (i.e., "why didn't I think of
> that?!").  Todd Gamblin has stated that the goal is to allow vendors to
> contribute because it will be in their interest to contribute.  This
> should have been done years ago.
>
> Regarding whether we could do better:  Now would actually be a good time
> to contribute while the project is young, but I don't have the time
> (like everyone else which is why this is a perennial problem).   It
> would also be a good time to create a separate project if this one is
> too annoying for folks.  In general, like spack, they have done a good
> job on the interface so that part is important.
>
> Scott
>
>
>
>
> >   Thanks,
> >
> >  Matt
> >
> >
> > > This is a *great* idea and eventually BuildSystem should incorporate
> it as
> > > the standard way of doing things; however, it is been focused mostly on
> > > the CPU issues, and is still under active development (my understanding
> > > is that the pulling it out of spack and getting those interop issues
> > > sorted out is tangled up in how spack handles dependencies and
> > > compilers).  It'd be nice if someone would go in and port the Kokkos
> gpu
> > > mappings to archspec as there is some great knowledge on these mapping
> > > buried in the Kokkos build system (not volunteering); i.e., translating
> > > that webpage to some real code (even if it is in make) is valuable.
> > >
> > > TL;DR:  It's a known problem with currently no good solution AFAIK.
> > > Waiting until archspec gets further along seems like the best solution.
> > >
> > > Scott
> > >
> > > P.S. ROCm has rocminfo which also doesn't solve the problem but is at
> > > least sane.
> > >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> their
> > experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/ <
> http://www.cse.buffalo.edu/~knepley/>
>
> --
> Scott Kruger
> Tech-X Corporation   kru...@txcorp.com
> 5621 Arapahoe Ave, Suite A   Phone: (720) 466-3196
> Boulder, CO 80303Fax:   (303) 448-7756
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] -with-kokkos-cuda-arch=AMPERE80 nonsense

[Scott Kruger]

On 2021-04-06 14:44, Matthew Knepley did write:
> > > Does spack have some magic for this we could use?
> > >
> >
> > spack developed the archspec repo to abstract all of these issues:
> > https://github.com/archspec/archspec
> 
> 
> I do not love it. Besides the actual code (you can always complain about
> code),
> they do not really do any tests. They go look in a few places that data
> should be.
> We can do the same thing in probably 10x less code. It would be great to
> actually
> test the hardware to verify.
> 

My impression is that the current project is languishing because they
are focusing on the spack side right now.   But if this is the project
that is the ECP-annointed solution, then it has the best chance of
succeeding through sheer resources.   

The thing I like the best is that having a stand-alone project to handle
these issues is a real forehead-slapper (i.e., "why didn't I think of
that?!").  Todd Gamblin has stated that the goal is to allow vendors to
contribute because it will be in their interest to contribute.  This
should have been done years ago.

Regarding whether we could do better:  Now would actually be a good time
to contribute while the project is young, but I don't have the time
(like everyone else which is why this is a perennial problem).   It
would also be a good time to create a separate project if this one is
too annoying for folks.  In general, like spack, they have done a good
job on the interface so that part is important.

Scott




>   Thanks,
> 
>  Matt
> 
> 
> > This is a *great* idea and eventually BuildSystem should incorporate it as
> > the standard way of doing things; however, it is been focused mostly on
> > the CPU issues, and is still under active development (my understanding
> > is that the pulling it out of spack and getting those interop issues
> > sorted out is tangled up in how spack handles dependencies and
> > compilers).  It'd be nice if someone would go in and port the Kokkos gpu
> > mappings to archspec as there is some great knowledge on these mapping
> > buried in the Kokkos build system (not volunteering); i.e., translating
> > that webpage to some real code (even if it is in make) is valuable.
> >
> > TL;DR:  It's a known problem with currently no good solution AFAIK.
> > Waiting until archspec gets further along seems like the best solution.
> >
> > Scott
> >
> > P.S. ROCm has rocminfo which also doesn't solve the problem but is at
> > least sane.
> >
> 
> 
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ 

-- 
Scott Kruger
Tech-X Corporation   kru...@txcorp.com
5621 Arapahoe Ave, Suite A   Phone: (720) 466-3196
Boulder, CO 80303Fax:   (303) 448-7756

Re: [petsc-dev] (arm64) could not find object file symbol for symbol ___muldc3

[Barry Smith]

   Ok, thanks. I am not sure how to handle it. libgcc_s.1.dylib never appears 
in the configure process and we don't actually test that the Fortran compiler 
can link C; for 25 years it just always could. 

   Barry


> On Apr 7, 2021, at 10:49 AM, Blaise A Bourdin  wrote:
> 
> No luck with -lgcc but it looks like this symbol is defined in 
> /usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libgcc_s.1.dylib
> 39b0 T ___muldc3
> 
> So adding -lgcc_s.1 eliminates the warning.
> I assume that the issue is with gcc-10 on macOS ARM. I can try to report it 
> to upstream to although building a small test may be a bit challenging.
> 
> Regards,
> Blaise
> 
>> On Apr 6, 2021, at 5:32 PM, Barry Smith > > wrote:
>> 
>> 
>> Blaise,
>> 
>> Thanks, can you try cd src/snes/tutorials and then cut and paste the 
>> huge line that you sent earlier that begins with mpif90 -Wl,-bind_at_load 
>> .  and add at the end -lgcc and see if that line successfully links the 
>> Fortran code?
>> 
>>Thanks
>> 
>>Barry
>> 
>> 
>> 
>>> On Apr 6, 2021, at 12:22 PM, Blaise A Bourdin >> > wrote:
>>> 
>>> Hi Barry,
>>> 
 
   Please send configure.log 
>>> See attached
>>> 
 
   It would also be helpful if you could run 
 
   nm -o /usr/lib/lib* | grep muldc3 
>>> 
>>> That is uneventful (I have no idea how these broken symbolic link got there 
>>> and can’t delete them because of SIP)…
>>> SiMini:lib $  nm -o /usr/lib/lib* | grep muldc3 
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm:
>>>  error: /usr/lib/libhunspell-1.2.0.dylib: No such file or directory.
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm:
>>>  error: /usr/lib/libhunspell-1.2.dylib: No such file or directory.
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm:
>>>  error: /usr/lib/libiodbc.2.dylib: No such file or directory.
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm:
>>>  error: /usr/lib/libiodbc.dylib: No such file or directory.
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm:
>>>  error: /usr/lib/libiodbcinst.2.dylib: No such file or directory.
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm:
>>>  error: /usr/lib/libiodbcinst.dylib: No such file or directory.
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm:
>>>  error: /usr/lib/libstdc++.6.dylib: No such file or directory.
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm:
>>>  error: /usr/lib/libstdc++.dylib: No such file or directory.
>>> 
>>> Blaise
>>> 
>>> 
>>> 
 
   Our cross language linker tests are not adding all the libraries they 
 need to add when using Fortran for linking.
 
   Barry
 
 
 
> On Apr 6, 2021, at 11:00 AM, Matthew Knepley  > wrote:
> 
> On Tue, Apr 6, 2021 at 11:04 AM Blaise A Bourdin  > wrote:
> Hi,
> 
> I am having the following warning when compiling any fortran example 
> (currently on main, but  it’s been going on for a little while) on a ARM 
> mac.
> 
> 
> ***Error detected during compile or 
> link!***
> See http://www.mcs.anl.gov/petsc/documentation/faq.html 
> 
> /opt/HPC/petsc-main/src/snes/tutorials ex5f
> *
> mpif90 -Wl,-bind_at_load -Wl,-multiply_defined,suppress 
> -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs 
> -Wl,-search_paths_first -Wl,-no_compact_unwind -ffree-line-length-none 
> -fallow-argument-mismatch -fPIC -g   -ffree-line-length-none 
> -fallow-argument-mismatch -fPIC -g-I/opt/HPC/petsc-main/include 
> -I/opt/HPC/petsc-main/bigsur-gcc10.2-arm64-reallybasic-g/include 
> -I/opt/X11/include ex5f.F90  
> -Wl,-rpath,/opt/HPC/petsc-main/bigsur-gcc10.2-arm64-reallybasic-g/lib 
> -L/opt/HPC/petsc-main/bigsur-gcc10.2-arm64-reallybasic-g/lib 
> -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib 
> -Wl,-rpath,/opt/homebrew/Cellar/mpich/3.4.1_2/lib 
> -L/opt/homebrew/Cellar/mpich/3.4.1_2/lib 
> -Wl,-rpath,/opt/homebrew/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/aarch64-apple-darwin20/10.2.1
>  
> 

Re: [petsc-dev] ?????? Petsc: Error code 1

[Satish Balay via petsc-dev]

There are always diffs (between different compilers, versions, hardware) - 
doesn't necessarily mean failures.

Very large diffs [as Barry mentioned] might require a closer look.

Depending upon your purpose - you might consider that you have a valid petsc 
install and start using it.

Obviously - based on the problem that you are solving - you might need to 
re-visit this issue [checking how your application behaves between a debug 
build and optimized build - and play with different optimization flags]

Satish

On Wed, 7 Apr 2021, Chen Gang wrote:

> ARM: configure with the following: -ffp-contract=off
> 
> 
> ./configure -with-debugging=0 COPTFLAGS='-O3 -ffp-contract=off 
> -march=armv8.2-a -mtune=tsv110' CXXOPTFLAGS='-O3 -ffp-contract=off 
> -march=armv8.2-a -mtune=tsv110' FOPTFLAGS='-O3 -ffp-contract=off 
> -march=armv8.2-a -mtune=tsv110' --with-x=1 -download-fblaslapack 
> PETSC-KERNEL-USE-UNROLL-4
> 
> 
> There are 3 failures, all related to FC code
> 
> 
> 
> 
> # -
> # Summary
> # -
> # FAILED diff-tao_bound_tutorials-plate2f_2 
> diff-tao_bound_tutorials-plate2f_1 diff-vec_is_is_tutorials-ex2f_1
> # success 7547/9528 tests (79.2%)
> # failed 3/9528 tests (0.0%)
> # todo 225/9528 tests (2.4%)
> # skip 1753/9528 tests (18.4%)
> #
> # Wall clock time for tests: 1371 sec
> # Approximate CPU time (not incl. build time): 106722.629 sec
> #
> # To rerun failed tests:
> #  /opt/rh/devtoolset-9/root/usr/bin/gmake -f gmakefile 
> test test-fail=1
> #
> # Timing summary (actual test time / total CPU time):
> # dm_tests-ex36_3dp1: 467.68 sec / 477.71 sec
> # dm_impls_stag_tests-ex1_multidof_3: 395.16 sec / 398.45 sec
> # ts_tutorials-ex29_1: 236.54 sec / 238.30 sec
> # dm_impls_stag_tests-ex1_basic_2: 205.41 sec / 207.34 sec
> # dm_tests-ex34_1: 178.94 sec / 180.26 sec
> 
> 
> 
> 
> 
> 
> 
> ----
> ??:   
>  "Chen Gang"  
>   
> <569615...@qq.com;
> :2021??4??7??(??) 10:49
> ??:"petsc-dev" 
> :?? [petsc-dev] Petsc: Error code 1
> 
> 
> 
> ARM: configure with the following: -ffp-contract=off
> 
> 
> ./configure -with-debugging=0 COPTFLAGS='-O3 -ffp-contract=off 
> -march=armv8.2-a -mtune=tsv110' CXXOPTFLAGS='-O3 -ffp-contract=off 
> -march=armv8.2-a -mtune=tsv110' FOPTFLAGS='-O3 -ffp-contract=off 
> -march=armv8.2-a -mtune=tsv110' --with-x=1 -download-fblaslapack 
> PETSC-KERNEL-USE-UNROLL-4
> 
> 
> There are 3 failures, all related to FC code
> 
> 
> 
> 
> # -
> # Summary
> # -
> # FAILED diff-tao_bound_tutorials-plate2f_2 
> diff-tao_bound_tutorials-plate2f_1 diff-vec_is_is_tutorials-ex2f_1
> # success 7547/9528 tests (79.2%)
> # failed 3/9528 tests (0.0%)
> # todo 225/9528 tests (2.4%)
> # skip 1753/9528 tests (18.4%)
> #
> # Wall clock time for tests: 1371 sec
> # Approximate CPU time (not incl. build time): 106722.629 sec
> #
> # To rerun failed tests:
> #  /opt/rh/devtoolset-9/root/usr/bin/gmake -f gmakefile 
> test test-fail=1
> #
> # Timing summary (actual test time / total CPU time):
> # dm_tests-ex36_3dp1: 467.68 sec / 477.71 sec
> # dm_impls_stag_tests-ex1_multidof_3: 395.16 sec / 398.45 sec
> # ts_tutorials-ex29_1: 236.54 sec / 238.30 sec
> # dm_impls_stag_tests-ex1_basic_2: 205.41 sec / 207.34 sec
> # dm_tests-ex34_1: 178.94 sec / 180.26 sec
> 
> 
> 
> --  --
> ??:   
>  "petsc-dev"  
>   
>  :2021??4??7??(??) 1:06
> ??:"Barry Smith" :"Chen Gang"<569615...@qq.com;"Alp 
> Dener" :Re: [petsc-dev] Petsc: Error code 1
> 
> 
> 
>  See the attachements. alltest.log is on a machine with 96 cores, ARM, 
> with FC,gcc9.3.5,mpich3.4.1,fblaslapack; 6 failures
> 
> Perhaps this is an issue with ARM - and such diffs are expected - as we 
> already have multiple alt files for some of these tests
> 
> $ ls -lt src/tao/bound/tutorials/output/plate2f_*
> -rw-r--r--. 1 balay balay 1029 Mar 23 19:48 
> src/tao/bound/tutorials/output/plate2f_1_alt.out
> -rw-r--r--. 1 balay balay 1071 Mar 23 19:48 
> src/tao/bound/tutorials/output/plate2f_1.out
> -rw-r--r--. 1 balay balay 1029 Mar 23 19:48 
> src/tao/bound/tutorials/output/plate2f_2_alt.out
> -rw-r--r--. 1 balay balay 1071 Mar 23 19:48 
> src/tao/bound/tutorials/output/plate2f_2.out
> 
> 
> not ok diff-vec_is_is_tutorials-ex2f_1 # Error code: 1
> #  16,24d15
> #  <  5
> #  <  7
> #  <  9
> #  < 11
> #  < 13
> #  < 15
> #  < 17
> #  < 19
> #  < 21
> <
> 
> This one is puzzling - missing fortran stdout? Perhaps compile issue on ARM? 
> [its a sequential example - so can't 

Re: [petsc-dev] DMNetwork static sizing

[Matthew Knepley]

On Tue, Apr 6, 2021 at 7:44 PM Abhyankar, Shrirang G <
shrirang.abhyan...@pnnl.gov> wrote:

> Hong,
>
>   It was just to keep things simple. We can have an API (and a run-time
> option) for setting the max. number of components instead of a configure
> option. I can work on making this change.
>

I would advocate just carrying the size of those arrays in the struct. That
way it can be completely flexible, and you can just check at runtime if
there is a chance of out of bounds indexing.

  Thanks,

Matt


>
>
> Thanks,
>
> Shri
>
> *From: *petsc-dev  on behalf of PETSc
> Development 
> *Reply-To: *"Zhang, Hong" 
> *Date: *Tuesday, April 6, 2021 at 4:46 PM
> *To: *Matthew Knepley , PETSc Development <
> petsc-dev@mcs.anl.gov>
> *Cc: *"Abhyankar, Shrirang G" 
> *Subject: *Re: [petsc-dev] DMNetwork static sizing
>
>
>
> Shri,
>
> You designed this approach. Is it intended or out of implementation
> convenience at the time?
>
> Hong
> --
>
> *From:* petsc-dev  on behalf of Matthew
> Knepley 
> *Sent:* Monday, April 5, 2021 5:47 AM
> *To:* PETSc 
> *Subject:* [petsc-dev] DMNetwork static sizing
>
>
>
> Dowe really need a configure time constant for
>
>
>
> struct _p_DMNetworkComponentHeader {
>   PetscInt index;/* index for user input global edge and vertex */
>   PetscInt subnetid; /* Id for subnetwork */
>   PetscInt ndata;/* number of components */
>   PetscInt size[PETSC_DMNETWORK_MAXIMUM_COMPONENTS_PER_POINT];
>   PetscInt key[PETSC_DMNETWORK_MAXIMUM_COMPONENTS_PER_POINT];
>   PetscInt offset[PETSC_DMNETWORK_MAXIMUM_COMPONENTS_PER_POINT];
>   PetscInt nvar[PETSC_DMNETWORK_MAXIMUM_COMPONENTS_PER_POINT]; /* Number
> of variables */
>   PetscInt offsetvarrel[PETSC_DMNETWORK_MAXIMUM_COMPONENTS_PER_POINT]; /*
> offset from the first variable of the network point */
> } PETSC_ATTRIBUTEALIGNED(PetscMax(sizeof(double),sizeof(PetscScalar)));
>
>
>
> Can't we just allocate this struct when needed and carry the size along?
>
>
>
> This design seem to go against the rest of what we do in PETSc?
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> https://www.cse.buffalo.edu/~knepley/
> 
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/