Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-11-02 Thread Mark Adams via petsc-dev
I did not configure hypre manually, so I guess it is not using GPUs. On Fri, Nov 2, 2018 at 2:40 PM Smith, Barry F. wrote: > > > > On Nov 2, 2018, at 1:25 PM, Mark Adams wrote: > > > > And I just tested it with GAMG and it seems fine. And hypre ran, but it > is not clear that it used GPUs

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-11-02 Thread Smith, Barry F. via petsc-dev
> On Nov 2, 2018, at 1:25 PM, Mark Adams wrote: > > And I just tested it with GAMG and it seems fine. And hypre ran, but it is > not clear that it used GPUs Presumably hyper must be configured to use GPUs. Currently the PETSc hyper download installer hypre.py doesn't have any

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-11-02 Thread Mark Adams via petsc-dev
And I just tested it with GAMG and it seems fine. And hypre ran, but it is not clear that it used GPUs 14:13 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19 -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type hypre -ksp_type fgmres -snes_monitor_short -snes_rtol 1.e-5

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-11-02 Thread Mark Adams via petsc-dev
FYI, I seem to have the new GPU machine at ORNL (summitdev) working with GPUs. That is good enough for now. Thanks, 14:00 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19 -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type none -ksp_type fgmres -snes_monitor_short -snes_rtol 1.e-5

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-11-01 Thread Mark Adams via petsc-dev
On Wed, Oct 31, 2018 at 12:30 PM Mark Adams wrote: > > > On Wed, Oct 31, 2018 at 6:59 AM Karl Rupp wrote: > >> Hi Mark, >> >> ah, I was confused by the Python information at the beginning of >> configure.log. So it is picking up the correct compiler. >> >> Have you tried uncommenting the check

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-31 Thread Karl Rupp via petsc-dev
Hi Mark, ah, I was confused by the Python information at the beginning of configure.log. So it is picking up the correct compiler. Have you tried uncommenting the check for GNU? Best regards, Karli On 10/31/18 11:40 AM, Mark Adams wrote: It looks like configure is not finding the correct

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-31 Thread Mark Adams via petsc-dev
It looks like configure is not finding the correct cc. It does not seem hard to find. 06:37 master= /lustre/atlas/proj-shared/geo127/petsc$ cc --version gcc (GCC) 6.3.0 20161221 (Cray Inc.) Copyright (C) 2016 Free Software Foundation, Inc. This is free software; see the source for copying

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-31 Thread Mark Adams via petsc-dev
On Wed, Oct 31, 2018 at 5:05 AM Karl Rupp wrote: > Hi Mark, > > please comment or remove lines 83 and 84 in > config/BuildSystem/config/packages/cuda.py > > Is there a compiler newer than GCC 4.3 available? > You mean 6.3? 06:33 ~$ module avail gcc

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-31 Thread Karl Rupp via petsc-dev
Hi Mark, please comment or remove lines 83 and 84 in config/BuildSystem/config/packages/cuda.py Is there a compiler newer than GCC 4.3 available? Best regards, Karli On 10/31/18 8:15 AM, Mark Adams via petsc-dev wrote: After loading a cuda module ... On Wed, Oct 31, 2018 at 2:58 AM Mark

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-30 Thread Smith, Barry F. via petsc-dev
--with-cudac=1 should be --with-cuda=1 > On Oct 30, 2018, at 12:35 PM, Smith, Barry F. via petsc-dev > wrote: > > > >> On Oct 29, 2018, at 8:09 PM, Mark Adams wrote: >> >> And a debug build seems to work: > >Well ok. > >Are there newer versions of the Gnu compiler for this

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-30 Thread Mark Adams via petsc-dev
> > > > Are there newer versions of the Gnu compiler for this system? Yes: -- /opt/modulefiles --

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-30 Thread Smith, Barry F. via petsc-dev
> On Oct 29, 2018, at 8:09 PM, Mark Adams wrote: > > And a debug build seems to work: Well ok. Are there newer versions of the Gnu compiler for this system? Are there any other compilers on the system that would likely be less buggy? IBM compilers? If this simple code generates a

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-29 Thread Mark Adams via petsc-dev
And a debug build seems to work: 21:04 1 master= /lustre/atlas/proj-shared/geo127/petsc$ make PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda PETSC_ARCH="" test Running test examples to verify correct installation Using

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-29 Thread Mark Adams via petsc-dev
On Mon, Oct 29, 2018 at 6:55 PM Smith, Barry F. wrote: > >Here is the code > > > PetscStackCallBLAS("LAPACKgeev",LAPACKgeev_("N","N",,R,,realpart,imagpart,work,,)); > if (lierr) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"Error in LAPACK > routine %d",(int)lierr); > >What is

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-29 Thread Smith, Barry F. via petsc-dev
Here is the code PetscStackCallBLAS("LAPACKgeev",LAPACKgeev_("N","N",,R,,realpart,imagpart,work,,)); if (lierr) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"Error in LAPACK routine %d",(int)lierr); What is unfathomable is that it prints (int) lierr of 0 but then the if () test

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-29 Thread Mark Adams via petsc-dev
Still getting this error with downloaded lapack. I sent the logs on the other thread. 18:02 master= /lustre/atlas/proj-shared/geo127/petsc$ make PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda PETSC_ARCH="" test Running test examples to verify correct installation Using

Re: [petsc-dev] Error running on Titan with GPUs & GNU

2018-10-29 Thread Mark Adams via petsc-dev
On Mon, Oct 29, 2018 at 5:07 PM Matthew Knepley wrote: > On Mon, Oct 29, 2018 at 5:01 PM Mark Adams via petsc-dev < > petsc-dev@mcs.anl.gov> wrote: > >> I get this error running the tests using GPUs. An error in an LAPACK >> routine. >> > > From the command line, it does not look like GPUs are