Re: [petsc-dev] configure issues with new MKL and --with-mkl_sparse_optimize=0

gracefully and configure to continue. That is: I don't want the fact that I am looking for this package automatically to imply that it HAS to be there. Is there any easy way to get this behavior? --Richard On 11/27/18 2:15 PM, Smith, Barry F. wrote: On Nov 26, 2018, at 6:12 PM, Mills, Richard Tran

Re: [petsc-dev] configure issues with new MKL and --with-mkl_sparse_optimize=0

On 12/3/18 5:43 PM, Matthew Knepley wrote: On Mon, Dec 3, 2018 at 8:35 PM Mills, Richard Tran via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: Sorry, everyone, for how long it took me to have time to get back to this. I tried the change I suggested in my previous message, but configure

Re: [petsc-dev] configure issues with new MKL and --with-mkl_sparse_optimize=0

Hi Stefano, Apologies for the slow reply; I was out for the US Thanksgiving Holiday. You've found an error in the logic for the MKL configure tests: Your MKL does indeed have mkl_sparse_sp2m(), but since this depends on having mkl_sparse_optimize() and you have specified

Re: [petsc-dev] configure issues with new MKL and --with-mkl_sparse_optimize=0

Oh, I should also have asked: Why are you telling PETSc to not use mkl_sparse_optimize()? Does it cause some sort of problem if you build with that on? --Richard On 11/21/18 10:59 PM, Stefano Zampini wrote: Richard, I just noticed that PETSc master does not build with the options

Re: [petsc-dev] Is there a good reason that BuildSystem's cuda.py requires GNU compilers?

to employ the right combination of find and grep to figure out where PCC_FLAGS and CCPPFLAGS come from. --Richard --Richard Satish On Fri, 22 Mar 2019, Mills, Richard Tran via petsc-dev wrote: On 3/18/19 7:29 PM, Balay, Satish wrote: On Tue, 19 Mar 2019, Mills, Richard Tran via petsc-dev wro

Re: [petsc-dev] Is there a good reason that BuildSystem's cuda.py requires GNU compilers?

values of and ? I'm not sure what modules I need to import from BuildSystem... --Richard Satish On Fri, 22 Mar 2019, Mills, Richard Tran via petsc-dev wrote: On 3/18/19 7:29 PM, Balay, Satish wrote: On Tue, 19 Mar 2019, Mills, Richard Tran via petsc-dev wrote: Colleagues, It took me a while t

Re: [petsc-dev] Is there a good reason that BuildSystem's cuda.py requires GNU compilers?

CPPFLAGS come from. --Richard --Richard Satish On Fri, 22 Mar 2019, Mills, Richard Tran via petsc-dev wrote: On 3/18/19 7:29 PM, Balay, Satish wrote: On Tue, 19 Mar 2019, Mills, Richard Tran via petsc-dev wrote: Colleagues, It took me a while to get PETSc to build at all with anything

Re: [petsc-dev] Note about CPR preconditioner in PCFIELDSPLIT man page?

On 3/3/19 8:21 PM, Jed Brown wrote: "Mills, Richard Tran" writes: Alright, I see the source of my confusion: I've now seen three different that are all called a "Constrained Pressure Residual" preconditioner. What I have generally seen is exactly what Matt

Re: [petsc-dev] Note about CPR preconditioner in PCFIELDSPLIT man page?

Alright, I see the source of my confusion: I've now seen three different that are all called a "Constrained Pressure Residual" preconditioner. What I have generally seen is exactly what Matt describes: R is just a restriction to a subset of DoFs (pressure, and sometimes saturation), and P is

[petsc-dev] Note about CPR preconditioner in PCFIELDSPLIT man page?

Folks, There is a note in the PCFIELDSPLIT manual page about implementing a so-called "Constrained Pressure Residual" (CPR) preconditioner: "The Constrained Pressure Preconditioner (CPR) can be implemented using

[petsc-dev] Errors in MatSetValuesBlockedLocal() when using AIJ, but not BAIJ!

Folks, I'm working with some PFLOTRAN examples, and I've been scratching my head for a while at errors that are being generated in MatSetValuesBlockedLocal(), related to new nonzeros requiring a malloc. E.g., [0]PETSC ERROR: Argument out of range [0]PETSC ERROR: New nonzero at (2,0) caused a

Re: [petsc-dev] Errors in MatSetValuesBlockedLocal() when using AIJ, but not BAIJ!

On 2/21/19 1:48 PM, Smith, Barry F. wrote: On Feb 20, 2019, at 12:50 PM, Mills, Richard Tran via petsc-dev <mailto:petsc-dev@mcs.anl.gov> wrote: Folks, I'm working with some PFLOTRAN examples, and I've been scratching my head for a while at errors that are being gen

Re: [petsc-dev] Is there a good reason that BuildSystem's cuda.py requires GNU compilers?

variable, strangely.) I will see if setting '-ccbin' to the path for whatever C++ compiler has been specified to configure.py works. --Richard Satish On Wed, 13 Mar 2019, Mills, Richard Tran via petsc-dev wrote: Fellow PETSc developers, If I try to configure PETSc with CUDA support on the ORNL

Re: [petsc-dev] Is there a good reason that BuildSystem's cuda.py requires GNU compilers?

g the Nvidia tools to use the compiler you want it to use. No one has made a serious effort to use any other compilers but Gnu (at least not publicly). Barry On Mar 12, 2019, at 10:40 PM, Mills, Richard Tran via petsc-dev <mailto:petsc-dev@mcs.anl.gov> wrote: Fellow PETSc develope

[petsc-dev] Is there a good reason that BuildSystem's cuda.py requires GNU compilers?

Fellow PETSc developers, If I try to configure PETSc with CUDA support on the ORNL Summit system using non-GNU compilers, I run into an error due to the following code in packages/cuda.py: def configureTypes(self): import config.setCompilers if not

Re: [petsc-dev] Is there a good reason that BuildSystem's cuda.py requires GNU compilers?

On Mar 12, 2019, at 10:40 PM, Mills, Richard Tran via petsc-dev <mailto:petsc-dev@mcs.anl.gov> wrote: Fellow PETSc developers, If I try to configure PETSc with CUDA support on the ORNL Summit system using non-GNU compilers, I run into an error due to the following code in packages

Re: [petsc-dev] https://www.dursi.ca/post/hpc-is-dying-and-mpi-is-killing-it.html

I've seen this quite some time ago. Others in this thread have already articulated many of the same criticisms I have with the material in this blog post, as well as some of the problems that I have with MPI, so I'll content myself by asking the following: If HPC is as dying as this guy says

Re: [petsc-dev] Is there a good reason that BuildSystem's cuda.py requires GNU compilers?

On 3/18/19 7:29 PM, Balay, Satish wrote: On Tue, 19 Mar 2019, Mills, Richard Tran via petsc-dev wrote: Colleagues, It took me a while to get PETSc to build at all with anything on Summit other than the GNU compilers, but, once this was accomplished, editing out the isGNU() test

Re: [petsc-dev] https://www.dursi.ca/post/hpc-is-dying-and-mpi-is-killing-it.html

n Mar 19, 2019 3:45 PM, "Smith, Barry F. via petsc-dev" <mailto:petsc-dev@mcs.anl.gov> wrote: > On Mar 19, 2019, at 12:27 AM, Mills, Richard Tran via petsc-dev > <mailto:petsc-dev@mcs.anl.gov> wrote: > > I've seen this quite some time ago. Others in this threa

[petsc-dev] User(s) manual sections field in manual pages?

Colleagues, I have noticed that we have a "Users manual sections" section in the MatNullSpaceCreate() manual page, and an empty "User manual sections" section (which I suppose should be corrected to "Users manual sections", since it is officially the "PETSc Users Manual"). Those appear to be

[petsc-dev] Controlling matrix type on different levels of multigrid hierarchy? (Motivation is GPUs)

Colleagues, I think we ought to have a way to control which levels of a PETSc multigrid solve happen on the GPU vs. the CPU, as I'd like to keep coarse levels on the CPU, but run the calculations for finer levels on the GPU. Currently, for a code that is using a DM to manage its grid, one can

Re: [petsc-dev] alternatives to cygwin on Windows with PETSc

I played around with WSL1 quite some time ago and it seemed pretty promising. I have not tried WSL2, but I'm guessing that it may be the best option for building PETSc on a Windows 10 machine. I've got a Windows 10 machine (it basically just runs my television/media center) and I'll give it a

Re: [petsc-dev] alternatives to cygwin on Windows with PETSc

a "regular" Windows users). Barry On Jul 1, 2019, at 3:59 PM, Mills, Richard Tran via petsc-dev <mailto:petsc-dev@mcs.anl.gov> wrote: I played around with WSL1 quite some time ago and it seemed pretty promising. I have not tried WSL2, but I'm guessing that it may be the best

[petsc-dev] Should we add something about GPU support to the user manual?

Fellow PETSc developers, I've had a few people recently ask me something along the lines of "Where do I look in the user manual for information about how to use GPUs with PETSc?", and then I have to give them the slightly embarrassing answer that there is nothing in there. Since we officially

Re: [petsc-dev] Gitlab notifications

On 9/12/19 6:33 AM, Jed Brown via petsc-dev wrote: [...] Can you explain CODEOWNERS to me? I cannot find it on the GItlab site. I want to see every MR. https://docs.gitlab.com/ee/user/project/code_owners.html We currently require approval from Integration (of which you are a member) and a

Re: [petsc-dev] Configure hangs on Summit

Hi Junchao, Glad you've found a workaround, but I don't know why you are hitting this problem. The last time I built PETSc on Summit (just a couple days ago), I didn't have this problem. I'm working from the example template that's in the PETSc repo at config/examples/arch-olcf-summit-opt.py.

Re: [petsc-dev] MatMult on Summit

Junchao, Can you share your 'jsrun' command so that we can see how you are mapping things to resource sets? --Richard On 9/20/19 11:22 PM, Zhang, Junchao via petsc-dev wrote: I downloaded a sparse matrix (HV15R) from Florida Sparse Matrix Collection. Its

Re: [petsc-dev] Configure hangs on Summit

tch system and automatically add the batch option. That would be a nice little feature for someone to add. Probably just a few lines of code. On Sep 20, 2019, at 8:59 PM, Mills, Richard Tran via petsc-dev <mailto:petsc-dev@mcs.anl.gov> wrote: Hi Junchao, Glad you've found a workaro

Re: [petsc-dev] MatMult on Summit

t the default layout for CPU cores is not good. > >>> You will want 3 cores on each socket then 12 on each. > >>> > >>> Thanks > >>> > >>> Barry > >>> > >>> Since Tim is one of our reviewers next we

Re: [petsc-dev] MatMult on Summit

ng. For completeness please run also 24 and 42 CPUs > >>> without the GPUs. Note that the default layout for CPU cores is not good. > >>> You will want 3 cores on each socket then 12 on each. > >>> > >>> Thanks > >>> > >>&

Re: [petsc-dev] MatMult on Summit

120 MB of L3 embedded DRAM (eDRAM)." --Junchao Zhang On Mon, Sep 23, 2019 at 11:58 AM Mills, Richard Tran via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: L3 and L2 are shared between cores, actually. See the attached 'lstopo' PDF output from a Summit compute node to see an illust

Re: [petsc-dev] MatMult on Summit

In MatMultAdd_SeqAIJCUSPARSE, before Junchao's changes, towards the end of the function it had if (!yy) { /* MatMult */ if (!cusparsestruct->stream) { ierr = WaitForGPU();CHKERRCUDA(ierr); } } I assume we don't need the logic to do this only in the MatMult() with no add case

Re: [petsc-dev] MatMult on Summit

PM, Zhang, Junchao wrote: It looks cusparsestruct->stream is always created (not NULL). I don't know logic of the "if (!cusparsestruct->stream)". --Junchao Zhang On Mon, Sep 23, 2019 at 5:04 PM Mills, Richard Tran via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: In M

Re: [petsc-dev] CUDA STREAMS

Mark, It looks like you are missing some critical CUDA library (or libraries) in your link line. I know you will at least need the CUDA runtime "-lcudart". Look at something like PETSC_WITH_EXTERNAL_LIB for one of your CUDA-enabled PETSc builds in $PETSC_ARCH/lib/petsc/conf/petscvariables to

[petsc-dev] Why no SpGEMM support in AIJCUSPARSE and AIJVIENNACL?

Fellow PETSc developers, I am wondering why the AIJCUSPARSE and AIJVIENNACL matrix types do not support the sparse matrix-matrix multiplication (SpGEMM, or MatMatMult() in PETSc parlance) routines provided by cuSPARSE and ViennaCL, respectively. Is there a good reason that I shouldn't add

Re: [petsc-dev] MatMult on Summit

), unless one calls WaitForGPU() after each operation. Some of the non-deterministic tests can likely be explained by this. I'll clean this up in the next few hours if there are no objections. Best regards, Karli On 9/24/19 1:05 AM, Mills, Richard Tran via petsc-dev wrote: I'm no CUDA expert (not yet

Re: [petsc-dev] Should we add something about GPU support to the user manual?

r post them on the online? Thanks, -Gautam On Sep 12, 2019, at 10:18 AM, Mills, Richard Tran via petsc-dev mailto:petsc-dev@mcs.anl.gov>> wrote: Fellow PETSc developers, I've had a few people recently ask me something along the lines of "Where do I look in the user manual for inform

Re: [petsc-dev] error with karlrupp/fix-cuda-streams

On 9/25/19 11:38 AM, Mark Adams via petsc-dev wrote: [...] > jsrun does take -n. It just has other args. I am trying to check if it > requires other args. I thought it did but let me check. https://www.olcf.ornl.gov/for-users/system-user-guides/summitdev-quickstart-guide/ -n --nrs Number

Re: [petsc-dev] MR seemed to vanish

Mark, I see it at https://gitlab.com/petsc/petsc/merge_requests/2130? Is that not the right MR? --Richard On 10/10/19 3:51 PM, Mark Adams via petsc-dev wrote: My MR mark/gamg-eigest-sa-cheby seems to have vanished. It does not seem to be in master. Anyone know where it is? Thanks, Mark

Re: [petsc-dev] Why no SpGEMM support in AIJCUSPARSE and AIJVIENNACL?

: http://viennacl.sourceforge.net/viennacl-benchmarks-spmm.html I agree that it would be good to have a GPU-MatMatMult for the sake of experiments. Under these performance constraints it's not top priority, though. Best regards, Karli On 10/3/19 12:00 AM, Mills, Richard Tran via petsc-dev wrote: Fellow

[petsc-dev] Postdoctoral position at Argonne: Numerical Solvers for Next Generation High Performance Computing Architectures

Dear PETSc Users and Developers, The PETSc/TAO team at Argonne National Laboratory has an opening for a postdoctoral researcher to work on development of robust and efficient algebraic solvers and related technologies targeting exascale-class supercomputers -- such as the Aurora machine slated

[petsc-dev] Meaning of PETSc matrices with zero rows but nonzero columns?

All, I'm working on adding support for matrix products to AIJMKL, and I'm uncertain about some issues surrounding empty matrices. PETSc will happily let me multiply an empty matrix with another (and this arises in the sequential matrix-matrix multiplication routines when running with multiple

Re: [petsc-dev] Meaning of PETSc matrices with zero rows but nonzero columns?

Thanks for the replies, everyone. It suppose is not actually that hard for me to handle these dimensions properly -- I just hadn't personally encountered or thought much about when such operations with empty matrices might arise, and was initially puzzled about what multiplication by an "empty"

[petsc-dev] Argonne National Laboratory hiring for staff position in Numerical PDEs and Scientific Computing

Dear PETSc Users and Developers, The Laboratory for Applied Mathematics, Numerical Software, and Statistics (LANS, https://www.anl.gov/mcs/lans) in the Mathematics and Computer Science Division at Argonne National Laboratory -- which has served as the "home" for PETSc development for over two

Re: [petsc-dev] Job openings

petsc-announce is used so infrequently that I've often found it a little odd that petsc-users isn't just used for the announcements. Perhap we could repurpose petsc-announce somewhat by making it something that users *can* post to, but posts are moderated. Then we can make sure that job posting

Re: [petsc-dev] -with-kokkos-cuda-arch=AMPERE80 nonsense

You raise a good point, Barry. I've been completely mystified by what some of these names even mean. What does "PASCAL60" vs. "PASCAL61" even mean? Do you know of where this is even documented? I can't really find anything about it in the Kokkos documentation. The only thing I can really find

Re: [petsc-dev] -with-kokkos-cuda-arch=AMPERE80 nonsense

Hmm, OK, I found a table at https://sparta.sandia.gov/doc/accelerate_kokkos.html and it tells me that "PASCAL60" refers to "NVIDIA Pascal generation CC 6.0 GPU" and "PASCAL61" refers to "NVIDIA Pascal generation CC 6.1 GPU". But I have no idea what those 6.0 vs 6.1 version numbers mean, and