Re: [petsc-users] How to install in /usr/lib64 instead of /usr/lib?

[Jed Brown]

The make convention would be to respond to `libdir`, which is probably the 
simplest if we can defer that choice until install time. It probably needs to 
be known at build time, thus should go in configure.

https://www.gnu.org/software/make/manual/html_node/Directory-Variables.html

Satish Balay via petsc-users  writes:

> For now - perhaps the following patch...
>
> Satish
>
> ---
>
> diff --git a/config/install.py b/config/install.py
> index 017bb736542..00f857f939e 100755
> --- a/config/install.py
> +++ b/config/install.py
> @@ -76,9 +76,9 @@ class Installer(script.Script):
>  self.archBinDir= os.path.join(self.rootDir, self.arch, 'bin')
>  self.archLibDir= os.path.join(self.rootDir, self.arch, 'lib')
>  self.destIncludeDir= os.path.join(self.destDir, 'include')
> -self.destConfDir   = os.path.join(self.destDir, 'lib','petsc','conf')
> -self.destLibDir= os.path.join(self.destDir, 'lib')
> -self.destBinDir= os.path.join(self.destDir, 'lib','petsc','bin')
> +self.destConfDir   = os.path.join(self.destDir, 
> 'lib64','petsc','conf')
> +self.destLibDir= os.path.join(self.destDir, 'lib64')
> +self.destBinDir= os.path.join(self.destDir, 
> 'lib64','petsc','bin')
>  self.installIncludeDir = os.path.join(self.installDir, 'include')
>  self.installBinDir = os.path.join(self.installDir, 
> 'lib','petsc','bin')
>  self.rootShareDir  = os.path.join(self.rootDir, 'share')
>
> On Thu, 5 Jan 2023, Fellype via petsc-users wrote:
>
>> Hi,
>> I'm building petsc from sources on a 64-bit Slackware Linux and I would like 
>> to know how to install the libraries in /usr/lib64 instead of /usr/lib. Is 
>> it possible? I've not found an option like --libdir=DIR to pass to 
>> ./configure.
>> 
>> Regards,
>> Fellype

Re: [petsc-users] MatCreateSeqAIJWithArrays for GPU / cusparse

[Junchao Zhang]

On Fri, Jan 6, 2023 at 7:35 PM Mark Lohry  wrote:

> Well, I think it's a moderately crazy idea unless it's less painful to
> implement than I'm thinking. Is there a use case for a mixed device system
> where one petsc executable might be addressing both a HIP and CUDA device
> beyond some frankenstein test system somebody cooked up? In all my code I
> implicitly assume I have either have one host with one device or one host
> with zero devices. I guess you can support these weird scenarios, but why?
> Life is hard enough supporting one device compiler with one host compiler.
>
> Many thanks Junchao -- with combinations of SetPreallocation I was able to
> grab allocated pointers out of petsc. Now I have all the jacobian
> construction on device with no copies.
>
Hi, Mark, could you say a few words about how you assemble matrices on
GPUs?  We ported MatSetValues like routines to GPUs but did not continue
this approach since we have to resolve data races between GPU threads.


>
> On Fri, Jan 6, 2023 at 12:27 AM Barry Smith  wrote:
>
>>
>>   So Jed's "everyone" now consists of "no one" and Jed can stop
>> complaining that "everyone" thinks it is a bad idea.
>>
>>
>>
>> On Jan 5, 2023, at 11:50 PM, Junchao Zhang 
>> wrote:
>>
>>
>>
>>
>> On Thu, Jan 5, 2023 at 10:32 PM Barry Smith  wrote:
>>
>>>
>>>
>>> > On Jan 5, 2023, at 3:42 PM, Jed Brown  wrote:
>>> >
>>> > Mark Adams  writes:
>>> >
>>> >> Support of HIP and CUDA hardware together would be crazy,
>>> >
>>> > I don't think it's remotely crazy. libCEED supports both together and
>>> it's very convenient when testing on a development machine that has one of
>>> each brand GPU and simplifies binary distribution for us and every package
>>> that uses us. Every day I wish PETSc could build with both simultaneously,
>>> but everyone tells me it's silly.
>>>
>>>   Not everyone at all; just a subset of everyone. Junchao is really the
>>> hold-out :-)
>>>
>> I am not, instead I think we should try (I fully agree it can ease binary
>> distribution).  But satish needs to install such a machine first :)
>> There are issues out of our control if we want to mix GPUs in execution.
>> For example, how to do VexAXPY on a cuda vector and a hip vector? Shall we
>> do it on the host? Also, there are no gpu-aware MPI implementations
>> supporting messages between cuda memory and hip memory.
>>
>>>
>>>   I just don't care about "binary packages" :-); I think they are an
>>> archaic and bad way of thinking about code distribution (but yes the
>>> alternatives need lots of work to make them flawless, but I think that is
>>> where the work should go in the packaging world.)
>>>
>>>I go further and think one should be able to automatically use a CUDA
>>> vector on a HIP device as well, it is not hard in theory but requires
>>> thinking about how we handle classes and subclasses a little to make it
>>> straightforward; or perhaps Jacob has fixed that also?
>>
>>
>>
>>
>>

Re: [petsc-users] MatCreateSeqAIJWithArrays for GPU / cusparse

[Mark Lohry]

Well, I think it's a moderately crazy idea unless it's less painful to
implement than I'm thinking. Is there a use case for a mixed device system
where one petsc executable might be addressing both a HIP and CUDA device
beyond some frankenstein test system somebody cooked up? In all my code I
implicitly assume I have either have one host with one device or one host
with zero devices. I guess you can support these weird scenarios, but why?
Life is hard enough supporting one device compiler with one host compiler.

Many thanks Junchao -- with combinations of SetPreallocation I was able to
grab allocated pointers out of petsc. Now I have all the jacobian
construction on device with no copies.

On Fri, Jan 6, 2023 at 12:27 AM Barry Smith  wrote:

>
>   So Jed's "everyone" now consists of "no one" and Jed can stop
> complaining that "everyone" thinks it is a bad idea.
>
>
>
> On Jan 5, 2023, at 11:50 PM, Junchao Zhang 
> wrote:
>
>
>
>
> On Thu, Jan 5, 2023 at 10:32 PM Barry Smith  wrote:
>
>>
>>
>> > On Jan 5, 2023, at 3:42 PM, Jed Brown  wrote:
>> >
>> > Mark Adams  writes:
>> >
>> >> Support of HIP and CUDA hardware together would be crazy,
>> >
>> > I don't think it's remotely crazy. libCEED supports both together and
>> it's very convenient when testing on a development machine that has one of
>> each brand GPU and simplifies binary distribution for us and every package
>> that uses us. Every day I wish PETSc could build with both simultaneously,
>> but everyone tells me it's silly.
>>
>>   Not everyone at all; just a subset of everyone. Junchao is really the
>> hold-out :-)
>>
> I am not, instead I think we should try (I fully agree it can ease binary
> distribution).  But satish needs to install such a machine first :)
> There are issues out of our control if we want to mix GPUs in execution.
> For example, how to do VexAXPY on a cuda vector and a hip vector? Shall we
> do it on the host? Also, there are no gpu-aware MPI implementations
> supporting messages between cuda memory and hip memory.
>
>>
>>   I just don't care about "binary packages" :-); I think they are an
>> archaic and bad way of thinking about code distribution (but yes the
>> alternatives need lots of work to make them flawless, but I think that is
>> where the work should go in the packaging world.)
>>
>>I go further and think one should be able to automatically use a CUDA
>> vector on a HIP device as well, it is not hard in theory but requires
>> thinking about how we handle classes and subclasses a little to make it
>> straightforward; or perhaps Jacob has fixed that also?
>
>
>
>
>

Re: [petsc-users] Getting correct local size using VecScatterCreateToAll

[Matthew Knepley]

On Fri, Jan 6, 2023 at 6:22 PM Venugopal, Vysakh (venugovh) via petsc-users
 wrote:

> Hello,
>
>
>
> I have created a global vector V using DMCreateGlobalVector of size m. For
> n processes, the local size of V is m/n.
>
>
>
> Subsequently, I am using VecScatterCreateToAll to get a sequential copy of
> the V, let's call it V_seq of local size m. It passes through a function
> and outputs the vector V_seq_hat (of local size m).
>
>
>
> Is there a way for me to substitute the values of V_seq_hat (at the
> correct indices) to the original V (with local size m/n)?
>
>
>
> When I use VecScatterCreateToAll (with SCATTER_REVERSE), I am getting a
> vector V with new values from V_seq_hat but with local size m. This is
> causing issues in the rest of my code where the size of V needs to be m/n.
>

You do not create a new scatter. You use the same scatter you got in the
first place, but with SCATTER_REVERSE.

  Thanks,

 Matt


> Any help would be really appreciated!
>
>
>
> Thanks,
>
>
>
> ---
>
> Vysakh Venugopal
>
> Ph.D. Candidate
>
> Department of Mechanical Engineering
>
> University of Cincinnati, Cincinnati, OH 45221-0072
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Error running configure on HDF5 in PETSc-3.18.3

[Danyang Su]

Hi All,

Problem is resolved by Homebrew Gfortran. I use GNU Fortran (GCC) 8.2.0 before. 
Conda does not cause the problem. 

Thanks,

Danyang

On 2023-01-06, 2:24 PM, "Satish Balay" mailto:ba...@mcs.anl.gov>> wrote:


Likely your installed gfortran is incompatible with hdf5



Executing: gfortran --version
stdout:
GNU Fortran (GCC) 8.2.0



We generally use brew gfortran - and that works with hdf5 aswell


balay@ypro ~ % gfortran --version
GNU Fortran (Homebrew GCC 11.2.0_1) 11.2.0


Satish


On Fri, 6 Jan 2023, Danyang Su wrote:


> Hi Pierre,
> 
> 
> 
> I have tried to exclude Conda related environment variables but it does not 
> work. Instead, if I include ‘--download-hdf5=yes’ but exclude 
> ‘--with-hdf5-fortran-bindings’ in the configuration, PETSc can be configured 
> and installed without problem, even with Conda related environment activated. 
> However, since my code requires fortran interface to HDF5, I do need 
> ‘--with-hdf5-fortran-bindings’, otherwise, my code cannot be compiled.
> 
> 
> 
> Any other suggestions?
> 
> 
> 
> Thanks,
> 
> 
> 
> Danyang
> 
> 
> 
> From: Pierre Jolivet mailto:pie...@joliv.et>>
> Date: Friday, January 6, 2023 at 7:59 AM
> To: Danyang Su mailto:danyang...@gmail.com>>
> Cc: mailto:petsc-users@mcs.anl.gov>>
> Subject: Re: [petsc-users] Error running configure on HDF5 in PETSc-3.18.3
> 
> 
> 
> 
> 
> 
> 
> On 6 Jan 2023, at 4:49 PM, Danyang Su  > wrote:
> 
> 
> 
> Hi All,
> 
> 
> 
> I get ‘Error running configure on HDF5’ in PETSc-3.18.3 on MacOS, but no 
> problem on Ubuntu. Attached is the configuration log file. 
> 
> 
> 
> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mumps 
> --download-scalapack --download-parmetis --download-metis --download-ptscotch 
> --download-fblaslapack --download-mpich --download-hypre 
> --download-superlu_dist --download-hdf5=yes --with-debugging=0 
> --download-cmake --with-hdf5-fortran-bindings
> 
> 
> 
> Any idea on this?
> 
> 
> 
> Could you try to reconfigure in a shell without conda being activated?
> 
> You have 
> PATH=/Users/danyangsu/Soft/Anaconda3/bin:/Users/danyangsu/Soft/Anaconda3/condabin:[…]
>  which typically results in a broken configuration.
> 
> 
> 
> Thanks,
> 
> Pierre
> 
> 
> 
> Thanks,
> 
> 
> 
> Danyang
> 
> 
> 
> 
> 

[petsc-users] Getting correct local size using VecScatterCreateToAll

[Venugopal, Vysakh (venugovh) via petsc-users]

Hello,

I have created a global vector V using DMCreateGlobalVector of size m. For n 
processes, the local size of V is m/n.

Subsequently, I am using VecScatterCreateToAll to get a sequential copy of the 
V, let's call it V_seq of local size m. It passes through a function and 
outputs the vector V_seq_hat (of local size m).

Is there a way for me to substitute the values of V_seq_hat (at the correct 
indices) to the original V (with local size m/n)?

When I use VecScatterCreateToAll (with SCATTER_REVERSE), I am getting a vector 
V with new values from V_seq_hat but with local size m. This is causing issues 
in the rest of my code where the size of V needs to be m/n.

Any help would be really appreciated!

Thanks,

---
Vysakh Venugopal
Ph.D. Candidate
Department of Mechanical Engineering
University of Cincinnati, Cincinnati, OH 45221-0072

Re: [petsc-users] Getting global indices of vector distributed among different processes.

[Venugopal, Vysakh (venugovh) via petsc-users]

Thank you, Matthew!

From: Matthew Knepley 
Sent: Wednesday, January 4, 2023 10:52 AM
To: Venugopal, Vysakh (venugovh) 
Cc: petsc-users@mcs.anl.gov
Subject: Re: [petsc-users] Getting global indices of vector distributed among 
different processes.


External Email: Use Caution


On Wed, Jan 4, 2023 at 10:48 AM Venugopal, Vysakh (venugovh) via petsc-users 
mailto:petsc-users@mcs.anl.gov>> wrote:
Hello,

Is there a way to get the global indices from a vector created from 
DMCreateGlobalVector? Example:

If global vector V (of size 10) has indices {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} and 
they are divided into 2 processes. Is there a way to get information such as 
(process 1: {0,1,2,3,4}, process 2: {5,6,7,8,9})?

https://petsc.org/main/docs/manualpages/Vec/VecGetOwnershipRange/

  Thanks,

 Matt

The reason I need this information is that I need to query the values of a 
different vector Q of size 10 and place those values in V. Example: Q(1) --- 
V(1) @ process 1, Q(7) - V(7) @ process 2, etc.. If there are smarter ways to 
do this, I am happy to pursue that.

Thank you,

Vysakh V.

---
Vysakh Venugopal
Ph.D. Candidate
Department of Mechanical Engineering
University of Cincinnati, Cincinnati, OH 45221-0072



--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

Re: [petsc-users] Error running configure on HDF5 in PETSc-3.18.3

[Satish Balay via petsc-users]

Likely your installed gfortran is incompatible with hdf5


Executing: gfortran --version
stdout:
GNU Fortran (GCC) 8.2.0


We generally use brew gfortran - and that works with hdf5 aswell

balay@ypro ~ % gfortran --version
GNU Fortran (Homebrew GCC 11.2.0_1) 11.2.0

Satish

On Fri, 6 Jan 2023, Danyang Su wrote:

> Hi Pierre,
> 
>  
> 
> I have tried to exclude Conda related environment variables but it does not 
> work. Instead, if I include ‘--download-hdf5=yes’ but exclude 
> ‘--with-hdf5-fortran-bindings’ in the configuration, PETSc can be configured 
> and installed without problem, even with Conda related environment activated. 
> However, since my code requires fortran interface to HDF5, I do need 
> ‘--with-hdf5-fortran-bindings’, otherwise, my code cannot be compiled.
> 
>  
> 
> Any other suggestions?
> 
>  
> 
> Thanks,
> 
>  
> 
> Danyang
> 
>  
> 
> From: Pierre Jolivet 
> Date: Friday, January 6, 2023 at 7:59 AM
> To: Danyang Su 
> Cc: 
> Subject: Re: [petsc-users] Error running configure on HDF5 in PETSc-3.18.3
> 
>  
> 
>  
> 
> 
> 
> On 6 Jan 2023, at 4:49 PM, Danyang Su  wrote:
> 
>  
> 
> Hi All,
> 
>  
> 
> I get ‘Error running configure on HDF5’ in PETSc-3.18.3 on MacOS, but no 
> problem on Ubuntu. Attached is the configuration log file. 
> 
>  
> 
> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mumps 
> --download-scalapack --download-parmetis --download-metis --download-ptscotch 
> --download-fblaslapack --download-mpich --download-hypre 
> --download-superlu_dist --download-hdf5=yes --with-debugging=0 
> --download-cmake --with-hdf5-fortran-bindings
> 
>  
> 
> Any idea on this?
> 
>  
> 
> Could you try to reconfigure in a shell without conda being activated?
> 
> You have 
> PATH=/Users/danyangsu/Soft/Anaconda3/bin:/Users/danyangsu/Soft/Anaconda3/condabin:[…]
>  which typically results in a broken configuration.
> 
>  
> 
> Thanks,
> 
> Pierre
> 
> 
> 
> Thanks,
> 
>  
> 
> Danyang
> 
> 
> 
> 
> 

Re: [petsc-users] Error running configure on HDF5 in PETSc-3.18.3

[Barry Smith]

  Please email your latest configure.log  (with no Conda stuff) to 
petsc-ma...@mcs.anl.gov 

  The configuration of HDF5 (done by HDF5) is objecting to some particular 
aspect of your current Fortran compiler, we need to figure out the exact 
objection.

  Barry


> On Jan 6, 2023, at 5:14 PM, Danyang Su  wrote:
> 
> Hi Pierre,
>  
> I have tried to exclude Conda related environment variables but it does not 
> work. Instead, if I include ‘--download-hdf5=yes’ but exclude 
> ‘--with-hdf5-fortran-bindings’ in the configuration, PETSc can be configured 
> and installed without problem, even with Conda related environment activated. 
> However, since my code requires fortran interface to HDF5, I do need 
> ‘--with-hdf5-fortran-bindings’, otherwise, my code cannot be compiled.
>  
> Any other suggestions?
>  
> Thanks,
>  
> Danyang
>  
> From: Pierre Jolivet mailto:pie...@joliv.et>>
> Date: Friday, January 6, 2023 at 7:59 AM
> To: Danyang Su mailto:danyang...@gmail.com>>
> Cc: mailto:petsc-users@mcs.anl.gov>>
> Subject: Re: [petsc-users] Error running configure on HDF5 in PETSc-3.18.3
>  
>  
> 
> 
>> On 6 Jan 2023, at 4:49 PM, Danyang Su > > wrote:
>>  
>> Hi All,
>>  
>> I get ‘Error running configure on HDF5’ in PETSc-3.18.3 on MacOS, but no 
>> problem on Ubuntu. Attached is the configuration log file. 
>>  
>> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mumps 
>> --download-scalapack --download-parmetis --download-metis 
>> --download-ptscotch --download-fblaslapack --download-mpich --download-hypre 
>> --download-superlu_dist --download-hdf5=yes --with-debugging=0 
>> --download-cmake --with-hdf5-fortran-bindings
>>  
>> Any idea on this?
>  
> Could you try to reconfigure in a shell without conda being activated?
> You have 
> PATH=/Users/danyangsu/Soft/Anaconda3/bin:/Users/danyangsu/Soft/Anaconda3/condabin:[…]
>  which typically results in a broken configuration.
>  
> Thanks,
> Pierre
> 
> 
>> Thanks,
>>  
>> Danyang

Re: [petsc-users] Error running configure on HDF5 in PETSc-3.18.3

[Danyang Su]

Hi Pierre,

 

I have tried to exclude Conda related environment variables but it does not 
work. Instead, if I include ‘--download-hdf5=yes’ but exclude 
‘--with-hdf5-fortran-bindings’ in the configuration, PETSc can be configured 
and installed without problem, even with Conda related environment activated. 
However, since my code requires fortran interface to HDF5, I do need 
‘--with-hdf5-fortran-bindings’, otherwise, my code cannot be compiled.

 

Any other suggestions?

 

Thanks,

 

Danyang

 

From: Pierre Jolivet 
Date: Friday, January 6, 2023 at 7:59 AM
To: Danyang Su 
Cc: 
Subject: Re: [petsc-users] Error running configure on HDF5 in PETSc-3.18.3

 

 



On 6 Jan 2023, at 4:49 PM, Danyang Su  wrote:

 

Hi All,

 

I get ‘Error running configure on HDF5’ in PETSc-3.18.3 on MacOS, but no 
problem on Ubuntu. Attached is the configuration log file. 

 

./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mumps 
--download-scalapack --download-parmetis --download-metis --download-ptscotch 
--download-fblaslapack --download-mpich --download-hypre 
--download-superlu_dist --download-hdf5=yes --with-debugging=0 --download-cmake 
--with-hdf5-fortran-bindings

 

Any idea on this?

 

Could you try to reconfigure in a shell without conda being activated?

You have 
PATH=/Users/danyangsu/Soft/Anaconda3/bin:/Users/danyangsu/Soft/Anaconda3/condabin:[…]
 which typically results in a broken configuration.

 

Thanks,

Pierre



Thanks,

 

Danyang

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 12:18 PM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> I am doing that as well (although not for vertex dofs). And I had it
> working quite well for purely cell-associated DOFs. But I realized later
> that I also wanted to transmit some DOFs associated with faces so I suspect
> I'm messing something up there.
>
> Something we discussed back on 12/26 (email subject:Getting a vector from
> a DM to output VTK)  was associating a vector generated this way with the
> DM so it can be more easily visualized using vtk files is
>
> VecSetOperation(state_dist, VECOP_VIEW, (void(*)(void))VecView_Plex);
>
> If there is an easy way to get this working in Fortran I would very much
> appreciate it as it was very helpful when I was debugging in C.
>

This is a workaround for Fortran (the problem is that Fortran cannot easily
refer to the function pointers). You can

  1) Set the new section into the distributed DM

  2) Make a new global or local vector

  3) Call VecCopy() to move the data

The new vector will have the right viewer.

  Thanks,

 Matt


> Thanks
> Nicholas
>
> On Fri, Jan 6, 2023 at 11:37 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 11:32 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> This was generated using the DMPlexDistributeField which we discussed a
>>> while back. Everything seemed to be working fine when I only had cells dofs
>>> but I recently added face dofs, which seems to have caused some issues.
>>> Whats weird is that I'm feeding the same distribution SF and inputting a
>>> vector and section that are consistent to the DMPlexDistributeField so I'd
>>> expect the vectors and section output to be consistent. I'll take a closer
>>> look at that.
>>>
>>
>> The way I use DistributeField(), for example to distribute coordinates,
>> is that I give it the migrationSF, the local coordinate section, and the
>> local coordinate vector. It gives me back the new local coordinate section
>> (which I set into the distributed DM), and the local coordinate vector. It
>> seems like you are doing something else. Maybe the documentation is
>> misleading.
>>
>>   Thanks,
>>
>> Matt
>>
>>
>>> Thanks
>>> Nicholas
>>>
>>> On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Matt
>
> I appreciate the help. The section view is quite extensive because
> each cell has 55 dofs located at the cells and on certain faces. I've
> appended the first of these which corresponds with the output in the first
> email, to save space. The following 54 are exactly the same but offset
> incremented by 1. (or negative 1 for negative offsets)
>

 Okay, from the output it is clear that this vector does not match your
 global section. Did you get stateVec by calling DMCreateGlobalVector()?

   Thanks,

  Matt


> Thanks for your time
> Nicholas
>
> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley 
> wrote:
>
>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> I apologize for any lack of clarity in the initial email.
>>>
>>> looking at the initial output on rank 1
>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>> cell   0 offset2475 oStart2640-165
>>>  cell   1 offset2530 oStart2640-110
>>>  cell   2 offset2585 oStart2640 -55
>>>  cell   3 offset2640 oStart2640   0
>>> .
>>> cell  15 offset-771 oStart2640   -3411
>>>
>>>
>>> cell 15 provides a negative offset because it is the overlap cell
>>> (that is unowned)
>>> The remained of cells are all owned. However, the first 3 cells
>>> (0,1,2) return an offset that is less than the starting ownership 
>>> range. I
>>> would expect cell 0 to start at offset 2640 at minimum.
>>>
>>
>> Send the output for this section
>>
>>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Sincerely
>>> Nicholas
>>>
>>>
>>>
>>>
>>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Apologies. If it helps, there is one cell of overlap in this small
> test case for a 2D mesh that is 1 cell in height and a number of 
> cells in
> length. .
>
> process 0
>  Petsc VecGetLocalSize2750
>  size(stateVecV)2750
>
> process 1
> 

Re: [petsc-users] cuda gpu eager initialization error cudaErrorNotSupported

[Jacob Faibussowitsch]

Hmm I suspect the problem is that GPU is simply too old yes, but perhaps there is a simple enough workaround available in the code as you suggest. I will investigate further on Monday.Best regards,Jacob Faibussowitsch(Jacob Fai - booss - oh - vitch)On Jan 6, 2023, at 09:55, Mark Lohry  wrote:These cards do indeed not support cudaDeviceGetMemPool -- cudaDeviceGetAttribute on cudaDevAttrMemoryPoolsSupported return false, meaning it doesn't support cudaMallocAsync, so the first point of failure is the call to cudaDeviceGetMemPool in the initialization.Would a workaround be to replace the cudaMallocAsync call to cudaMalloc and skip the mempool or is that a bad idea?On Fri, Jan 6, 2023 at 9:17 AM Mark Lohry  wrote:It built+ran fine on a different system with an sm75 arch. Is there a documented minimum version if that indeed is the cause?One minor hiccup FYI -- compilation of hypre fails with cuda toolkit 12, due to cusprase removing csrsv2Info_t (although it's still referenced in their docs...) in favor of bsrsv2Info_t. Rolling back to cuda toolkit 11.8 worked.On Thu, Jan 5, 2023 at 6:37 PM Junchao Zhang  wrote:Jacob, is it because the cuda arch is too old? --Junchao ZhangOn Thu, Jan 5, 2023 at 4:30 PM Mark Lohry  wrote:I'm seeing the same thing on latest main with a different machine and -sm52 card, cuda 11.8. make check fails with the below, where the indicated line 249 corresponds to PetscCallCUPM(cupmDeviceGetMemPool(, static_cast(device->deviceId)));   in the initialize function. Running check examples to verify correct installationUsing PETSC_DIR=/home/mlohry/dev/petsc and PETSC_ARCH=arch-linux-c-debugC/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI processC/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes2,17c2,46<   0 SNES Function norm 2.391552133017e-01 <     0 KSP Residual norm 2.928487269734e-01 <     1 KSP Residual norm 1.876489580142e-02 <     2 KSP Residual norm 3.291394847944e-03 <     3 KSP Residual norm 2.456493072124e-04 <     4 KSP Residual norm 1.161647147715e-05 <     5 KSP Residual norm 1.285648407621e-06 <   1 SNES Function norm 6.846805706142e-05 <     0 KSP Residual norm 2.292783790384e-05 <     1 KSP Residual norm 2.100673631699e-06 <     2 KSP Residual norm 2.121341386147e-07 <     3 KSP Residual norm 2.455932678957e-08 <     4 KSP Residual norm 1.753095730744e-09 <     5 KSP Residual norm 7.489214418904e-11 <   2 SNES Function norm 2.103908447865e-10 < Number of SNES iterations = 2---> [0]PETSC ERROR: - Error Message --> [0]PETSC ERROR: GPU error> [0]PETSC ERROR: cuda error 801 (cudaErrorNotSupported) : operation not supported> [0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!> [0]PETSC ERROR: Option left: name:-mg_levels_ksp_max_it value: 3 source: command line> [0]PETSC ERROR: Option left: name:-nox (no value) source: environment> [0]PETSC ERROR: Option left: name:-nox_warning (no value) source: environment> [0]PETSC ERROR: Option left: name:-pc_gamg_esteig_ksp_max_it value: 10 source: command line> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.> [0]PETSC ERROR: Petsc Development GIT revision: v3.18.3-352-g91c56366cb  GIT Date: 2023-01-05 17:22:48 +> [0]PETSC ERROR: ./ex19 on a arch-linux-c-debug named osprey by mlohry Thu Jan  5 17:25:17 2023> [0]PETSC ERROR: Configure options --with-cuda --with-mpi=1> [0]PETSC ERROR: #1 initialize() at /home/mlohry/dev/petsc/src/sys/objects/device/impls/cupm/cuda/../cupmcontext.hpp:249> [0]PETSC ERROR: #2 PetscDeviceContextCreate_CUDA() at /home/mlohry/dev/petsc/src/sys/objects/device/impls/cupm/cuda/cupmcontext.cu:10> [0]PETSC ERROR: #3 PetscDeviceContextSetDevice_Private() at /home/mlohry/dev/petsc/src/sys/objects/device/interface/dcontext.cxx:247> [0]PETSC ERROR: #4 PetscDeviceContextSetDefaultDeviceForType_Internal() at /home/mlohry/dev/petsc/src/sys/objects/device/interface/dcontext.cxx:260> [0]PETSC ERROR: #5 PetscDeviceContextSetupGlobalContext_Private() at /home/mlohry/dev/petsc/src/sys/objects/device/interface/global_dcontext.cxx:52> [0]PETSC ERROR: #6 PetscDeviceContextGetCurrentContext() at /home/mlohry/dev/petsc/src/sys/objects/device/interface/global_dcontext.cxx:84> [0]PETSC ERROR: #7 GetHandleDispatch_() at /home/mlohry/dev/petsc/include/petsc/private/veccupmimpl.h:499> [0]PETSC ERROR: #8 create() at /home/mlohry/dev/petsc/include/petsc/private/veccupmimpl.h:1069> [0]PETSC ERROR: #9 VecCreate_SeqCUDA() at /home/mlohry/dev/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:10> [0]PETSC ERROR: #10 VecSetType() at /home/mlohry/dev/petsc/src/vec/vec/interface/vecreg.c:89> [0]PETSC ERROR: #11 DMCreateGlobalVector_DA() at /home/mlohry/dev/petsc/src/dm/impls/da/dadist.c:31> [0]PETSC ERROR: #12 DMCreateGlobalVector() at 

Re: [petsc-users] Definition of threshold(s) in gamg and boomerAMG

[Mark Adams]

These thresholds are for completely different coursening algorithms.
GAMG just drops edges below the threshold and hypre (classical AMG) does
something completely different.

Mark

On Fri, Jan 6, 2023 at 10:43 AM Edoardo Centofanti <
edoardo.centofant...@universitadipavia.it> wrote:

> Hi PETSc users,
>
> I was looking for the exact definitions of the threshold parameter
> (-pc_gamg_threshold) for gamg and of the strong threshold
> (-pc_hypre_boomeramg_strong_threshold) for Hypre BoomerAMG. My curiosity
> comes from the fact that the suggested parameters (apparently acting on the
> same aspect of the algorithms) differ of around one order of magnitude (if
> I remember right from the PETSc manual, the gamg threshold for 3D problems
> is around 0.05, while for boomerAMG ranges from 0.25 to 0.5).
>
> Thank you in advance,
> Edoardo
>

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

I am doing that as well (although not for vertex dofs). And I had it
working quite well for purely cell-associated DOFs. But I realized later
that I also wanted to transmit some DOFs associated with faces so I suspect
I'm messing something up there.

Something we discussed back on 12/26 (email subject:Getting a vector from a
DM to output VTK)  was associating a vector generated this way with the DM
so it can be more easily visualized using vtk files is

VecSetOperation(state_dist, VECOP_VIEW, (void(*)(void))VecView_Plex);

If there is an easy way to get this working in Fortran I would very much
appreciate it as it was very helpful when I was debugging in C.

Thanks
Nicholas

On Fri, Jan 6, 2023 at 11:37 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 11:32 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> This was generated using the DMPlexDistributeField which we discussed a
>> while back. Everything seemed to be working fine when I only had cells dofs
>> but I recently added face dofs, which seems to have caused some issues.
>> Whats weird is that I'm feeding the same distribution SF and inputting a
>> vector and section that are consistent to the DMPlexDistributeField so I'd
>> expect the vectors and section output to be consistent. I'll take a closer
>> look at that.
>>
>
> The way I use DistributeField(), for example to distribute coordinates, is
> that I give it the migrationSF, the local coordinate section, and the local
> coordinate vector. It gives me back the new local coordinate section (which
> I set into the distributed DM), and the local coordinate vector. It seems
> like you are doing something else. Maybe the documentation is misleading.
>
>   Thanks,
>
> Matt
>
>
>> Thanks
>> Nicholas
>>
>> On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Matt

 I appreciate the help. The section view is quite extensive because each
 cell has 55 dofs located at the cells and on certain faces. I've appended
 the first of these which corresponds with the output in the first email, to
 save space. The following 54 are exactly the same but offset incremented by
 1. (or negative 1 for negative offsets)

>>>
>>> Okay, from the output it is clear that this vector does not match your
>>> global section. Did you get stateVec by calling DMCreateGlobalVector()?
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 Thanks for your time
 Nicholas

 On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I apologize for any lack of clarity in the initial email.
>>
>> looking at the initial output on rank 1
>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>> cell   0 offset2475 oStart2640-165
>>  cell   1 offset2530 oStart2640-110
>>  cell   2 offset2585 oStart2640 -55
>>  cell   3 offset2640 oStart2640   0
>> .
>> cell  15 offset-771 oStart2640   -3411
>>
>>
>> cell 15 provides a negative offset because it is the overlap cell
>> (that is unowned)
>> The remained of cells are all owned. However, the first 3 cells
>> (0,1,2) return an offset that is less than the starting ownership range. 
>> I
>> would expect cell 0 to start at offset 2640 at minimum.
>>
>
> Send the output for this section
>
>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>
>   Thanks,
>
>  Matt
>
>
>> Sincerely
>> Nicholas
>>
>>
>>
>>
>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Apologies. If it helps, there is one cell of overlap in this small
 test case for a 2D mesh that is 1 cell in height and a number of cells 
 in
 length. .

 process 0
  Petsc VecGetLocalSize2750
  size(stateVecV)2750

 process 1
  Petsc VecGetLocalSize2640
  size(stateVecV)2640

>>>
>>> The offsets shown below are well-within these sizes. I do not
>>> understand the problem.
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I made a typo on the line statVecV(offset) = 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 11:32 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Hi Matt
>
> This was generated using the DMPlexDistributeField which we discussed a
> while back. Everything seemed to be working fine when I only had cells dofs
> but I recently added face dofs, which seems to have caused some issues.
> Whats weird is that I'm feeding the same distribution SF and inputting a
> vector and section that are consistent to the DMPlexDistributeField so I'd
> expect the vectors and section output to be consistent. I'll take a closer
> look at that.
>

The way I use DistributeField(), for example to distribute coordinates, is
that I give it the migrationSF, the local coordinate section, and the local
coordinate vector. It gives me back the new local coordinate section (which
I set into the distributed DM), and the local coordinate vector. It seems
like you are doing something else. Maybe the documentation is misleading.

  Thanks,

Matt


> Thanks
> Nicholas
>
> On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> I appreciate the help. The section view is quite extensive because each
>>> cell has 55 dofs located at the cells and on certain faces. I've appended
>>> the first of these which corresponds with the output in the first email, to
>>> save space. The following 54 are exactly the same but offset incremented by
>>> 1. (or negative 1 for negative offsets)
>>>
>>
>> Okay, from the output it is clear that this vector does not match your
>> global section. Did you get stateVec by calling DMCreateGlobalVector()?
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Thanks for your time
>>> Nicholas
>>>
>>> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Matt
>
> I apologize for any lack of clarity in the initial email.
>
> looking at the initial output on rank 1
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
> cell   0 offset2475 oStart2640-165
>  cell   1 offset2530 oStart2640-110
>  cell   2 offset2585 oStart2640 -55
>  cell   3 offset2640 oStart2640   0
> .
> cell  15 offset-771 oStart2640   -3411
>
>
> cell 15 provides a negative offset because it is the overlap cell
> (that is unowned)
> The remained of cells are all owned. However, the first 3 cells
> (0,1,2) return an offset that is less than the starting ownership range. I
> would expect cell 0 to start at offset 2640 at minimum.
>

 Send the output for this section

   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);

   Thanks,

  Matt


> Sincerely
> Nicholas
>
>
>
>
> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
> wrote:
>
>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Apologies. If it helps, there is one cell of overlap in this small
>>> test case for a 2D mesh that is 1 cell in height and a number of cells 
>>> in
>>> length. .
>>>
>>> process 0
>>>  Petsc VecGetLocalSize2750
>>>  size(stateVecV)2750
>>>
>>> process 1
>>>  Petsc VecGetLocalSize2640
>>>  size(stateVecV)2640
>>>
>>
>> The offsets shown below are well-within these sizes. I do not
>> understand the problem.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Matt
>
> I made a typo on the line statVecV(offset) =  in
> my example, I agree. (I wrote that offhand since the actual 
> assignment is
> much larger) I should be statVecV(offset+1) =  so I'm 
> confident
> it's not a 1 0 indexing thing.
>
> My question is more related to what is happening in the offsets.
> c0 and c1 are pulled using DMplexgetheight stratum, so they are
> zero-indexed (which is why I loop from c0 to (c1-1)).
>
> For the size inquiries. on processor 0
>  Petsc VecGetSize(stateVec)5390
>

 I need to see VecGetLocalSize()

   Matt


>  size(stateVecV)   2640
>
> on processor 1
> Petsc VecGetSize 5390
> size(stateVecV)2750
>
> It's quite weird to me that processor one can have a positive
> 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Hi Matt

This was generated using the DMPlexDistributeField which we discussed a
while back. Everything seemed to be working fine when I only had cells dofs
but I recently added face dofs, which seems to have caused some issues.
Whats weird is that I'm feeding the same distribution SF and inputting a
vector and section that are consistent to the DMPlexDistributeField so I'd
expect the vectors and section output to be consistent. I'll take a closer
look at that.

Thanks
Nicholas

On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I appreciate the help. The section view is quite extensive because each
>> cell has 55 dofs located at the cells and on certain faces. I've appended
>> the first of these which corresponds with the output in the first email, to
>> save space. The following 54 are exactly the same but offset incremented by
>> 1. (or negative 1 for negative offsets)
>>
>
> Okay, from the output it is clear that this vector does not match your
> global section. Did you get stateVec by calling DMCreateGlobalVector()?
>
>   Thanks,
>
>  Matt
>
>
>> Thanks for your time
>> Nicholas
>>
>> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Matt

 I apologize for any lack of clarity in the initial email.

 looking at the initial output on rank 1
 write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
 cell   0 offset2475 oStart2640-165
  cell   1 offset2530 oStart2640-110
  cell   2 offset2585 oStart2640 -55
  cell   3 offset2640 oStart2640   0
 .
 cell  15 offset-771 oStart2640   -3411


 cell 15 provides a negative offset because it is the overlap cell (that
 is unowned)
 The remained of cells are all owned. However, the first 3 cells (0,1,2)
 return an offset that is less than the starting ownership range. I would
 expect cell 0 to start at offset 2640 at minimum.

>>>
>>> Send the output for this section
>>>
>>>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 Sincerely
 Nicholas




 On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Apologies. If it helps, there is one cell of overlap in this small
>> test case for a 2D mesh that is 1 cell in height and a number of cells in
>> length. .
>>
>> process 0
>>  Petsc VecGetLocalSize2750
>>  size(stateVecV)2750
>>
>> process 1
>>  Petsc VecGetLocalSize2640
>>  size(stateVecV)2640
>>
>
> The offsets shown below are well-within these sizes. I do not
> understand the problem.
>
>   Thanks,
>
>  Matt
>
>
>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Matt

 I made a typo on the line statVecV(offset) =  in
 my example, I agree. (I wrote that offhand since the actual assignment 
 is
 much larger) I should be statVecV(offset+1) =  so I'm 
 confident
 it's not a 1 0 indexing thing.

 My question is more related to what is happening in the offsets. c0
 and c1 are pulled using DMplexgetheight stratum, so they are 
 zero-indexed
 (which is why I loop from c0 to (c1-1)).

 For the size inquiries. on processor 0
  Petsc VecGetSize(stateVec)5390

>>>
>>> I need to see VecGetLocalSize()
>>>
>>>   Matt
>>>
>>>
  size(stateVecV)   2640

 on processor 1
 Petsc VecGetSize 5390
 size(stateVecV)2750

 It's quite weird to me that processor one can have a positive
 offset that is less than its starting ownership index (in the initial 
 email
 output).

 Thanks for the assistance
 Nicholas


 On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Petsc Users,
>>
>> I'm working with a dmplex system with a subsampled mesh
>> distributed with an overlap of 1.
>>
>> I'm encountering unusual situations when using
>> VecGetOwnershipRange to adjust 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Hi Matt
>
> I appreciate the help. The section view is quite extensive because each
> cell has 55 dofs located at the cells and on certain faces. I've appended
> the first of these which corresponds with the output in the first email, to
> save space. The following 54 are exactly the same but offset incremented by
> 1. (or negative 1 for negative offsets)
>

Okay, from the output it is clear that this vector does not match your
global section. Did you get stateVec by calling DMCreateGlobalVector()?

  Thanks,

 Matt


> Thanks for your time
> Nicholas
>
> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> I apologize for any lack of clarity in the initial email.
>>>
>>> looking at the initial output on rank 1
>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>> cell   0 offset2475 oStart2640-165
>>>  cell   1 offset2530 oStart2640-110
>>>  cell   2 offset2585 oStart2640 -55
>>>  cell   3 offset2640 oStart2640   0
>>> .
>>> cell  15 offset-771 oStart2640   -3411
>>>
>>>
>>> cell 15 provides a negative offset because it is the overlap cell (that
>>> is unowned)
>>> The remained of cells are all owned. However, the first 3 cells (0,1,2)
>>> return an offset that is less than the starting ownership range. I would
>>> expect cell 0 to start at offset 2640 at minimum.
>>>
>>
>> Send the output for this section
>>
>>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Sincerely
>>> Nicholas
>>>
>>>
>>>
>>>
>>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Apologies. If it helps, there is one cell of overlap in this small
> test case for a 2D mesh that is 1 cell in height and a number of cells in
> length. .
>
> process 0
>  Petsc VecGetLocalSize2750
>  size(stateVecV)2750
>
> process 1
>  Petsc VecGetLocalSize2640
>  size(stateVecV)2640
>

 The offsets shown below are well-within these sizes. I do not
 understand the problem.

   Thanks,

  Matt


> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
> wrote:
>
>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> I made a typo on the line statVecV(offset) =  in
>>> my example, I agree. (I wrote that offhand since the actual assignment 
>>> is
>>> much larger) I should be statVecV(offset+1) =  so I'm 
>>> confident
>>> it's not a 1 0 indexing thing.
>>>
>>> My question is more related to what is happening in the offsets. c0
>>> and c1 are pulled using DMplexgetheight stratum, so they are 
>>> zero-indexed
>>> (which is why I loop from c0 to (c1-1)).
>>>
>>> For the size inquiries. on processor 0
>>>  Petsc VecGetSize(stateVec)5390
>>>
>>
>> I need to see VecGetLocalSize()
>>
>>   Matt
>>
>>
>>>  size(stateVecV)   2640
>>>
>>> on processor 1
>>> Petsc VecGetSize 5390
>>> size(stateVecV)2750
>>>
>>> It's quite weird to me that processor one can have a positive offset
>>> that is less than its starting ownership index (in the initial email
>>> output).
>>>
>>> Thanks for the assistance
>>> Nicholas
>>>
>>>
>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Petsc Users,
>
> I'm working with a dmplex system with a subsampled mesh
> distributed with an overlap of 1.
>
> I'm encountering unusual situations when using
> VecGetOwnershipRange to adjust the offset received from a global 
> section.
> The logic of the following code is first to get the offset needed to 
> index
> a global vector while still being able to check if it is an 
> overlapped cell
> and skip if needed while counting the owned cells.
>


>
> call DMGetGlobalSection(dmplex,section,ierr)
> call VecGetArrayF90(stateVec,stateVecV,ierr)
> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
> do i = c0, (c1-1)
>
> call PetscSectionGetOffset(section,i,offset,ierr)
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
> 

Re: [petsc-users] Error running configure on HDF5 in PETSc-3.18.3

[Pierre Jolivet]

> On 6 Jan 2023, at 4:49 PM, Danyang Su  wrote:
> 
> Hi All,
>  
> I get ‘Error running configure on HDF5’ in PETSc-3.18.3 on MacOS, but no 
> problem on Ubuntu. Attached is the configuration log file. 
>  
> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mumps 
> --download-scalapack --download-parmetis --download-metis --download-ptscotch 
> --download-fblaslapack --download-mpich --download-hypre 
> --download-superlu_dist --download-hdf5=yes --with-debugging=0 
> --download-cmake --with-hdf5-fortran-bindings
>  
> Any idea on this?

Could you try to reconfigure in a shell without conda being activated?
You have 
PATH=/Users/danyangsu/Soft/Anaconda3/bin:/Users/danyangsu/Soft/Anaconda3/condabin:[…]
 which typically results in a broken configuration.

Thanks,
Pierre

> Thanks,
>  
> Danyang

[petsc-users] Definition of threshold(s) in gamg and boomerAMG

[Edoardo Centofanti]

Hi PETSc users,

I was looking for the exact definitions of the threshold parameter
(-pc_gamg_threshold) for gamg and of the strong threshold
(-pc_hypre_boomeramg_strong_threshold) for Hypre BoomerAMG. My curiosity
comes from the fact that the suggested parameters (apparently acting on the
same aspect of the algorithms) differ of around one order of magnitude (if
I remember right from the PETSc manual, the gamg threshold for 3D problems
is around 0.05, while for boomerAMG ranges from 0.25 to 0.5).

Thank you in advance,
Edoardo

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Hi Matt

I appreciate the help. The section view is quite extensive because each
cell has 55 dofs located at the cells and on certain faces. I've appended
the first of these which corresponds with the output in the first email, to
save space. The following 54 are exactly the same but offset incremented by
1. (or negative 1 for negative offsets)

Thanks for your time
Nicholas

On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I apologize for any lack of clarity in the initial email.
>>
>> looking at the initial output on rank 1
>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>> cell   0 offset2475 oStart2640-165
>>  cell   1 offset2530 oStart2640-110
>>  cell   2 offset2585 oStart2640 -55
>>  cell   3 offset2640 oStart2640   0
>> .
>> cell  15 offset-771 oStart2640   -3411
>>
>>
>> cell 15 provides a negative offset because it is the overlap cell (that
>> is unowned)
>> The remained of cells are all owned. However, the first 3 cells (0,1,2)
>> return an offset that is less than the starting ownership range. I would
>> expect cell 0 to start at offset 2640 at minimum.
>>
>
> Send the output for this section
>
>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>
>   Thanks,
>
>  Matt
>
>
>> Sincerely
>> Nicholas
>>
>>
>>
>>
>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Apologies. If it helps, there is one cell of overlap in this small test
 case for a 2D mesh that is 1 cell in height and a number of cells in
 length. .

 process 0
  Petsc VecGetLocalSize2750
  size(stateVecV)2750

 process 1
  Petsc VecGetLocalSize2640
  size(stateVecV)2640

>>>
>>> The offsets shown below are well-within these sizes. I do not understand
>>> the problem.
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I made a typo on the line statVecV(offset) =  in my
>> example, I agree. (I wrote that offhand since the actual assignment is 
>> much
>> larger) I should be statVecV(offset+1) =  so I'm confident 
>> it's
>> not a 1 0 indexing thing.
>>
>> My question is more related to what is happening in the offsets. c0
>> and c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
>> (which is why I loop from c0 to (c1-1)).
>>
>> For the size inquiries. on processor 0
>>  Petsc VecGetSize(stateVec)5390
>>
>
> I need to see VecGetLocalSize()
>
>   Matt
>
>
>>  size(stateVecV)   2640
>>
>> on processor 1
>> Petsc VecGetSize 5390
>> size(stateVecV)2750
>>
>> It's quite weird to me that processor one can have a positive offset
>> that is less than its starting ownership index (in the initial email
>> output).
>>
>> Thanks for the assistance
>> Nicholas
>>
>>
>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Petsc Users,

 I'm working with a dmplex system with a subsampled mesh distributed
 with an overlap of 1.

 I'm encountering unusual situations when using VecGetOwnershipRange
 to adjust the offset received from a global section. The logic of the
 following code is first to get the offset needed to index a global 
 vector
 while still being able to check if it is an overlapped cell and skip if
 needed while counting the owned cells.

>>>
>>>

 call DMGetGlobalSection(dmplex,section,ierr)
 call VecGetArrayF90(stateVec,stateVecV,ierr)
 call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
 do i = c0, (c1-1)

 call PetscSectionGetOffset(section,i,offset,ierr)
 write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
 oStart

 if(offset<0) then
 cycle
 endif
 offset=offset-oStart
 plexcells=plexcells+1
 stateVecV(offset)=  enddo

 I'm noticing some very weird results that I've appended below. The
 GetOffset documentation notes that a negative offset indicates an 
 unowned
 point (which I use to cycle). However, the offset subtraction with 
 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Hi Matt
>
> I apologize for any lack of clarity in the initial email.
>
> looking at the initial output on rank 1
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
> cell   0 offset2475 oStart2640-165
>  cell   1 offset2530 oStart2640-110
>  cell   2 offset2585 oStart2640 -55
>  cell   3 offset2640 oStart2640   0
> .
> cell  15 offset-771 oStart2640   -3411
>
>
> cell 15 provides a negative offset because it is the overlap cell (that is
> unowned)
> The remained of cells are all owned. However, the first 3 cells (0,1,2)
> return an offset that is less than the starting ownership range. I would
> expect cell 0 to start at offset 2640 at minimum.
>

Send the output for this section

  call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);

  Thanks,

 Matt


> Sincerely
> Nicholas
>
>
>
>
> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Apologies. If it helps, there is one cell of overlap in this small test
>>> case for a 2D mesh that is 1 cell in height and a number of cells in
>>> length. .
>>>
>>> process 0
>>>  Petsc VecGetLocalSize2750
>>>  size(stateVecV)2750
>>>
>>> process 1
>>>  Petsc VecGetLocalSize2640
>>>  size(stateVecV)2640
>>>
>>
>> The offsets shown below are well-within these sizes. I do not understand
>> the problem.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Matt
>
> I made a typo on the line statVecV(offset) =  in my
> example, I agree. (I wrote that offhand since the actual assignment is 
> much
> larger) I should be statVecV(offset+1) =  so I'm confident 
> it's
> not a 1 0 indexing thing.
>
> My question is more related to what is happening in the offsets. c0
> and c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
> (which is why I loop from c0 to (c1-1)).
>
> For the size inquiries. on processor 0
>  Petsc VecGetSize(stateVec)5390
>

 I need to see VecGetLocalSize()

   Matt


>  size(stateVecV)   2640
>
> on processor 1
> Petsc VecGetSize 5390
> size(stateVecV)2750
>
> It's quite weird to me that processor one can have a positive offset
> that is less than its starting ownership index (in the initial email
> output).
>
> Thanks for the assistance
> Nicholas
>
>
> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
> wrote:
>
>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Petsc Users,
>>>
>>> I'm working with a dmplex system with a subsampled mesh distributed
>>> with an overlap of 1.
>>>
>>> I'm encountering unusual situations when using VecGetOwnershipRange
>>> to adjust the offset received from a global section. The logic of the
>>> following code is first to get the offset needed to index a global 
>>> vector
>>> while still being able to check if it is an overlapped cell and skip if
>>> needed while counting the owned cells.
>>>
>>
>>
>>>
>>> call DMGetGlobalSection(dmplex,section,ierr)
>>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>>> do i = c0, (c1-1)
>>>
>>> call PetscSectionGetOffset(section,i,offset,ierr)
>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
>>> oStart
>>>
>>> if(offset<0) then
>>> cycle
>>> endif
>>> offset=offset-oStart
>>> plexcells=plexcells+1
>>> stateVecV(offset)=  enddo
>>>
>>> I'm noticing some very weird results that I've appended below. The
>>> GetOffset documentation notes that a negative offset indicates an 
>>> unowned
>>> point (which I use to cycle). However, the offset subtraction with 
>>> oStart
>>> will yield an illegal index for the Vector access. I see that on the
>>> documentation for GetOwnershipRange, it notes that this may be
>>> "ill-defined"  but I wanted to see if this is type of ill-defined I can
>>> expect or there is just something terribly wrong with my 
>>> PetscSection.(both
>>> the Vec and Section were produced from DMPlexDistributeField so should 
>>> by
>>> definition have synchronized section information) I was wondering if 
>>> there
>>> is a possible output and/or the best way to index the vector. I'm 
>>> 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Hi Matt

I apologize for any lack of clarity in the initial email.

looking at the initial output on rank 1
write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
cell   0 offset2475 oStart2640-165
 cell   1 offset2530 oStart2640-110
 cell   2 offset2585 oStart2640 -55
 cell   3 offset2640 oStart2640   0
.
cell  15 offset-771 oStart2640   -3411


cell 15 provides a negative offset because it is the overlap cell (that is
unowned)
The remained of cells are all owned. However, the first 3 cells (0,1,2)
return an offset that is less than the starting ownership range. I would
expect cell 0 to start at offset 2640 at minimum.

Sincerely
Nicholas




On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Apologies. If it helps, there is one cell of overlap in this small test
>> case for a 2D mesh that is 1 cell in height and a number of cells in
>> length. .
>>
>> process 0
>>  Petsc VecGetLocalSize2750
>>  size(stateVecV)2750
>>
>> process 1
>>  Petsc VecGetLocalSize2640
>>  size(stateVecV)2640
>>
>
> The offsets shown below are well-within these sizes. I do not understand
> the problem.
>
>   Thanks,
>
>  Matt
>
>
>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley  wrote:
>>
>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Matt

 I made a typo on the line statVecV(offset) =  in my
 example, I agree. (I wrote that offhand since the actual assignment is much
 larger) I should be statVecV(offset+1) =  so I'm confident it's
 not a 1 0 indexing thing.

 My question is more related to what is happening in the offsets. c0 and
 c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
 (which is why I loop from c0 to (c1-1)).

 For the size inquiries. on processor 0
  Petsc VecGetSize(stateVec)5390

>>>
>>> I need to see VecGetLocalSize()
>>>
>>>   Matt
>>>
>>>
  size(stateVecV)   2640

 on processor 1
 Petsc VecGetSize 5390
 size(stateVecV)2750

 It's quite weird to me that processor one can have a positive offset
 that is less than its starting ownership index (in the initial email
 output).

 Thanks for the assistance
 Nicholas


 On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
 wrote:

> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Petsc Users,
>>
>> I'm working with a dmplex system with a subsampled mesh distributed
>> with an overlap of 1.
>>
>> I'm encountering unusual situations when using VecGetOwnershipRange
>> to adjust the offset received from a global section. The logic of the
>> following code is first to get the offset needed to index a global vector
>> while still being able to check if it is an overlapped cell and skip if
>> needed while counting the owned cells.
>>
>
>
>>
>> call DMGetGlobalSection(dmplex,section,ierr)
>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>> do i = c0, (c1-1)
>>
>> call PetscSectionGetOffset(section,i,offset,ierr)
>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
>> oStart
>>
>> if(offset<0) then
>> cycle
>> endif
>> offset=offset-oStart
>> plexcells=plexcells+1
>> stateVecV(offset)=  enddo
>>
>> I'm noticing some very weird results that I've appended below. The
>> GetOffset documentation notes that a negative offset indicates an unowned
>> point (which I use to cycle). However, the offset subtraction with oStart
>> will yield an illegal index for the Vector access. I see that on the
>> documentation for GetOwnershipRange, it notes that this may be
>> "ill-defined"  but I wanted to see if this is type of ill-defined I can
>> expect or there is just something terribly wrong with my 
>> PetscSection.(both
>> the Vec and Section were produced from DMPlexDistributeField so should by
>> definition have synchronized section information) I was wondering if 
>> there
>> is a possible output and/or the best way to index the vector. I'm 
>> thinking
>> of subtracting the offset of cell 0 perhaps?
>>
>
> Can you show your vector sizes? Are you sure it is not the fact that
> F90 arrays use 1-based indices, but these are 0-based offsets?
>
>   Thanks,
>
>  Matt
>
>
>> on rank 0
>>
>>  cell   0 offset   0 oStart   0   0
>>  cell   1 offset  55 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Apologies. If it helps, there is one cell of overlap in this small test
> case for a 2D mesh that is 1 cell in height and a number of cells in
> length. .
>
> process 0
>  Petsc VecGetLocalSize2750
>  size(stateVecV)2750
>
> process 1
>  Petsc VecGetLocalSize2640
>  size(stateVecV)2640
>

The offsets shown below are well-within these sizes. I do not understand
the problem.

  Thanks,

 Matt


> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> I made a typo on the line statVecV(offset) =  in my
>>> example, I agree. (I wrote that offhand since the actual assignment is much
>>> larger) I should be statVecV(offset+1) =  so I'm confident it's
>>> not a 1 0 indexing thing.
>>>
>>> My question is more related to what is happening in the offsets. c0 and
>>> c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
>>> (which is why I loop from c0 to (c1-1)).
>>>
>>> For the size inquiries. on processor 0
>>>  Petsc VecGetSize(stateVec)5390
>>>
>>
>> I need to see VecGetLocalSize()
>>
>>   Matt
>>
>>
>>>  size(stateVecV)   2640
>>>
>>> on processor 1
>>> Petsc VecGetSize 5390
>>> size(stateVecV)2750
>>>
>>> It's quite weird to me that processor one can have a positive offset
>>> that is less than its starting ownership index (in the initial email
>>> output).
>>>
>>> Thanks for the assistance
>>> Nicholas
>>>
>>>
>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley 
>>> wrote:
>>>
 On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
 narno...@umich.edu> wrote:

> Hi Petsc Users,
>
> I'm working with a dmplex system with a subsampled mesh distributed
> with an overlap of 1.
>
> I'm encountering unusual situations when using VecGetOwnershipRange to
> adjust the offset received from a global section. The logic of the
> following code is first to get the offset needed to index a global vector
> while still being able to check if it is an overlapped cell and skip if
> needed while counting the owned cells.
>


>
> call DMGetGlobalSection(dmplex,section,ierr)
> call VecGetArrayF90(stateVec,stateVecV,ierr)
> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
> do i = c0, (c1-1)
>
> call PetscSectionGetOffset(section,i,offset,ierr)
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>
> if(offset<0) then
> cycle
> endif
> offset=offset-oStart
> plexcells=plexcells+1
> stateVecV(offset)=  enddo
>
> I'm noticing some very weird results that I've appended below. The
> GetOffset documentation notes that a negative offset indicates an unowned
> point (which I use to cycle). However, the offset subtraction with oStart
> will yield an illegal index for the Vector access. I see that on the
> documentation for GetOwnershipRange, it notes that this may be
> "ill-defined"  but I wanted to see if this is type of ill-defined I can
> expect or there is just something terribly wrong with my 
> PetscSection.(both
> the Vec and Section were produced from DMPlexDistributeField so should by
> definition have synchronized section information) I was wondering if there
> is a possible output and/or the best way to index the vector. I'm thinking
> of subtracting the offset of cell 0 perhaps?
>

 Can you show your vector sizes? Are you sure it is not the fact that
 F90 arrays use 1-based indices, but these are 0-based offsets?

   Thanks,

  Matt


> on rank 0
>
>  cell   0 offset   0 oStart   0   0
>  cell   1 offset  55 oStart   0  55
>  cell   2 offset 110 oStart   0 110
>  cell   3 offset 165 oStart   0 165
>  cell   4 offset 220 oStart   0 220
>  cell   5 offset 275 oStart   0 275
>  cell   6 offset 330 oStart   0 330
>  cell   7 offset 385 oStart   0 385
>  cell   8 offset 440 oStart   0 440
>  cell   9 offset 495 oStart   0 495
>  cell  10 offset 550 oStart   0 550
>  cell  11 offset 605 oStart   0 605
>  cell  12 offset 660 oStart   0 660
>  cell  13 offset 715 oStart   0 715
>
> and on rank one
> cell   0 offset2475 oStart2640-165
>  cell 

[petsc-users] Petsc DMLabel Fortran Stub request

[Nicholas Arnold-Medabalimi]

Hi Petsc Users

I am trying to use the sequence of
call DMLabelPropagateBegin(synchLabel,sf,ierr)
call
DMLabelPropagatePush(synchLabel,sf,PETSC_NULL_OPTIONS,PETSC_NULL_INTEGER,ierr)
call DMLabelPropagateEnd(synchLabel,sf, ierr)
in fortran.

I apologize if I messed something up, it appears as if the
DMLabelPropagatePush command doesn't have an appropriate Fortran interface
as I get an undefined reference when it is called.

I would appreciate any assistance.

As a side note in practice, what is the proper Fortran NULL pointer to use
for void arguments? I used an integer one temporarily to get to the
undefined reference error but I assume it doesn't matter?


Sincerely
Nicholas

-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Apologies. If it helps, there is one cell of overlap in this small test
case for a 2D mesh that is 1 cell in height and a number of cells in
length. .

process 0
 Petsc VecGetLocalSize2750
 size(stateVecV)2750

process 1
 Petsc VecGetLocalSize2640
 size(stateVecV)2640

On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Matt
>>
>> I made a typo on the line statVecV(offset) =  in my
>> example, I agree. (I wrote that offhand since the actual assignment is much
>> larger) I should be statVecV(offset+1) =  so I'm confident it's
>> not a 1 0 indexing thing.
>>
>> My question is more related to what is happening in the offsets. c0 and
>> c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
>> (which is why I loop from c0 to (c1-1)).
>>
>> For the size inquiries. on processor 0
>>  Petsc VecGetSize(stateVec)5390
>>
>
> I need to see VecGetLocalSize()
>
>   Matt
>
>
>>  size(stateVecV)   2640
>>
>> on processor 1
>> Petsc VecGetSize 5390
>> size(stateVecV)2750
>>
>> It's quite weird to me that processor one can have a positive offset that
>> is less than its starting ownership index (in the initial email output).
>>
>> Thanks for the assistance
>> Nicholas
>>
>>
>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley  wrote:
>>
>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>>> narno...@umich.edu> wrote:
>>>
 Hi Petsc Users,

 I'm working with a dmplex system with a subsampled mesh distributed
 with an overlap of 1.

 I'm encountering unusual situations when using VecGetOwnershipRange to
 adjust the offset received from a global section. The logic of the
 following code is first to get the offset needed to index a global vector
 while still being able to check if it is an overlapped cell and skip if
 needed while counting the owned cells.

>>>
>>>

 call DMGetGlobalSection(dmplex,section,ierr)
 call VecGetArrayF90(stateVec,stateVecV,ierr)
 call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
 do i = c0, (c1-1)

 call PetscSectionGetOffset(section,i,offset,ierr)
 write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart

 if(offset<0) then
 cycle
 endif
 offset=offset-oStart
 plexcells=plexcells+1
 stateVecV(offset)=  enddo

 I'm noticing some very weird results that I've appended below. The
 GetOffset documentation notes that a negative offset indicates an unowned
 point (which I use to cycle). However, the offset subtraction with oStart
 will yield an illegal index for the Vector access. I see that on the
 documentation for GetOwnershipRange, it notes that this may be
 "ill-defined"  but I wanted to see if this is type of ill-defined I can
 expect or there is just something terribly wrong with my PetscSection.(both
 the Vec and Section were produced from DMPlexDistributeField so should by
 definition have synchronized section information) I was wondering if there
 is a possible output and/or the best way to index the vector. I'm thinking
 of subtracting the offset of cell 0 perhaps?

>>>
>>> Can you show your vector sizes? Are you sure it is not the fact that F90
>>> arrays use 1-based indices, but these are 0-based offsets?
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>
>>>
 on rank 0

  cell   0 offset   0 oStart   0   0
  cell   1 offset  55 oStart   0  55
  cell   2 offset 110 oStart   0 110
  cell   3 offset 165 oStart   0 165
  cell   4 offset 220 oStart   0 220
  cell   5 offset 275 oStart   0 275
  cell   6 offset 330 oStart   0 330
  cell   7 offset 385 oStart   0 385
  cell   8 offset 440 oStart   0 440
  cell   9 offset 495 oStart   0 495
  cell  10 offset 550 oStart   0 550
  cell  11 offset 605 oStart   0 605
  cell  12 offset 660 oStart   0 660
  cell  13 offset 715 oStart   0 715

 and on rank one
 cell   0 offset2475 oStart2640-165
  cell   1 offset2530 oStart2640-110
  cell   2 offset2585 oStart2640 -55
  cell   3 offset2640 oStart2640   0
  cell   4 offset2695 oStart2640  55
  cell   5 offset2750 oStart2640 110

Re: [petsc-users] cuda gpu eager initialization error cudaErrorNotSupported

[Mark Lohry]

These cards do indeed not support cudaDeviceGetMemPool --
cudaDeviceGetAttribute on cudaDevAttrMemoryPoolsSupported return false,
meaning it doesn't support cudaMallocAsync, so the first point of failure
is the call to cudaDeviceGetMemPool in the initialization.

Would a workaround be to replace the cudaMallocAsync call to cudaMalloc and
skip the mempool or is that a bad idea?

On Fri, Jan 6, 2023 at 9:17 AM Mark Lohry  wrote:

> It built+ran fine on a different system with an sm75 arch. Is there a
> documented minimum version if that indeed is the cause?
>
> One minor hiccup FYI -- compilation of hypre fails with cuda toolkit 12,
> due to cusprase removing csrsv2Info_t (although it's still referenced in
> their docs...) in favor of bsrsv2Info_t. Rolling back to cuda toolkit 11.8
> worked.
>
>
> On Thu, Jan 5, 2023 at 6:37 PM Junchao Zhang 
> wrote:
>
>> Jacob, is it because the cuda arch is too old?
>>
>> --Junchao Zhang
>>
>>
>> On Thu, Jan 5, 2023 at 4:30 PM Mark Lohry  wrote:
>>
>>> I'm seeing the same thing on latest main with a different machine and
>>> -sm52 card, cuda 11.8. make check fails with the below, where the indicated
>>> line 249 corresponds to PetscCallCUPM(cupmDeviceGetMemPool(,
>>> static_cast(device->deviceId)));   in the initialize function.
>>>
>>>
>>> Running check examples to verify correct installation
>>> Using PETSC_DIR=/home/mlohry/dev/petsc and PETSC_ARCH=arch-linux-c-debug
>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI
>>> processes
>>> 2,17c2,46
>>> <   0 SNES Function norm 2.391552133017e-01
>>> < 0 KSP Residual norm 2.928487269734e-01
>>> < 1 KSP Residual norm 1.876489580142e-02
>>> < 2 KSP Residual norm 3.291394847944e-03
>>> < 3 KSP Residual norm 2.456493072124e-04
>>> < 4 KSP Residual norm 1.161647147715e-05
>>> < 5 KSP Residual norm 1.285648407621e-06
>>> <   1 SNES Function norm 6.846805706142e-05
>>> < 0 KSP Residual norm 2.292783790384e-05
>>> < 1 KSP Residual norm 2.100673631699e-06
>>> < 2 KSP Residual norm 2.121341386147e-07
>>> < 3 KSP Residual norm 2.455932678957e-08
>>> < 4 KSP Residual norm 1.753095730744e-09
>>> < 5 KSP Residual norm 7.489214418904e-11
>>> <   2 SNES Function norm 2.103908447865e-10
>>> < Number of SNES iterations = 2
>>> ---
>>> > [0]PETSC ERROR: - Error Message
>>> --
>>> > [0]PETSC ERROR: GPU error
>>> > [0]PETSC ERROR: cuda error 801 (cudaErrorNotSupported) : operation not
>>> supported
>>> > [0]PETSC ERROR: WARNING! There are option(s) set that were not used!
>>> Could be the program crashed before they were used or a spelling mistake,
>>> etc!
>>> > [0]PETSC ERROR: Option left: name:-mg_levels_ksp_max_it value: 3
>>> source: command line
>>> > [0]PETSC ERROR: Option left: name:-nox (no value) source: environment
>>> > [0]PETSC ERROR: Option left: name:-nox_warning (no value) source:
>>> environment
>>> > [0]PETSC ERROR: Option left: name:-pc_gamg_esteig_ksp_max_it value: 10
>>> source: command line
>>> > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble
>>> shooting.
>>> > [0]PETSC ERROR: Petsc Development GIT revision:
>>> v3.18.3-352-g91c56366cb  GIT Date: 2023-01-05 17:22:48 +
>>> > [0]PETSC ERROR: ./ex19 on a arch-linux-c-debug named osprey by mlohry
>>> Thu Jan  5 17:25:17 2023
>>> > [0]PETSC ERROR: Configure options --with-cuda --with-mpi=1
>>> > [0]PETSC ERROR: #1 initialize() at
>>> /home/mlohry/dev/petsc/src/sys/objects/device/impls/cupm/cuda/../cupmcontext.hpp:249
>>> > [0]PETSC ERROR: #2 PetscDeviceContextCreate_CUDA() at
>>> /home/mlohry/dev/petsc/src/sys/objects/device/impls/cupm/cuda/
>>> cupmcontext.cu:10
>>> > [0]PETSC ERROR: #3 PetscDeviceContextSetDevice_Private() at
>>> /home/mlohry/dev/petsc/src/sys/objects/device/interface/dcontext.cxx:247
>>> > [0]PETSC ERROR: #4
>>> PetscDeviceContextSetDefaultDeviceForType_Internal() at
>>> /home/mlohry/dev/petsc/src/sys/objects/device/interface/dcontext.cxx:260
>>> > [0]PETSC ERROR: #5 PetscDeviceContextSetupGlobalContext_Private() at
>>> /home/mlohry/dev/petsc/src/sys/objects/device/interface/global_dcontext.cxx:52
>>> > [0]PETSC ERROR: #6 PetscDeviceContextGetCurrentContext() at
>>> /home/mlohry/dev/petsc/src/sys/objects/device/interface/global_dcontext.cxx:84
>>> > [0]PETSC ERROR: #7 GetHandleDispatch_() at
>>> /home/mlohry/dev/petsc/include/petsc/private/veccupmimpl.h:499
>>> > [0]PETSC ERROR: #8 create() at
>>> /home/mlohry/dev/petsc/include/petsc/private/veccupmimpl.h:1069
>>> > [0]PETSC ERROR: #9 VecCreate_SeqCUDA() at
>>> /home/mlohry/dev/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:10
>>> > [0]PETSC ERROR: #10 VecSetType() at
>>> /home/mlohry/dev/petsc/src/vec/vec/interface/vecreg.c:89
>>> > [0]PETSC ERROR: #11 DMCreateGlobalVector_DA() at
>>> /home/mlohry/dev/petsc/src/dm/impls/da/dadist.c:31
>>> > [0]PETSC ERROR: #12 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Hi Matt
>
> I made a typo on the line statVecV(offset) =  in my
> example, I agree. (I wrote that offhand since the actual assignment is much
> larger) I should be statVecV(offset+1) =  so I'm confident it's
> not a 1 0 indexing thing.
>
> My question is more related to what is happening in the offsets. c0 and c1
> are pulled using DMplexgetheight stratum, so they are zero-indexed (which
> is why I loop from c0 to (c1-1)).
>
> For the size inquiries. on processor 0
>  Petsc VecGetSize(stateVec)5390
>

I need to see VecGetLocalSize()

  Matt


>  size(stateVecV)   2640
>
> on processor 1
> Petsc VecGetSize 5390
> size(stateVecV)2750
>
> It's quite weird to me that processor one can have a positive offset that
> is less than its starting ownership index (in the initial email output).
>
> Thanks for the assistance
> Nicholas
>
>
> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley  wrote:
>
>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>> narno...@umich.edu> wrote:
>>
>>> Hi Petsc Users,
>>>
>>> I'm working with a dmplex system with a subsampled mesh distributed with
>>> an overlap of 1.
>>>
>>> I'm encountering unusual situations when using VecGetOwnershipRange to
>>> adjust the offset received from a global section. The logic of the
>>> following code is first to get the offset needed to index a global vector
>>> while still being able to check if it is an overlapped cell and skip if
>>> needed while counting the owned cells.
>>>
>>
>>
>>>
>>> call DMGetGlobalSection(dmplex,section,ierr)
>>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>>> do i = c0, (c1-1)
>>>
>>> call PetscSectionGetOffset(section,i,offset,ierr)
>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>>
>>> if(offset<0) then
>>> cycle
>>> endif
>>> offset=offset-oStart
>>> plexcells=plexcells+1
>>> stateVecV(offset)=  enddo
>>>
>>> I'm noticing some very weird results that I've appended below. The
>>> GetOffset documentation notes that a negative offset indicates an unowned
>>> point (which I use to cycle). However, the offset subtraction with oStart
>>> will yield an illegal index for the Vector access. I see that on the
>>> documentation for GetOwnershipRange, it notes that this may be
>>> "ill-defined"  but I wanted to see if this is type of ill-defined I can
>>> expect or there is just something terribly wrong with my PetscSection.(both
>>> the Vec and Section were produced from DMPlexDistributeField so should by
>>> definition have synchronized section information) I was wondering if there
>>> is a possible output and/or the best way to index the vector. I'm thinking
>>> of subtracting the offset of cell 0 perhaps?
>>>
>>
>> Can you show your vector sizes? Are you sure it is not the fact that F90
>> arrays use 1-based indices, but these are 0-based offsets?
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> on rank 0
>>>
>>>  cell   0 offset   0 oStart   0   0
>>>  cell   1 offset  55 oStart   0  55
>>>  cell   2 offset 110 oStart   0 110
>>>  cell   3 offset 165 oStart   0 165
>>>  cell   4 offset 220 oStart   0 220
>>>  cell   5 offset 275 oStart   0 275
>>>  cell   6 offset 330 oStart   0 330
>>>  cell   7 offset 385 oStart   0 385
>>>  cell   8 offset 440 oStart   0 440
>>>  cell   9 offset 495 oStart   0 495
>>>  cell  10 offset 550 oStart   0 550
>>>  cell  11 offset 605 oStart   0 605
>>>  cell  12 offset 660 oStart   0 660
>>>  cell  13 offset 715 oStart   0 715
>>>
>>> and on rank one
>>> cell   0 offset2475 oStart2640-165
>>>  cell   1 offset2530 oStart2640-110
>>>  cell   2 offset2585 oStart2640 -55
>>>  cell   3 offset2640 oStart2640   0
>>>  cell   4 offset2695 oStart2640  55
>>>  cell   5 offset2750 oStart2640 110
>>>  cell   6 offset2805 oStart2640 165
>>>  cell   7 offset2860 oStart2640 220
>>>  cell   8 offset2915 oStart2640 275
>>>  cell   9 offset2970 oStart2640 330
>>>  cell  10 offset3025 oStart2640 385
>>>  cell  11 offset3080 oStart2640 440
>>>  cell  12 offset3135 oStart 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Hi Matt

I made a typo on the line statVecV(offset) =  in my
example, I agree. (I wrote that offhand since the actual assignment is much
larger) I should be statVecV(offset+1) =  so I'm confident it's
not a 1 0 indexing thing.

My question is more related to what is happening in the offsets. c0 and c1
are pulled using DMplexgetheight stratum, so they are zero-indexed (which
is why I loop from c0 to (c1-1)).

For the size inquiries. on processor 0
 Petsc VecGetSize(stateVec)5390
 size(stateVecV)   2640

on processor 1
Petsc VecGetSize 5390
size(stateVecV)2750

It's quite weird to me that processor one can have a positive offset that
is less than its starting ownership index (in the initial email output).

Thanks for the assistance
Nicholas


On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley  wrote:

> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
> narno...@umich.edu> wrote:
>
>> Hi Petsc Users,
>>
>> I'm working with a dmplex system with a subsampled mesh distributed with
>> an overlap of 1.
>>
>> I'm encountering unusual situations when using VecGetOwnershipRange to
>> adjust the offset received from a global section. The logic of the
>> following code is first to get the offset needed to index a global vector
>> while still being able to check if it is an overlapped cell and skip if
>> needed while counting the owned cells.
>>
>
>
>>
>> call DMGetGlobalSection(dmplex,section,ierr)
>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>> do i = c0, (c1-1)
>>
>> call PetscSectionGetOffset(section,i,offset,ierr)
>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>
>> if(offset<0) then
>> cycle
>> endif
>> offset=offset-oStart
>> plexcells=plexcells+1
>> stateVecV(offset)=  enddo
>>
>> I'm noticing some very weird results that I've appended below. The
>> GetOffset documentation notes that a negative offset indicates an unowned
>> point (which I use to cycle). However, the offset subtraction with oStart
>> will yield an illegal index for the Vector access. I see that on the
>> documentation for GetOwnershipRange, it notes that this may be
>> "ill-defined"  but I wanted to see if this is type of ill-defined I can
>> expect or there is just something terribly wrong with my PetscSection.(both
>> the Vec and Section were produced from DMPlexDistributeField so should by
>> definition have synchronized section information) I was wondering if there
>> is a possible output and/or the best way to index the vector. I'm thinking
>> of subtracting the offset of cell 0 perhaps?
>>
>
> Can you show your vector sizes? Are you sure it is not the fact that F90
> arrays use 1-based indices, but these are 0-based offsets?
>
>   Thanks,
>
>  Matt
>
>
>> on rank 0
>>
>>  cell   0 offset   0 oStart   0   0
>>  cell   1 offset  55 oStart   0  55
>>  cell   2 offset 110 oStart   0 110
>>  cell   3 offset 165 oStart   0 165
>>  cell   4 offset 220 oStart   0 220
>>  cell   5 offset 275 oStart   0 275
>>  cell   6 offset 330 oStart   0 330
>>  cell   7 offset 385 oStart   0 385
>>  cell   8 offset 440 oStart   0 440
>>  cell   9 offset 495 oStart   0 495
>>  cell  10 offset 550 oStart   0 550
>>  cell  11 offset 605 oStart   0 605
>>  cell  12 offset 660 oStart   0 660
>>  cell  13 offset 715 oStart   0 715
>>
>> and on rank one
>> cell   0 offset2475 oStart2640-165
>>  cell   1 offset2530 oStart2640-110
>>  cell   2 offset2585 oStart2640 -55
>>  cell   3 offset2640 oStart2640   0
>>  cell   4 offset2695 oStart2640  55
>>  cell   5 offset2750 oStart2640 110
>>  cell   6 offset2805 oStart2640 165
>>  cell   7 offset2860 oStart2640 220
>>  cell   8 offset2915 oStart2640 275
>>  cell   9 offset2970 oStart2640 330
>>  cell  10 offset3025 oStart2640 385
>>  cell  11 offset3080 oStart2640 440
>>  cell  12 offset3135 oStart2640 495
>>  cell  13 offset3190 oStart2640 550
>>  cell  14 offset3245 oStart2640 605
>>  cell  15 offset-771 oStart2640   -3411
>>
>>
>> Sincerely
>> 

Re: [petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
narno...@umich.edu> wrote:

> Hi Petsc Users,
>
> I'm working with a dmplex system with a subsampled mesh distributed with
> an overlap of 1.
>
> I'm encountering unusual situations when using VecGetOwnershipRange to
> adjust the offset received from a global section. The logic of the
> following code is first to get the offset needed to index a global vector
> while still being able to check if it is an overlapped cell and skip if
> needed while counting the owned cells.
>


>
> call DMGetGlobalSection(dmplex,section,ierr)
> call VecGetArrayF90(stateVec,stateVecV,ierr)
> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
> do i = c0, (c1-1)
>
> call PetscSectionGetOffset(section,i,offset,ierr)
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>
> if(offset<0) then
> cycle
> endif
> offset=offset-oStart
> plexcells=plexcells+1
> stateVecV(offset)=  enddo
>
> I'm noticing some very weird results that I've appended below. The
> GetOffset documentation notes that a negative offset indicates an unowned
> point (which I use to cycle). However, the offset subtraction with oStart
> will yield an illegal index for the Vector access. I see that on the
> documentation for GetOwnershipRange, it notes that this may be
> "ill-defined"  but I wanted to see if this is type of ill-defined I can
> expect or there is just something terribly wrong with my PetscSection.(both
> the Vec and Section were produced from DMPlexDistributeField so should by
> definition have synchronized section information) I was wondering if there
> is a possible output and/or the best way to index the vector. I'm thinking
> of subtracting the offset of cell 0 perhaps?
>

Can you show your vector sizes? Are you sure it is not the fact that F90
arrays use 1-based indices, but these are 0-based offsets?

  Thanks,

 Matt


> on rank 0
>
>  cell   0 offset   0 oStart   0   0
>  cell   1 offset  55 oStart   0  55
>  cell   2 offset 110 oStart   0 110
>  cell   3 offset 165 oStart   0 165
>  cell   4 offset 220 oStart   0 220
>  cell   5 offset 275 oStart   0 275
>  cell   6 offset 330 oStart   0 330
>  cell   7 offset 385 oStart   0 385
>  cell   8 offset 440 oStart   0 440
>  cell   9 offset 495 oStart   0 495
>  cell  10 offset 550 oStart   0 550
>  cell  11 offset 605 oStart   0 605
>  cell  12 offset 660 oStart   0 660
>  cell  13 offset 715 oStart   0 715
>
> and on rank one
> cell   0 offset2475 oStart2640-165
>  cell   1 offset2530 oStart2640-110
>  cell   2 offset2585 oStart2640 -55
>  cell   3 offset2640 oStart2640   0
>  cell   4 offset2695 oStart2640  55
>  cell   5 offset2750 oStart2640 110
>  cell   6 offset2805 oStart2640 165
>  cell   7 offset2860 oStart2640 220
>  cell   8 offset2915 oStart2640 275
>  cell   9 offset2970 oStart2640 330
>  cell  10 offset3025 oStart2640 385
>  cell  11 offset3080 oStart2640 440
>  cell  12 offset3135 oStart2640 495
>  cell  13 offset3190 oStart2640 550
>  cell  14 offset3245 oStart2640 605
>  cell  15 offset-771 oStart2640   -3411
>
>
> Sincerely
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] cuda gpu eager initialization error cudaErrorNotSupported

[Mark Lohry]

It built+ran fine on a different system with an sm75 arch. Is there a
documented minimum version if that indeed is the cause?

One minor hiccup FYI -- compilation of hypre fails with cuda toolkit 12,
due to cusprase removing csrsv2Info_t (although it's still referenced in
their docs...) in favor of bsrsv2Info_t. Rolling back to cuda toolkit 11.8
worked.


On Thu, Jan 5, 2023 at 6:37 PM Junchao Zhang 
wrote:

> Jacob, is it because the cuda arch is too old?
>
> --Junchao Zhang
>
>
> On Thu, Jan 5, 2023 at 4:30 PM Mark Lohry  wrote:
>
>> I'm seeing the same thing on latest main with a different machine and
>> -sm52 card, cuda 11.8. make check fails with the below, where the indicated
>> line 249 corresponds to PetscCallCUPM(cupmDeviceGetMemPool(,
>> static_cast(device->deviceId)));   in the initialize function.
>>
>>
>> Running check examples to verify correct installation
>> Using PETSC_DIR=/home/mlohry/dev/petsc and PETSC_ARCH=arch-linux-c-debug
>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI
>> processes
>> 2,17c2,46
>> <   0 SNES Function norm 2.391552133017e-01
>> < 0 KSP Residual norm 2.928487269734e-01
>> < 1 KSP Residual norm 1.876489580142e-02
>> < 2 KSP Residual norm 3.291394847944e-03
>> < 3 KSP Residual norm 2.456493072124e-04
>> < 4 KSP Residual norm 1.161647147715e-05
>> < 5 KSP Residual norm 1.285648407621e-06
>> <   1 SNES Function norm 6.846805706142e-05
>> < 0 KSP Residual norm 2.292783790384e-05
>> < 1 KSP Residual norm 2.100673631699e-06
>> < 2 KSP Residual norm 2.121341386147e-07
>> < 3 KSP Residual norm 2.455932678957e-08
>> < 4 KSP Residual norm 1.753095730744e-09
>> < 5 KSP Residual norm 7.489214418904e-11
>> <   2 SNES Function norm 2.103908447865e-10
>> < Number of SNES iterations = 2
>> ---
>> > [0]PETSC ERROR: - Error Message
>> --
>> > [0]PETSC ERROR: GPU error
>> > [0]PETSC ERROR: cuda error 801 (cudaErrorNotSupported) : operation not
>> supported
>> > [0]PETSC ERROR: WARNING! There are option(s) set that were not used!
>> Could be the program crashed before they were used or a spelling mistake,
>> etc!
>> > [0]PETSC ERROR: Option left: name:-mg_levels_ksp_max_it value: 3
>> source: command line
>> > [0]PETSC ERROR: Option left: name:-nox (no value) source: environment
>> > [0]PETSC ERROR: Option left: name:-nox_warning (no value) source:
>> environment
>> > [0]PETSC ERROR: Option left: name:-pc_gamg_esteig_ksp_max_it value: 10
>> source: command line
>> > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble
>> shooting.
>> > [0]PETSC ERROR: Petsc Development GIT revision: v3.18.3-352-g91c56366cb
>>  GIT Date: 2023-01-05 17:22:48 +
>> > [0]PETSC ERROR: ./ex19 on a arch-linux-c-debug named osprey by mlohry
>> Thu Jan  5 17:25:17 2023
>> > [0]PETSC ERROR: Configure options --with-cuda --with-mpi=1
>> > [0]PETSC ERROR: #1 initialize() at
>> /home/mlohry/dev/petsc/src/sys/objects/device/impls/cupm/cuda/../cupmcontext.hpp:249
>> > [0]PETSC ERROR: #2 PetscDeviceContextCreate_CUDA() at
>> /home/mlohry/dev/petsc/src/sys/objects/device/impls/cupm/cuda/
>> cupmcontext.cu:10
>> > [0]PETSC ERROR: #3 PetscDeviceContextSetDevice_Private() at
>> /home/mlohry/dev/petsc/src/sys/objects/device/interface/dcontext.cxx:247
>> > [0]PETSC ERROR: #4 PetscDeviceContextSetDefaultDeviceForType_Internal()
>> at /home/mlohry/dev/petsc/src/sys/objects/device/interface/dcontext.cxx:260
>> > [0]PETSC ERROR: #5 PetscDeviceContextSetupGlobalContext_Private() at
>> /home/mlohry/dev/petsc/src/sys/objects/device/interface/global_dcontext.cxx:52
>> > [0]PETSC ERROR: #6 PetscDeviceContextGetCurrentContext() at
>> /home/mlohry/dev/petsc/src/sys/objects/device/interface/global_dcontext.cxx:84
>> > [0]PETSC ERROR: #7 GetHandleDispatch_() at
>> /home/mlohry/dev/petsc/include/petsc/private/veccupmimpl.h:499
>> > [0]PETSC ERROR: #8 create() at
>> /home/mlohry/dev/petsc/include/petsc/private/veccupmimpl.h:1069
>> > [0]PETSC ERROR: #9 VecCreate_SeqCUDA() at
>> /home/mlohry/dev/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:10
>> > [0]PETSC ERROR: #10 VecSetType() at
>> /home/mlohry/dev/petsc/src/vec/vec/interface/vecreg.c:89
>> > [0]PETSC ERROR: #11 DMCreateGlobalVector_DA() at
>> /home/mlohry/dev/petsc/src/dm/impls/da/dadist.c:31
>> > [0]PETSC ERROR: #12 DMCreateGlobalVector() at
>> /home/mlohry/dev/petsc/src/dm/interface/dm.c:1023
>> > [0]PETSC ERROR: #13 main() at ex19.c:149
>>
>>
>> On Thu, Jan 5, 2023 at 3:42 PM Mark Lohry  wrote:
>>
>>> I'm trying to compile the cuda example
>>>
>>> ./config/examples/arch-ci-linux-cuda-double-64idx.py
>>> --with-cudac=/usr/local/cuda-11.5/bin/nvcc
>>>
>>> and running make test passes the test ok
>>> diff-sys_objects_device_tests-ex1_host_with_device+nsize-1device_enable-lazy
>>> but the eager variant fails, pasted below.
>>>
>>> I get a similar 

Re: [petsc-users] error when trying to compile with HPDDM

[Jose E. Roman]

This happens because you have typed 'make -j128'. If you just do 'make', PETSc 
will choose a reasonable value (-j59 in your case).

Satish: do we want to support this use case? Then a possible fix is:

diff --git a/config/BuildSystem/config/packages/slepc.py 
b/config/BuildSystem/config/packages/slepc.py
index b7c80930750..2688403f908 100644
--- a/config/BuildSystem/config/packages/slepc.py
+++ b/config/BuildSystem/config/packages/slepc.py
@@ -63,7 +63,7 @@ class Configure(config.package.Package):
 self.addMakeRule('slepcbuild','', \
['@echo "*** Building SLEPc ***"',\
   '@${RM} 
${PETSC_ARCH}/lib/petsc/conf/slepc.errorflg',\
-  '@(cd '+self.packageDir+' && \\\n\
+  '+@(cd '+self.packageDir+' && \\\n\
'+carg+self.python.pyexe+' ./configure --prefix='+prefix+' 
'+configargs+' && \\\n\
'+barg+'${OMAKE} '+barg+') || \\\n\
  (echo 
"**ERROR*" && \\\n\


> El 5 ene 2023, a las 21:06, Alfredo Jaramillo  
> escribió:
> 
> Hi Pierre, no, I don't really need that flag. I removed it and the 
> installation process went well. I just noticed a "minor" detail when building 
> SLEPc:
> 
> "gmake[3]: warning: jobserver unavailable: using -j1.  Add '+' to parent make 
> rule"
> 
> so the compilation of that library went slow.
> 
> Thanks,
> Alfredo