Re: [petsc-users] DMPlex: Mapping cells before and after partitioning

[Bisht, Gautam via petsc-users]

On Jan 9, 2020, at 4:25 PM, Matthew Knepley 
mailto:knep...@gmail.com>> wrote:

On Thu, Jan 9, 2020 at 1:35 PM 'Bisht, Gautam' via tdycores-dev 
mailto:tdycores-...@googlegroups.com>> wrote:

> On Jan 9, 2020, at 2:58 PM, Jed Brown 
> mailto:j...@jedbrown.org>> wrote:
>
> "'Bisht, Gautam' via tdycores-dev" 
> mailto:tdycores-...@googlegroups.com>> writes:
>
>>> Do you need to rely on the element number, or would coordinates (of a
>>> centroid?) be sufficient for your purposes?
>>
>> I do need to rely on the element number.  In my case, I have a mapping file 
>> that remaps data from one grid onto another grid. Though I’m currently 
>> creating a hexahedron mesh, in the future I would be reading in an 
>> unstructured grid from a file for which I cannot rely on coordinates.
>
> How does the mapping file work and how is it generated?

In CESM/E3SM, the mapping file is used to map fluxes or states between grids of 
two components (e.g. land & atmosphere). The mapping method can be 
conservative, nearest neighbor, bilinear, etc. While CESM/E3SM uses 
ESMF_RegridWeightGen to generate the mapping file, I’m using by own MATLAB 
script to create the mapping file.

I’m surprised that this is not an issue for other codes that are using DMPlex. 
E.g In PFLOTRAN, when a user creates a custom unstructured grid, they can 
specify material property for each grid cell. So, there should be a way to 
create a vectorscatter that will scatter material property read in the 
“application”-order (i.e. order before calling DMPlexDistribute() ) to 
ghosted-order (i.e. order after calling DMPlexDistribute()).

We did build something specific for this because some people wanted it. I wish 
I could purge this from all simulations. Its
definitely destructive, but this is the way the world currently is.

You want this:

  
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexNaturalToGlobalBegin.html

Perfect.

Thanks.
-Gautam



Thanks,

 Matt

> We can locate points and create interpolation with unstructured grids.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "tdycores-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to 
> tdycores-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://protect2.fireeye.com/v1/url?k=b265c01b-eed0fed4-b265ea0e-0cc47adc5e60-1707adbf1790c7e4=1=0962f8e1-9155-4d9c-abdf-2b6481141cd0=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F8736come4e.fsf%2540jedbrown.org.

--
You received this message because you are subscribed to the Google Groups 
"tdycores-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
tdycores-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tdycores-dev/9AB001AF-8857-446A-AE69-E8D6A25CB8FA%40pnnl.gov.


--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

--
You received this message because you are subscribed to the Google Groups 
"tdycores-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
tdycores-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tdycores-dev/CAMYG4Gm%3DSY%3DyDiYOdBm1j_KZO5NYhu80ZhbFTV23O%2Bv-zVvFnA%40mail.gmail.com.

Re: [petsc-users] DMPlex: Mapping cells before and after partitioning

[Matthew Knepley]

On Thu, Jan 9, 2020 at 1:35 PM 'Bisht, Gautam' via tdycores-dev <
tdycores-...@googlegroups.com> wrote:

>
> > On Jan 9, 2020, at 2:58 PM, Jed Brown  wrote:
> >
> > "'Bisht, Gautam' via tdycores-dev" 
> writes:
> >
> >>> Do you need to rely on the element number, or would coordinates (of a
> >>> centroid?) be sufficient for your purposes?
> >>
> >> I do need to rely on the element number.  In my case, I have a mapping
> file that remaps data from one grid onto another grid. Though I’m currently
> creating a hexahedron mesh, in the future I would be reading in an
> unstructured grid from a file for which I cannot rely on coordinates.
> >
> > How does the mapping file work and how is it generated?
>
> In CESM/E3SM, the mapping file is used to map fluxes or states between
> grids of two components (e.g. land & atmosphere). The mapping method can be
> conservative, nearest neighbor, bilinear, etc. While CESM/E3SM uses
> ESMF_RegridWeightGen to generate the mapping file, I’m using by own MATLAB
> script to create the mapping file.
>
> I’m surprised that this is not an issue for other codes that are using
> DMPlex. E.g In PFLOTRAN, when a user creates a custom unstructured grid,
> they can specify material property for each grid cell. So, there should be
> a way to create a vectorscatter that will scatter material property read in
> the “application”-order (i.e. order before calling DMPlexDistribute() ) to
> ghosted-order (i.e. order after calling DMPlexDistribute()).
>

We did build something specific for this because some people wanted it. I
wish I could purge this from all simulations. Its
definitely destructive, but this is the way the world currently is.

You want this:


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexNaturalToGlobalBegin.html

Thanks,

 Matt


> > We can locate points and create interpolation with unstructured grids.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "tdycores-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to tdycores-dev+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://protect2.fireeye.com/v1/url?k=b265c01b-eed0fed4-b265ea0e-0cc47adc5e60-1707adbf1790c7e4=1=0962f8e1-9155-4d9c-abdf-2b6481141cd0=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F8736come4e.fsf%2540jedbrown.org
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "tdycores-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tdycores-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tdycores-dev/9AB001AF-8857-446A-AE69-E8D6A25CB8FA%40pnnl.gov
> .
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] DMPlex: Mapping cells before and after partitioning

[Bisht, Gautam via petsc-users]

> On Jan 9, 2020, at 2:58 PM, Jed Brown  wrote:
> 
> "'Bisht, Gautam' via tdycores-dev"  writes:
> 
>>> Do you need to rely on the element number, or would coordinates (of a
>>> centroid?) be sufficient for your purposes?  
>> 
>> I do need to rely on the element number.  In my case, I have a mapping file 
>> that remaps data from one grid onto another grid. Though I’m currently 
>> creating a hexahedron mesh, in the future I would be reading in an 
>> unstructured grid from a file for which I cannot rely on coordinates.
> 
> How does the mapping file work and how is it generated?

In CESM/E3SM, the mapping file is used to map fluxes or states between grids of 
two components (e.g. land & atmosphere). The mapping method can be 
conservative, nearest neighbor, bilinear, etc. While CESM/E3SM uses 
ESMF_RegridWeightGen to generate the mapping file, I’m using by own MATLAB 
script to create the mapping file.

I’m surprised that this is not an issue for other codes that are using DMPlex. 
E.g In PFLOTRAN, when a user creates a custom unstructured grid, they can 
specify material property for each grid cell. So, there should be a way to 
create a vectorscatter that will scatter material property read in the 
“application”-order (i.e. order before calling DMPlexDistribute() ) to 
ghosted-order (i.e. order after calling DMPlexDistribute()).


> We can locate points and create interpolation with unstructured grids.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "tdycores-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to tdycores-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://protect2.fireeye.com/v1/url?k=b265c01b-eed0fed4-b265ea0e-0cc47adc5e60-1707adbf1790c7e4=1=0962f8e1-9155-4d9c-abdf-2b6481141cd0=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F8736come4e.fsf%2540jedbrown.org.

Re: [petsc-users] DMPlex: Mapping cells before and after partitioning

[Jed Brown]

"'Bisht, Gautam' via tdycores-dev"  writes:

>> Do you need to rely on the element number, or would coordinates (of a
>> centroid?) be sufficient for your purposes?  
>
> I do need to rely on the element number.  In my case, I have a mapping file 
> that remaps data from one grid onto another grid. Though I’m currently 
> creating a hexahedron mesh, in the future I would be reading in an 
> unstructured grid from a file for which I cannot rely on coordinates.

How does the mapping file work and how is it generated?

We can locate points and create interpolation with unstructured grids.

Re: [petsc-users] DMPlex: Mapping cells before and after partitioning

[Bisht, Gautam via petsc-users]

> On Jan 9, 2020, at 2:38 PM, Jed Brown  wrote:
> 
> "'Bisht, Gautam' via tdycores-dev"  writes:
> 
>> Hi Matt,
>> 
>> Here is the situation that I’m running into and hoping you could provide 
>> some guidance. I create a mesh using DMPlexCreateBoxMesh() in which cells 
>> are ordered such that cells with increasing x coordinates are first, 
>> followed by cells with increasing y-corodinate and so forth. Next, I call 
>> DMPlexDistribute() that rearranges the cells after partitioning. How can I 
>> map the cell order after partitioning to cell order before partitioning? (In 
>> PFLOTRAN, we call this type of mapping as Ghosted—to—Application or 
>> Ghosted—to—Natural).
> 
> Do you need to rely on the element number, or would coordinates (of a
> centroid?) be sufficient for your purposes?  

I do need to rely on the element number.  In my case, I have a mapping file 
that remaps data from one grid onto another grid. Though I’m currently creating 
a hexahedron mesh, in the future I would be reading in an unstructured grid 
from a file for which I cannot rely on coordinates.


> I think DMLabel can be used
> to track that numbering, but I also think it's brittle and probably best
> avoided.

I will look into DMLabel.

> 
> Note that you should be able to get columns to appear on the same
> process after partitioning by setting large weights for the vertical
> coupling.  Alternatively, we could partition a 2D mesh and extrude it
> (see DMPlexExtrude).
> 

Thanks,
-Gautam

PS: I also asked the same question on petsc-users mailing list, so I’m 
including it in this email thread.


> -- 
> You received this message because you are subscribed to the Google Groups 
> "tdycores-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to tdycores-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://protect2.fireeye.com/v1/url?k=7fbfa880-230a9739-7fbf8295-0cc47adc5fce-47455a25d1827a20=1=0edcbe39-9671-477e-b05b-0d3bf6a363e4=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Ftdycores-dev%2F875zhkmf0z.fsf%2540jedbrown.org.

[petsc-users] DMPlex: Mapping cells before and after partitioning

[Bisht, Gautam via petsc-users]

Hi All,

Here is the situation that I’m running into and am hoping you could provide 
some guidance. I created a mesh using DMPlexCreateBoxMesh() in which cells are 
ordered such that cells with increasing x coordinates are first, followed by 
cells with increasing y-corodinate and so forth. Next, I call 
DMPlexDistribute() that rearranges the cells after partitioning. How can I map 
cells after partitioning to cells before partitioning? 

Thanks,
-Gautam

Re: [petsc-users] killed 9 signal after upgrade from petsc 3.9.4 to 3.12.2

[Dave May]

This kind of issue is difficult to untangle because you have potentially
three pieces of software which might have changed between v3.9 and v3.12,
namely
PETSc, SLEPC and SuperLU_dist.
You need to isolate which software component is responsible for the 2x
increase in memory.

When I look at the memory usage in the log files, things look very very
similar for the raw PETSc objects.

[v3.9]
--- Event Stage 0: Main Stage


  Viewer 4  3 2520 0.

  Matrix15 15125236536 0.

  Vector22 22 19713856 0.

   Index Set10 10   995280 0.

 Vec Scatter 4  4 4928 0.

  EPS Solver 1  1 2276 0.

  Spectral Transform 1  1  848 0.

   Basis Vectors 1  1 2168 0.

 PetscRandom 1  1  662 0.

  Region 1  1  672 0.

   Direct Solver 1  117440 0.

   Krylov Solver 1  1 1176 0.

  Preconditioner 1  1 1000 0.

versus

[v3.12]

--- Event Stage 0: Main Stage


  Viewer 4  3 2520 0.

  Matrix15 15125237144 0.

  Vector22 22 19714528 0.

   Index Set10 10   995096 0.

 Vec Scatter 4  4 3168 0.

   Star Forest Graph 4  4 3936 0.

  EPS Solver 1  1 2292 0.

  Spectral Transform 1  1  848 0.

   Basis Vectors 1  1 2184 0.

 PetscRandom 1  1  662 0.

  Region 1  1  672 0.

   Direct Solver 1  117456 0.

   Krylov Solver 1  1 1400 0.

  Preconditioner 1  1 1000 0.

Certainly there is no apparent factor 2x increase in memory usage in the
underlying petsc objects themselves.
Furthermore, the counts of creations of petsc objects in toobig.log and
justfine.log match, indicating that none of the implementations used in
either PETSc or SLEPc have fundamentally changed wrt the usage of the
native petsc objects.

It is also curious that VecNorm is called 3 times in "justfine.log" and 19
times in "toobig.log" - although I don't see how that could be related to
you problem...

The above at least gives me the impression that issue of memory increase is
likely not coming from PETSc.
I just read Barry's useful email which is even more compelling and also
indicates SLEPc is not the likely culprit either as it uses PetscMalloc()
internally.

Some options to identify the problem:

1/ Eliminate SLEPc as a possible culprit by not calling EPSSolve() and
rather just call KSPSolve() with some RHS vector.
* If you still see a 2x increase, switch the preconditioner to using
-pc_type bjacobi -ksp_max_it 10 rather than superlu_dist.
If the memory usage is good, you can be pretty certain the issue arises
internally to superl_dist.

2/ Leave your code as is and perform your profiling using mumps rather than
superlu_dist.
This is a less reliable test than 1/ since the mumps implementation used
with v3.9 and v3.12 may differ...

Thanks
Dave

On Thu, 9 Jan 2020 at 20:17, Santiago Andres Triana 
wrote:

> Dear all,
>
> I think parmetis is not involved since I still run out of memory if I use
> the following options:
> export opts='-st_type sinvert -st_ksp_type preonly -st_pc_type lu
> -st_pc_factor_mat_solver_type superlu_dist -eps_true_residual 1'
> and  issuing:
> mpiexec -n 24 ./ex7 -f1 A.petsc -f2 B.petsc -eps_nev 1 -eps_target
> -4.008e-3+1.57142i $opts -eps_target_magnitude -eps_tol 1e-14 -memory_view
>
> Bottom line is that the memory usage of petsc-3.9.4 / slepc-3.9.2 is much
> lower than current version. I can only solve relatively small problems
> using the 3.12 series :(
> I have an example with smaller matrices that will likely fail in a 32 Gb
> ram machine with petsc-3.12 but runs just fine with petsc-3.9. The
> -memory_view output is
>
> with petsc-3.9.4: (log 'justfine.log' attached)
>
> Summary of Memory Usage in PETSc
> Maximum (over computational time) process memory:total 1.6665e+10
> max 7.5674e+08 min 6.4215e+08
> Current process memory:  total 1.5841e+10
> max 7.2881e+08 min 6.0905e+08
> Maximum (over computational time) space PetscMalloc()ed: total 3.1290e+09
> max 1.5868e+08 min 1.0179e+08
> Current space PetscMalloc()ed:   total 1.8808e+06
> max 7.8368e+04 min 7.8368e+04
>
>
> with petsc-3.12.2: (log 'toobig.log' attached)
>
> Summary of Memory Usage in PETSc
> Maximum (over 

Re: [petsc-users] killed 9 signal after upgrade from petsc 3.9.4 to 3.12.2

[Smith, Barry F. via petsc-users]

with petsc-3.9.4: (log 'justfine.log' attached)

Summary of Memory Usage in PETSc
Maximum (over computational time) process memory:total 1.6665e+10 max 
7.5674e+08 min 6.4215e+08
Current process memory:  total 1.5841e+10 max 
7.2881e+08 min 6.0905e+08

Below is the space allocated by PETSc

Maximum (over computational time) space PetscMalloc()ed: total 3.1290e+09 max 
1.5868e+08 min 1.0179e+08
Current space PetscMalloc()ed:   total 1.8808e+06 max 
7.8368e+04 min 7.8368e+04


with petsc-3.12.2: (log 'toobig.log' attached)

Summary of Memory Usage in PETSc
Maximum (over computational time) process memory:total 3.1564e+10 max 
1.3662e+09 min 1.2604e+09
Current process memory:  total 3.0355e+10 max 
1.3082e+09 min 1.2254e+09

Below is the space allocated by PETSc.  Note that 2.7618e+09 max is actually a 
bit lower than 3.1290e+09

Maximum (over computational time) space PetscMalloc()ed: total 2.7618e+09 max 
1.4339e+08 min 8.6493e+07
Current space PetscMalloc()ed:   total 3.6127e+06 max 
1.5053e+05 min 1.5053e+05

So it is not PETSc that allocating more memory than before.

Use the Massif option of valgrind to see where the large chunk of memory is 
actually used in the simulation.

  Barry



On Jan 9, 2020, at 2:16 PM, Santiago Andres Triana 
mailto:rep...@gmail.com>> wrote:

Dear all,

I think parmetis is not involved since I still run out of memory if I use the 
following options:
export opts='-st_type sinvert -st_ksp_type preonly -st_pc_type lu 
-st_pc_factor_mat_solver_type superlu_dist -eps_true_residual 1'
and  issuing:
mpiexec -n 24 ./ex7 -f1 A.petsc -f2 B.petsc -eps_nev 1 -eps_target 
-4.008e-3+1.57142i $opts -eps_target_magnitude -eps_tol 1e-14 -memory_view

Bottom line is that the memory usage of petsc-3.9.4 / slepc-3.9.2 is much lower 
than current version. I can only solve relatively small problems using the 3.12 
series :(
I have an example with smaller matrices that will likely fail in a 32 Gb ram 
machine with petsc-3.12 but runs just fine with petsc-3.9. The -memory_view 
output is

with petsc-3.9.4: (log 'justfine.log' attached)

Summary of Memory Usage in PETSc
Maximum (over computational time) process memory:total 1.6665e+10 max 
7.5674e+08 min 6.4215e+08
Current process memory:  total 1.5841e+10 max 
7.2881e+08 min 6.0905e+08
Maximum (over computational time) space PetscMalloc()ed: total 3.1290e+09 max 
1.5868e+08 min 1.0179e+08
Current space PetscMalloc()ed:   total 1.8808e+06 max 
7.8368e+04 min 7.8368e+04


with petsc-3.12.2: (log 'toobig.log' attached)

Summary of Memory Usage in PETSc
Maximum (over computational time) process memory:total 3.1564e+10 max 
1.3662e+09 min 1.2604e+09
Current process memory:  total 3.0355e+10 max 
1.3082e+09 min 1.2254e+09
Maximum (over computational time) space PetscMalloc()ed: total 2.7618e+09 max 
1.4339e+08 min 8.6493e+07
Current space PetscMalloc()ed:   total 3.6127e+06 max 
1.5053e+05 min 1.5053e+05

Strangely, monitoring with 'top' I can see *appreciably higher* peak memory 
use, usually twice what -memory_view ends up reporting, both for petsc-3.9.4 
and current. Program fails usually at this peak if not enough ram available

The matrices for the example quoted above can be downloaded here (I use slepc's 
tutorial ex7.c to solve the problem):
https://www.dropbox.com/s/as9bec9iurjra6r/A.petsc?dl=0  (about 600 Mb)
https://www.dropbox.com/s/u2bbmng23rp8l91/B.petsc?dl=0  (about 210 Mb)

I haven't been able to use a debugger successfully since I am using a compute 
node without the possibility of an xterm ... note that I have no experience 
using a debugger so any help on that will also be appreciated!
Hope I can switch to the current petsc/slepc version for my production runs 
soon...

Thanks again!
Santiago



On Thu, Jan 9, 2020 at 4:25 PM Stefano Zampini  
wrote:
Can you reproduce the issue with smaller matrices? Or with a debug build (i.e. 
using —with-debugging=1 and compilation flags -02 -g)?

The only changes in parmetis between the two PETSc releases are these below, 
but I don’t see how they could cause issues

kl-18448:pkg-parmetis szampini$ git log -2
commit ab4fedc6db1f2e3b506be136e3710fcf89ce16ea (HEAD -> master, tag: 
v4.0.3-p5, origin/master, origin/dalcinl/random, origin/HEAD)
Author: Lisandro Dalcin 
Date:   Thu May 9 18:44:10 2019 +0300

GKLib: Make FPRFX##randInRange() portable for 32bit/64bit indices

commit 2b4afc79a79ef063f369c43da2617fdb64746dd7
Author: Lisandro Dalcin 
Date:   Sat May 4 17:22:19 2019 +0300

GKlib: Use gk_randint32() to define the RandomInRange() macro



On Jan 9, 2020, at 4:31 AM, Smith, Barry F. via petsc-users 
 wrote:


 This is extremely worrisome:

==23361== Use of uninitialised value of size 8
==23361==at 0x847E939: gk_randint64 

Re: [petsc-users] killed 9 signal after upgrade from petsc 3.9.4 to 3.12.2

[Santiago Andres Triana]

Dear all,

I think parmetis is not involved since I still run out of memory if I use
the following options:
export opts='-st_type sinvert -st_ksp_type preonly -st_pc_type lu
-st_pc_factor_mat_solver_type superlu_dist -eps_true_residual 1'
and  issuing:
mpiexec -n 24 ./ex7 -f1 A.petsc -f2 B.petsc -eps_nev 1 -eps_target
-4.008e-3+1.57142i $opts -eps_target_magnitude -eps_tol 1e-14 -memory_view

Bottom line is that the memory usage of petsc-3.9.4 / slepc-3.9.2 is much
lower than current version. I can only solve relatively small problems
using the 3.12 series :(
I have an example with smaller matrices that will likely fail in a 32 Gb
ram machine with petsc-3.12 but runs just fine with petsc-3.9. The
-memory_view output is

with petsc-3.9.4: (log 'justfine.log' attached)

Summary of Memory Usage in PETSc
Maximum (over computational time) process memory:total 1.6665e+10
max 7.5674e+08 min 6.4215e+08
Current process memory:  total 1.5841e+10
max 7.2881e+08 min 6.0905e+08
Maximum (over computational time) space PetscMalloc()ed: total 3.1290e+09
max 1.5868e+08 min 1.0179e+08
Current space PetscMalloc()ed:   total 1.8808e+06
max 7.8368e+04 min 7.8368e+04


with petsc-3.12.2: (log 'toobig.log' attached)

Summary of Memory Usage in PETSc
Maximum (over computational time) process memory:total 3.1564e+10
max 1.3662e+09 min 1.2604e+09
Current process memory:  total 3.0355e+10
max 1.3082e+09 min 1.2254e+09
Maximum (over computational time) space PetscMalloc()ed: total 2.7618e+09
max 1.4339e+08 min 8.6493e+07
Current space PetscMalloc()ed:   total 3.6127e+06
max 1.5053e+05 min 1.5053e+05

Strangely, monitoring with 'top' I can see *appreciably higher* peak memory
use, usually twice what -memory_view ends up reporting, both for
petsc-3.9.4 and current. Program fails usually at this peak if not enough
ram available

The matrices for the example quoted above can be downloaded here (I use
slepc's tutorial ex7.c to solve the problem):
https://www.dropbox.com/s/as9bec9iurjra6r/A.petsc?dl=0  (about 600 Mb)
https://www.dropbox.com/s/u2bbmng23rp8l91/B.petsc?dl=0  (about 210 Mb)

I haven't been able to use a debugger successfully since I am using a
compute node without the possibility of an xterm ... note that I have no
experience using a debugger so any help on that will also be appreciated!
Hope I can switch to the current petsc/slepc version for my production runs
soon...

Thanks again!
Santiago



On Thu, Jan 9, 2020 at 4:25 PM Stefano Zampini 
wrote:

> Can you reproduce the issue with smaller matrices? Or with a debug build
> (i.e. using —with-debugging=1 and compilation flags -02 -g)?
>
> The only changes in parmetis between the two PETSc releases are these
> below, but I don’t see how they could cause issues
>
> kl-18448:pkg-parmetis szampini$ git log -2
> commit ab4fedc6db1f2e3b506be136e3710fcf89ce16ea (*HEAD -> **master*, *tag:
> v4.0.3-p5*, *origin/master*, *origin/dalcinl/random*, *origin/HEAD*)
> Author: Lisandro Dalcin 
> Date:   Thu May 9 18:44:10 2019 +0300
>
> GKLib: Make FPRFX##randInRange() portable for 32bit/64bit indices
>
> commit 2b4afc79a79ef063f369c43da2617fdb64746dd7
> Author: Lisandro Dalcin 
> Date:   Sat May 4 17:22:19 2019 +0300
>
> GKlib: Use gk_randint32() to define the RandomInRange() macro
>
>
>
> On Jan 9, 2020, at 4:31 AM, Smith, Barry F. via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
>
>  This is extremely worrisome:
>
> ==23361== Use of uninitialised value of size 8
> ==23361==at 0x847E939: gk_randint64 (random.c:99)
> ==23361==by 0x847EF88: gk_randint32 (random.c:128)
> ==23361==by 0x81EBF0B: libparmetis__Match_Global (in
> /space/hpc-home/trianas/petsc-3.12.3/arch-linux2-c-debug/lib/libparmetis.so)
>
> do you get that with PETSc-3.9.4 or only with 3.12.3?
>
>   This may result in Parmetis using non-random numbers and then giving
> back an inappropriate ordering that requires more memory for SuperLU_DIST.
>
>  Suggest looking at the code, or running in the debugger to see what is
> going on there. We use parmetis all the time and don't see this.
>
>  Barry
>
>
>
>
>
>
> On Jan 8, 2020, at 4:34 PM, Santiago Andres Triana 
> wrote:
>
> Dear Matt, petsc-users:
>
> Finally back after the holidays to try to solve this issue, thanks for
> your patience!
> I compiled the latest petsc (3.12.3) with debugging enabled, the same
> problem appears: relatively large matrices result in out of memory errors.
> This is not the case for petsc-3.9.4, all fine there.
> This is a non-hermitian, generalized eigenvalue problem, I generate the A
> and B matrices myself and then I use example 7 (from the slepc tutorial at
> $SLEPC_DIR/src/eps/examples/tutorials/ex7.c ) to solve the problem:
>
> mpiexec -n 24 valgrind --tool=memcheck -q --num-callers=20
> --log-file=valgrind.log.%p ./ex7 -malloc off -f1 A.petsc -f2 B.petsc
> -eps_nev 1 -eps_target -2.5e-4+1.56524i 

Re: [petsc-users] set petsc matrix using input array

[Jed Brown]

Note that PETSc's formats are more space-efficient and faster than the
COO (triplet) format.  If you can produce triplet chunks instead of the
full matrix, you can add them incrementally to reduce the peak memory
usage.  Note that many preconditioners use storage similar to (or
greater than) a single assembled matrix, so copying (which is done
before preconditioner setup) may not increase the peak memory usage
(which is all that matters for capability).

"Smith, Barry F. via petsc-users"  writes:

>   Since PETSc does not use that format there, of course, has to be a time 
> when you have duplicate memory. 
>
>   Barry
>
>
>
>
>
>> On Jan 9, 2020, at 12:47 PM, Sam Guo  wrote:
>> 
>> Dear PETSc dev team,
>>Suppose I have the matrix already in triplet format int int[] I, int[] J, 
>> double[] A, Is possible to create petsc matrix using A without copying? I 
>> like to avoid the duplicate memory if possible.
>> 
>> Thanks,
>> Sam

Re: [petsc-users] set petsc matrix using input array

[Smith, Barry F. via petsc-users]

  Since PETSc does not use that format there, of course, has to be a time when 
you have duplicate memory. 

  Barry





> On Jan 9, 2020, at 12:47 PM, Sam Guo  wrote:
> 
> Dear PETSc dev team,
>Suppose I have the matrix already in triplet format int int[] I, int[] J, 
> double[] A, Is possible to create petsc matrix using A without copying? I 
> like to avoid the duplicate memory if possible.
> 
> Thanks,
> Sam

[petsc-users] set petsc matrix using input array

[Sam Guo]

Dear PETSc dev team,
   Suppose I have the matrix already in triplet format int int[] I, int[]
J, double[] A, Is possible to create petsc matrix using A without copying?
I like to avoid the duplicate memory if possible.

Thanks,
Sam

Re: [petsc-users] killed 9 signal after upgrade from petsc 3.9.4 to 3.12.2

[Stefano Zampini]

Can you reproduce the issue with smaller matrices? Or with a debug build (i.e. 
using —with-debugging=1 and compilation flags -02 -g)? 

The only changes in parmetis between the two PETSc releases are these below, 
but I don’t see how they could cause issues

kl-18448:pkg-parmetis szampini$ git log -2
commit ab4fedc6db1f2e3b506be136e3710fcf89ce16ea (HEAD -> master, tag: 
v4.0.3-p5, origin/master, origin/dalcinl/random, origin/HEAD)
Author: Lisandro Dalcin 
Date:   Thu May 9 18:44:10 2019 +0300

GKLib: Make FPRFX##randInRange() portable for 32bit/64bit indices

commit 2b4afc79a79ef063f369c43da2617fdb64746dd7
Author: Lisandro Dalcin 
Date:   Sat May 4 17:22:19 2019 +0300

GKlib: Use gk_randint32() to define the RandomInRange() macro



> On Jan 9, 2020, at 4:31 AM, Smith, Barry F. via petsc-users 
>  wrote:
> 
> 
>  This is extremely worrisome:
> 
> ==23361== Use of uninitialised value of size 8
> ==23361==at 0x847E939: gk_randint64 (random.c:99)
> ==23361==by 0x847EF88: gk_randint32 (random.c:128)
> ==23361==by 0x81EBF0B: libparmetis__Match_Global (in 
> /space/hpc-home/trianas/petsc-3.12.3/arch-linux2-c-debug/lib/libparmetis.so)
> 
> do you get that with PETSc-3.9.4 or only with 3.12.3?  
> 
>   This may result in Parmetis using non-random numbers and then giving back 
> an inappropriate ordering that requires more memory for SuperLU_DIST.
> 
>  Suggest looking at the code, or running in the debugger to see what is going 
> on there. We use parmetis all the time and don't see this.
> 
>  Barry
> 
> 
> 
> 
> 
> 
>> On Jan 8, 2020, at 4:34 PM, Santiago Andres Triana  wrote:
>> 
>> Dear Matt, petsc-users:
>> 
>> Finally back after the holidays to try to solve this issue, thanks for your 
>> patience!
>> I compiled the latest petsc (3.12.3) with debugging enabled, the same 
>> problem appears: relatively large matrices result in out of memory errors. 
>> This is not the case for petsc-3.9.4, all fine there.
>> This is a non-hermitian, generalized eigenvalue problem, I generate the A 
>> and B matrices myself and then I use example 7 (from the slepc tutorial at 
>> $SLEPC_DIR/src/eps/examples/tutorials/ex7.c ) to solve the problem:
>> 
>> mpiexec -n 24 valgrind --tool=memcheck -q --num-callers=20 
>> --log-file=valgrind.log.%p ./ex7 -malloc off -f1 A.petsc -f2 B.petsc 
>> -eps_nev 1 -eps_target -2.5e-4+1.56524i -eps_target_magnitude -eps_tol 1e-14 
>> $opts
>> 
>> where the $opts variable is:
>> export opts='-st_type sinvert -st_ksp_type preonly -st_pc_type lu 
>> -eps_error_relative ::ascii_info_detail -st_pc_factor_mat_solver_type 
>> superlu_dist -mat_superlu_dist_iterrefine 1 -mat_superlu_dist_colperm 
>> PARMETIS -mat_superlu_dist_parsymbfact 1 -eps_converged_reason -eps_conv_rel 
>> -eps_monitor_conv -eps_true_residual 1'
>> 
>> the output from valgrind (sample from one processor) and from the program 
>> are attached.
>> If it's of any use the matrices are here (might need at least 180 Gb of ram 
>> to solve the problem succesfully under petsc-3.9.4):
>> 
>> https://www.dropbox.com/s/as9bec9iurjra6r/A.petsc?dl=0
>> https://www.dropbox.com/s/u2bbmng23rp8l91/B.petsc?dl=0
>> 
>> WIth petsc-3.9.4 and slepc-3.9.2 I can use matrices up to 10Gb (with 240 Gb 
>> ram), but only up to 3Gb with the latest petsc/slepc.
>> Any suggestions, comments or any other help are very much appreciated!
>> 
>> Cheers,
>> Santiago
>> 
>> 
>> 
>> On Mon, Dec 23, 2019 at 11:19 PM Matthew Knepley  wrote:
>> On Mon, Dec 23, 2019 at 3:14 PM Santiago Andres Triana  
>> wrote:
>> Dear all,
>> 
>> After upgrading to petsc 3.12.2 my solver program crashes consistently. 
>> Before the upgrade I was using petsc 3.9.4 with no problems.
>> 
>> My application deals with a complex-valued, generalized eigenvalue problem. 
>> The matrices involved are relatively large, typically 2 to 10 Gb in size, 
>> which is no problem for petsc 3.9.4.
>> 
>> Are you sure that your indices do not exceed 4B? If so, you need to 
>> configure using
>> 
>>  --with-64-bit-indices
>> 
>> Also, it would be nice if you ran with the debugger so we can get a stack 
>> trace for the SEGV.
>> 
>>  Thanks,
>> 
>>Matt
>> 
>> However, after the upgrade I can only obtain solutions when the matrices are 
>> small, the solver crashes when the matrices' size exceed about 1.5 Gb:
>> 
>> [0]PETSC ERROR: 
>> 
>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the 
>> batch system) has told this process to end
>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>> [0]PETSC ERROR: or see 
>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X 
>> to find memory corruption errors
>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and 
>> run 
>> [0]PETSC ERROR: to get more information on the crash.
>> 
>> and so on for each cpu.
>>