Re: [petsc-users] Question about petsc4py createWithArray function

[Samar Khatiwala]

Hi Stefano,

Thank you for this. Your example was invaluable in helping me figure out what 
is going on. (I did not know about about sys.getrefcount - I’m only just 
starting to learn python.) It was indeed the refcount. Until the Vec is 
destroyed it can’t go to zero and my array destructor method won’t be triggered 
(which is not really a problem because the memory is freed elsewhere). I take 
back what I said about there being a memory leak in petsc4py. Sorry.

Thanks again for your help!

Best,

Samar

On May 6, 2024, at 2:09 PM, Stefano Zampini  wrote:

Samar

After a second look, I believe the petsc4py code is correct. You can test it 
using the script below.
After destroy is called (or del), the reference count of the numpy array is 
back to its initial state.
Maybe you are not calling del or destroy? a MWE would help to understand your 
use case

kl-18448:~ szampini$ cat t.py
import sys
import numpy as np
from petsc4py import PETSc

x = np.zeros(4, dtype=PETSc.ScalarType)
print('initial ref count',sys.getrefcount(x))

v = PETSc.Vec().createWithArray(x)
print('after create',sys.getrefcount(x))

# check if they share memory
v.view()
x[1] = 2
v.view()

# free
v.destroy()
# you can also call del
# del v
print('after destroy',sys.getrefcount(x))

kl-18448:~ szampini$ python t.py
initial ref count 2
after create 3
Vec Object: 1 MPI process
  type: seq
0.
0.
0.
0.
Vec Object: 1 MPI process
  type: seq
0.
2.
0.
0.
after destroy 2



Il giorno lun 6 mag 2024 alle ore 08:49 Samar Khatiwala 
mailto:samar.khatiw...@earth.ox.ac.uk>> ha 
scritto:
Hi Stefano,

Thanks for looking into this. Since createWithArray calls VecCreateMPIWithArray 
which, as Matt noted and is documented 
(https://urldefense.us/v3/__https://petsc.org/main/manualpages/Vec/VecCreateMPIWithArray/__;!!G_uCfscf7eWS!fl-7I7xkuEmTFH8M9EaARIZL5vhGFWkIk5bVIGt0bkt_cYcuwPSwgOWaec1-DvYwiKWxA5q0HS0VTLBmD4NnSe1PfG057Y4SBd-8qs8$
 ) doesn’t free the memory, then there’s a memory leak (and, furthermore, 
calling del on the original array will have no effect).

Lisandro: would be great if you can provide some guidance.

Thanks,

Samar

On May 3, 2024, at 12:45 PM, Stefano Zampini 
mailto:stefano.zamp...@gmail.com>> wrote:

While waiting for our Python wizard to shed light on this, I note that, from 
the documentation of PyArray_FROM_OTF 
https://urldefense.us/v3/__https://numpy.org/devdocs/user/c-info.how-to-extend.html*converting-an-arbitrary-sequence-object__;Iw!!G_uCfscf7eWS!fl-7I7xkuEmTFH8M9EaARIZL5vhGFWkIk5bVIGt0bkt_cYcuwPSwgOWaec1-DvYwiKWxA5q0HS0VTLBmD4NnSe1PfG057Y4S1ro4qy0$
 , we have

The object can be any Python object convertible to an ndarray. If the object is 
already (a subclass of) the ndarray that satisfies the requirements then a new 
reference is returned.

I guess we should call "del" on the ndarray returned by iarray_s after having 
called  self.set_attr('__array__', array) in this case, but let's wait for 
Lisandro to confirm




Il giorno ven 3 mag 2024 alle ore 11:42 Samar Khatiwala 
mailto:samar.khatiw...@earth.ox.ac.uk>> ha 
scritto:
This Message Is From an External Sender
This message came from outside your organization.

Hi Matt,

Thanks so much for the quick reply!

Regarding #2, I put some debug statement in my code and what I find is that 
when I use createWithArray on my Cython-allocated numpy array, the destructor I 
set for it is no longer called when I delete the array. (If I don’t use 
createWithArray then the destructor is triggered.) I interpret that to suggest 
that the petsc4py Vec is somehow ’taking over’ management of the numpy array. 
But I don’t understand where that could be happening. (I don’t think it has to 
do with the actual freeing of memory by PETSc's VecDestroy.)

createWithArray calls iarray_s which in turn calls PyArray_FROM_OTF. Could it 
be there’s something going on there? The numpy documentation is unclear.

Lisandro: do you have any thoughts on this?

Thanks,

Samar

On May 2, 2024, at 11:56 PM, Matthew Knepley 
mailto:knep...@gmail.com>> wrote:

On Thu, May 2, 2024 at 12:53 PM Samar Khatiwala 
mailto:samar.khatiw...@earth.ox.ac.uk>> wrote:
This Message Is From an External Sender
This message came from outside your organization.


Hello,

I have a couple of questions about createWithArray in petsc4py:

1) What is the correct usage for creating a standard MPI Vec with it? Something 
like this seems to work but is it right?:

On each rank do:
a = np.zeros(localSize)
v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD)

Is that all it takes?

That looks right to me.

2) Who ‘owns’ the underlying memory for a Vec created with the createWithArray 
method, i.e., who is responsible for managing it and doing garbage collection? 
In my problem, the numpy array is created in a Cython module where memory is 
allocated, and a pointer to it is associated with a numpy ndarray via 
PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have a deallocator 
method of my

Re: [petsc-users] Question about petsc4py createWithArray function

[Samar Khatiwala]

Hi Stefano,

Thanks for looking into this. Since createWithArray calls VecCreateMPIWithArray 
which, as Matt noted and is documented 
(https://urldefense.us/v3/__https://petsc.org/main/manualpages/Vec/VecCreateMPIWithArray/__;!!G_uCfscf7eWS!YpH_OGif6W8Ql9KZDYBiRMKyALSRhMZOHRnkCkBHqBcVB4ef_UBHFbT_EuTgx8ivGmDNQU8bRVaOc-B_tqQ2heGjdvoE9tp8ZbPi3QQ$
 ) doesn’t free the memory, then there’s a memory leak (and, furthermore, 
calling del on the original array will have no effect).

Lisandro: would be great if you can provide some guidance.

Thanks,

Samar

On May 3, 2024, at 12:45 PM, Stefano Zampini  wrote:

While waiting for our Python wizard to shed light on this, I note that, from 
the documentation of PyArray_FROM_OTF 
https://urldefense.us/v3/__https://numpy.org/devdocs/user/c-info.how-to-extend.html*converting-an-arbitrary-sequence-object__;Iw!!G_uCfscf7eWS!YpH_OGif6W8Ql9KZDYBiRMKyALSRhMZOHRnkCkBHqBcVB4ef_UBHFbT_EuTgx8ivGmDNQU8bRVaOc-B_tqQ2heGjdvoE9tp8_QyxtQk$
 , we have

The object can be any Python object convertible to an ndarray. If the object is 
already (a subclass of) the ndarray that satisfies the requirements then a new 
reference is returned.

I guess we should call "del" on the ndarray returned by iarray_s after having 
called  self.set_attr('__array__', array) in this case, but let's wait for 
Lisandro to confirm




Il giorno ven 3 mag 2024 alle ore 11:42 Samar Khatiwala 
mailto:samar.khatiw...@earth.ox.ac.uk>> ha 
scritto:
Hi Matt, Thanks so much for the quick reply! Regarding #2, I put some debug 
statement in my code and what I find is that when I use createWithArray on my 
Cython-allocated numpy array, the destructor I set for it is no longer called 
when I delete
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd
Hi Matt,

Thanks so much for the quick reply!

Regarding #2, I put some debug statement in my code and what I find is that 
when I use createWithArray on my Cython-allocated numpy array, the destructor I 
set for it is no longer called when I delete the array. (If I don’t use 
createWithArray then the destructor is triggered.) I interpret that to suggest 
that the petsc4py Vec is somehow ’taking over’ management of the numpy array. 
But I don’t understand where that could be happening. (I don’t think it has to 
do with the actual freeing of memory by PETSc's VecDestroy.)

createWithArray calls iarray_s which in turn calls PyArray_FROM_OTF. Could it 
be there’s something going on there? The numpy documentation is unclear.

Lisandro: do you have any thoughts on this?

Thanks,

Samar

On May 2, 2024, at 11:56 PM, Matthew Knepley 
mailto:knep...@gmail.com>> wrote:

On Thu, May 2, 2024 at 12:53 PM Samar Khatiwala 
mailto:samar.khatiw...@earth.ox.ac.uk>> wrote:
This Message Is From an External Sender
This message came from outside your organization.


Hello,

I have a couple of questions about createWithArray in petsc4py:

1) What is the correct usage for creating a standard MPI Vec with it? Something 
like this seems to work but is it right?:

On each rank do:
a = np.zeros(localSize)
v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD)

Is that all it takes?

That looks right to me.

2) Who ‘owns’ the underlying memory for a Vec created with the createWithArray 
method, i.e., who is responsible for managing it and doing garbage collection? 
In my problem, the numpy array is created in a Cython module where memory is 
allocated, and a pointer to it is associated with a numpy ndarray via 
PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have a deallocator 
method of my own that is called when the numpy array is deleted/goes out of 
scope/whenever python does garbage collection. All of that works fine. But if I 
use this array to create a Vec with createWithArray what happens when the Vec 
is, e.g., destroyed? Will my deallocator be called?

No. The PETSc struct will be deallocated, but the storage will not be touched.

  Thanks,

 Matt

Or does petsc4py know that it doesn’t own the memory and won’t attempt to free 
it? I can’t quite figure out from the petsc4py code what is going on. And help 
would be appreciated.

Thanks very much.

Samar




--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YpH_OGif6W8Ql9KZDYBiRMKyALSRhMZOHRnkCkBHqBcVB4ef_UBHFbT_EuTgx8ivGmDNQU8bRVaOc-B_tqQ2heGjdvoE9tp8YS3M6pE$
 
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dXZuKR18IO2hZ4uPEqHaG0Jb0ALPgw1T0dm4HAWfyBfsCvkYTmq7aG7yw3yc-5CVhcFMaUL3WzGM8YHlBTTA_JMs_dYUCTX81-bKyrM$>



--
Stefano

Re: [petsc-users] Question about petsc4py createWithArray function

[Samar Khatiwala]

Hi Matt,

Thanks so much for the quick reply!

Regarding #2, I put some debug statement in my code and what I find is that 
when I use createWithArray on my Cython-allocated numpy array, the destructor I 
set for it is no longer called when I delete the array. (If I don’t use 
createWithArray then the destructor is triggered.) I interpret that to suggest 
that the petsc4py Vec is somehow ’taking over’ management of the numpy array. 
But I don’t understand where that could be happening. (I don’t think it has to 
do with the actual freeing of memory by PETSc's VecDestroy.)

createWithArray calls iarray_s which in turn calls PyArray_FROM_OTF. Could it 
be there’s something going on there? The numpy documentation is unclear.

Lisandro: do you have any thoughts on this?

Thanks,

Samar

On May 2, 2024, at 11:56 PM, Matthew Knepley  wrote:

On Thu, May 2, 2024 at 12:53 PM Samar Khatiwala 
mailto:samar.khatiw...@earth.ox.ac.uk>> wrote:
This Message Is From an External Sender
This message came from outside your organization.


Hello,

I have a couple of questions about createWithArray in petsc4py:

1) What is the correct usage for creating a standard MPI Vec with it? Something 
like this seems to work but is it right?:

On each rank do:
a = np.zeros(localSize)
v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD)

Is that all it takes?

That looks right to me.

2) Who ‘owns’ the underlying memory for a Vec created with the createWithArray 
method, i.e., who is responsible for managing it and doing garbage collection? 
In my problem, the numpy array is created in a Cython module where memory is 
allocated, and a pointer to it is associated with a numpy ndarray via 
PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have a deallocator 
method of my own that is called when the numpy array is deleted/goes out of 
scope/whenever python does garbage collection. All of that works fine. But if I 
use this array to create a Vec with createWithArray what happens when the Vec 
is, e.g., destroyed? Will my deallocator be called?

No. The PETSc struct will be deallocated, but the storage will not be touched.

  Thanks,

 Matt

Or does petsc4py know that it doesn’t own the memory and won’t attempt to free 
it? I can’t quite figure out from the petsc4py code what is going on. And help 
would be appreciated.

Thanks very much.

Samar




--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dXZuKR18IO2hZ4uPEqHaG0Jb0ALPgw1T0dm4HAWfyBfsCvkYTmq7aG7yw3yc-5CVhcFMaUL3WzGM8YHlBTTA_JMs_dYUCTX8wgHdY_Y$
 
<https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dXZuKR18IO2hZ4uPEqHaG0Jb0ALPgw1T0dm4HAWfyBfsCvkYTmq7aG7yw3yc-5CVhcFMaUL3WzGM8YHlBTTA_JMs_dYUCTX81-bKyrM$
 >

[petsc-users] Question about petsc4py createWithArray function

[Samar Khatiwala]

 Hello, I have a couple of questions about createWithArray in petsc4py: 1) What is the correct usage for creating a standard MPI Vec with it? Something like this seems to work but is it right?: On each rank do: a = np. zeros(localSize) v = PETSc. Vec(). createWithArray(a,




ZjQcmQRYFpfptBannerStart




  

  
	This Message Is From an External Sender
  
  
This message came from outside your organization.
  



 
  


ZjQcmQRYFpfptBannerEnd




Hello,

I have a couple of questions about createWithArray in petsc4py:

1) What is the correct usage for creating a standard MPI Vec with it? Something like this seems to work but is it right?:

On each rank do:
a = np.zeros(localSize)
v = PETSc.Vec().createWithArray(a, comm=PETSc.COMM_WORLD)

Is that all it takes?

2) Who ‘owns’ the underlying memory for a Vec created with the createWithArray method, i.e., who is responsible for managing it and doing garbage collection? In my problem, the numpy array is created in a Cython module where memory is allocated, and a pointer to it is associated with a numpy ndarray via PyArray_SimpleNewFromData and PyArray_SetBaseObject. I have a deallocator method of my own that is called when the numpy array is deleted/goes out of scope/whenever python does garbage collection. All of that works fine. But if I use this array to create a Vec with createWithArray what happens when the Vec is, e.g., destroyed? Will my deallocator be called? Or does petsc4py know that it doesn’t own the memory and won’t attempt to free it? I can’t quite figure out from the petsc4py code what is going on. And help would be appreciated.

Thanks very much.

Samar

Re: [petsc-users] M2 macs

[Samar Khatiwala]

Hi Sanjay,

I’ve done this with ifort on my M2 MacBook Air and reported on my experience in 
this list some months ago. I attach the message below.

Samar

Begin forwarded message:

From: Samar Khatiwala 
Subject: Re: [petsc-users] PETSc build asks for network connections
Date: April 28, 2023 at 5:43:44 PM GMT+1
To: Pierre Jolivet 
Cc: petsc-users 

Hi,

I realize this is an old thread but I have some recent experience based on 
setting up an M2 Mac that might be relevant.

I was dreading moving to Apple Silicon Macs because of issues like these but I 
actually did not run into this particular problem.
While I can’t be certain I think it is because in the process of installing 
another piece of software I had to modify Apple’s security
restrictions to make them more permissive. Details of how to do this are in the 
following and it takes only a minute to implement:

https://rogueamoeba.com/support/knowledgebase/?showArticle=ACE-StepByStep=Audio+Hijack

Incidentally, I built mpich from source followed by PETSc in the usual way.

Something else that might be helpful for others is my experience getting ifort 
to work. (My needs were somewhat specific: mixed
fortran/C code, preferably ifort, and avoid package managers.) The intel OneAPI 
installer ran smoothly (via rosetta) but when
building mpich (or PETSc) I ran into an obvious problem: clang produces arm64 
object files while ifort produces x86 ones. I couldn’t
manage to set the correct CFLAGS to tell clang to target x86. Instead, the 
(simpler) solution turned out to be (1) the fact that all the
executables in Apple’s toolchain are universal binaries, and (2) the ‘arch’ 
command can let you run programs for any of the two
architectures. Specifically, executing in the terminal:

arch -x86_64 bash

starts a bash shell and *every* program that is then run from that shell is 
automatically the x86 version. So I could then do:
FC=ifort
./configure --prefix=/usr/local/mpichx86 --enable-two-level-namespace
make
sudo make install

and get an x86 build of mpich which I could then use (from the same shell or a 
new one started as above) to build [x86] PETSc.
Except for some annoying warnings from MKL (I think because it is confused what 
architecture it is running on) everything runs
smoothly and - even in emulation - surprisingly fast.

Sorry if this is all well know and already documented on PETSc’s install page.

Samar




On Jan 9, 2024, at 9:46 PM, Sanjay Govindjee via petsc-users 
 wrote:

I was wondering if anyone has build experience with PETSc + FORTRAN on an 
M2-based MAC?  In particular, I am looking for compiler recommendations.

-sanjay

Re: [petsc-users] UNABLE to read '.mat' file with 'PetscViewerHDF5Open' (WSL-Ubuntu22.04)

[Samar Khatiwala]

Another possibility is that the Matlab file wasn’t saved in HDF5. Try resaving 
it with the “-v7.3” option.

Samar

On Jul 25, 2023, at 10:18 AM, Matthew Knepley  wrote:

On Tue, Jul 25, 2023 at 3:12 AM maitri ksh 
mailto:maitri@gmail.com>> wrote:
Hi,
I am new to Petsc, here are some details of the relevant softwares I am using:
1. petsc-3.19.3 (on wsl-ubuntu22.04 platform)
2. MATLAB-R2022a
3. hdf5-1.10.7 (checked using 'pkg-config --modversion hdf5')
4. configured using:  './configure --with-cc=gcc --with-cxx=g++ 
--with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab 
--with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1'

I am trying to read a '.mat' file ('myfile.mat') using 'PetscViewerHDF5Open'  
(the source code 'read_Matrix.c' is attached herewith). I could compile the 
source code with no error, but I am getting an error ('ERROR.txt') indicating 
that there is an issue opening the HDF5 file. Could you please help me to 
resolve this issue

HDF5 thinks that this is not an HDF5 file. Since PETSc built HDF5 for you, you 
can check this using

  $PETSC_DIR/$PETSC_ARCH/bin/h5dump myfile.mat

It could be a version incompatibility between the HDF5 on your machine and the 
one you asked PETSc to download.

  Thanks,

Matt

Maitri



--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

Re: [petsc-users] Compiling PETSC with Intel OneAPI compilers and OpenMPI

[Samar Khatiwala]

Hi Marcos,

Yes, I compiled with clang instead of icc (no particular reason for this; I 
tend to use gcc/clang). I use mpich4.1.1, which I first built with clang and 
ifort:


FC=ifort

./configure --prefix=/usr/local/mpich4 --enable-two-level-namespace


Samar

On May 15, 2023, at 6:07 PM, Vanella, Marcos (Fed)  
wrote:

Hi Samar, what MPI library do you use? Did you compile it with clang instead of 
icc?
Thanks,
Marcos

From: Samar Khatiwala 
Sent: Monday, May 15, 2023 1:05 PM
To: Matthew Knepley 
Cc: Vanella, Marcos (Fed) ; petsc-users@mcs.anl.gov 

Subject: Re: [petsc-users] Compiling PETSC with Intel OneAPI compilers and 
OpenMPI

Hi, for what it’s worth, clang + ifort from OneAPI 2023 update 1 works fine for 
me on both Intel and M2 Macs. So it might just be a matter of upgrading.

Samar

On May 15, 2023, at 5:53 PM, Matthew Knepley  wrote:

Send us

  $PETSC_ARCH/include/petscconf.h

  Thanks,

 Matt

On Mon, May 15, 2023 at 12:49 PM Vanella, Marcos (Fed) 
mailto:marcos.vane...@nist.gov>> wrote:
Hi Matt, I configured the lib like this:

$ ./configure --with-blaslapack-dir=/opt/intel/oneapi/mkl/2022.2.1 
--with-debugging=0 --with-shared-libraries=0 --download-make

and compiled. I still get some check segfault error. See below:

$ make PETSC_DIR=/Users/mnv/Documents/Software/petsc-3.19.1 
PETSC_ARCH=arch-darwin-c-opt check
Running check examples to verify correct installation
Using PETSC_DIR=/Users/mnv/Documents/Software/petsc-3.19.1 and 
PETSC_ARCH=arch-darwin-c-opt
***Error detected during compile or link!***
See https://petsc.org/release/faq/
/Users/mnv/Documents/Software/petsc-3.19.1/src/snes/tutorials ex19
*
mpicc -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined 
-Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first 
-Wl,-no_compact_unwind  -fPIC -wd1572 -Wno-unknown-pragmas -g -O3  
-I/Users/mnv/Documents/Software/petsc-3.19.1/include 
-I/Users/mnv/Documents/Software/petsc-3.19.1/arch-darwin-c-opt/include 
-I/opt/X11/include  -std=c99ex19.c  
-L/Users/mnv/Documents/Software/petsc-3.19.1/arch-darwin-c-opt/lib 
-Wl,-rpath,/opt/intel/oneapi/mkl/2022.2.1/lib 
-L/opt/intel/oneapi/mkl/2022.2.1/lib -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib 
-L/opt/openmpi414_oneapi22u3/lib 
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.2.1/mac/compiler/lib 
-L/opt/intel/oneapi/tbb/2021.7.1/lib -L/opt/intel/oneapi/ippcp/2021.6.2/lib 
-L/opt/intel/oneapi/ipp/2021.6.2/lib 
-L/opt/intel/oneapi/dnnl/2022.2.1/cpu_iomp/lib 
-L/opt/intel/oneapi/dal/2021.7.1/lib 
-L/opt/intel/oneapi/compiler/2022.2.1/mac/compiler/lib 
-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib 
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.2.1/mac/bin/intel64/../../compiler/lib
 -L/opt/intel/oneapi/compiler/2022.2.1/mac/bin/intel64/../../compiler/lib 
-Wl,-rpath,/Library/Developer/CommandLineTools/usr/lib/clang/14.0.3/lib/darwin 
-L/Library/Developer/CommandLineTools/usr/lib/clang/14.0.3/lib/darwin -lpetsc 
-lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -lX11 -lmpi_usempif08 
-lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -limf -lm -lz 
-lifport -lifcoremt -lsvml -lipgo -lirc -lpthread -lclang_rt.osx -lmpi 
-lopen-rte -lopen-pal -limf -lm -lz -lsvml -lirng -lc++ -lipgo -ldecimal -lirc 
-lclang_rt.osx -lmpi -lopen-rte -lopen-pal -limf -lm -lz -lsvml -lirng -lc++ 
-lipgo -ldecimal -lirc -lclang_rt.osx -o ex19
icc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and 
will be removed from product release in the second half of 2023. The Intel(R) 
oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. 
Please transition to use this compiler. Use '-diag-disable=10441' to disable 
this message.
In file included from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscsys.h(44),
 from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscvec.h(9),
 from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscmat.h(7),
 from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscpc.h(7),
 from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscksp.h(7),
 from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscsnes.h(7),
 from ex19.c(68):
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscsystypes.h(68): warning 
#2621: attribute "warn_unused_result" does not apply here
  PETSC_ERROR_CODE_TYPEDEF enum PETSC_ERROR_CODE_NODISCARD {
^

Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process
See https://petsc.org/release/faq/
[excess:37807] *** Process received signal ***
[excess:37807] Signal: Segmentation fault: 11 (11)
[excess:37807] Signal code: Address not mapped (1)
[excess:37807] Failing at address: 0x7f
[excess:37807] *** End 

Re: [petsc-users] Compiling PETSC with Intel OneAPI compilers and OpenMPI

[Samar Khatiwala]

Hi, for what it’s worth, clang + ifort from OneAPI 2023 update 1 works fine for 
me on both Intel and M2 Macs. So it might just be a matter of upgrading.

Samar

On May 15, 2023, at 5:53 PM, Matthew Knepley  wrote:

Send us

  $PETSC_ARCH/include/petscconf.h

  Thanks,

 Matt

On Mon, May 15, 2023 at 12:49 PM Vanella, Marcos (Fed) 
mailto:marcos.vane...@nist.gov>> wrote:
Hi Matt, I configured the lib like this:

$ ./configure --with-blaslapack-dir=/opt/intel/oneapi/mkl/2022.2.1 
--with-debugging=0 --with-shared-libraries=0 --download-make

and compiled. I still get some check segfault error. See below:

$ make PETSC_DIR=/Users/mnv/Documents/Software/petsc-3.19.1 
PETSC_ARCH=arch-darwin-c-opt check
Running check examples to verify correct installation
Using PETSC_DIR=/Users/mnv/Documents/Software/petsc-3.19.1 and 
PETSC_ARCH=arch-darwin-c-opt
***Error detected during compile or link!***
See https://petsc.org/release/faq/
/Users/mnv/Documents/Software/petsc-3.19.1/src/snes/tutorials ex19
*
mpicc -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined 
-Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first 
-Wl,-no_compact_unwind  -fPIC -wd1572 -Wno-unknown-pragmas -g -O3  
-I/Users/mnv/Documents/Software/petsc-3.19.1/include 
-I/Users/mnv/Documents/Software/petsc-3.19.1/arch-darwin-c-opt/include 
-I/opt/X11/include  -std=c99ex19.c  
-L/Users/mnv/Documents/Software/petsc-3.19.1/arch-darwin-c-opt/lib 
-Wl,-rpath,/opt/intel/oneapi/mkl/2022.2.1/lib 
-L/opt/intel/oneapi/mkl/2022.2.1/lib -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib 
-L/opt/openmpi414_oneapi22u3/lib 
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.2.1/mac/compiler/lib 
-L/opt/intel/oneapi/tbb/2021.7.1/lib -L/opt/intel/oneapi/ippcp/2021.6.2/lib 
-L/opt/intel/oneapi/ipp/2021.6.2/lib 
-L/opt/intel/oneapi/dnnl/2022.2.1/cpu_iomp/lib 
-L/opt/intel/oneapi/dal/2021.7.1/lib 
-L/opt/intel/oneapi/compiler/2022.2.1/mac/compiler/lib 
-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib 
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.2.1/mac/bin/intel64/../../compiler/lib
 -L/opt/intel/oneapi/compiler/2022.2.1/mac/bin/intel64/../../compiler/lib 
-Wl,-rpath,/Library/Developer/CommandLineTools/usr/lib/clang/14.0.3/lib/darwin 
-L/Library/Developer/CommandLineTools/usr/lib/clang/14.0.3/lib/darwin -lpetsc 
-lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -lX11 -lmpi_usempif08 
-lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -limf -lm -lz 
-lifport -lifcoremt -lsvml -lipgo -lirc -lpthread -lclang_rt.osx -lmpi 
-lopen-rte -lopen-pal -limf -lm -lz -lsvml -lirng -lc++ -lipgo -ldecimal -lirc 
-lclang_rt.osx -lmpi -lopen-rte -lopen-pal -limf -lm -lz -lsvml -lirng -lc++ 
-lipgo -ldecimal -lirc -lclang_rt.osx -o ex19
icc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and 
will be removed from product release in the second half of 2023. The Intel(R) 
oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. 
Please transition to use this compiler. Use '-diag-disable=10441' to disable 
this message.
In file included from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscsys.h(44),
 from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscvec.h(9),
 from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscmat.h(7),
 from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscpc.h(7),
 from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscksp.h(7),
 from 
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscsnes.h(7),
 from ex19.c(68):
/Users/mnv/Documents/Software/petsc-3.19.1/include/petscsystypes.h(68): warning 
#2621: attribute "warn_unused_result" does not apply here
  PETSC_ERROR_CODE_TYPEDEF enum PETSC_ERROR_CODE_NODISCARD {
^

Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process
See https://petsc.org/release/faq/
[excess:37807] *** Process received signal ***
[excess:37807] Signal: Segmentation fault: 11 (11)
[excess:37807] Signal code: Address not mapped (1)
[excess:37807] Failing at address: 0x7f
[excess:37807] *** End of error message ***
--
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--
--
mpiexec noticed that process rank 0 with PID 0 on node excess exited on signal 
11 (Segmentation fault: 11).
--
Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes
See https://petsc.org/release/faq/
[excess:37831] *** Process 

Re: [petsc-users] PETSc build asks for network connections

[Samar Khatiwala]

Hi,

I realize this is an old thread but I have some recent experience based on 
setting up an M2 Mac that might be relevant.

I was dreading moving to Apple Silicon Macs because of issues like these but I 
actually did not run into this particular problem.
While I can’t be certain I think it is because in the process of installing 
another piece of software I had to modify Apple’s security
restrictions to make them more permissive. Details of how to do this are in the 
following and it takes only a minute to implement:

https://rogueamoeba.com/support/knowledgebase/?showArticle=ACE-StepByStep=Audio+Hijack

Incidentally, I built mpich from source followed by PETSc in the usual way.

Something else that might be helpful for others is my experience getting ifort 
to work. (My needs were somewhat specific: mixed
fortran/C code, preferably ifort, and avoid package managers.) The intel OneAPI 
installer ran smoothly (via rosetta) but when
building mpich (or PETSc) I ran into an obvious problem: clang produces arm64 
object files while ifort produces x86 ones. I couldn’t
manage to set the correct CFLAGS to tell clang to target x86. Instead, the 
(simpler) solution turned out to be (1) the fact that all the
executables in Apple’s toolchain are universal binaries, and (2) the ‘arch’ 
command can let you run programs for any of the two
architectures. Specifically, executing in the terminal:

arch -x86_64 bash

starts a bash shell and *every* program that is then run from that shell is 
automatically the x86 version. So I could then do:
FC=ifort
./configure --prefix=/usr/local/mpichx86 --enable-two-level-namespace
make
sudo make install

and get an x86 build of mpich which I could then use (from the same shell or a 
new one started as above) to build [x86] PETSc.
Except for some annoying warnings from MKL (I think because it is confused what 
architecture it is running on) everything runs
smoothly and - even in emulation - surprisingly fast.

Sorry if this is all well know and already documented on PETSc’s install page.

Samar

On Mar 20, 2023, at 6:39 AM, Pierre Jolivet 
mailto:pie...@joliv.et>> wrote:


On 20 Mar 2023, at 2:45 AM, Barry Smith 
mailto:bsm...@petsc.dev>> wrote:


  I found a bit more information in gmakefile.test which has the magic sauce 
used by make test to stop the firewall popups while running the test suite.

# MACOS FIREWALL HANDLING
# - if run with MACOS_FIREWALL=1
#   (automatically set in $PETSC_ARCH/lib/petsc/conf/petscvariables if 
configured --with-macos-firewall-rules),
#   ensure mpiexec and test executable is on firewall list
#
ifeq ($(MACOS_FIREWALL),1)
FW := /usr/libexec/ApplicationFirewall/socketfilterfw
# There is no reliable realpath command in macOS without need for 3rd party 
tools like homebrew coreutils
# Using Python's realpath seems like the most robust way here
realpath-py = $(shell $(PYTHON) -c 'import os, sys; 
print(os.path.realpath(sys.argv[1]))' $(1))
#
define macos-firewall-register
  @APP=$(call realpath-py, $(1)); \
if ! sudo -n true 2>/dev/null; then printf "Asking for sudo password to add 
new firewall rule for\n  $$APP\n"; fi; \
sudo $(FW) --remove $$APP --add $$APP --blockapp $$APP
endef
endif

and below. When building each executable it automatically calls socketfilterfw 
on that executable so it won't popup.

From this I think you can reverse engineer how to turn it off for your 
executables.

Perhaps PETSc's make ex1 etc should also apply this magic sauce, Pierre?

This configure option was added in 
https://gitlab.com/petsc/petsc/-/merge_requests/3131 but it never worked on my 
machines.
I just tried again this morning a make check with MACOS_FIREWALL=1, it’s asking 
for my password to register MPICH in the firewall, but the popups are still 
appearing afterwards.
That’s why I’ve never used that configure option and why I’m not sure if I can 
trust this code from makefile.test, but I’m probably being paranoid.
Prior to Ventura, when I was running the test suite, I manually disabled the 
firewall https://support.apple.com/en-gb/guide/mac-help/mh11783/12.0/mac/12.0
Apple has done yet again Apple things, and even if you disable the firewall on 
Ventura (https://support.apple.com/en-gb/guide/mac-help/mh11783/13.0/mac/13.0), 
the popups are still appearing.
Right now, I don’t have a solution, except for not using my machine while the 
test suite runs…
I don’t recall whether this has been mentioned by any of the other devs, but 
this is a completely harmless (though frustrating) message: MPI and/or PETSc 
cannot be used without an action from the user to allow others to get access to 
your machine.

Thanks,
Pierre

On Mar 19, 2023, at 8:10 PM, Amneet Bhalla 
mailto:mail2amn...@gmail.com>> wrote:

This helped only during the configure stage, and not during the check stage and 
during executing the application built on PETSc. Do you think it is because I 
built mpich locally and not with PETSc?

On Sun, Mar 19, 2023 at 3:51 PM Barry Smith 

Re: [petsc-users] Creating multiple Vecs with petsc4py

[Samar Khatiwala]

Thanks Matt and Jed for the quick replies! That answers my immediate questions.

Best,

Samar

> On Feb 12, 2022, at 1:33 PM, Jed Brown  wrote:
> 
> VecDuplicateVecs isn't implemented in petsc4py, but it internally just loops 
> over VecDuplicate so you can use
> 
> qs = [x.duplicate() for i in range(4)]
> 
> y.maxpy(alphas, qs)
> 
> 
> where the Python binding here handles qs being a Python array.
> 
> Samar Khatiwala  writes:
> 
>> Hello,
>> 
>> I’d like to create an array of Vecs in petsc4py by calling VecDuplicateVecs 
>> but I can’t find the corresponding method (I’ve tried various iterations 
>> such as q = x.duplicateVecs(4), etc). 
>> Is this not implemented in petsc4py? One workaround I’ve come up with is 
>> something like:
>> 
>> q={}
>> for i in range(0, 3):
>>q[i]=x.duplicate()
>> 
>> Is there another/better way? And how do I then use PETSc functions that 
>> operate on Vecs (e.g., VecMAXPY)? Would I just call VecAXPY in a loop as 
>> above?
>> 
>> Ultimately, what I really want to do is wrap my own C functions with Cython 
>> that take an array of Vecs as an argument and then operate on them. (The 
>> function needs the entire array 
>> of Vecs to do its thing so I can’t loop over the elements of the array.) For 
>> instance, I want to pass the q above to myCfunc(Vec *q, Vec *qout). How do I 
>> go about doing that?
>> 
>> Thanks very much!
>> 
>> Best,
>> 
>> Samar

[petsc-users] Creating multiple Vecs with petsc4py

[Samar Khatiwala]

Hello,

I’d like to create an array of Vecs in petsc4py by calling VecDuplicateVecs but 
I can’t find the corresponding method (I’ve tried various iterations such as q 
= x.duplicateVecs(4), etc). 
Is this not implemented in petsc4py? One workaround I’ve come up with is 
something like:

q={}
for i in range(0, 3):
q[i]=x.duplicate()

Is there another/better way? And how do I then use PETSc functions that operate 
on Vecs (e.g., VecMAXPY)? Would I just call VecAXPY in a loop as above?

Ultimately, what I really want to do is wrap my own C functions with Cython 
that take an array of Vecs as an argument and then operate on them. (The 
function needs the entire array 
of Vecs to do its thing so I can’t loop over the elements of the array.) For 
instance, I want to pass the q above to myCfunc(Vec *q, Vec *qout). How do I go 
about doing that?

Thanks very much!

Best,

Samar

Re: [petsc-users] PETSc configuration error on macOS Monterey with Intel oneAPI

[Samar Khatiwala]

Hi Danyang,

Just to reiterate, the presence of -Wl,-flat_namespace *is* the problem. I got 
rid of it by configuring mpich with --enable-two-level-namespace. I reported 
this problem to the PETSc 
folks a few weeks ago and they were going to patch MPICH.py (under 
config/BuildSystem/config/packages) to pass this flag. So you could try 
configuring with —download-mpich 
(or build your own mpich, which is pretty straightforward). If you’re wedded to 
openmpi, you could patch up OpenMPI.py yourself (maybe 
--enable-two-level-namespace is called 
something else for openmpi).

Best,

Samar

> On Jan 13, 2022, at 6:01 AM, Danyang Su  wrote:
> 
> Hi Samar,
>  
> Thanks for your suggestion. Unfortunately, it does not work. I checked the 
> mpif90 wrapper and the option "-Wl,-flat_namespace” is present. 
>  
> (base) ➜  bin ./mpif90 -show
> ifort -I/Users/danyangsu/Soft/PETSc/petsc-3.16.3/macos-intel-dbg/include 
> -Wl,-flat_namespace -Wl,-commons,use_dylibs 
> -I/Users/danyangsu/Soft/PETSc/petsc-3.16.3/macos-intel-dbg/lib 
> -L/Users/danyangsu/Soft/PETSc/petsc-3.16.3/macos-intel-dbg/lib 
> -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
>  
> Thanks anyway,
>  
> Danyang
> From: Samar Khatiwala 
> Date: Wednesday, January 12, 2022 at 2:01 PM
> To: Danyang Su 
> Cc: PETSc 
> Subject: Re: [petsc-users] PETSc configuration error on macOS Monterey with 
> Intel oneAPI
>  
> Hi Danyang,
>  
> I had trouble configuring PETSc on MacOS Monterey with ifort when using mpich 
> (which I was building myself). I tracked it down to an errant 
> "-Wl,-flat_namespace” 
> option in the mpif90 wrapper. I rebuilt mpich with the 
> "--enable-two-level-namespace” configuration option and the problem went 
> away. I don’t know if there’s a similar 
> issue with openmpi but you could check the corresponding mpif90 wrapper 
> (mpif90 -show) whether "-Wl,-flat_namespace” is present or not. If so, 
> perhaps passing 
> "--enable-two-level-namespace” to PETSc configure might fix the problem 
> (although I don’t know how you would set this flag *just* for building 
> openmpi).
>  
> Samar
> 
> 
>> On Jan 12, 2022, at 9:41 PM, Danyang Su > <mailto:danyang...@gmail.com>> wrote:
>>  
>> Hi All,
>> 
>> I got an error in PETSc configuration on macOS Monterey with Intel oneAPI 
>> using the following options:
>> 
>>  
>> ./configure --with-cc=icc --with-cxx=icpc --with-fc=ifort 
>> --with-blas-lapack-dir=/opt/intel/oneapi/mkl/2022.0.0/lib/ 
>> --with-debugging=1 PETSC_ARCH=macos-intel-dbg --download-mumps 
>> --download-parmetis --download-metis --download-hypre --download-superlu 
>> --download-hdf5=yes --download-openmpi
>> 
>>  
>> Error with downloaded OpenMPI: Cannot compile/link FC with 
>> /Users/danyangsu/Soft/PETSc/petsc-3.16.3/macos-intel-dbg/bin/mpif90.
>> 
>>  
>> Any suggestions for that?
>> 
>>  
>> There is no problem if I use GNU compiler and MPICH.
>> 
>>  
>> Thanks,
>> 
>>  
>> Danyang

Re: [petsc-users] PETSc configuration error on macOS Monterey with Intel oneAPI

[Samar Khatiwala]

Hi Danyang,

I had trouble configuring PETSc on MacOS Monterey with ifort when using mpich 
(which I was building myself). I tracked it down to an errant 
"-Wl,-flat_namespace” 
option in the mpif90 wrapper. I rebuilt mpich with the 
"--enable-two-level-namespace” configuration option and the problem went away. 
I don’t know if there’s a similar 
issue with openmpi but you could check the corresponding mpif90 wrapper (mpif90 
-show) whether "-Wl,-flat_namespace” is present or not. If so, perhaps passing 
"--enable-two-level-namespace” to PETSc configure might fix the problem 
(although I don’t know how you would set this flag *just* for building openmpi).

Samar

> On Jan 12, 2022, at 9:41 PM, Danyang Su  wrote:
> 
> Hi All,
> 
> I got an error in PETSc configuration on macOS Monterey with Intel oneAPI 
> using the following options:
> 
>  
> ./configure --with-cc=icc --with-cxx=icpc --with-fc=ifort 
> --with-blas-lapack-dir=/opt/intel/oneapi/mkl/2022.0.0/lib/ --with-debugging=1 
> PETSC_ARCH=macos-intel-dbg --download-mumps --download-parmetis 
> --download-metis --download-hypre --download-superlu --download-hdf5=yes 
> --download-openmpi
> 
>  
> Error with downloaded OpenMPI: Cannot compile/link FC with 
> /Users/danyangsu/Soft/PETSc/petsc-3.16.3/macos-intel-dbg/bin/mpif90.
> 
>  
> Any suggestions for that?
> 
>  
> There is no problem if I use GNU compiler and MPICH.
> 
>  
> Thanks,
> 
>  
> Danyang

Re: [petsc-users] Installation problems on IBM machine

[Samar Khatiwala]

Hi Barry,

That solved the problem! Thanks so much for the fast and helpful reply!

Best,

Samar

On May 24, 2014, at 9:22 AM, Barry Smith bsm...@mcs.anl.gov wrote:

 
 
  Try using make all-legacy
 
   Barry
 
 That is some cmake problem; fortunately we are abandoning cmake for the 
 future. The reason the problem persists is likely because cmake has cached 
 something somewhere that doesn’t get rebuilt.
 
 
 On May 24, 2014, at 7:04 AM, Samar Khatiwala s...@ldeo.columbia.edu wrote:
 
 Hello,
 
 I'm having trouble installing PETSc on an IBM machine (power6). After some 
 trial and error I managed to configure but the 'make all' step fails with: 
 
 ==
 Building PETSc using CMake with 21 build threads
 ==
 make: 1254-002 Cannot find a rule to create target 21 from dependencies.
 Stop.
 make: 1254-004 The error code from the last command is 2.
 …
 
 Please see attached logs. I configured with:
 
 config/configure.py --with-cc=mpcc --with-cxx=mpCC --with-clanguage=c 
 --with-fc=mpxlf90 --with-debugging=0 FFLAGS=-qextname --with-batch=1 
 --known-mpi-shared-libraries=0 --with-blas-lapack-lib=[libessl.a]
 
 What is odd is that this worked once but 'make test' failed because of a 
 missing LAPACK routine in ESSL. I reconfigured with 
 --with-blas-lib=libessl.a 
 --with-lapack-lib=/sw/aix53/lapack-3.2.0/lib/liblapack.a but then 'make all' 
 failed with the same error as I now 
 get *even after* reverting to the original (above) configure options. I've 
 now tried this several times with a fresh copy of PETSc to no 
 avail.
 
 Any help would be appreciated. Thanks very much!
 
 Samar
 
 
 
 make.log.gzconfigure.log.gz
 

Re: [petsc-users] possible performance issues with PETSc on Cray

[Samar Khatiwala]

Hi Jed et al.,

Just wanted to report back on the resolution of this issue. The computing 
support people at HLRN in Germany 
submitted a test case to CRAY re. performance on their XC30. CRAY has finally 
gotten back with a solution, 
which is to use the run-time option  -vecscatter_alltoall. Apparently this is a 
known issue and according to the 
HLRN folks passing this command line option to PETSc seems to work nicely.

Thanks again for your help.

Samar

On Apr 11, 2014, at 7:44 AM, Jed Brown j...@jedbrown.org wrote:

 Samar Khatiwala s...@ldeo.columbia.edu writes:
 
 Hello,
 
 This is a somewhat vague query but I and a colleague have been running PETSc 
 (3.4.3.0) on a Cray 
 XC30 in Germany (https://www.hlrn.de/home/view/System3/WebHome) and the 
 system administrators 
 alerted us to some anomalies with our jobs that may or may not be related to 
 PETSc but I thought I'd ask 
 here in case others have noticed something similar.
 
 First, there was a large variation in run-time for identical jobs, sometimes 
 as much as 50%. We didn't 
 really pick up on this but other users complained to the IT people that 
 their jobs were taking a performance 
 hit with a similar variation in run-time. At that point we're told the IT 
 folks started monitoring jobs and 
 carrying out tests to see what was going on. They discovered that (1) this 
 always happened when we were 
 running our jobs and (2) the problem got worse with physical proximity to 
 the nodes on which our jobs were 
 running (what they described as a strong interaction between our jobs and 
 others presumably through the 
 communication network).
 
 It sounds like you are strong scaling (smallish subdomains) so that your
 application is sensitive to network latency.  I see significant
 performance variability on XC-30 with this Full Multigrid solver that is
 not using PETSc.
 
 http://59a2.org/files/hopper-vs-edison.3semilogx.png
 
 See the factor of 2 performance variability for the samples of the ~15M
 element case.  This operation is limited by instruction issue rather
 than bandwidth (indeed, it is several times faster than doing the same
 operations with assembled matrices).  Here the variability is within the
 same application performing repeated solves.  If you get a different
 partition on a different run, you can see larger variation.
 
 If your matrices are large enough, your performance will be limited by
 memory bandwidth.  (This is the typical case, but sufficiently small
 matrices can fit in cache.)  I once encountered a batch system that did
 not properly reset nodes between runs, leaving a partially-filled
 ramdisk distributed asymmetrically across the memory busses.  This led
 to 3x performance reduction on 4-socket nodes because much of the memory
 demanded by the application would be faulted onto one memory bus.
 Presumably your machine has a resource manager that would not allow such
 things to happen.

Re: [petsc-users] possible performance issues with PETSc on Cray

[Samar Khatiwala]

Hi Jed,

This is useful to know. My matrices are all very sparse but just may not be 
ordered 
optimally (there's a problem-specific reason why I order them in a certain 
way). That 
said, this is the first time in many years of similar computations with similar 
matrices 
that I've encountered this problem. It may just be peculiar to the XC30's.

Thanks,

Samar

On May 9, 2014, at 8:30 AM, Jed Brown j...@jedbrown.org wrote:

 Samar Khatiwala s...@ldeo.columbia.edu writes:
 CRAY has finally gotten back with a solution,
 which is to use the run-time option -vecscatter_alltoall. Apparently
 this is a known issue and according to the HLRN folks passing this
 command line option to PETSc seems to work nicely.
 
 This option is good when you have nearly-dense rows or columns (in terms
 of processors depended on).  For problems with actual dense rows or
 columns, it is good to formulate as a sparse matrix plus a low-rank
 correction.  The other cases are usually poor dof layout, and reordering
 will make the graph sparser.  Sparse problems with good layout usually
 run faster with the default VecScatter, though there are exceptions
 (mostly non-PDE problems).

Re: [petsc-users] possible performance issues with PETSc on Cray

[Samar Khatiwala]

Hi Jed,

Thanks for the quick reply. This is very helpful. You may well be right that my 
matrices are not large enough 
(~2. 5e6 x 2.5e6 and I'm running on 360 cores = 15 nodes x 24 cores/node on 
this XC-30) and my runs are 
therefore sensitive to network latency. Would this, though, impact other people 
running jobs on nearby nodes? 
(I suppose it would if I'm passing too many messages because of the small size 
of the matrices.)

I'm going to pass on your reply to the system administrators. They will be able 
to understand the technical content 
better than I am capable of.

Thanks again!

Best,

Samar

On Apr 11, 2014, at 7:44 AM, Jed Brown j...@jedbrown.org wrote:

 Samar Khatiwala s...@ldeo.columbia.edu writes:
 
 Hello,
 
 This is a somewhat vague query but I and a colleague have been running PETSc 
 (3.4.3.0) on a Cray 
 XC30 in Germany (https://www.hlrn.de/home/view/System3/WebHome) and the 
 system administrators 
 alerted us to some anomalies with our jobs that may or may not be related to 
 PETSc but I thought I'd ask 
 here in case others have noticed something similar.
 
 First, there was a large variation in run-time for identical jobs, sometimes 
 as much as 50%. We didn't 
 really pick up on this but other users complained to the IT people that 
 their jobs were taking a performance 
 hit with a similar variation in run-time. At that point we're told the IT 
 folks started monitoring jobs and 
 carrying out tests to see what was going on. They discovered that (1) this 
 always happened when we were 
 running our jobs and (2) the problem got worse with physical proximity to 
 the nodes on which our jobs were 
 running (what they described as a strong interaction between our jobs and 
 others presumably through the 
 communication network).
 
 It sounds like you are strong scaling (smallish subdomains) so that your
 application is sensitive to network latency.  I see significant
 performance variability on XC-30 with this Full Multigrid solver that is
 not using PETSc.
 
 http://59a2.org/files/hopper-vs-edison.3semilogx.png
 
 See the factor of 2 performance variability for the samples of the ~15M
 element case.  This operation is limited by instruction issue rather
 than bandwidth (indeed, it is several times faster than doing the same
 operations with assembled matrices).  Here the variability is within the
 same application performing repeated solves.  If you get a different
 partition on a different run, you can see larger variation.
 
 If your matrices are large enough, your performance will be limited by
 memory bandwidth.  (This is the typical case, but sufficiently small
 matrices can fit in cache.)  I once encountered a batch system that did
 not properly reset nodes between runs, leaving a partially-filled
 ramdisk distributed asymmetrically across the memory busses.  This led
 to 3x performance reduction on 4-socket nodes because much of the memory
 demanded by the application would be faulted onto one memory bus.
 Presumably your machine has a resource manager that would not allow such
 things to happen.

Re: [petsc-users] possible performance issues with PETSc on Cray

[Samar Khatiwala]

Thanks again Jed. This has definitely helped narrow down the possibilities.

Best,

Samar

On Apr 11, 2014, at 8:41 AM, Jed Brown j...@jedbrown.org wrote:

 Samar Khatiwala s...@ldeo.columbia.edu writes:
 
 Hi Jed,
 
 Thanks for the quick reply. This is very helpful. You may well be right that 
 my matrices are not large enough 
 (~2. 5e6 x 2.5e6 and I'm running on 360 cores = 15 nodes x 24 cores/node on 
 this XC-30) and my runs are 
 therefore sensitive to network latency. Would this, though, impact other 
 people running jobs on nearby nodes? 
 (I suppose it would if I'm passing too many messages because of the small 
 size of the matrices.)
 
 It depends on your partition.  The Aries network on XC-30 is a
 high-radix low-diameter network.  There should be many routes between
 nodes, but the routing algorithm likely does not know which wires to
 avoid.  This leads to performance variation, though I think it should
 tend to be less extreme than when you obtain disconnected partitions on
 Gemini.
 
 The gold standard of reproducible performance is Blue Gene, where the
 network is reconfigured to give you an isolated 5D torus.  A Blue Gene
 may or may not be available or cost effective (reproducible performance
 does not imply high performance/efficiency for a given workload).

Re: [petsc-users] Error using MUMPS to solve large linear system

[Samar Khatiwala]

Hi Sherry,

Thanks for the offer to help!

I tried superlu_dist again and it crashes even more quickly than MUMPS with 
just the following error:

ERROR: 0031-250  task 128: Killed

Absolutely nothing else is written out to either stderr or stdout. This is with 
-mat_superlu_dist_statprint. 
The program works fine on a smaller matrix.

This is the sequence of calls:

KSPSetType(ksp,KSPPREONLY);
PCSetType(pc,PCLU);
PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU_DIST);
KSPSetFromOptions(ksp);
PCSetFromOptions(pc);
KSPSolve(ksp,b,x);

All of these successfully return *except* the very last one to KSPSolve.

Any help would be appreciated. Thanks!

Samar

On Feb 24, 2014, at 3:58 PM, Xiaoye S. Li x...@lbl.gov wrote:

 Samar:
 If you include the error message while crashing using superlu_dist, I 
 probably know the reason.  (better yet, include the printout before the 
 crash. )
 
 Sherry
 
 
 On Mon, Feb 24, 2014 at 9:56 AM, Hong Zhang hzh...@mcs.anl.gov wrote:
 Samar :
 There are limitations for direct solvers. 
 Do not expect any solver can be used on arbitrarily large problems.
 Since superlu_dist also crashes, direct solvers may not be able to work on 
 your application. 
 This is why I suggest to increase size incrementally. 
 You may have to experiment other type of solvers.
 
 Hong
 
 Hi Hong and Jed,
 
 Many thanks for replying. It would indeed be nice if the error messages from 
 MUMPS were less cryptic!
 
 1) I have tried smaller matrices although given how my problem is set up a 
 jump is difficult to avoid. But a good idea 
 that I will try.
 
 2) I did try various ordering but not the one you suggested.
 
 3) Tracing the error through the MUMPS code suggest a rather abrupt 
 termination of the program (there should be more 
 error messages if, for example, memory was a problem). I therefore thought it 
 might be an interface problem rather than 
 one with mumps and turned to the petsc-users group first. 
 
 4) I've tried superlu_dist but it also crashes (also unclear as to why) at 
 which point I decided to try mumps. The fact that both 
 crash would again indicate a common (memory?) problem.
 
 I'll try a few more things before asking the MUMPS developers.
 
 Thanks again for your help!
 
 Samar
 
 On Feb 24, 2014, at 11:47 AM, Hong Zhang hzh...@mcs.anl.gov wrote:
 
 Samar:
 The crash occurs in 
 ...
 [161]PETSC ERROR: Error in external library!
 [161]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: 
 INFO(1)=-1, INFO(2)=48
  
 for very large matrix, likely memory problem as you suspected. 
 I would suggest 
 1. run problems with increased sizes (not jump from a small one to a very 
 large one) and observe memory usage using
 '-ksp_view'.
I see you use '-mat_mumps_icntl_14 1000', i.e., percentage of estimated 
 workspace increase. Is it too large?
Anyway, this input should not cause the crash, I guess.
 2. experimenting with different matrix ordering -mat_mumps_icntl_7  (I 
 usually use sequential ordering 2) 
 I see you use parallel ordering -mat_mumps_icntl_29 2.
 3. send bug report to mumps developers for their suggestion.
 
 4. try other direct solvers, e.g., superlu_dist.
  
 …
 
 etc etc. The above error I can tell has something to do with processor 48 
 (INFO(2)) and so forth but not the previous one.
 
 The full output enabled with -mat_mumps_icntl_4 3 looks as in the attached 
 file. Any hints as to what could be giving this
 error would be very much appreciated.
  
 I do not know how to interpret this  output file. mumps developer would give 
 you better suggestion on it.
 I would appreciate to learn as well :-)
 
 Hong
 
 
 

Re: [petsc-users] Error using MUMPS to solve large linear system

[Samar Khatiwala]

Hi Barry,

You're probably right. I note that the error occurs almost instantly and I've 
tried increasing the number of CPUs 
(as many as ~1000 on Yellowstone) to no avail. I know this is a big problem but 
I didn't think it was that big!

Sherry: Is there any way to write out more diagnostic info? E.g.,how much 
memory superlu thinks it needs/is attempting 
to allocate.

Thanks,

Samar

On Feb 25, 2014, at 10:57 AM, Barry Smith bsm...@mcs.anl.gov wrote:
 
 
 I tried superlu_dist again and it crashes even more quickly than MUMPS with 
 just the following error:
 
 ERROR: 0031-250  task 128: Killed
 
   This is usually a symptom of running out of memory.
 
 
 Absolutely nothing else is written out to either stderr or stdout. This is 
 with -mat_superlu_dist_statprint. 
 The program works fine on a smaller matrix.
 
 This is the sequence of calls:
 
 KSPSetType(ksp,KSPPREONLY);
 PCSetType(pc,PCLU);
 PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU_DIST);
 KSPSetFromOptions(ksp);
 PCSetFromOptions(pc);
 KSPSolve(ksp,b,x);
 
 All of these successfully return *except* the very last one to KSPSolve.
 
 Any help would be appreciated. Thanks!
 
 Samar
 
 On Feb 24, 2014, at 3:58 PM, Xiaoye S. Li x...@lbl.gov wrote:
 
 Samar:
 If you include the error message while crashing using superlu_dist, I 
 probably know the reason.  (better yet, include the printout before the 
 crash. )
 
 Sherry
 
 
 On Mon, Feb 24, 2014 at 9:56 AM, Hong Zhang hzh...@mcs.anl.gov wrote:
 Samar :
 There are limitations for direct solvers. 
 Do not expect any solver can be used on arbitrarily large problems.
 Since superlu_dist also crashes, direct solvers may not be able to work on 
 your application. 
 This is why I suggest to increase size incrementally. 
 You may have to experiment other type of solvers.
 
 Hong
 
 Hi Hong and Jed,
 
 Many thanks for replying. It would indeed be nice if the error messages 
 from MUMPS were less cryptic!
 
 1) I have tried smaller matrices although given how my problem is set up a 
 jump is difficult to avoid. But a good idea 
 that I will try.
 
 2) I did try various ordering but not the one you suggested.
 
 3) Tracing the error through the MUMPS code suggest a rather abrupt 
 termination of the program (there should be more 
 error messages if, for example, memory was a problem). I therefore thought 
 it might be an interface problem rather than 
 one with mumps and turned to the petsc-users group first. 
 
 4) I've tried superlu_dist but it also crashes (also unclear as to why) at 
 which point I decided to try mumps. The fact that both 
 crash would again indicate a common (memory?) problem.
 
 I'll try a few more things before asking the MUMPS developers.
 
 Thanks again for your help!
 
 Samar
 
 On Feb 24, 2014, at 11:47 AM, Hong Zhang hzh...@mcs.anl.gov wrote:
 
 Samar:
 The crash occurs in 
 ...
 [161]PETSC ERROR: Error in external library!
 [161]PETSC ERROR: Error reported by MUMPS in numerical factorization 
 phase: INFO(1)=-1, INFO(2)=48
 
 for very large matrix, likely memory problem as you suspected. 
 I would suggest 
 1. run problems with increased sizes (not jump from a small one to a very 
 large one) and observe memory usage using
 '-ksp_view'.
   I see you use '-mat_mumps_icntl_14 1000', i.e., percentage of estimated 
 workspace increase. Is it too large?
   Anyway, this input should not cause the crash, I guess.
 2. experimenting with different matrix ordering -mat_mumps_icntl_7  (I 
 usually use sequential ordering 2) 
I see you use parallel ordering -mat_mumps_icntl_29 2.
 3. send bug report to mumps developers for their suggestion.
 
 4. try other direct solvers, e.g., superlu_dist.
 
 …
 
 etc etc. The above error I can tell has something to do with processor 48 
 (INFO(2)) and so forth but not the previous one.
 
 The full output enabled with -mat_mumps_icntl_4 3 looks as in the attached 
 file. Any hints as to what could be giving this
 error would be very much appreciated.
 
 I do not know how to interpret this  output file. mumps developer would 
 give you better suggestion on it.
 I would appreciate to learn as well :-)
 
 Hong
 
 
 
 
 

Re: [petsc-users] Error using MUMPS to solve large linear system

[Samar Khatiwala]

Hi Sherry,

Thanks! I tried your suggestions and it worked!

For the record I added these flags: -mat_superlu_dist_colperm PARMETIS 
-mat_superlu_dist_parsymbfact 1 

Also, for completeness and since you asked:

size: 2346346 x 2346346
nnz:  60856894
unsymmetric

The hardware (http://www2.cisl.ucar.edu/resources/yellowstone/hardware) specs 
are: 2 GB/core, 32 GB/node (27 GB usable), (16 cores per node)
I've been running on 8 nodes (so 8 x 27 ~ 216 GB).

Thanks again for your help!

Samar

On Feb 25, 2014, at 1:00 PM, Xiaoye S. Li x...@lbl.gov wrote:

 I didn't follow the discussion thread closely ... How large is your matrix 
 dimension, and number of nonzeros?
 How large is the memory per core (or per node)?  
 
 The default setting in superlu_dist is to use serial symbolic factorization. 
 You can turn on parallel symbolic factorization by:
 
 options.ParSymbFact = YES;
 options.ColPerm = PARMETIS;
 
 Is your matrix symmetric?  if so, you need to give both upper and lower half 
 of matrix A to superlu, which doesn't exploit symmetry.
 
 Do you know whether you need numerical pivoting?  If not, you can turn off 
 pivoting by:
 
 options.RowPerm = NATURAL;
 
 This avoids some other serial bottleneck.
 
 All these options can be turned on in the petsc interface. Please check out 
 the syntax there.
 
 
 Sherry
 
 
 
 On Tue, Feb 25, 2014 at 8:07 AM, Samar Khatiwala s...@ldeo.columbia.edu 
 wrote:
 Hi Barry,
 
 You're probably right. I note that the error occurs almost instantly and I've 
 tried increasing the number of CPUs
 (as many as ~1000 on Yellowstone) to no avail. I know this is a big problem 
 but I didn't think it was that big!
 
 Sherry: Is there any way to write out more diagnostic info? E.g.,how much 
 memory superlu thinks it needs/is attempting
 to allocate.
 
 Thanks,
 
 Samar
 
 On Feb 25, 2014, at 10:57 AM, Barry Smith bsm...@mcs.anl.gov wrote:
 
 
  I tried superlu_dist again and it crashes even more quickly than MUMPS 
  with just the following error:
 
  ERROR: 0031-250  task 128: Killed
 
This is usually a symptom of running out of memory.
 
 
  Absolutely nothing else is written out to either stderr or stdout. This is 
  with -mat_superlu_dist_statprint.
  The program works fine on a smaller matrix.
 
  This is the sequence of calls:
 
  KSPSetType(ksp,KSPPREONLY);
  PCSetType(pc,PCLU);
  PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU_DIST);
  KSPSetFromOptions(ksp);
  PCSetFromOptions(pc);
  KSPSolve(ksp,b,x);
 
  All of these successfully return *except* the very last one to KSPSolve.
 
  Any help would be appreciated. Thanks!
 
  Samar
 
  On Feb 24, 2014, at 3:58 PM, Xiaoye S. Li x...@lbl.gov wrote:
 
  Samar:
  If you include the error message while crashing using superlu_dist, I 
  probably know the reason.  (better yet, include the printout before the 
  crash. )
 
  Sherry
 
 
  On Mon, Feb 24, 2014 at 9:56 AM, Hong Zhang hzh...@mcs.anl.gov wrote:
  Samar :
  There are limitations for direct solvers.
  Do not expect any solver can be used on arbitrarily large problems.
  Since superlu_dist also crashes, direct solvers may not be able to work 
  on your application.
  This is why I suggest to increase size incrementally.
  You may have to experiment other type of solvers.
 
  Hong
 
  Hi Hong and Jed,
 
  Many thanks for replying. It would indeed be nice if the error messages 
  from MUMPS were less cryptic!
 
  1) I have tried smaller matrices although given how my problem is set up 
  a jump is difficult to avoid. But a good idea
  that I will try.
 
  2) I did try various ordering but not the one you suggested.
 
  3) Tracing the error through the MUMPS code suggest a rather abrupt 
  termination of the program (there should be more
  error messages if, for example, memory was a problem). I therefore 
  thought it might be an interface problem rather than
  one with mumps and turned to the petsc-users group first.
 
  4) I've tried superlu_dist but it also crashes (also unclear as to why) 
  at which point I decided to try mumps. The fact that both
  crash would again indicate a common (memory?) problem.
 
  I'll try a few more things before asking the MUMPS developers.
 
  Thanks again for your help!
 
  Samar
 
  On Feb 24, 2014, at 11:47 AM, Hong Zhang hzh...@mcs.anl.gov wrote:
 
  Samar:
  The crash occurs in
  ...
  [161]PETSC ERROR: Error in external library!
  [161]PETSC ERROR: Error reported by MUMPS in numerical factorization 
  phase: INFO(1)=-1, INFO(2)=48
 
  for very large matrix, likely memory problem as you suspected.
  I would suggest
  1. run problems with increased sizes (not jump from a small one to a 
  very large one) and observe memory usage using
  '-ksp_view'.
I see you use '-mat_mumps_icntl_14 1000', i.e., percentage of 
  estimated workspace increase. Is it too large?
Anyway, this input should not cause the crash, I guess.
  2. experimenting with different matrix ordering -mat_mumps_icntl_7  (I 
  usually use sequential ordering 2

[petsc-users] Error using MUMPS to solve large linear system

[Samar Khatiwala]

Hello,

I'm trying to solve a linear system with MUMPS and keep getting an error that 
ends with:
 On return from DMUMPS, INFOG(1)=-100
 On return from DMUMPS, INFOG(2)=  -32766

I've looked at the MUMPS documentation but can't figure out what that means. 
This is a large (2346346 x 2346346) sparse 
matrix (read from file) and the code works fine on a (much) smaller one leading 
me to think this is related to memory and 
this problem is simply too big to solve with a sparse direct solver. Throwing 
more CPUs at the problem doesn't solve the 
problem or change the above error.

This is with PETSc 3.4.3 on Yellowstone. The standard error looks like this:

...
[161]PETSC ERROR: - Error Message 

[161]PETSC ERROR: Error in external library!
[161]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: 
INFO(1)=-1, INFO(2)=48
!
[161]PETSC ERROR: 

[161]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013 
[161]PETSC ERROR: See docs/changes/index.html for recent updates.
[161]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[161]PETSC ERROR: See docs/index.html for manual pages.
[161]PETSC ERROR: 

[161]PETSC ERROR: ./linearsolve on a arch-linux2-c-opt named ys0805 by spk Mon 
Feb 24 08:20:15 2014
[161]PETSC ERROR: Libraries linked from 
/glade/u/home/spk/petsc-3.4.3/arch-linux2-c-opt/lib
[161]PETSC ERROR: Configure run at Sun Feb 16 05:17:20 2014
[161]PETSC ERROR: Configure options 
--with-mpi-dir=/ncar/opt/intel/12.1.0.233/impi/4.0.3.008/intel64 
--with-debugging=0 --with-shared-libraries=0 --download-superlu=1 
--download-superlu_dist=1 --download-blacs=1 --download-scalapack=1 
--download-mumps=1 --download-parmetis=1 --download-metis 
--with-blas-lapack-dir=/ncar/opt/intel/12.1.0.233/composer_xe_2013.1.117/mkl 
FFLAGS=-convert big_endian -assume byterecl
[161]PETSC ERROR: 

[161]PETSC ERROR: MatFactorNumeric_MUMPS() line 722 in 
/glade/u/home/spk/petsc-3.4.3/src/mat/impls/aij/mpi/mumps/mumps.c
[161]PETSC ERROR: MatLUFactorNumeric() line 2889 in 
/glade/u/home/spk/petsc-3.4.3/src/mat/interface/matrix.c
[161]PETSC ERROR: [168]PETSC ERROR: KSPSetUp() line 278 in 
/glade/u/home/spk/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[168]PETSC ERROR: KSPSolve() line 399 in 
/glade/u/home/spk/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[168]PETSC ERROR: main() line 123 in linearsolve.c
Abort(76) on node 168 (rank 168 in comm 1140850688): application called 
MPI_Abort(MPI_COMM_WORLD, 76) - process 168
PCSetUp_LU() line 152 in 
/glade/u/home/spk/petsc-3.4.3/src/ksp/pc/impls/factor/lu/lu.c
[161]PETSC ERROR: PCSetUp() line 890 in 
/glade/u/home/spk/petsc-3.4.3/src/ksp/pc/interface/precon.c
[161]PETSC ERROR: KSPSetUp() line 278 in 
/glade/u/home/spk/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[161]PETSC ERROR: KSPSolve() line 399 in 
/glade/u/home/spk/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c
[161]PETSC ERROR: main() line 123 in linearsolve.c
…

etc etc. The above error I can tell has something to do with processor 48 
(INFO(2)) and so forth but not the previous one.

The full output enabled with -mat_mumps_icntl_4 3 looks as in the attached 
file. Any hints as to what could be giving this 
error would be very much appreciated.

Thanks very much!

Samar



log.gz
Description: GNU Zip compressed data

Re: [petsc-users] Error using MUMPS to solve large linear system

[Samar Khatiwala]

Hi Hong and Jed,

Many thanks for replying. It would indeed be nice if the error messages from 
MUMPS were less cryptic!

1) I have tried smaller matrices although given how my problem is set up a jump 
is difficult to avoid. But a good idea 
that I will try.

2) I did try various ordering but not the one you suggested.

3) Tracing the error through the MUMPS code suggest a rather abrupt termination 
of the program (there should be more 
error messages if, for example, memory was a problem). I therefore thought it 
might be an interface problem rather than 
one with mumps and turned to the petsc-users group first. 

4) I've tried superlu_dist but it also crashes (also unclear as to why) at 
which point I decided to try mumps. The fact that both 
crash would again indicate a common (memory?) problem.

I'll try a few more things before asking the MUMPS developers.

Thanks again for your help!

Samar

On Feb 24, 2014, at 11:47 AM, Hong Zhang hzh...@mcs.anl.gov wrote:

 Samar:
 The crash occurs in 
 ...
 [161]PETSC ERROR: Error in external library!
 [161]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: 
 INFO(1)=-1, INFO(2)=48
  
 for very large matrix, likely memory problem as you suspected. 
 I would suggest 
 1. run problems with increased sizes (not jump from a small one to a very 
 large one) and observe memory usage using
 '-ksp_view'.
I see you use '-mat_mumps_icntl_14 1000', i.e., percentage of estimated 
 workspace increase. Is it too large?
Anyway, this input should not cause the crash, I guess.
 2. experimenting with different matrix ordering -mat_mumps_icntl_7  (I 
 usually use sequential ordering 2) 
 I see you use parallel ordering -mat_mumps_icntl_29 2.
 3. send bug report to mumps developers for their suggestion.
 
 4. try other direct solvers, e.g., superlu_dist.
  
 …
 
 etc etc. The above error I can tell has something to do with processor 48 
 (INFO(2)) and so forth but not the previous one.
 
 The full output enabled with -mat_mumps_icntl_4 3 looks as in the attached 
 file. Any hints as to what could be giving this
 error would be very much appreciated.
  
 I do not know how to interpret this  output file. mumps developer would give 
 you better suggestion on it.
 I would appreciate to learn as well :-)
 
 Hong