Re: [petsc-users] MPI barrier issue using MatZeroRows

[Amneet Bhalla]

I added that option but the code still gets stuck at the same call
MatZeroRows with 3 processors.

On Tue, Nov 28, 2023 at 7:23 PM Amneet Bhalla  wrote:

>
>
> On Tue, Nov 28, 2023 at 6:42 PM Barry Smith  wrote:
>
>>
>>   for (int comp = 0; comp < 2; ++comp)
>> {
>> ...
>> for (Box::Iterator bc(bc_coef_box); bc; bc++)
>> {
>>..
>> if (IBTK::abs_equal_eps(b, 0.0))
>> {
>> const double diag_value = a;
>> ierr = MatZeroRows(mat, 1, _dof_index,
>> diag_value, NULL, NULL);
>> IBTK_CHKERRQ(ierr);
>> }
>> }
>> }
>>
>> In general, this code will not work because each process calls
>> MatZeroRows a different number of times, so it cannot match up with all the
>> processes.
>>
>> If u_dof_index is always local to the current process, you can call
>> MatSetOption(mat, MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE) above the for loop
>> and
>> the MatZeroRows will not synchronize across the MPI processes (since it
>> does not need to and you told it that).
>>
>
> Yes, u_dof_index is going to be local and I put a check on it a few lines
> before calling MatZeroRows.
>
> Can MatSetOption() be called after the matrix has been assembled?
>
>
>> If the u_dof_index will not always be local, then you need, on each
>> process, to list all the u_dof_index for each process in an array and then
>> call MatZeroRows()
>> once after the loop so it can exchange the needed information with the
>> other MPI processes to get the row indices to the right place.
>>
>> Barry
>>
>>
>>
>>
>> On Nov 28, 2023, at 6:44 PM, Amneet Bhalla  wrote:
>>
>>
>> Hi Folks,
>>
>> I am using MatZeroRows() to set Dirichlet boundary conditions. This works
>> fine for the serial run and the solver produces correct results (verified
>> through analytical solution). However, when I run the case in parallel, the
>> simulation gets stuck at MatZeroRows(). My understanding is that this
>> function needs to be called after the MatAssemblyBegin{End}() has been
>> called, and should be called by all processors. Here is that bit of the
>> code which calls MatZeroRows() after the matrix has been assembled
>>
>>
>> https://github.com/IBAMR/IBAMR/blob/amneetb/acoustically-driven-flows/src/acoustic_streaming/AcousticStreamingPETScMatUtilities.cpp#L724-L801
>>
>> I ran the parallel code (on 3 processors) in the debugger
>> (-start_in_debugger). Below is the call stack from the processor that gets
>> stuck
>>
>> amneetb@APSB-MBP-16:~$ lldb  -p 4307
>> (lldb) process attach --pid 4307
>> Process 4307 stopped
>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>> SIGSTOP
>> frame #0: 0x00018a2d750c libsystem_kernel.dylib`__semwait_signal
>> + 8
>> libsystem_kernel.dylib`:
>> ->  0x18a2d750c <+8>:  b.lo   0x18a2d752c   ; <+40>
>> 0x18a2d7510 <+12>: pacibsp
>> 0x18a2d7514 <+16>: stpx29, x30, [sp, #-0x10]!
>> 0x18a2d7518 <+20>: movx29, sp
>> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
>> Executable module set to
>> "/Users/amneetb/Softwares/IBAMR-Git/objs-dbg/tests/IBTK/fo_acoustic_streaming_solver_2d".
>> Architecture set to: arm64-apple-macosx-.
>> (lldb) cont
>> Process 4307 resuming
>> Process 4307 stopped
>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>> SIGSTOP
>> frame #0: 0x000109d281b8
>> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 400
>> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather:
>> ->  0x109d281b8 <+400>: ldrw9, [x24]
>> 0x109d281bc <+404>: cmpw8, w9
>> 0x109d281c0 <+408>: b.lt   0x109d281a0   ; <+376>
>> 0x109d281c4 <+412>: bl 0x109d28e64   ;
>> MPID_Progress_test
>> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
>> (lldb) bt
>> * thread #1, queue = 'com.apple.main-thread', stop reason = signal
>> SIGSTOP
>>   * frame #0: 0x000109d281b8
>> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 400
>> frame #1: 0x000109d27d14 libpmpi.12.dylib`MPIDI_SHM_mpi_barrier
>> + 224
>> frame #2: 0x000109d27b60
>> libpmpi.12.dylib`MPIDI_Barrier_intra_composition_alpha + 44
>> frame #3: 0x000109d0d490 libpmpi.12.dylib`MPIR_Barrier + 900
>> frame #4: 0x00010224d030 libmpi.12.dylib`MPI_Barrier + 684
>> frame #5: 0x0001045ea638
>> libpetsc.3.17.dylib`PetscCommDuplicate(comm_in=-2080374782,
>> comm_out=0x00010300bcb0, first_tag=0x00010300bce4) at tagm.c:235:
>> 5
>> frame #6: 0x0001045f2910
>> libpetsc.3.17.dylib`PetscHeaderCreate_Private(h=0x00010300bc70,
>> classid=1211227, class_name="PetscSF", descr="Star Forest",
>> mansec="PetscSF", comm=-2080374782,
>> destroy=(libpetsc.3.17.dylib`PetscSFDestroy at sf.c:224),
>> view=(libpetsc.3.17.dylib`PetscSFView at 

Re: [petsc-users] MPI barrier issue using MatZeroRows

[Amneet Bhalla]

On Tue, Nov 28, 2023 at 6:42 PM Barry Smith  wrote:

>
>   for (int comp = 0; comp < 2; ++comp)
> {
> ...
> for (Box::Iterator bc(bc_coef_box); bc; bc++)
> {
>..
> if (IBTK::abs_equal_eps(b, 0.0))
> {
> const double diag_value = a;
> ierr = MatZeroRows(mat, 1, _dof_index,
> diag_value, NULL, NULL);
> IBTK_CHKERRQ(ierr);
> }
> }
> }
>
> In general, this code will not work because each process calls MatZeroRows
> a different number of times, so it cannot match up with all the processes.
>
> If u_dof_index is always local to the current process, you can call
> MatSetOption(mat, MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE) above the for loop
> and
> the MatZeroRows will not synchronize across the MPI processes (since it
> does not need to and you told it that).
>

Yes, u_dof_index is going to be local and I put a check on it a few lines
before calling MatZeroRows.

Can MatSetOption() be called after the matrix has been assembled?


> If the u_dof_index will not always be local, then you need, on each
> process, to list all the u_dof_index for each process in an array and then
> call MatZeroRows()
> once after the loop so it can exchange the needed information with the
> other MPI processes to get the row indices to the right place.
>
> Barry
>
>
>
>
> On Nov 28, 2023, at 6:44 PM, Amneet Bhalla  wrote:
>
>
> Hi Folks,
>
> I am using MatZeroRows() to set Dirichlet boundary conditions. This works
> fine for the serial run and the solver produces correct results (verified
> through analytical solution). However, when I run the case in parallel, the
> simulation gets stuck at MatZeroRows(). My understanding is that this
> function needs to be called after the MatAssemblyBegin{End}() has been
> called, and should be called by all processors. Here is that bit of the
> code which calls MatZeroRows() after the matrix has been assembled
>
>
> https://github.com/IBAMR/IBAMR/blob/amneetb/acoustically-driven-flows/src/acoustic_streaming/AcousticStreamingPETScMatUtilities.cpp#L724-L801
>
> I ran the parallel code (on 3 processors) in the debugger
> (-start_in_debugger). Below is the call stack from the processor that gets
> stuck
>
> amneetb@APSB-MBP-16:~$ lldb  -p 4307
> (lldb) process attach --pid 4307
> Process 4307 stopped
> * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
> frame #0: 0x00018a2d750c libsystem_kernel.dylib`__semwait_signal
> + 8
> libsystem_kernel.dylib`:
> ->  0x18a2d750c <+8>:  b.lo   0x18a2d752c   ; <+40>
> 0x18a2d7510 <+12>: pacibsp
> 0x18a2d7514 <+16>: stpx29, x30, [sp, #-0x10]!
> 0x18a2d7518 <+20>: movx29, sp
> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
> Executable module set to
> "/Users/amneetb/Softwares/IBAMR-Git/objs-dbg/tests/IBTK/fo_acoustic_streaming_solver_2d".
> Architecture set to: arm64-apple-macosx-.
> (lldb) cont
> Process 4307 resuming
> Process 4307 stopped
> * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
> frame #0: 0x000109d281b8
> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 400
> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather:
> ->  0x109d281b8 <+400>: ldrw9, [x24]
> 0x109d281bc <+404>: cmpw8, w9
> 0x109d281c0 <+408>: b.lt   0x109d281a0   ; <+376>
> 0x109d281c4 <+412>: bl 0x109d28e64   ;
> MPID_Progress_test
> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
> (lldb) bt
> * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
>   * frame #0: 0x000109d281b8
> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 400
> frame #1: 0x000109d27d14 libpmpi.12.dylib`MPIDI_SHM_mpi_barrier +
> 224
> frame #2: 0x000109d27b60
> libpmpi.12.dylib`MPIDI_Barrier_intra_composition_alpha + 44
> frame #3: 0x000109d0d490 libpmpi.12.dylib`MPIR_Barrier + 900
> frame #4: 0x00010224d030 libmpi.12.dylib`MPI_Barrier + 684
> frame #5: 0x0001045ea638
> libpetsc.3.17.dylib`PetscCommDuplicate(comm_in=-2080374782,
> comm_out=0x00010300bcb0, first_tag=0x00010300bce4) at tagm.c:235:5
> frame #6: 0x0001045f2910
> libpetsc.3.17.dylib`PetscHeaderCreate_Private(h=0x00010300bc70,
> classid=1211227, class_name="PetscSF", descr="Star Forest",
> mansec="PetscSF", comm=-2080374782,
> destroy=(libpetsc.3.17.dylib`PetscSFDestroy at sf.c:224),
> view=(libpetsc.3.17.dylib`PetscSFView at sf.c:841)) at inherit.c:62:3
> frame #7: 0x0001049cf820
> libpetsc.3.17.dylib`PetscSFCreate(comm=-2080374782, sf=0x00016f911a50)
> at sf.c:62:3
> frame #8: 0x000104cd3024
> libpetsc.3.17.dylib`MatZeroRowsMapLocal_Private(A=0x0001170c1270, N=1,
> 

Re: [petsc-users] MPI barrier issue using MatZeroRows

[Barry Smith]

  for (int comp = 0; comp < 2; ++comp)
{
...
for (Box::Iterator bc(bc_coef_box); bc; bc++)
{
   ..
if (IBTK::abs_equal_eps(b, 0.0))
{
const double diag_value = a;
ierr = MatZeroRows(mat, 1, _dof_index, 
diag_value, NULL, NULL);
IBTK_CHKERRQ(ierr);
}
}
}

In general, this code will not work because each process calls MatZeroRows a 
different number of times, so it cannot match up with all the processes.

If u_dof_index is always local to the current process, you can call 
MatSetOption(mat, MAT_NO_OFF_PROC_ENTRIES,PETSC_TRUE) above the for loop and 
the MatZeroRows will not synchronize across the MPI processes (since it does 
not need to and you told it that).

If the u_dof_index will not always be local, then you need, on each process, to 
list all the u_dof_index for each process in an array and then call 
MatZeroRows()
once after the loop so it can exchange the needed information with the other 
MPI processes to get the row indices to the right place.

Barry




> On Nov 28, 2023, at 6:44 PM, Amneet Bhalla  wrote:
> 
> 
> Hi Folks, 
> 
> I am using MatZeroRows() to set Dirichlet boundary conditions. This works 
> fine for the serial run and the solver produces correct results (verified 
> through analytical solution). However, when I run the case in parallel, the 
> simulation gets stuck at MatZeroRows(). My understanding is that this 
> function needs to be called after the MatAssemblyBegin{End}() has been 
> called, and should be called by all processors. Here is that bit of the code 
> which calls MatZeroRows() after the matrix has been assembled
> 
> https://github.com/IBAMR/IBAMR/blob/amneetb/acoustically-driven-flows/src/acoustic_streaming/AcousticStreamingPETScMatUtilities.cpp#L724-L801
> 
> I ran the parallel code (on 3 processors) in the debugger 
> (-start_in_debugger). Below is the call stack from the processor that gets 
> stuck
> 
> amneetb@APSB-MBP-16:~$ lldb  -p 4307 
> (lldb) process attach --pid 4307
> Process 4307 stopped
> * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
> frame #0: 0x00018a2d750c libsystem_kernel.dylib`__semwait_signal + 8
> libsystem_kernel.dylib`:
> ->  0x18a2d750c <+8>:  b.lo   0x18a2d752c   ; <+40>
> 0x18a2d7510 <+12>: pacibsp 
> 0x18a2d7514 <+16>: stpx29, x30, [sp, #-0x10]!
> 0x18a2d7518 <+20>: movx29, sp
> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
> Executable module set to 
> "/Users/amneetb/Softwares/IBAMR-Git/objs-dbg/tests/IBTK/fo_acoustic_streaming_solver_2d".
> Architecture set to: arm64-apple-macosx-.
> (lldb) cont
> Process 4307 resuming
> Process 4307 stopped
> * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
> frame #0: 0x000109d281b8 
> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 400
> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather:
> ->  0x109d281b8 <+400>: ldrw9, [x24]
> 0x109d281bc <+404>: cmpw8, w9
> 0x109d281c0 <+408>: b.lt    0x109d281a0   ; 
> <+376>
> 0x109d281c4 <+412>: bl 0x109d28e64   ; MPID_Progress_test
> Target 0: (fo_acoustic_streaming_solver_2d) stopped.
> (lldb) bt
> * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
>   * frame #0: 0x000109d281b8 
> libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 400
> frame #1: 0x000109d27d14 libpmpi.12.dylib`MPIDI_SHM_mpi_barrier + 224
> frame #2: 0x000109d27b60 
> libpmpi.12.dylib`MPIDI_Barrier_intra_composition_alpha + 44
> frame #3: 0x000109d0d490 libpmpi.12.dylib`MPIR_Barrier + 900
> frame #4: 0x00010224d030 libmpi.12.dylib`MPI_Barrier + 684
> frame #5: 0x0001045ea638 
> libpetsc.3.17.dylib`PetscCommDuplicate(comm_in=-2080374782, 
> comm_out=0x00010300bcb0, first_tag=0x00010300bce4) at tagm.c:235:5
> frame #6: 0x0001045f2910 
> libpetsc.3.17.dylib`PetscHeaderCreate_Private(h=0x00010300bc70, 
> classid=1211227, class_name="PetscSF", descr="Star Forest", mansec="PetscSF", 
> comm=-2080374782, destroy=(libpetsc.3.17.dylib`PetscSFDestroy at sf.c:224), 
> view=(libpetsc.3.17.dylib`PetscSFView at sf.c:841)) at inherit.c:62:3
> frame #7: 0x0001049cf820 
> libpetsc.3.17.dylib`PetscSFCreate(comm=-2080374782, sf=0x00016f911a50) at 
> sf.c:62:3
> frame #8: 0x000104cd3024 
> libpetsc.3.17.dylib`MatZeroRowsMapLocal_Private(A=0x0001170c1270, N=1, 
> rows=0x00016f912cb4, nr=0x00016f911df8, olrows=0x00016f911e00) at 
> zerorows.c:36:5
> frame #9: 0x00010504ea50 
> libpetsc.3.17.dylib`MatZeroRows_MPIAIJ(A=0x0001170c1270, N=1, 
> rows=0x00016f912cb4, diag=1, x=0x, 

Re: [petsc-users] [EXTERNAL] Re: Unexpected performance losses switching to COO interface

[Jed Brown]

"Fackler, Philip via petsc-users"  writes:

> That makes sense. Here are the arguments that I think are relevant:
>
> -fieldsplit_1_pc_type redundant -fieldsplit_0_pc_type sor -pc_type fieldsplit 
> -pc_fieldsplit_detect_coupling​

What sort of physics are in splits 0 and 1?

SOR is not a good GPU algorithm, so we'll want to change that one way or 
another. Are the splits of similar size or very different?

> What would you suggest to make this better?
>
> Also, note that the cases marked "serial" are running on CPU only, that is, 
> using only the SERIAL backend for kokkos.
>
> Philip Fackler
> Research Software Engineer, Application Engineering Group
> Advanced Computing Systems Research Section
> Computer Science and Mathematics Division
> Oak Ridge National Laboratory
> 
> From: Junchao Zhang 
> Sent: Tuesday, November 28, 2023 15:51
> To: Fackler, Philip 
> Cc: petsc-users@mcs.anl.gov ; 
> xolotl-psi-developm...@lists.sourceforge.net 
> 
> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses 
> switching to COO interface
>
> Hi, Philip,
>I opened hpcdb-PSI_9-serial and it seems you used PCLU.  Since Kokkos does 
> not have a GPU LU implementation, we do it on CPU via 
> MatLUFactorNumeric_SeqAIJ(). Perhaps you can try other PC types?
>
> [Screenshot 2023-11-28 at 2.43.03 PM.png]
> --Junchao Zhang
>
>
> On Wed, Nov 22, 2023 at 10:43 AM Fackler, Philip 
> mailto:fackle...@ornl.gov>> wrote:
> I definitely dropped the ball on this. I'm sorry for that. I have new 
> profiling data using the latest (as of yesterday) of petsc/main. I've put 
> them in a single google drive folder linked here:
>
> https://drive.google.com/drive/folders/14ScvyfxOzc4OzXs9HZVeQDO-g6FdIVAI?usp=drive_link
>
> Have a happy holiday weekend!
>
> Thanks,
>
> Philip Fackler
> Research Software Engineer, Application Engineering Group
> Advanced Computing Systems Research Section
> Computer Science and Mathematics Division
> Oak Ridge National Laboratory
> 
> From: Junchao Zhang mailto:junchao.zh...@gmail.com>>
> Sent: Monday, October 16, 2023 15:24
> To: Fackler, Philip mailto:fackle...@ornl.gov>>
> Cc: petsc-users@mcs.anl.gov 
> mailto:petsc-users@mcs.anl.gov>>; 
> xolotl-psi-developm...@lists.sourceforge.net
>  
> mailto:xolotl-psi-developm...@lists.sourceforge.net>>
> Subject: Re: [EXTERNAL] Re: [petsc-users] Unexpected performance losses 
> switching to COO interface
>
> Hi, Philip,
>That branch was merged to petsc/main today. Let me know once you have new 
> profiling results.
>
>Thanks.
> --Junchao Zhang
>
>
> On Mon, Oct 16, 2023 at 9:33 AM Fackler, Philip 
> mailto:fackle...@ornl.gov>> wrote:
> Junchao,
>
> I've attached updated timing plots (red and blue are swapped from before; 
> yellow is the new one). There is an improvement for the NE_3 case only with 
> CUDA. Serial stays the same, and the PSI cases stay the same. In the PSI 
> cases, MatShift doesn't show up (I assume because we're using different 
> preconditioner arguments). So, there must be some other primary culprit. I'll 
> try to get updated profiling data to you soon.
>
> Thanks,
>
> Philip Fackler
> Research Software Engineer, Application Engineering Group
> Advanced Computing Systems Research Section
> Computer Science and Mathematics Division
> Oak Ridge National Laboratory
> 
> From: Fackler, Philip via Xolotl-psi-development 
> mailto:xolotl-psi-developm...@lists.sourceforge.net>>
> Sent: Wednesday, October 11, 2023 11:31
> To: Junchao Zhang mailto:junchao.zh...@gmail.com>>
> Cc: petsc-users@mcs.anl.gov 
> mailto:petsc-users@mcs.anl.gov>>; 
> xolotl-psi-developm...@lists.sourceforge.net
>  
> mailto:xolotl-psi-developm...@lists.sourceforge.net>>
> Subject: Re: [Xolotl-psi-development] [EXTERNAL] Re: [petsc-users] Unexpected 
> performance losses switching to COO interface
>
> I'm on it.
>
> Philip Fackler
> Research Software Engineer, Application Engineering Group
> Advanced Computing Systems Research Section
> Computer Science and Mathematics Division
> Oak Ridge National Laboratory
> 
> From: Junchao Zhang mailto:junchao.zh...@gmail.com>>
> Sent: Wednesday, October 11, 2023 10:14
> To: Fackler, Philip mailto:fackle...@ornl.gov>>
> Cc: petsc-users@mcs.anl.gov 
> mailto:petsc-users@mcs.anl.gov>>; 
> 

[petsc-users] MPI barrier issue using MatZeroRows

[Amneet Bhalla]

Hi Folks,

I am using MatZeroRows() to set Dirichlet boundary conditions. This works
fine for the serial run and the solver produces correct results (verified
through analytical solution). However, when I run the case in parallel, the
simulation gets stuck at MatZeroRows(). My understanding is that this
function needs to be called after the MatAssemblyBegin{End}() has been
called, and should be called by all processors. Here is that bit of the
code which calls MatZeroRows() after the matrix has been assembled

https://github.com/IBAMR/IBAMR/blob/amneetb/acoustically-driven-flows/src/acoustic_streaming/AcousticStreamingPETScMatUtilities.cpp#L724-L801

I ran the parallel code (on 3 processors) in the debugger
(-start_in_debugger). Below is the call stack from the processor that gets
stuck

amneetb@APSB-MBP-16:~$ lldb  -p 4307

(lldb) process attach --pid 4307

Process 4307 stopped

* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP

frame #0: 0x00018a2d750c libsystem_kernel.dylib`__semwait_signal + 8

libsystem_kernel.dylib`:

->  0x18a2d750c <+8>:  b.lo   0x18a2d752c   ; <+40>

0x18a2d7510 <+12>: pacibsp

0x18a2d7514 <+16>: stpx29, x30, [sp, #-0x10]!

0x18a2d7518 <+20>: movx29, sp

Target 0: (fo_acoustic_streaming_solver_2d) stopped.

Executable module set to
"/Users/amneetb/Softwares/IBAMR-Git/objs-dbg/tests/IBTK/fo_acoustic_streaming_solver_2d".

Architecture set to: arm64-apple-macosx-.

(lldb) cont

Process 4307 resuming

Process 4307 stopped

* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP

frame #0: 0x000109d281b8
libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 400

libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather:

->  0x109d281b8 <+400>: ldrw9, [x24]

0x109d281bc <+404>: cmpw8, w9

0x109d281c0 <+408>: b.lt   0x109d281a0   ; <+376>

0x109d281c4 <+412>: bl 0x109d28e64   ;
MPID_Progress_test

Target 0: (fo_acoustic_streaming_solver_2d) stopped.

(lldb) bt

* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP

  * frame #0: 0x000109d281b8
libpmpi.12.dylib`MPIDI_POSIX_mpi_barrier_release_gather + 400

frame #1: 0x000109d27d14 libpmpi.12.dylib`MPIDI_SHM_mpi_barrier +
224

frame #2: 0x000109d27b60
libpmpi.12.dylib`MPIDI_Barrier_intra_composition_alpha + 44

frame #3: 0x000109d0d490 libpmpi.12.dylib`MPIR_Barrier + 900

frame #4: 0x00010224d030 libmpi.12.dylib`MPI_Barrier + 684

frame #5: 0x0001045ea638
libpetsc.3.17.dylib`PetscCommDuplicate(comm_in=-2080374782,
comm_out=0x00010300bcb0, first_tag=0x00010300bce4) at tagm.c:235:5

frame #6: 0x0001045f2910
libpetsc.3.17.dylib`PetscHeaderCreate_Private(h=0x00010300bc70,
classid=1211227, class_name="PetscSF", descr="Star Forest",
mansec="PetscSF", comm=-2080374782,
destroy=(libpetsc.3.17.dylib`PetscSFDestroy at sf.c:224),
view=(libpetsc.3.17.dylib`PetscSFView at sf.c:841)) at inherit.c:62:3

frame #7: 0x0001049cf820
libpetsc.3.17.dylib`PetscSFCreate(comm=-2080374782, sf=0x00016f911a50)
at sf.c:62:3

frame #8: 0x000104cd3024
libpetsc.3.17.dylib`MatZeroRowsMapLocal_Private(A=0x0001170c1270, N=1,
rows=0x00016f912cb4, nr=0x00016f911df8, olrows=0x00016f911e00)
at zerorows.c:36:5

frame #9: 0x00010504ea50
libpetsc.3.17.dylib`MatZeroRows_MPIAIJ(A=0x0001170c1270, N=1,
rows=0x00016f912cb4, diag=1, x=0x,
b=0x) at mpiaij.c:768:3

frame #10: 0x000104d95fac
libpetsc.3.17.dylib`MatZeroRows(mat=0x0001170c1270, numRows=1,
rows=0x00016f912cb4, diag=1, x=0x,
b=0x) at matrix.c:5935:3

frame #11: 0x00010067d320
fo_acoustic_streaming_solver_2d`IBAMR::AcousticStreamingPETScMatUtilities::constructPatchLevelFOAcousticStreamingOp(mat=0x00016f91c178,
omega=1, sound_speed=1, rho_idx=3, mu_idx=2, lambda_idx=4,
u_bc_coefs=0x00016f91c3a8, data_time=NaN, num_dofs_per_proc=size=3,
u_dof_index_idx=27, p_dof_index_idx=28,
patch_level=Pointer > @ 0x00016f914ed0,
mu_interp_type=VC_HARMONIC_INTERP) at AcousticStreamingPETScMatUtilities.cpp
:794:36

frame #12: 0x000100694bdc
fo_acoustic_streaming_solver_2d`IBAMR::FOAcousticStreamingPETScLevelSolver::initializeSolverStateSpecialized(this=0x00016f91c028,
x=0x00016f91d788, (null)=0x00016f91d690) at
FOAcousticStreamingPETScLevelSolver.cpp:149:5

frame #13: 0x00010083232c
fo_acoustic_streaming_solver_2d`IBTK::PETScLevelSolver::initializeSolverState(this=0x00016f91c028,
x=0x00016f91d788, b=0x00016f91d690) at PETScLevelSolver.cpp:340:5

frame #14: 0x0001004eb230
fo_acoustic_streaming_solver_2d`main(argc=11, argv=0x00016f91f460) at
fo_acoustic_streaming_solver.cpp:400:22

frame #15: 0x000189fbbf28 dyld`start + 2236


Any suggestions on how to avoid this barrier? Here are all MAT options I am
using (in 

Re: [petsc-users] Fortran problem MatGetValuesLocal

[Mark Adams]

It looks like we don't have a Fortran interface for this. In the
source code:

2366:   Developer Note:2367:   This is labeled with C so does not
automatically generate Fortran stubs and interfaces2368:   because it
requires multiple Fortran interfaces depending on which arguments are
scalar or arrays.

Someone else might be able to offer a workaround.

Thanks,
Mark

On Tue, Nov 28, 2023 at 5:46 AM Frank Bramkamp  wrote:

> Dear PETSc team,
>
>
> We are using the latest petsc version 3.20.1, intel compiler 2023,
> and we found the following problem:
>
> We want to call the function MatGetValuesLocal to extract a block
> sub-matrix
> from an assembled matrix (e.g. a 5x5 blocked sub matrix). We use the
> matrix format MatCreateBAIJ in parallel.
> In particular we try to call MatGetValuesLocal in Fortran.
>
> It seems that the linked does not find the subroutine MatGetValuesLocal.
> The subroutine MatGetValues seems to be fine.
> I guess that the fortran stubs/fortran interface is missing for this
> routine.
> On the documentation side, you also write a note for developers that the
> fortran stubs and interface
> Is not automatically generated for MatGetValuesLocal. So maybe that has
> been forgotten to do.
>
>
> Unfortunately I do not have any small test example, since we just
> incorporated the function call into our own software.
> Otherwise I would first have to set a small test example for the parallel
> case.
>
> I think there is also an include file where one can check the fortran
> interfaces ?!
> I forgot where to look this up.
>
>
> Greetings, Frank Bramkamp
>
>
>
>
>
>
>
>
>
>
>

Re: [petsc-users] Fortran problem MatGetValuesLocal

[Barry Smith]

  This is fixed in branch 
barry/2023-11-28/add-matsetvalueslocal-fortran/release see also 
https://gitlab.com/petsc/petsc/-/merge_requests/7065

> On Nov 28, 2023, at 8:46 AM, Frank Bramkamp  wrote:
> 
> Dear PETSc team,
> 
> 
> We are using the latest petsc version 3.20.1, intel compiler 2023,
> and we found the following problem:
> 
> We want to call the function MatGetValuesLocal to extract a block sub-matrix
> from an assembled matrix (e.g. a 5x5 blocked sub matrix). We use the matrix 
> format MatCreateBAIJ in parallel.
> In particular we try to call MatGetValuesLocal in Fortran.
> 
> It seems that the linked does not find the subroutine MatGetValuesLocal.
> The subroutine MatGetValues seems to be fine.
> I guess that the fortran stubs/fortran interface is missing for this routine.
> On the documentation side, you also write a note for developers that the 
> fortran stubs and interface
> Is not automatically generated for MatGetValuesLocal. So maybe that has been 
> forgotten to do.
> 
> 
> Unfortunately I do not have any small test example, since we just 
> incorporated the function call into our own software.
> Otherwise I would first have to set a small test example for the parallel 
> case.
> 
> I think there is also an include file where one can check the fortran 
> interfaces ?!
> I forgot where to look this up.
> 
> 
> Greetings, Frank Bramkamp
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

[petsc-users] Fortran problem MatGetValuesLocal

[Frank Bramkamp]

Dear PETSc team,


We are using the latest petsc version 3.20.1, intel compiler 2023,
and we found the following problem:

We want to call the function MatGetValuesLocal to extract a block sub-matrix
from an assembled matrix (e.g. a 5x5 blocked sub matrix). We use the matrix 
format MatCreateBAIJ in parallel.
In particular we try to call MatGetValuesLocal in Fortran.

It seems that the linked does not find the subroutine MatGetValuesLocal.
The subroutine MatGetValues seems to be fine.
I guess that the fortran stubs/fortran interface is missing for this routine.
On the documentation side, you also write a note for developers that the 
fortran stubs and interface
Is not automatically generated for MatGetValuesLocal. So maybe that has been 
forgotten to do.


Unfortunately I do not have any small test example, since we just incorporated 
the function call into our own software.
Otherwise I would first have to set a small test example for the parallel case.

I think there is also an include file where one can check the fortran 
interfaces ?!
I forgot where to look this up.


Greetings, Frank Bramkamp

[petsc-users] msh4 geenerated from neper

[Sharan Roongta]

Hello,
 
I have problem loading the msh4 file generated from neper. The mesh format is 
4.1
The error is:
[0]PETSC ERROR: - Error Message 
--
[0]PETSC ERROR: Unexpected data in file
[0]PETSC ERROR: File is not a valid Gmsh file, expecting $Entities
[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be 
the program crashed before they were used or a spelling mistake, etc!
[0]PETSC ERROR: Option left: name:-g value: test.msh4
[0]PETSC ERROR: Option left: name:-l value: tensionX.load
[0]PETSC ERROR: Option left: name:-m value: material.yaml
[0]PETSC ERROR: Option left: name:-n value: numerics.yaml
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.18.5, unknown 
[0]PETSC ERROR: DAMASK_mesh on a gfortran named mamc57x by work Mon Nov 27 
21:50:06 2023
[0]PETSC ERROR: Configure options --with-mpi-f90module-visibility=0 
--download-hdf5 --with-hdf5-fortran-bindings=1 --download-fftw --download-mpich 
--with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 
--download-ml --download-mumps --download-scalapack --download-hypre 
--download-zlib
[0]PETSC ERROR: #1 GmshExpect() at 
/home/work/petsc/src/dm/impls/plex/plexgmsh.c:270
[0]PETSC ERROR: #2 DMPlexCreateGmsh() at 
/home/work/petsc/src/dm/impls/plex/plexgmsh.c:1548
[0]PETSC ERROR: #3 DMPlexCreateGmshFromFile() at 
/home/work/petsc/src/dm/impls/plex/plexgmsh.c:1433
[0]PETSC ERROR: #4 DMPlexCreateFromFile() at 
/home/work/petsc/src/dm/impls/plex/plexcreate.c:5207
[0]PETSC ERROR: #5 /home/work/DAMASK/src/mesh/discretization_mesh.f90:112
[0]PETSC ERROR: 

[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably 
memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and 
https://petsc.org/release/faq/
[0]PETSC ERROR: -  Stack Frames 

[0]PETSC ERROR: No error traceback is available, the problem could be in the 
main program. 
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59
 
The file does contain the $Entities section (file attached). 
Has anyone encountered this before?

Regards,
Sharan
P.S Sorry for the earlier mail that was sent to the incorrect mailing list.
 
--- 
Max-Planck-Institut für Eisenforschung GmbH 
Max-Planck-Straße 1 
D-40237 Düsseldorf 

Handelsregister B 2533 
Amtsgericht Düsseldorf 
   
Geschäftsführung 
Prof. Dr. Gerhard Dehm 
Prof. Dr. Jörg Neugebauer 
Prof. Dr. Dierk Raabe 
Dr. Kai de Weldige 

Ust.-Id.-Nr.: DE 11 93 58 514 
Steuernummer: 105 5891 1000 
- 
Please consider that invitations and e-mails of our institute are 
only valid if they end with ...@mpie.de. 
If you are not sure of the validity please contact r...@mpie.de 

Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails 
aus unserem Haus nur mit der Endung ...@mpie.de gültig sind. 
In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de

-
Stay up to date and follow us on LinkedIn, Twitter and YouTube.

Max-Planck-Institut für Eisenforschung GmbH
Max-Planck-Straße 1
D-40237 Düsseldorf
 
Handelsregister B 2533 
Amtsgericht Düsseldorf
 
Geschäftsführung
Prof. Dr. Gerhard Dehm
Prof. Dr. Jörg Neugebauer
Prof. Dr. Dierk Raabe
Dr. Kai de Weldige
 
Ust.-Id.-Nr.: DE 11 93 58 514 
Steuernummer: 105 5891 1000


Please consider that invitations and e-mails of our institute are 
only valid if they end with …@mpie.de. 
If you are not sure of the validity please contact r...@mpie.de

Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
aus unserem Haus nur mit der Endung …@mpie.de gültig sind. 
In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
-
<>