Re: [petsc-users] MatAssemblyEnd hangs during a parallel calculation with PETSc>3.3

[Projet_TRIOU]

Sorry a typo, it is really MatMPISBAIJSetPreallocation()
Yes, I call MatMPISBAIJSetPreallocation() on ALL processes and sometimes
a local part of the matrix has zero rows. It worked well with Petsc 3.3 
and before
in this particular case.

Sorry, it will difficult to me to send you the whole code to reproduce 
the problem.
May be we could reproduce the hang with an example test of PETSc ?

Pierre
   Hmm, sometimes you talk about MatMPIAIJSetPreallocation() and sometimes 
MatMPISBAIJSetPreallocation() which one is it?

   If a process has zero rows that should not affect the nonew variable.

    It is crucial the the SetPreallocation routines be called on ALL processes 
that share the matrix, even if they have zero rows, but it shouldn’t matter 
what the values are for a matrix with zero rows.

    Can you send use your test case so we can track down the problem?

     Barry



On Jan 15, 2014, at 5:57 AM, Projet_TRIOU <tr...@cea.fr> wrote:

Hi all,

I switched from PETSc 3.3 to PETSc 3.4.3 and all was fine except for
one of my test case on 3 processors where one processor was
dealing an empty local part of the global matrix.

My code hangs just during the call at MatAssemblyEnd:

ierr = MatMPIAIJSetPreallocation(MatricePetsc, PETSC_DEFAULT, d_nnz.addr(), 
PETSC_DEFAULT, o_nnz.addr());
...
ierr = MatAssemblyEnd(MatricePetsc, MAT_FINAL_ASSEMBLY);

When I debugged, I notice on the empty processor, that in
src/mat/impls/aij/mpi/mpiaij.c:

  if (!((Mat_SeqAIJ*)aij->B->data)->nonew) {
    ierr = 
MPI_Allreduce(&mat->was_assembled,&other_disassembled,1,MPIU_BOOL,MPI_PROD,PetscObjectComm((PetscObject)mat));CHKERRQ(ierr);
    if (mat->was_assembled && !other_disassembled) {
      ierr = MatDisAssemble_MPIAIJ(mat);CHKERRQ(ierr);
    }

((Mat_SeqAIJ*)aij->B->data)->nonew was 0 on the "empty" processor
and -2 on the 2 others...

I bypassed my problem with a different call to MatMPIAIJSetPreallocation():

      if (nb_rows==0) // Local matrix is empty
         ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, 0, NULL, 
0, NULL);
      else
         ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, 
PETSC_DEFAULT, d_nnz.addr(), PETSC_DEFAULT, o_nnz.addr());

Now, it runs well. So I don't know if it is a PETSc regression or if I was 
abusively
calling MatMPISBAIJSetPreallocation with d_nnz/o_nnz empty arrays....

Thanks,

Pierre



--
*Trio_U support team*
Marthe ROUX (Saclay)
Pierre LEDAC (Grenoble)