On Jan 15, 2014, at 11:20 AM, Matthew Knepley <[email protected]> wrote:
> This is a bug in the setup phase for Mat. I did not put in the logic, so I
> might be missing the point. Here is
> my outline:
>
> 1) When MatSetPreallocation_*() is called, we set the
> NEW_NONZERO_ALLOCATION_ERR option, but ONLY
> if the user passed a nonzero nz or nnz.
Matt,
Ahhh.
The background is we wanted PETSc to automatically YELL if someone
preallocated but did not preallocate enough.
But if the user did not preallocate we automatically call
MatMPIXXSetPreallocation_MPIXXX() with default values that do not setup the
yelling.
If might be that some process has some rows, but they are all empty so our
current model is totally incorrect
1) We could do a global reduction in MatMPIXXXXSetPreallocation_MPIXXX() and
if ANY process allocates space then mark the nonew. We definitely don’t want
to do a global reduction on every mat assembly. For efficiency we could not
automatically set the nonew for optimized builds but that is a dangerous game.
2) Alternative, when MatMPIXXXXSetPreallocation_MPIXXX() is called, ALWAYS set
the nonew flag, but when it is called because the user never called it like in
PetscErrorCode MatSetUp_MPIAIJ(Mat A)
{
PetscErrorCode ierr;
PetscFunctionBegin;
ierr =
MatMPIAIJSetPreallocation(A,PETSC_DEFAULT,0,PETSC_DEFAULT,0);CHKERRQ(ierr);
PetscFunctionReturn(0);
}
we immediately change the flag after the call by added a MatSetOption() after
the call to MatMPIXXXSetPreallocation()
I think 2 is ok and suggest doing that
Barry
>
> 2) During assembly, we check A->B->nonew, and do a collective call if it is
> 0, meaning NEW_NONZERO_LOCATIONS
>
> I understand that we would like to avoid collective calls in MatAssembly(),
> but to guarantee correctness then we would
> need to make the option collective, which is not true right now. I see
>
> Slightly Dangerous: Change SetPreallocation to make the option value
> collective
>
> Slightly Slow: Always check for B disassembly
>
> What do you think?
>
> Matt
>
> ---------- Forwarded message ----------
> From: Projet_TRIOU <[email protected]>
> Date: Wed, Jan 15, 2014 at 5:57 AM
> Subject: [petsc-users] MatAssemblyEnd hangs during a parallel calculation
> with PETSc>3.3
> To: [email protected]
>
>
> Hi all,
>
> I switched from PETSc 3.3 to PETSc 3.4.3 and all was fine except for
> one of my test case on 3 processors where one processor was
> dealing an empty local part of the global matrix.
>
> My code hangs just during the call at MatAssemblyEnd:
>
> ierr = MatMPIAIJSetPreallocation(MatricePetsc, PETSC_DEFAULT, d_nnz.addr(),
> PETSC_DEFAULT, o_nnz.addr());
> ...
> ierr = MatAssemblyEnd(MatricePetsc, MAT_FINAL_ASSEMBLY);
>
> When I debugged, I notice on the empty processor, that in
> src/mat/impls/aij/mpi/mpiaij.c:
>
> if (!((Mat_SeqAIJ*)aij->B->data)->nonew) {
> ierr =
> MPI_Allreduce(&mat->was_assembled,&other_disassembled,1,MPIU_BOOL,MPI_PROD,PetscObjectComm((PetscObject)mat));CHKERRQ(ierr);
> if (mat->was_assembled && !other_disassembled) {
> ierr = MatDisAssemble_MPIAIJ(mat);CHKERRQ(ierr);
> }
>
> ((Mat_SeqAIJ*)aij->B->data)->nonew was 0 on the "empty" processor
> and -2 on the 2 others...
>
> I bypassed my problem with a different call to MatMPIAIJSetPreallocation():
>
> if (nb_rows==0) // Local matrix is empty
> ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_, 0,
> NULL, 0, NULL);
> else
> ierr = MatMPISBAIJSetPreallocation(MatricePetsc, block_size_,
> PETSC_DEFAULT, d_nnz.addr(), PETSC_DEFAULT, o_nnz.addr());
>
> Now, it runs well. So I don't know if it is a PETSc regression or if I was
> abusively
> calling MatMPISBAIJSetPreallocation with d_nnz/o_nnz empty arrays....
>
> Thanks,
>
> Pierre
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener
