Patrick,
We may have gone overboard with our assertions in this routine.
Some MatOptions really should be collective while others like the one you
are using need not be collective.
Simple solution just remove those two collective asserts. Harder solution,
use the asserts for only the options that should be checked. We could for
example use less then some number for collective asserts and greater than
number for for non collective. A little hacky but would work?
Barry
On Sep 20, 2012, at 11:42 AM, Patrick Lacasse <patrick.m.lacasse at gmail.com>
wrote:
> Hi,
>
> I just discovered that the way we assemble our matrix cause a deadlock in
> MatSetOption
> when petsc is compiled with debugging option enabled.
>
> I resume :
> - We loop over elements.
> - For each element we create a small elementary matrix.
> - We assemble the elementary matrix with MatSetValues into the big one.
>
> Sometimes, we have some rectangular elementary matrix that needs to be added
> twice,
> the second time it is transposed. The big matrix is not symmetric for many
> possible reasons.
> We are essentially doing it for every elementary matrix that needs it :
>
> MatSetOption(mat,MAT_ROW_ORIENTED,true)
> MatSetValues(...)
> MatSetOption(mat,MAT_ROW_ORIENTED,false)
> MatSetValues(...)
>
> Of course, it fails (deadlock) in MatSetOption at assertion
> PetscValidLogicalCollectiveEnum(mat,op,2);
> because different process don't loop over the same (number of) elements.
> However, it is working fine without the assertion.
>
> I have two question :
> Is there really a risk to call MatSetOption(mat, MAT_ROW_ORIENTED, flg) on
> only one process?
> What can I do now?
>
> Some possible answer are
> - Close our eyes and stop compiling petsc with debugging option enabled.
> - We could copy our elementary matrix into a transposed one before to
> MatSetValues. We don't want do this cause we think it would slow down our
> code.
> - Could MatSetValues has a bonus argument row_oriented?
> - Could the assertions in MatSetOption be skipped for some options?
> - Could we add a function MatSetRowOriented which would not be collective?
>
> thanks,
>
> Patrick Lacasse
