Re: [petsc-users] Problem with MPI, MatAXPY and SAME_NONZERO_PATTERN

[Klaus Kaiser]

Hallo Matthew,

thanks for your fast response. With "no beside the nonzero structure" I 
meant, that I do not create a different non-zero structure while I'm 
adding values to my matrices.

I also tried MAT_COPY_VALUES and the result was the same.

I do not get a error message but the resulting matrix is wrong. Here I 
have a short example. The first 9 rows and first 3 colomn of the 
matrices Ah, At and Ah+At on the first processor (of 8):

Ah:
0:                   60.3553                -0.249975 2.77556e-17
1:                         0 60.3553                        0
2:                         0 0                  60.3553
3:                   17.6777 0.374962                 0.124987
4:                         0 17.6777                        0
5:                         0 0                  17.6777
6:                  -7.32233 -0.124987                 0.374962
7:                         0 -7.32233                        0
8:                         0 0                 -7.32233


At:
0:                         0 0                        0
1:                      2500 0                        0
2:              -4.54747e-13 0                        0
3:                         0 0                        0
4:                      1250 0                        0
5:                      1250 0                        0
6:                         0 0                        0
7:                      1250 0                        0
8:                      3750 0                        0

Ah+At
0:                   60.3553                -0.249975 2.77556e-17
1:                      2500 60.3553                        0
2:              -4.54747e-13 0                  60.3553
3:                   17.6777 0.374962                 0.124987
4:                         0 17.6777                        0
5:                         0 0                  17.6777
6:                  -7.32233 -0.124987                 0.374962
7:                         0 -7.32233                        0
8:                         0 0                 -7.32233

you can see the first 3 rows of the resulting matrix looks exactly like 
what I would expect, but the last 6 rows only consists of the values of 
Ah. When you would also look on the matrix A and A+Ah+At you would also 
see, that values where both matrices have an nonzero entry are not sum 
correctly.

Best

Klaus

On 03/27/2015 03:59 PM, Matthew Knepley wrote:
On Fri, Mar 27, 2015 at 9:48 AM, Klaus Kaiser 
<kai...@igpm.rwth-aachen.de <mailto:kai...@igpm.rwth-aachen.de>> wrote:

    Hallo,

    I have a strange behavior in my code concerning the function
    MatAXPY. I create 3 different Matrices

        ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
    local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&A);
        ierr = MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
        ierr = MatSetOption(A,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
        ierr = MatSetOption(A,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);

        ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
    local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&At);
        ierr = MatSetOption(At,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
        ierr = MatSetOption(At,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
        ierr = MatSetOption(At,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);

        ierr = MatCreateBAIJ(PETSC_COMM_WORLD, block_size, local_size,
    local_size, system_size, system_size, 0, d_nnz, 0, o_nnz,&Ah);
        ierr = MatSetOption(Ah,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);
        ierr = MatSetOption(Ah,MAT_KEEP_NONZERO_PATTERN,PETSC_TRUE);
        ierr = MatSetOption(Ah,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE);


These creations are superfluous since you use MatDuplicate() below.

    and want to sum these three matrixes with different factors. First
    I fill the Matrix A with some values, and duplicate the structure
    of A to the other two matrices:

        MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
        MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
        MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&Ah);
        MatDuplicate(A,MAT_DO_NOT_COPY_VALUES,&At);
        MatAssemblyBegin(Ah,MAT_FINAL_ASSEMBLY);
        MatAssemblyEnd(Ah,MAT_FINAL_ASSEMBLY);
        MatAssemblyBegin(At,MAT_FINAL_ASSEMBLY);
        MatAssemblyEnd(At,MAT_FINAL_ASSEMBLY);


    After this I fill the matrices At and Ah with some other values,
    which are not beside the non zero structure (I also tried with
    just copying the Matrix A). Now after another MatAssembly


I do not understand "no beside the nonzero structure". Do you mean 
that the nonzero structure is the same?

Can you first test with MAT_COPY_VALUES?

    I want to add these Matrices in the form A+c*(Ah+d*At): 


        MatAXPY(Ah,c,At,SAME_NONZERO_PATTERN);
        MatAXPY(A,d,Ah,SAME_NONZERO_PATTERN);

    When I run the method with mpi and one core everything works fine.
    Starting the same method with more cores, the sum of the matrices
    fails. It seems like some values are added


Please send the full output of a failure when you use MAT_COPY_VALUES 
on 2 procs.

  Thanks,

     Matt

    correctly and many values are missed. Using
    DIFFERENT_NONZERO_STRUCTURE leads to the right behavior in the
    multi-core case, but is very slow. I checked with a viewer if all
    matrices have the same nonzero structure and this is the case.

    Does anyone know why this fails, or do I have made any wrong thoughts?

    I'm corrently working with a petsc version (Petsc Release Version
    3.3.0, Patch 5, Sat Dec  1 15:10:41 CST 2012), I looked into the
    changelogs up to the current version and did not find any note
    about MatAXPY or MatAYPX.


    Best and Thanks a lot for your help

    Klaus




--
What most experimenters take for granted before they begin their 
experiments is infinitely more interesting than any results to which 
their experiments lead.
-- Norbert Wiener