Re: [petsc-users] error: Petsc has generated inconsistent data, MPI_Allreduce() called in different locations (code lines) on different processors

2019-04-07 Thread Mark Adams via petsc-users
You have MatSetOption inside of a loop for the number of rows. This is
apparently a collective operation. Move it to line 156.

On Sun, Apr 7, 2019 at 5:44 AM Eda Oktay via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Dear Barry,
>
> Thank you for answering. I am sending my code, makefile and 2 PETSc Binary
> File where airfoil1_binary has odd numbered size (the one I got error) and
> gr_30_30_binary has even numbered size (I got correct result). I got the
> error for lines 164 and 169.
>
> By the way, I think I have done something wrong at line 320 (I tried to
> allocate an array twice) but I just need to reduce the error in 164 and 169
> then I will get to reduce the error in line 320.
>
> Thanks,
>
> Eda
>
>
> Smith, Barry F. , 6 Nis 2019 Cmt, 09:29 tarihinde
> şunu yazdı:
>
>>
>>   Eda,
>>
>>Can you send us your code (and any needed data files)? We certainly
>> expect PETSc to perform correctly if the size of the matrix cannot be
>> divide by the number of processors. It is possible the problem is due to
>> bugs either in MatStashScatterBegin_BTS() or in your code.
>>
>>Thanks
>>
>> Barry
>>
>>
>> > On Apr 5, 2019, at 7:20 AM, Matthew Knepley via petsc-users <
>> petsc-users@mcs.anl.gov> wrote:
>> >
>> > On Fri, Apr 5, 2019 at 3:20 AM Eda Oktay via petsc-users <
>> petsc-users@mcs.anl.gov> wrote:
>> > Hello,
>> >
>> > I am trying to calculate unweighted Laplacian of a matrix by using 2
>> cores. If the size of matrix is in even number then my program works.
>> However, when I try to use a matrix having odd number for size, I guess
>> since size of the matrix cannot be divided into processors correctly, I get
>> the following error:
>> >
>> > [0]PETSC ERROR: - Error Message
>> --
>> > [1]PETSC ERROR: - Error Message
>> --
>> > [1]PETSC ERROR: Petsc has generated inconsistent data
>> > [1]PETSC ERROR: MPI_Allreduce() called in different locations (code
>> lines) on different processors
>> > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> > [1]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> > [0]PETSC ERROR: Petsc has generated inconsistent data
>> > [0]PETSC ERROR: MPI_Allreduce() called in different locations (code
>> lines) on different processors
>> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
>> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
>> 09:50:54 2019
>> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> > [1]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
>> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
>> 09:50:54 2019
>> > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
>> --download-metis --download-parmetis --download-superlu_dist
>> --download-slepc --download-mpich
>> > [1]PETSC ERROR: [0]PETSC ERROR: #1 MatSetOption() line 5505 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > #1 MatStashScatterBegin_BTS() line 843 in
>> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
>> > [1]PETSC ERROR: [0]PETSC ERROR: #2 MatSetOption() line 5505 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > #2 MatStashScatterBegin_BTS() line 843 in
>> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
>> > [1]PETSC ERROR: #3 MatStashScatterBegin_Private() line 462 in
>> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
>> > [0]PETSC ERROR: #3 main() line 164 in
>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>> > [1]PETSC ERROR: #4 MatAssemblyBegin_MPIAIJ() line 774 in
>> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
>> > [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries:
>> > [0]PETSC ERROR: -f
>> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
>> > #5 MatAssemblyBegin() line 5251 in
>> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
>> > [1]PETSC ERROR: [0]PETSC ERROR: -mat_partitioning_type parmetis
>> > [0]PETSC ERROR: -unweighted
>> > #6 main() line 169 in
>> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
>> > [1]PETSC ERROR: PETSc Option Table entries:
>> > [1]PETSC ERROR: [0]PETSC ERROR: End of Error Message
>> ---send entire error message to petsc-ma...@mcs.anl.gov--
>> > -f
>> /home/edaoktay/petsc-3.10.3/sh

Re: [petsc-users] error: Petsc has generated inconsistent data, MPI_Allreduce() called in different locations (code lines) on different processors

2019-04-07 Thread Eda Oktay via petsc-users
Dear Barry,

Thank you for answering. I am sending my code, makefile and 2 PETSc Binary
File where airfoil1_binary has odd numbered size (the one I got error) and
gr_30_30_binary has even numbered size (I got correct result). I got the
error for lines 164 and 169.

By the way, I think I have done something wrong at line 320 (I tried to
allocate an array twice) but I just need to reduce the error in 164 and 169
then I will get to reduce the error in line 320.

Thanks,

Eda


Smith, Barry F. , 6 Nis 2019 Cmt, 09:29 tarihinde şunu
yazdı:

>
>   Eda,
>
>Can you send us your code (and any needed data files)? We certainly
> expect PETSc to perform correctly if the size of the matrix cannot be
> divide by the number of processors. It is possible the problem is due to
> bugs either in MatStashScatterBegin_BTS() or in your code.
>
>Thanks
>
> Barry
>
>
> > On Apr 5, 2019, at 7:20 AM, Matthew Knepley via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
> >
> > On Fri, Apr 5, 2019 at 3:20 AM Eda Oktay via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
> > Hello,
> >
> > I am trying to calculate unweighted Laplacian of a matrix by using 2
> cores. If the size of matrix is in even number then my program works.
> However, when I try to use a matrix having odd number for size, I guess
> since size of the matrix cannot be divided into processors correctly, I get
> the following error:
> >
> > [0]PETSC ERROR: - Error Message
> --
> > [1]PETSC ERROR: - Error Message
> --
> > [1]PETSC ERROR: Petsc has generated inconsistent data
> > [1]PETSC ERROR: MPI_Allreduce() called in different locations (code
> lines) on different processors
> > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [1]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> > [0]PETSC ERROR: Petsc has generated inconsistent data
> > [0]PETSC ERROR: MPI_Allreduce() called in different locations (code
> lines) on different processors
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> > [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
> 09:50:54 2019
> > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> > [1]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
> 09:50:54 2019
> > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> > [1]PETSC ERROR: [0]PETSC ERROR: #1 MatSetOption() line 5505 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> > #1 MatStashScatterBegin_BTS() line 843 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> > [1]PETSC ERROR: [0]PETSC ERROR: #2 MatSetOption() line 5505 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> > #2 MatStashScatterBegin_BTS() line 843 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> > [1]PETSC ERROR: #3 MatStashScatterBegin_Private() line 462 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> > [0]PETSC ERROR: #3 main() line 164 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> > [1]PETSC ERROR: #4 MatAssemblyBegin_MPIAIJ() line 774 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> > [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries:
> > [0]PETSC ERROR: -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> > #5 MatAssemblyBegin() line 5251 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> > [1]PETSC ERROR: [0]PETSC ERROR: -mat_partitioning_type parmetis
> > [0]PETSC ERROR: -unweighted
> > #6 main() line 169 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> > [1]PETSC ERROR: PETSc Option Table entries:
> > [1]PETSC ERROR: [0]PETSC ERROR: End of Error Message
> ---send entire error message to petsc-ma...@mcs.anl.gov--
> > -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> > [1]PETSC ERROR: -mat_partitioning_type parmetis
> > [1]PETSC ERROR: -unweighted
> > [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 1) -
> process 0
> > End of Error Message ---send entire error message to
> petsc-ma...@mcs.anl.gov-

Re: [petsc-users] error: Petsc has generated inconsistent data, MPI_Allreduce() called in different locations (code lines) on different processors

2019-04-05 Thread Smith, Barry F. via petsc-users


  Eda,

   Can you send us your code (and any needed data files)? We certainly expect 
PETSc to perform correctly if the size of the matrix cannot be divide by the 
number of processors. It is possible the problem is due to bugs either in 
MatStashScatterBegin_BTS() or in your code.

   Thanks

Barry


> On Apr 5, 2019, at 7:20 AM, Matthew Knepley via petsc-users 
>  wrote:
> 
> On Fri, Apr 5, 2019 at 3:20 AM Eda Oktay via petsc-users 
>  wrote:
> Hello,
> 
> I am trying to calculate unweighted Laplacian of a matrix by using 2 cores. 
> If the size of matrix is in even number then my program works. However, when 
> I try to use a matrix having odd number for size, I guess since size of the 
> matrix cannot be divided into processors correctly, I get the following error:
> 
> [0]PETSC ERROR: - Error Message 
> --
> [1]PETSC ERROR: - Error Message 
> --
> [1]PETSC ERROR: Petsc has generated inconsistent data
> [1]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on 
> different processors
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
> trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on 
> different processors
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
> trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug 
> named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5 09:50:54 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ 
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas 
> --download-metis --download-parmetis --download-superlu_dist --download-slepc 
> --download-mpich
> [1]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a arch-linux2-c-debug 
> named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5 09:50:54 2019
> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ 
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas 
> --download-metis --download-parmetis --download-superlu_dist --download-slepc 
> --download-mpich
> [1]PETSC ERROR: [0]PETSC ERROR: #1 MatSetOption() line 5505 in 
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> #1 MatStashScatterBegin_BTS() line 843 in 
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [1]PETSC ERROR: [0]PETSC ERROR: #2 MatSetOption() line 5505 in 
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> #2 MatStashScatterBegin_BTS() line 843 in 
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [1]PETSC ERROR: #3 MatStashScatterBegin_Private() line 462 in 
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [0]PETSC ERROR: #3 main() line 164 in 
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [1]PETSC ERROR: #4 MatAssemblyBegin_MPIAIJ() line 774 in 
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -f 
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> #5 MatAssemblyBegin() line 5251 in 
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [1]PETSC ERROR: [0]PETSC ERROR: -mat_partitioning_type parmetis
> [0]PETSC ERROR: -unweighted
> #6 main() line 169 in 
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: [0]PETSC ERROR: End of Error Message 
> ---send entire error message to petsc-ma...@mcs.anl.gov--
> -f 
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> [1]PETSC ERROR: -mat_partitioning_type parmetis
> [1]PETSC ERROR: -unweighted
> [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
> End of Error Message ---send entire error message to 
> petsc-ma...@mcs.anl.gov--
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1
> 
> 
>  where line 164 in my main program is MatSetOption and line 169 is 
> MatAssemblyBegin. I am new to MPI usage so I did not understand why 
> MPI_Allreduce() causes a problem and how can I fix the problem.
> 
> You have to call collective methods on all processes in the same order. This 
> is not happening in your code. Beyond that,
> there is no way for us to tell how this happened.
> 
>   Thanks,
> 
>  Matt
>  
> Thank you,
> 
> Eda
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to whic

Re: [petsc-users] error: Petsc has generated inconsistent data, MPI_Allreduce() called in different locations (code lines) on different processors

2019-04-05 Thread Matthew Knepley via petsc-users
On Fri, Apr 5, 2019 at 3:20 AM Eda Oktay via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello,
>
> I am trying to calculate unweighted Laplacian of a matrix by using 2
> cores. If the size of matrix is in even number then my program works.
> However, when I try to use a matrix having odd number for size, I guess
> since size of the matrix cannot be divided into processors correctly, I get
> the following error:
>
> [0]PETSC ERROR: - Error Message
> --
> [1]PETSC ERROR: - Error Message
> --
> [1]PETSC ERROR: Petsc has generated inconsistent data
> [1]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
> on different processors
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines)
> on different processors
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
> 09:50:54 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> [1]PETSC ERROR: ./SON_YENI_DENEME_TEMIZ_ENYENI_FINAL on a
> arch-linux2-c-debug named dfa.wls.metu.edu.tr by edaoktay Fri Apr  5
> 09:50:54 2019
> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --with-cxx-dialect=C++11 --download-openblas
> --download-metis --download-parmetis --download-superlu_dist
> --download-slepc --download-mpich
> [1]PETSC ERROR: [0]PETSC ERROR: #1 MatSetOption() line 5505 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> #1 MatStashScatterBegin_BTS() line 843 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [1]PETSC ERROR: [0]PETSC ERROR: #2 MatSetOption() line 5505 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> #2 MatStashScatterBegin_BTS() line 843 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [1]PETSC ERROR: #3 MatStashScatterBegin_Private() line 462 in
> /home/edaoktay/petsc-3.10.3/src/mat/utils/matstash.c
> [0]PETSC ERROR: #3 main() line 164 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [1]PETSC ERROR: #4 MatAssemblyBegin_MPIAIJ() line 774 in
> /home/edaoktay/petsc-3.10.3/src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> #5 MatAssemblyBegin() line 5251 in
> /home/edaoktay/petsc-3.10.3/src/mat/interface/matrix.c
> [1]PETSC ERROR: [0]PETSC ERROR: -mat_partitioning_type parmetis
> [0]PETSC ERROR: -unweighted
> #6 main() line 169 in
> /home/edaoktay/petsc-3.10.3/arch-linux2-c-debug/share/slepc/examples/src/eda/SON_YENI_DENEME_TEMIZ_ENYENI_FINAL.c
> [1]PETSC ERROR: PETSc Option Table entries:
> [1]PETSC ERROR: [0]PETSC ERROR: End of Error Message
> ---send entire error message to petsc-ma...@mcs.anl.gov--
> -f
> /home/edaoktay/petsc-3.10.3/share/petsc/datafiles/matrices/binary_files/airfoil1_binary
> [1]PETSC ERROR: -mat_partitioning_type parmetis
> [1]PETSC ERROR: -unweighted
> [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
> End of Error Message ---send entire error message to
> petsc-ma...@mcs.anl.gov--
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1
>
>
>  where line 164 in my main program is MatSetOption and line 169
> is MatAssemblyBegin. I am new to MPI usage so I did not understand why
> MPI_Allreduce() causes a problem and how can I fix the problem.
>

You have to call collective methods on all processes in the same order.
This is not happening in your code. Beyond that,
there is no way for us to tell how this happened.

  Thanks,

 Matt


> Thank you,
>
> Eda
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/