Re: [petsc-users] Nodes coordinates in distributed dmplex

2015-07-14 Thread Jed Brown 
Alejandro D Otero  writes:
> But when I set the viewer to use VTK_VTU (in this case ASCII just for make
> it readable, but the same in native vtk format when read with paraview) I
> got:

The ASCII viewer is unrelated I never use it.  The VTU
(XML/binary-appended) format writes multiple blocks into the same file.
Those blocks include ghost points (thus some redundancy) because that's
how the format is specified.


signature.asc
Description: PGP signature

Re: [petsc-users] PETSc / GAMG user counts

2015-07-14 Thread Jed Brown 
Mark Adams  writes:

> Do we have any data with respect to the number of PETSc users?  Better yet
> number of GAMG users?

Mailing list statistics, downstream software packages, papers, and
commits.

We don't spy on users, so just measure products that are made public.


signature.asc
Description: PGP signature

Re: [petsc-users] Building a rectangular MPI matrix relating to PetscSections

2015-07-14 Thread Jed Brown 
Alejandro D Otero  writes:

> Hi all, is it posible to build a rectangular matrix that relates two fields
> with (posible) different number of DoF per point defining the nonzero
> structure from 2 petscsections representing one field each?

There isn't a "convenience function", but of course you can create this
rectangular matrix.

What you plan to do with it and your discretization will affect whether
or not representing it as an assembled matrix is a good choice.


signature.asc
Description: PGP signature

Re: [petsc-users] Fwd: Fwd: same petsc problem

2015-07-14 Thread Barry Smith 


  Version should not matter. Do other command line options work? You can also 
put 

   PetscOptionsSetValue("-ksp_view_mat","binary:filename"); 

in the code just before the call to KSPSolve().

   Or run in the debugger and put a break point in KSPSolve and then next/step 
through to see why the call to 
MatViewFromOptions() is not triggered.

  Barry


> On Jul 14, 2015, at 3:13 PM, Greg Miller  wrote:
> 
> Thank you for the suggestion, Barry.  I've tried using these options, but the 
> code generates no output.
> i.e., no saved binary files.  I'm using 3.5.3 - might that be the problem?  
> Do you recommend a 3.6 upgrade?
> Thank you,
> Greg
> 
> 
> On 07/13/2015 07:45 PM, Barry Smith wrote:
>> 
>>  You can save a (or multiple) linear system(s) with additional options
>> -ksp_view_mat  binary:filename -ksp_view_rhs binary:filename 
>> 
>> and then simply email the file named filename to petsc-ma...@mcs.anl.gov 
>> with number of processes you are using and solver options and we can run the 
>> same problem and thus hopefully reproduce the troublesome behavior.
>> 
>> 
>>  Barry
>> 
>>> On Jul 13, 2015, at 3:23 PM, Mark Adams  wrote:
>>> 
>>> Please respond to all, to get the petsc mailing list.
>>> 
>>> The out file here looks fine.  It would help of you run in debug mode (and 
>>> get -g) so that gdb can give line numbers.
>>> 
>>> If the error is in the dot product it looks like you are getting an Inf or 
>>> Nan someplace.  If the first evaluation of the norm was the problem then 
>>> run the bad code and add a VecView and MatView, and add a VecNorm and 
>>> MatNorm so that the error happens in your code, just to check.  If you had 
>>> 1e200 number in the vector then the 2 norm would give you this trap for 
>>> overflow.  You could test with the inf norm and it works then it is an 
>>> overflow problem.
>>> 
>>> 
>>> 
>>> 
>>> -- Forwarded message --
>>> From: Greg Miller 
>>> Date: Mon, Jul 13, 2015 at 3:54 PM
>>> Subject: Re: Fwd: same petsc problem
>>> To: Mark Adams 
>>> Cc: "t...@lbl.gov" 
>>> 
>>> 
>>> thanks.  here it is.
>>> 
>>> I'm not getting the nan problem with this code - not sure why.  NaN  was 
>>> showing up KSP_solve in the first evaluation of the vector norm.
>>> In going from my original example to the minimal code I mailed today the 
>>> matrix coefficients got rounded through asci printing.
>>> That's the only difference I can think of.  The gdb stack from the old code 
>>> is shown below.
>>> G
>>> 
>>> 
>>> 
>>> Program received signal SIGFPE, Arithmetic exception.
>>> 0x76c4ed15 in ddot_ () from /usr/lib/libblas.so.3
>>> (gdb) where
>>> #0  0x76c4ed15 in ddot_ () from /usr/lib/libblas.so.3
>>> #1  0x008f87da in VecNorm_Seq (xin=0x22e7ca0, type=NORM_2, 
>>> z=0x7fffba30)
>>>at /home/usr/local/src/petsc-3.5.3/src/vec/vec/impls/seq/bvec2.c:614
>>> #2  0x008c9eea in VecNorm (x=0x22e7ca0, type=NORM_2, 
>>> val=0x7fffba30)
>>>at /home/usr/local/src/petsc-3.5.3/src/vec/vec/interface/rvector.c:242
>>> #3  0x008cab40 in VecNormalize (x=0x22e7ca0, val=0x7fffbaa0) at 
>>> /home/usr/local/src/petsc-3.5.3/src/vec/vec/interface/rvector.c:337
>>> #4  0x00d44a9d in KSPGMRESCycle (itcount=0x7fffbb08, 
>>> ksp=0x2288ff0)
>>>at /home/usr/local/src/petsc-3.5.3/src/ksp/ksp/impls/gmres/gmres.c:161
>>> #5  0x00d453e3 in KSPSolve_GMRES (ksp=0x2288ff0) at 
>>> /home/usr/local/src/petsc-3.5.3/src/ksp/ksp/impls/gmres/gmres.c:235
>>> 
>>> 
>>> On 07/13/2015 11:54 AM, Mark Adams wrote:
 Greg, I am forwarding this to the PETSc mailing list.
 
 Please send the entire output from this run.  As I recall you were getting 
 a message that all values were not the same on all processors in GMRES.  I 
 have seen this when I get NaNs in the system.
 
 While you are doing this you should use a simple solver like change:
 
 -pressure_pc_type gamg
 
 to
 
 -pressure_pc_type jacobi
 
 
 And add:
 
 *-*pressure_*ksp_monitor_true_residual*
 
 Mark
 
 
 -- Forwarded message --
 From: *Greg Miller* mailto:grgmil...@ucdavis.edu>>
 Date: Mon, Jul 13, 2015 at 2:08 PM
 Subject: same petsc problem
 To: Mark Adams mailto:mfad...@lbl.gov>>
 Cc: David Trebotich mailto:t...@hpcrd.lbl.gov>>
 
 
 Hi Mark.  I'm still stuck on the same petsc problem.   Would you please 
 try the attached minimal example and advise me?
 
 I'm running this without MPI:
 make DIM=2 DEBUG=TRUE MPI=FALSE USE_PETSC=TRUE test
 
 There is no input file.
 
 Thank you,
 Greg
 
 --
 Greg Miller
 Department of Chemical Engineering and Materials Science
 University of California, Davis
 One Shields Avenue
 Davis, CA 95616
 grgmil...@ucdavis.edu 
 
>>> 
>>> --
>>> Greg Miller
>>> Department of Chemical Engineering and Materia

Re: [petsc-users] Question about Nested matrices

2015-07-14 Thread Jed Brown 
Alejandro D Otero  writes:

> Hi all,
> I am trying to nest to square matrices to form a rectangular one with dims
> 2n x n (with n the size of the square matrices)
> I want the two matrices to have interlaced rows so I passed the following 2
> list of indices to MatCreateNest:
>
> [[ 0  2  4  6  8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46
> 48]
> [ 1  3  5  7  9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
> 49]]
>
> And we get the following error message:
> Global sizes (25,25) of nested submatrix (0,0) do not agree with space
> defined by index sets (50,25)

Looks like the isglobal.row[i] index set has dimension 50 instead of 25.

#if defined(PETSC_USE_DEBUG)
  for (i=0; inr; i++) {
for (j=0; jnc; j++) {
  PetscInt m,n,M,N,mi,ni,Mi,Ni;
  Mat  B = vs->m[i][j];
  if (!B) continue;
  ierr = MatGetSize(B,&M,&N);CHKERRQ(ierr);
  ierr = MatGetLocalSize(B,&m,&n);CHKERRQ(ierr);
  ierr = ISGetSize(vs->isglobal.row[i],&Mi);CHKERRQ(ierr);
  ierr = ISGetSize(vs->isglobal.col[j],&Ni);CHKERRQ(ierr);
  ierr = ISGetLocalSize(vs->isglobal.row[i],&mi);CHKERRQ(ierr);
  ierr = ISGetLocalSize(vs->isglobal.col[j],&ni);CHKERRQ(ierr);
  if (M != Mi || N != Ni) 
SETERRQ6(PetscObjectComm((PetscObject)sub),PETSC_ERR_ARG_INCOMP,"Global sizes 
(%D,%D) of nested submatrix (%D,%D) do not agree with space defined by index 
sets (%D,%D)",M,N,i,j,Mi,Ni);
  if (m != mi || n != ni) 
SETERRQ6(PetscObjectComm((PetscObject)sub),PETSC_ERR_ARG_INCOMP,"Local sizes 
(%D,%D) of nested submatrix (%D,%D) do not agree with space defined by index 
sets (%D,%D)",m,n,i,j,mi,ni);
}
  }
#endif


signature.asc
Description: PGP signature

Re: [petsc-users] Fieldsplit PC null pointer error on getksp

2015-07-14 Thread Barry Smith 

> On Jul 14, 2015, at 11:37 AM, Sander Land  wrote:
> 
> Hi Matt,
> 
> Doing things to the PC like this has worked for me for a long time. It just 
> happened to be formerly KSPPREONLY/PCLU. Setting things like GMRES/PCNONE 
> works as well with no errors, though it converges far too slowly.
> 
> The actual preconditioner I am trying to form is one from recent literature 
> and uses a slightly modified Schur complement split embedded in another Schur 
> complement split with multiple solves of the inner system and very specific 
> choices for sub-preconditioners. I don't think command line options will work 
> very well here.  
> 
> I've since gone the way of just using PCSHELL and setting things up more 
> explicitly. This seems to work, but i'm not quite sure how to set up 
> boomeramg for one of the sub-problems.
> 
> boomeramg is used on the matrix A derived from:
> MatGetSubMatrix(J,is_x ,is_x ,MAT_INITIAL_MATRIX,&ctxp->A);
> The degrees of freedom for J are xxxyyyzzzpw, with the size of xxxyyyzzz 
> around 20k, p around 1k, and w 8 for my 'small' test problem.
> 
> I can see that HYPRE_BoomerAMGSetNumFunctions takes its value from the matrix 
> block size. This is 1 for J as it doesn't have a consistent block size. 
> However, the number of functions for boomeramg should be 3.
> Changing the block size is not allowed by petsc, and there doesn't seem to be 
> another way to reach HYPRE_BoomerAMGSetNumFunctions.
> What is the best way to handle this, and does it matter that the dofs are 
> ordererd xxxyyyzzz rather than xyzxyzxyz ?

   I don't think BoomerAMG supports xxxyyyzzz, I think it will just get very 
confused trying to do algebraic multigrid with this form. Read up on it but my 
guess is you need to rewrite the code that prepares the system to do the 
interlacing.

  Barry

> 
> 
> Thanks,
> Sander
> 
> 
> 
> On Fri, Jul 10, 2015 at 7:44 PM, Matthew Knepley  wrote:
> On Tue, Jun 30, 2015 at 3:10 PM, Sander Land  wrote:
> This gives
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Matrix must be set first
> 
> The PC is from a SNESGetKSP, and I have called SNESSetJacobian before this.
> 
> If this is embedded in a SNES, it is very messy to yank it out and do thing 
> manually. However, if you
> set the matrix in the PC manually, it might work.
> 
> However, I would encourage you to try an alternate scheme and see if you 
> think it is easier. We have
> started to use the DM object to hold this kind of information. For 
> FieldSplit, I think it is easier to tell
> a DMShell about your field division, and then have it pass this on to 
> PCFIELDSPLIT. So you would
> 
>   a) Create a DMSHELL
> 
>   b) Create a PetscSection and call DMSetDefaultSection()
> 
>   c) Call SNESSetDM()
> 
>   d) use command line options to configure the split -pc_fieldsplit_0_fields 
> 0 -pc_fieldsplit_1_fields 1
> 
> Then everything should work, and can work in any embedded context. Does this 
> make sense?
> 
>   Thanks,
> 
> Matt
> 
> Thanks,
> Sander
> 
> On Tue, Jun 30, 2015 at 2:09 PM, Matthew Knepley  wrote:
> On Tue, Jun 30, 2015 at 11:43 AM, Sander Land  wrote:
> I am trying to use the Schur complement preconditioner in petsc, but am 
> encountering a null argument error calling PCFieldSplitGetSubKSP.
> This only happens on PC_COMPOSITE_SCHUR, the multiplicative/additive options 
> do return a KSP array.
> 
> Error message:
> 
> [0]PETSC ERROR: - Error Message 
> --
> [0]PETSC ERROR: Null argument, when expecting valid pointer
> [0]PETSC ERROR: Null Object: Parameter # 1
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
> trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 
> [0]PETSC ERROR: ./elecmech on a arch-linux2-cxx-debug
> [0]PETSC ERROR: Configure options --with-shared-libraries=1 
> --with-debugging=1 --download-openmpi=1 --with-clanguage=c++ 
> --download-fblaslapack=1 --download-scalapack=1 --download-blacs=1 
> --download-suitesparse=1 --download-pastix=1 --download-superlu=1 
> --dowload-essl=1 --download-ptscotch=1 --download-mumps=1 --download-lusol=1
> [0]PETSC ERROR: #1 MatSchurComplementGetKSP() line 317 in 
> petsc-3.6/src/ksp/ksp/utils/schurm.c
> [0]PETSC ERROR: #2 PCFieldSplitGetSubKSP_FieldSplit_Schur() line 1264 in 
> petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> [0]PETSC ERROR: #3 PCFieldSplitGetSubKSP() line 1668 in 
> petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> 
> Code snippet:
> 
> ISCreateGeneral(PETSC_COMM_SELF,xp_dofs,&ii[0],PETSC_COPY_VALUES,&is_xp);
> ISCreateGeneral(PETSC_COMM_SELF,all_dofs - 
> xp_dofs,&ii[xp_dofs],PETSC_COPY_VALUES,&is_wk);
> 
> PCSetType(pc,PCFIELDSPLIT);
> PCFieldSplitSetType(pc,PC_COMPOSITE_SCHUR);
> PCFieldSplitSetIS(pc,"xp",is_xp);
> PCFieldSplitSetIS(pc,"wk",is_wk);
> int n;
> KSP* ksps; 
> PC   subpc;
> PCFieldSplitGetSubKSP(pc,&n,&ksps);
> 
> 
> ii here is simply an array with ii[i] = i

Re: [petsc-users] How to disable KSP message

2015-07-14 Thread Jin, Shuangshuang 
Yes, found it in GridPACK's math module Petsc linear solver part.

Thanks,
Shuangshuang

From: Abhyankar, Shrirang G. [mailto:abhy...@anl.gov]
Sent: Tuesday, July 14, 2015 10:00 AM
To: Jin, Shuangshuang; petsc-users@mcs.anl.gov
Cc: Palmer, Bruce J
Subject: Re: [petsc-users] How to disable KSP message

This seems to be a result of some print statement in GridPACK, not PETSc.

Shri

From: , Shuangshuang 
mailto:shuangshuang@pnnl.gov>>
Date: Tuesday, July 14, 2015 at 10:55 AM
To: "petsc-users@mcs.anl.gov" 
mailto:petsc-users@mcs.anl.gov>>
Subject: [petsc-users] How to disable KSP message

Hi, does anyone know what is the command line option to disable the following 
printout message about KSP convergence?

0: PETSc KSP converged after 1 iterations, reason: 3
0: PETSc KSP converged after 0 iterations, reason: 3
0: PETSc KSP converged after 1 iterations, reason: 3
0: PETSc KSP converged after 0 iterations, reason: 3
0: PETSc KSP converged after 1 iterations, reason: 3

Thanks,
Shuangshuang

Re: [petsc-users] How to disable KSP message

2015-07-14 Thread Abhyankar, Shrirang G. 
This seems to be a result of some print statement in GridPACK, not PETSc.

Shri

From: , Shuangshuang 
mailto:shuangshuang@pnnl.gov>>
Date: Tuesday, July 14, 2015 at 10:55 AM
To: "petsc-users@mcs.anl.gov" 
mailto:petsc-users@mcs.anl.gov>>
Subject: [petsc-users] How to disable KSP message

Hi, does anyone know what is the command line option to disable the following 
printout message about KSP convergence?

0: PETSc KSP converged after 1 iterations, reason: 3
0: PETSc KSP converged after 0 iterations, reason: 3
0: PETSc KSP converged after 1 iterations, reason: 3
0: PETSc KSP converged after 0 iterations, reason: 3
0: PETSc KSP converged after 1 iterations, reason: 3

Thanks,
Shuangshuang

Re: [petsc-users] Fieldsplit PC null pointer error on getksp

2015-07-14 Thread Matthew Knepley 
On Tue, Jul 14, 2015 at 11:37 AM, Sander Land  wrote:

> Hi Matt,
>
> Doing things to the PC like this has worked for me for a long time. It
> just happened to be formerly KSPPREONLY/PCLU. Setting things like
> GMRES/PCNONE works as well with no errors, though it converges far too
> slowly.
>
> The actual preconditioner I am trying to form is one from recent
> literature and uses a slightly modified Schur complement split embedded in
> another Schur complement split with multiple solves of the inner system and
> very specific choices for sub-preconditioners. I don't think command line
> options will work very well here.
>
> I've since gone the way of just using PCSHELL and setting things up more
> explicitly. This seems to work, but i'm not quite sure how to set up
> boomeramg for one of the sub-problems.
>
> boomeramg is used on the matrix A derived from:
> MatGetSubMatrix(J,is_x ,is_x ,MAT_INITIAL_MATRIX,&ctxp->A);
> The degrees of freedom for J are xxxyyyzzzpw, with the size of xxxyyyzzz
> around 20k, p around 1k, and w 8 for my 'small' test problem.
>
> I can see that HYPRE_BoomerAMGSetNumFunctions takes its value from the
> matrix block size. This is 1 for J as it doesn't have a consistent block
> size. However, the number of functions for boomeramg should be 3.
> Changing the block size is not allowed by petsc, and there doesn't seem to
> be another way to reach HYPRE_BoomerAMGSetNumFunctions.
> What is the best way to handle this, and does it matter that the dofs are
> ordererd xxxyyyzzz rather than xyzxyzxyz ?
>

If you want to explicitly setup the near null space, I recommend using an
agglomeration multigrid, which can
take that in, like ML or GAMG, and using MatSetNearNullSpace(), which
functions just like MatSetNullSpace().

  Thanks,

 Matt


> Thanks,
> Sander
>
>
>
> On Fri, Jul 10, 2015 at 7:44 PM, Matthew Knepley 
> wrote:
>
>> On Tue, Jun 30, 2015 at 3:10 PM, Sander Land 
>> wrote:
>>
>>> This gives
>>> [0]PETSC ERROR: Object is in wrong state
>>> [0]PETSC ERROR: Matrix must be set first
>>>
>>> The PC is from a SNESGetKSP, and I have called SNESSetJacobian before
>>> this.
>>>
>>
>> If this is embedded in a SNES, it is very messy to yank it out and do
>> thing manually. However, if you
>> set the matrix in the PC manually, it might work.
>>
>> However, I would encourage you to try an alternate scheme and see if you
>> think it is easier. We have
>> started to use the DM object to hold this kind of information. For
>> FieldSplit, I think it is easier to tell
>> a DMShell about your field division, and then have it pass this on to
>> PCFIELDSPLIT. So you would
>>
>>   a) Create a DMSHELL
>>
>>   b) Create a PetscSection and call DMSetDefaultSection()
>>
>>   c) Call SNESSetDM()
>>
>>   d) use command line options to configure the split
>> -pc_fieldsplit_0_fields 0 -pc_fieldsplit_1_fields 1
>>
>> Then everything should work, and can work in any embedded context. Does
>> this make sense?
>>
>>   Thanks,
>>
>> Matt
>>
>> Thanks,
>>> Sander
>>>
>>> On Tue, Jun 30, 2015 at 2:09 PM, Matthew Knepley 
>>> wrote:
>>>
 On Tue, Jun 30, 2015 at 11:43 AM, Sander Land 
 wrote:

> I am trying to use the Schur complement preconditioner in petsc, but
> am encountering a null argument error calling PCFieldSplitGetSubKSP.
> This only happens on PC_COMPOSITE_SCHUR, the multiplicative/additive
> options do return a KSP array.
>
> Error message:
>
> [0]PETSC ERROR: - Error Message
> --
> [0]PETSC ERROR: Null argument, when expecting valid pointer
> [0]PETSC ERROR: Null Object: Parameter # 1
> [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
> shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015
> [0]PETSC ERROR: ./elecmech on a arch-linux2-cxx-debug
> [0]PETSC ERROR: Configure options --with-shared-libraries=1
> --with-debugging=1 --download-openmpi=1 --with-clanguage=c++
> --download-fblaslapack=1 --download-scalapack=1 --download-blacs=1
> --download-suitesparse=1 --download-pastix=1 --download-superlu=1
> --dowload-essl=1 --download-ptscotch=1 --download-mumps=1 
> --download-lusol=1
> [0]PETSC ERROR: #1 MatSchurComplementGetKSP() line 317 in
> petsc-3.6/src/ksp/ksp/utils/schurm.c
> [0]PETSC ERROR: #2 PCFieldSplitGetSubKSP_FieldSplit_Schur() line 1264
> in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> [0]PETSC ERROR: #3 PCFieldSplitGetSubKSP() line 1668 in
> petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c
>
> Code snippet:
>
>
> ISCreateGeneral(PETSC_COMM_SELF,xp_dofs,&ii[0],PETSC_COPY_VALUES,&is_xp);
> ISCreateGeneral(PETSC_COMM_SELF,all_dofs -
> xp_dofs,&ii[xp_dofs],PETSC_COPY_VALUES,&is_wk);
>
> PCSetType(pc,PCFIELDSPLIT);
> PCFieldSplitSetType(pc,PC_COMPOSITE_SCHUR);
> PCFieldSplitSetIS(pc,

Re: [petsc-users] How to disable KSP message

2015-07-14 Thread Matthew Knepley 
I believe this comes from your code rather than PETSc.

   Matt

On Tue, Jul 14, 2015 at 10:55 AM, Jin, Shuangshuang <
shuangshuang@pnnl.gov> wrote:

>  Hi, does anyone know what is the command line option to disable the
> following printout message about KSP convergence?
>
>
>
> 0: PETSc KSP converged after 1 iterations, reason: 3
>
> 0: PETSc KSP converged after 0 iterations, reason: 3
>
> 0: PETSc KSP converged after 1 iterations, reason: 3
>
> 0: PETSc KSP converged after 0 iterations, reason: 3
>
> 0: PETSc KSP converged after 1 iterations, reason: 3
>
>
>
> Thanks,
>
> Shuangshuang
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

Re: [petsc-users] Fieldsplit PC null pointer error on getksp

2015-07-14 Thread Sander Land 
Hi Matt,

Doing things to the PC like this has worked for me for a long time. It just
happened to be formerly KSPPREONLY/PCLU. Setting things like GMRES/PCNONE
works as well with no errors, though it converges far too slowly.

The actual preconditioner I am trying to form is one from recent literature
and uses a slightly modified Schur complement split embedded in another
Schur complement split with multiple solves of the inner system and very
specific choices for sub-preconditioners. I don't think command line
options will work very well here.

I've since gone the way of just using PCSHELL and setting things up more
explicitly. This seems to work, but i'm not quite sure how to set up
boomeramg for one of the sub-problems.

boomeramg is used on the matrix A derived from:
MatGetSubMatrix(J,is_x ,is_x ,MAT_INITIAL_MATRIX,&ctxp->A);
The degrees of freedom for J are xxxyyyzzzpw, with the size of xxxyyyzzz
around 20k, p around 1k, and w 8 for my 'small' test problem.

I can see that HYPRE_BoomerAMGSetNumFunctions takes its value from the
matrix block size. This is 1 for J as it doesn't have a consistent block
size. However, the number of functions for boomeramg should be 3.
Changing the block size is not allowed by petsc, and there doesn't seem to
be another way to reach HYPRE_BoomerAMGSetNumFunctions.
What is the best way to handle this, and does it matter that the dofs are
ordererd xxxyyyzzz rather than xyzxyzxyz ?


Thanks,
Sander



On Fri, Jul 10, 2015 at 7:44 PM, Matthew Knepley  wrote:

> On Tue, Jun 30, 2015 at 3:10 PM, Sander Land 
> wrote:
>
>> This gives
>> [0]PETSC ERROR: Object is in wrong state
>> [0]PETSC ERROR: Matrix must be set first
>>
>> The PC is from a SNESGetKSP, and I have called SNESSetJacobian before
>> this.
>>
>
> If this is embedded in a SNES, it is very messy to yank it out and do
> thing manually. However, if you
> set the matrix in the PC manually, it might work.
>
> However, I would encourage you to try an alternate scheme and see if you
> think it is easier. We have
> started to use the DM object to hold this kind of information. For
> FieldSplit, I think it is easier to tell
> a DMShell about your field division, and then have it pass this on to
> PCFIELDSPLIT. So you would
>
>   a) Create a DMSHELL
>
>   b) Create a PetscSection and call DMSetDefaultSection()
>
>   c) Call SNESSetDM()
>
>   d) use command line options to configure the split
> -pc_fieldsplit_0_fields 0 -pc_fieldsplit_1_fields 1
>
> Then everything should work, and can work in any embedded context. Does
> this make sense?
>
>   Thanks,
>
> Matt
>
> Thanks,
>> Sander
>>
>> On Tue, Jun 30, 2015 at 2:09 PM, Matthew Knepley 
>> wrote:
>>
>>> On Tue, Jun 30, 2015 at 11:43 AM, Sander Land 
>>> wrote:
>>>
 I am trying to use the Schur complement preconditioner in petsc, but am
 encountering a null argument error calling PCFieldSplitGetSubKSP.
 This only happens on PC_COMPOSITE_SCHUR, the multiplicative/additive
 options do return a KSP array.

 Error message:

 [0]PETSC ERROR: - Error Message
 --
 [0]PETSC ERROR: Null argument, when expecting valid pointer
 [0]PETSC ERROR: Null Object: Parameter # 1
 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
 for trouble shooting.
 [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015
 [0]PETSC ERROR: ./elecmech on a arch-linux2-cxx-debug
 [0]PETSC ERROR: Configure options --with-shared-libraries=1
 --with-debugging=1 --download-openmpi=1 --with-clanguage=c++
 --download-fblaslapack=1 --download-scalapack=1 --download-blacs=1
 --download-suitesparse=1 --download-pastix=1 --download-superlu=1
 --dowload-essl=1 --download-ptscotch=1 --download-mumps=1 
 --download-lusol=1
 [0]PETSC ERROR: #1 MatSchurComplementGetKSP() line 317 in
 petsc-3.6/src/ksp/ksp/utils/schurm.c
 [0]PETSC ERROR: #2 PCFieldSplitGetSubKSP_FieldSplit_Schur() line 1264
 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c
 [0]PETSC ERROR: #3 PCFieldSplitGetSubKSP() line 1668 in
 petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c

 Code snippet:


 ISCreateGeneral(PETSC_COMM_SELF,xp_dofs,&ii[0],PETSC_COPY_VALUES,&is_xp);
 ISCreateGeneral(PETSC_COMM_SELF,all_dofs -
 xp_dofs,&ii[xp_dofs],PETSC_COPY_VALUES,&is_wk);

 PCSetType(pc,PCFIELDSPLIT);
 PCFieldSplitSetType(pc,PC_COMPOSITE_SCHUR);
 PCFieldSplitSetIS(pc,"xp",is_xp);
 PCFieldSplitSetIS(pc,"wk",is_wk);
 int n;
 KSP* ksps;
 PC   subpc;
 PCFieldSplitGetSubKSP(pc,&n,&ksps);


 ii here is simply an array with ii[i] = i. There is probably a better
 way to simply indicate two blocks of different size, but I couldn't find 
 it.

>>>
>>> It looks like the PC is not setup. Can you try calling PCSetUp(pc)
>>> before GetSubKSP()?
>>>
>>>   Thanks,
>>>
>>>  Matt
>>>

[petsc-users] How to disable KSP message

2015-07-14 Thread Jin, Shuangshuang 
Hi, does anyone know what is the command line option to disable the following 
printout message about KSP convergence?

0: PETSc KSP converged after 1 iterations, reason: 3
0: PETSc KSP converged after 0 iterations, reason: 3
0: PETSc KSP converged after 1 iterations, reason: 3
0: PETSc KSP converged after 0 iterations, reason: 3
0: PETSc KSP converged after 1 iterations, reason: 3

Thanks,
Shuangshuang

Re: [petsc-users] PETSC MPI error on PetscGatherMessageLengths2

2015-07-14 Thread Hong 
Hui :
It is impossible to know what causes the problem from these error messages.
I notice that you use Petsc-3.5.2.
I would suggest
1. update to the latest petsc-3.6;
2. if problem remains, send us your matrices C and B in petsc binary
format, we can investigate. With matrix files, one can test
MatTransposeMatMult()
using
petsc/src/mat/examples/tests/ex94.c

Hong


 I'm running some code written by myself, using PETSC with MPI. It runs
> fine with less than or equal to 12 cores. However, if I ran it with 16
> cores, it gives me an error. By looking at the error message, it seems that
> the error happens at a line where I have something like this:
>
>   ierr = MatTransposeMatMult(C,B,MAT_INITIAL_MATRIX,1,&A);CHKERRQ(ierr);
>
>  And I have attached the error message here. Does anyone sort of having
> an idea of what might be causing the problem here? Or is there any hint of
> what might be a direction for looking into this bug?
>
>   [hus003@tscc-login2 test]$ cat /home/hus003/output/errFluid3d | grep
> '\[4\]'
>
> [4]PETSC ERROR:
> 
>
> [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
>
> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>
> [4]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC
> ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> memory corruption errors
>
> [4]PETSC ERROR: likely location of problem given in stack below
>
> [4]PETSC ERROR: -  Stack Frames
> 
>
> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
>
> [4]PETSC ERROR:   INSTEAD the line number of the start of the function
>
> [4]PETSC ERROR:   is given.
>
> [4]PETSC ERROR: [4] PetscGatherMessageLengths2 line 180
> /state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/sys/utils/mpimesg.c
>
> [4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ line 1573
> /state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/mat/impls/aij/mpi/mpimatmatmult.c
>
> [4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 812
> /state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/mat/impls/aij/mpi/mpimatmatmult.c
>
> [4]PETSC ERROR: [4] MatTransposeMatMult line 8945
> /state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/mat/interface/matrix.c
>
> [4]PETSC ERROR: [4] Schur_Prec line 2885
> /home/hus003/DISM/LSM_DISM3D/test/../Stokes3D.h
>
> [4]PETSC ERROR: [4] StokesSetup line 140
> /home/hus003/DISM/LSM_DISM3D/test/../Stokes3D.h
>
> [4]PETSC ERROR: [4] test line 102
> /home/hus003/DISM/LSM_DISM3D/test/test.cpp
>
> [4]PETSC ERROR: - Error Message
> --
>
> [4]PETSC ERROR: Signal received
>
> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
>
> [4]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014
>
> [4]PETSC ERROR: ./test on a intel-openmpi_ib named tscc-gpu-6-5.sdsc.edu
> by hus003 Mon Jul 13 23:57:09 2015
>
> [4]PETSC ERROR: Configure options CFLAGS= FFLAGS= CXXFLAGS=
> --prefix=/opt/petsc/intel/openmpi_ib --with-mpi=1
> --download-pastix=../pastix_5.2.2.12.tar.bz2
> --download-ptscotch=../scotch_6.0.0_esmumps.tar.gz
> --with-blas-lib="-Wl,--start-group
> /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_intel_lp64.a
>
> /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_sequential.a
> /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_core.a
> -Wl,--end-group -lpthread -lm" --with-lapack-lib="-Wl,--start-group
> /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_intel_lp64.a
>
> /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_sequential.a
> /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_core.a
> -Wl,--end-group -lpthread -lm"
> --with-superlu_dist-include=/opt/superlu/intel/openmpi_ib/include
> --with-superlu_dist-lib="-L/opt/superlu/intel/openmpi_ib/lib -lsuperlu"
> --with-parmetis-dir=/opt/parmetis/intel/openmpi_ib
> --with-metis-dir=/opt/parmetis/intel/openmpi_ib
> --with-mpi-dir=/opt/openmpi/intel/ib
> --with-scalapack-dir=/opt/scalapack/intel/openmpi_ib
> --download-mumps=../MUMPS_4.10.0-p3.tar.gz
> --download-blacs=../blacs-dev.tar.gz
> --download-fblaslapack=../fblaslapack-3.4.2.tar.gz --with-pic=1
> --with-shared-libraries=1
>
> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>
>
>
>

[petsc-users] Question about Nested matrices

2015-07-14 Thread Alejandro D Otero 
Hi all,
I am trying to nest to square matrices to form a rectangular one with dims
2n x n (with n the size of the square matrices)
I want the two matrices to have interlaced rows so I passed the following 2
list of indices to MatCreateNest:

[[ 0  2  4  6  8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46
48]
[ 1  3  5  7  9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
49]]

And we get the following error message:
Global sizes (25,25) of nested submatrix (0,0) do not agree with space
defined by index sets (50,25)

The size of the 2 square matrices is 25 x 25.

What am I doing wrong?

Thanks in advance,

Re: [petsc-users] A combined petsc4py, FEniCS, and Firedrake issue

2015-07-14 Thread Justin Chang 
Hi Lawrence,

I just resolved this issue, you're right I did have two petsc4py's
(manually removing the one in /usr/local/... did the trick) and I undid
every installation step i took for Firedrake. Seems the issue was me
installing FEniCS with sudo apt-get install fenics and installing Firedrake
dependencies via sudo pip install ... gave me problems.

Thanks,
Justin

PS - as for the other e-mail, I will get back to you shortly on that



On Tue, Jul 14, 2015 at 3:37 AM, Lawrence Mitchell <
lawrence.mitch...@imperial.ac.uk> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi Justin,
>
> On 14/07/15 09:24, Justin Chang wrote:
> > Hi everyone,
> >
> > As the title says, I have an issue regarding all three. Not sure
> > which mailing to post this to but i figured somebody here may
> > help.
> >
> > I was attempting to install Firedrake on my Ubuntu PC, so far it
> > has not worked yet. However, after my failed attempt, i tried to
> > run some of my old FEniCS programs, and something within the Dolfin
> > library has screwed up. Below is the error:
> >
> > Traceback (most recent call last): File "Idealized_reservoir.py",
> > line 8, in  from dolfin import * File
> > "/usr/lib/python2.7/dist-packages/dolfin/__init__.py", line 16, in
> >  from . import cpp File
> > "/usr/lib/python2.7/dist-packages/dolfin/cpp/__init__.py", line 42,
> > in  exec("from . import %s" % module_name) File "",
> > line 1, in  File
> > "/usr/lib/python2.7/dist-packages/dolfin/cpp/common.py", line 32,
> > in  _common = swig_import_helper() File
> > "/usr/lib/python2.7/dist-packages/dolfin/cpp/common.py", line 28,
> > in swig_import_helper _mod = imp.load_module('_common', fp,
> > pathname, description) File
> > "/usr/local/lib/python2.7/dist-packages/petsc4py/PETSc.py", line 4,
> > in  PETSc._initialize() File "PETSc/PETSc.pyx", line 407,
> > in petsc4py.PETSc._initialize (src/petsc4py.PETSc.c:232527) File
> > "PETSc/PETSc.pyx", line 395, in petsc4py.PETSc.register
> > (src/petsc4py.PETSc.c:232364) File "PETSc/Object.pyx", line 257,
> > in petsc4py.PETSc.PyPetscType_Register
> > (src/petsc4py.PETSc.c:64906) ValueError: key: 0, cannot register:
> > , already registered: None
> >
> > It seems I screwed something up when I attempted to installed
> > petsc4py. The following command was used to install petsc4py per
> > firedrake instructions:
> >
> > sudo pip install
> > git+https://bitbucket.org/mapdes/petsc4py.git@firedrake#egg=petsc4py
> >
> >  I tried uninstalling petsc4py via sudo pip uninstall but it says
> > "Not uninstalling petsc4py at /usr/lib/pymodules/python2.7, owned
> > by OS".
>
> Something funky is going on here, since the petsc4py in the backtrace
> above lives in /usr/local/lib/python2.7/... not
> /usr/lib/pymodules/python2.7
>
> I presume "pip list" doesn't list petsc4py anywhere.
>
>  I
> > reinstalled fenics/dolfin and all of its dependencies but i still
> > get the traceback errors above. I tried to upgrade petsc4py sudo
> > pip install -U --no-deps ... but it says "Can't roll back petsc4py;
> > was not uninstalled".
> >
> > Any idea what I can do here?
>
> This looks a bit like there are two petsc4py versions kicking around.
> Does just importing petsc4py work at all (from petsc4py import PETSc),
> and if so, where does it point to?  (print PETSc.__file__).
>
> Do things work if you manually remove petsc4py from
> /usr/local/lib/python2.7/dist-packages/petsc4py?
>
> Cheers,
>
> Lawrence
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1
>
> iQEcBAEBAgAGBQJVpMpVAAoJECOc1kQ8PEYvE+QH/R8V9n7a/v18eREO4t06h/X1
> jrYyU3AmYEQUaKHUnDyhS0G3ew4jVUWscSzxqqagnQFZLsegm2ubS5+q22tWn52k
> W7ExtrZ1EQl2D2KcivJH0o1cQmmufoV3A3M+VjW4yKhoBWacDK4FabFwu7xWdx5S
> RWUWbYHfsjjiTaVFneuFwNZUhwYs4Eofnx4OBRBhji704rUhcqLZhiJDBFJJrnCA
> bcRMusvU6Bfq1+xtrmLYZnJqN45vUK4tnXdMQUBJCQi3jBwh+yMB2MLTX2qmoQzO
> VvxRS4wC7Cl91tYC7rqh6GzVh5oZNocet+0OQ9+/AwdjDwVa7ppSplMnJAWeQrc=
> =Ksbp
> -END PGP SIGNATURE-
>

Re: [petsc-users] A combined petsc4py, FEniCS, and Firedrake issue

2015-07-14 Thread Lawrence Mitchell 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Justin,

On 14/07/15 09:24, Justin Chang wrote:
> Hi everyone,
> 
> As the title says, I have an issue regarding all three. Not sure
> which mailing to post this to but i figured somebody here may
> help.
> 
> I was attempting to install Firedrake on my Ubuntu PC, so far it
> has not worked yet. However, after my failed attempt, i tried to
> run some of my old FEniCS programs, and something within the Dolfin
> library has screwed up. Below is the error:
> 
> Traceback (most recent call last): File "Idealized_reservoir.py",
> line 8, in  from dolfin import * File
> "/usr/lib/python2.7/dist-packages/dolfin/__init__.py", line 16, in
>  from . import cpp File
> "/usr/lib/python2.7/dist-packages/dolfin/cpp/__init__.py", line 42,
> in  exec("from . import %s" % module_name) File "",
> line 1, in  File
> "/usr/lib/python2.7/dist-packages/dolfin/cpp/common.py", line 32, 
> in  _common = swig_import_helper() File
> "/usr/lib/python2.7/dist-packages/dolfin/cpp/common.py", line 28, 
> in swig_import_helper _mod = imp.load_module('_common', fp,
> pathname, description) File
> "/usr/local/lib/python2.7/dist-packages/petsc4py/PETSc.py", line 4,
> in  PETSc._initialize() File "PETSc/PETSc.pyx", line 407,
> in petsc4py.PETSc._initialize (src/petsc4py.PETSc.c:232527) File
> "PETSc/PETSc.pyx", line 395, in petsc4py.PETSc.register 
> (src/petsc4py.PETSc.c:232364) File "PETSc/Object.pyx", line 257,
> in petsc4py.PETSc.PyPetscType_Register
> (src/petsc4py.PETSc.c:64906) ValueError: key: 0, cannot register:
> , already registered: None
> 
> It seems I screwed something up when I attempted to installed
> petsc4py. The following command was used to install petsc4py per
> firedrake instructions:
> 
> sudo pip install
> git+https://bitbucket.org/mapdes/petsc4py.git@firedrake#egg=petsc4py
>
>  I tried uninstalling petsc4py via sudo pip uninstall but it says
> "Not uninstalling petsc4py at /usr/lib/pymodules/python2.7, owned
> by OS".

Something funky is going on here, since the petsc4py in the backtrace
above lives in /usr/local/lib/python2.7/... not
/usr/lib/pymodules/python2.7

I presume "pip list" doesn't list petsc4py anywhere.

 I
> reinstalled fenics/dolfin and all of its dependencies but i still
> get the traceback errors above. I tried to upgrade petsc4py sudo
> pip install -U --no-deps ... but it says "Can't roll back petsc4py;
> was not uninstalled".
> 
> Any idea what I can do here?

This looks a bit like there are two petsc4py versions kicking around.
Does just importing petsc4py work at all (from petsc4py import PETSc),
and if so, where does it point to?  (print PETSc.__file__).

Do things work if you manually remove petsc4py from
/usr/local/lib/python2.7/dist-packages/petsc4py?

Cheers,

Lawrence
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJVpMpVAAoJECOc1kQ8PEYvE+QH/R8V9n7a/v18eREO4t06h/X1
jrYyU3AmYEQUaKHUnDyhS0G3ew4jVUWscSzxqqagnQFZLsegm2ubS5+q22tWn52k
W7ExtrZ1EQl2D2KcivJH0o1cQmmufoV3A3M+VjW4yKhoBWacDK4FabFwu7xWdx5S
RWUWbYHfsjjiTaVFneuFwNZUhwYs4Eofnx4OBRBhji704rUhcqLZhiJDBFJJrnCA
bcRMusvU6Bfq1+xtrmLYZnJqN45vUK4tnXdMQUBJCQi3jBwh+yMB2MLTX2qmoQzO
VvxRS4wC7Cl91tYC7rqh6GzVh5oZNocet+0OQ9+/AwdjDwVa7ppSplMnJAWeQrc=
=Ksbp
-END PGP SIGNATURE-

[petsc-users] A combined petsc4py, FEniCS, and Firedrake issue

2015-07-14 Thread Justin Chang 
Hi everyone,

As the title says, I have an issue regarding all three. Not sure which
mailing to post this to but i figured somebody here may help.

I was attempting to install Firedrake on my Ubuntu PC, so far it has not
worked yet. However, after my failed attempt, i tried to run some of my old
FEniCS programs, and something within the Dolfin library has screwed up.
Below is the error:

Traceback (most recent call last):
  File "Idealized_reservoir.py", line 8, in 
from dolfin import *
  File "/usr/lib/python2.7/dist-packages/dolfin/__init__.py", line 16, in

from . import cpp
  File "/usr/lib/python2.7/dist-packages/dolfin/cpp/__init__.py", line 42,
in 
exec("from . import %s" % module_name)
  File "", line 1, in 
  File "/usr/lib/python2.7/dist-packages/dolfin/cpp/common.py", line 32, in

_common = swig_import_helper()
  File "/usr/lib/python2.7/dist-packages/dolfin/cpp/common.py", line 28, in
swig_import_helper
_mod = imp.load_module('_common', fp, pathname, description)
  File "/usr/local/lib/python2.7/dist-packages/petsc4py/PETSc.py", line 4,
in 
PETSc._initialize()
  File "PETSc/PETSc.pyx", line 407, in petsc4py.PETSc._initialize
(src/petsc4py.PETSc.c:232527)
  File "PETSc/PETSc.pyx", line 395, in petsc4py.PETSc.register
(src/petsc4py.PETSc.c:232364)
  File "PETSc/Object.pyx", line 257, in petsc4py.PETSc.PyPetscType_Register
(src/petsc4py.PETSc.c:64906)
ValueError: key: 0, cannot register: ,
already registered: None

It seems I screwed something up when I attempted to installed petsc4py. The
following command was used to install petsc4py per firedrake instructions:

sudo pip install
git+https://bitbucket.org/mapdes/petsc4py.git@firedrake#egg=petsc4py

I tried uninstalling petsc4py via sudo pip uninstall but it says "Not
uninstalling petsc4py at /usr/lib/pymodules/python2.7, owned by OS". I
reinstalled fenics/dolfin and all of its dependencies but i still get the
traceback errors above. I tried to upgrade petsc4py sudo pip install -U
--no-deps ... but it says "Can't roll back petsc4py; was not uninstalled".

Any idea what I can do here?

Thanks,
Justin

Re: [petsc-users] Double free in KSPDestroy

2015-07-14 Thread Florian Lindner 
Hello,

Am Montag, 13. Juli 2015, 12:26:21 schrieb Barry Smith:
> 
>   Run under valgrind first, see if it gives any more details about the memory 
> issue http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

I tried running it like that:

valgrind --tool=memcheck ./pmpi A -malloc off 

(pmpi is my application, no mpirun)

but it reported no errors at all.
 
>   Can you send the code that produces this problem?

I was not able to isolate that problem, you can of course have a look at our 
application:

git clone g...@github.com:precice/precice.git
MPI_CXX="clang++" scons -j 4 boost_inst=on python=off petsc=on mpi=on 
compiler=mpic++ build=debug

The test client:
git clone g...@github.com:floli/prempi.git
you need to adapt line 5 in SConstruct: preciceRoot
scons

Take one terminal run ./pmpi A, another to run ./pmpi B

Thanks for taking a look! Mail me if any problem with the build occurs.

Florian

> 
> > On Jul 13, 2015, at 10:56 AM, Florian Lindner  wrote:
> > 
> > Hello,
> > 
> > our petsc application suffers from a memory error (double free or 
> > corruption).
> > 
> > Situation is a like that:
> > 
> > A KSP is private member of a C++ class. In its constructor I call 
> > KSPCreate. Inbetween it may haben that I call KSPREset. In the class' 
> > destructor I call KSPDestroy. That's where the memory error appears:
> > 
> > gdb backtrace:
> > 
> > 
> > #4  0x7490b8db in _int_free () from /usr/lib/libc.so.6
> > #5  0x76188c9c in PetscFreeAlign (ptr=0xfcd990, line=258, 
> > func=0x7753c4c8 <__func__.20304> "KSPReset_GMRES", file=0x7753b8b0 
> > "/home/florian/software/petsc/src/ksp/ksp/impls/gmres/gmres.c")
> >at /home/florian/software/petsc/src/sys/memory/mal.c:72
> > #6  0x76ff6cdc in KSPReset_GMRES (ksp=0xf48470) at 
> > /home/florian/software/petsc/src/ksp/ksp/impls/gmres/gmres.c:258
> > #7  0x770ad804 in KSPReset (ksp=0xf48470) at 
> > /home/florian/software/petsc/src/ksp/ksp/interface/itfunc.c:885
> > #8  0x770ae2e8 in KSPDestroy (ksp=0xeb89d8) at 
> > /home/florian/software/petsc/src/ksp/ksp/interface/itfunc.c:933
> > 
> > #9  0x00599b24 in 
> > precice::mapping::PetRadialBasisFctMapping::~PetRadialBasisFctMapping
> >  (this=0xeb8960) at src/mapping/PetRadialBasisFctMapping.hpp:148
> > #10 0x00599bc9 in 
> > precice::mapping::PetRadialBasisFctMapping::~PetRadialBasisFctMapping
> >  (this=0xeb8960) at src/mapping/PetRadialBasisFctMapping.hpp:146
> > 
> > Complete backtrace at http://pastebin.com/ASjibeNF
> > 
> > 
> > Could it be a problem it objects set by KSPSetOperators are destroyed 
> > afterwards? I don't think so, since KSPReset is called before.
> > 
> > I've wrapped a class (just a bunch of helper function, no encapsulating 
> > wrapper) round Mat and Vec objects. Nothing fancy, the ctor calls 
> > MatCreate, the dtor MatDestroy, you can have a look at 
> > https://github.com/precice/precice/blob/develop/src/mapping/petnum.cpp / 
> > .hpp.
> > 
> > These objects are also members of the same class like KSP, so their dtor is 
> > called after KSPDestroy.
> > 
> > What could cause the memory corruption here?
> > 
> > Thanks a lot,
> > Florian
>

[petsc-users] PETSC MPI error on PetscGatherMessageLengths2

2015-07-14 Thread Sun, Hui 
I'm running some code written by myself, using PETSC with MPI. It runs fine 
with less than or equal to 12 cores. However, if I ran it with 16 cores, it 
gives me an error. By looking at the error message, it seems that the error 
happens at a line where I have something like this:


ierr = MatTransposeMatMult(C,B,MAT_INITIAL_MATRIX,1,&A);CHKERRQ(ierr);

And I have attached the error message here. Does anyone sort of having an idea 
of what might be causing the problem here? Or is there any hint of what might 
be a direction for looking into this bug?


[hus003@tscc-login2 test]$ cat /home/hus003/output/errFluid3d | grep '\[4\]'

[4]PETSC ERROR: 


[4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch 
system) has told this process to end

[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger

[4]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or 
try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory 
corruption errors

[4]PETSC ERROR: likely location of problem given in stack below

[4]PETSC ERROR: -  Stack Frames 


[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,

[4]PETSC ERROR:   INSTEAD the line number of the start of the function

[4]PETSC ERROR:   is given.

[4]PETSC ERROR: [4] PetscGatherMessageLengths2 line 180 
/state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/sys/utils/mpimesg.c

[4]PETSC ERROR: [4] MatTransposeMatMultSymbolic_MPIAIJ_MPIAIJ line 1573 
/state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/mat/impls/aij/mpi/mpimatmatmult.c

[4]PETSC ERROR: [4] MatTransposeMatMult_MPIAIJ_MPIAIJ line 812 
/state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/mat/impls/aij/mpi/mpimatmatmult.c

[4]PETSC ERROR: [4] MatTransposeMatMult line 8945 
/state/partition1/jjh/All-Rolls/math-roll/BUILD/sdsc-petsc_intel_openmpi_ib-3.5.2/petsc-3.5.2/src/mat/interface/matrix.c

[4]PETSC ERROR: [4] Schur_Prec line 2885 
/home/hus003/DISM/LSM_DISM3D/test/../Stokes3D.h

[4]PETSC ERROR: [4] StokesSetup line 140 
/home/hus003/DISM/LSM_DISM3D/test/../Stokes3D.h

[4]PETSC ERROR: [4] test line 102 /home/hus003/DISM/LSM_DISM3D/test/test.cpp

[4]PETSC ERROR: - Error Message 
--

[4]PETSC ERROR: Signal received

[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
trouble shooting.

[4]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014

[4]PETSC ERROR: ./test on a intel-openmpi_ib named tscc-gpu-6-5.sdsc.edu by 
hus003 Mon Jul 13 23:57:09 2015

[4]PETSC ERROR: Configure options CFLAGS= FFLAGS= CXXFLAGS= 
--prefix=/opt/petsc/intel/openmpi_ib --with-mpi=1 
--download-pastix=../pastix_5.2.2.12.tar.bz2 
--download-ptscotch=../scotch_6.0.0_esmumps.tar.gz 
--with-blas-lib="-Wl,--start-group 
/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_intel_lp64.a   
 /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_sequential.a 
/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_core.a
-Wl,--end-group -lpthread -lm" --with-lapack-lib="-Wl,--start-group 
/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_intel_lp64.a   
 /opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_sequential.a 
/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64/libmkl_core.a
-Wl,--end-group -lpthread -lm" 
--with-superlu_dist-include=/opt/superlu/intel/openmpi_ib/include 
--with-superlu_dist-lib="-L/opt/superlu/intel/openmpi_ib/lib -lsuperlu" 
--with-parmetis-dir=/opt/parmetis/intel/openmpi_ib 
--with-metis-dir=/opt/parmetis/intel/openmpi_ib 
--with-mpi-dir=/opt/openmpi/intel/ib 
--with-scalapack-dir=/opt/scalapack/intel/openmpi_ib 
--download-mumps=../MUMPS_4.10.0-p3.tar.gz --download-blacs=../blacs-dev.tar.gz 
--download-fblaslapack=../fblaslapack-3.4.2.tar.gz --with-pic=1 
--with-shared-libraries=1

[4]PETSC ERROR: #1 User provided function() line 0 in  unknown file