Re: [petsc-users] 3.16.4 amd blis libflame linking problem

[Anton Popov]

On 04.02.22 22:18, Satish Balay wrote:

On Fri, 4 Feb 2022, Anton Popov wrote:


Am 04.02.2022 um 18:54 schrieb Satish Balay:

On Fri, 4 Feb 2022, Anton Popov wrote:


Thanks Matt, Barry and Satish, for  your suggestions.

The problem was indeed in mismatch between the gfortran library version
(system has libgfortran.so.4 libflame needs libgfortran.so.5)

3.9.4 did not detect this during configure, but only gave error later
during
test.

3.16.4 detected this immediately.

After installing libgfortran.so.5, both PETSc versions install just fine,
however I get the warning mentioned by Satish.

Hm - you could install libgfortran.so.5 without the corresponding compiler?

Yes, apparently it is possible on Ubuntu 18.04.

Presumably it also comes with the corresponding gcc/gfortran [gcc-8, 
gfortran-8?]. Perhaps these compilers would avoid this issue.


/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so

Ah - this is a binary download - not something compiled locally?

Yes it is the latest binary from AMD website.

I wonder if the compiler gave both an error [return code] and a warning
previously [with the missing libgfortran.so.5] - as Barry was suggesting.

It was an error at runtime, configure was just fine.


As you say - you still get that warning [which is expected - as you are
using a different version of gfortran than what libflame.so was built with]

Now since libgfortran.so.5 coexists with libgfortran.so.4 it links against the
proper one, but gives a warning that both versions are available.

I think I will just compile BLIS and LibFLAME from sources on my system to
avoid these problems altogether.

Yeah - generally building everything with the same version of the compiler 
avoids these issues.


Actually this is the fastest way to go and it works: 
--download-f2cblaslapack --download-blis


Are there plans to also support  --download-libflame?

I know that most of performance boost comes from BLIS in this case. But 
maybe LibFLAME would also run a bit faster than a reference LAPACK.


Anton



Satish


Best,

Anton



Satish


Maybe it is indeed worth upgrading everything to the compatible versions.

Best,

Anton


On 04.02.22 18:33, Barry Smith wrote:

On Feb 4, 2022, at 12:27 PM, Satish Balay  wrote:

Probably best if you can use the same version of gfortran to build both
petsc and libflame/blis


/usr/bin/ld: warning: libgfortran.so.5, needed by
/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so, not found (try using
-rpath or -rpath-link):

This is probably an ignore-able warning - but configure defaults to
-Werror
mode here.

 Hmm, if it needs libgfortran.so.5 then it needs it and it cannot be
 ignored since a link cannot succeed. Flame presumably contains a lot
 of
 old Fortran code from Lapack so would normally need the fortran
 libraries.


Wrt forcing link with static libraries - you can try:

LIBS="/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.a
/opt/amd/amd-blis-3.1.0/lib/lp64/libblis.a"

[instead of --with-blas-lib= --with-lapack-lib= options].

Satish


On Fri, 4 Feb 2022, Barry Smith wrote:


Please do

ldd -O /opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so

You may need to list the gfortran library directory of libgfortran.so.5
it
needs to use in LDFLAGS passed to PETSc configure

Barry

Note: Even though you explicitly listed a static library of libflame to
use our configure is goofy and loses that information and wants to link
with the shared version



On Feb 4, 2022, at 12:00 PM, Anton Popov  wrote:



On 04.02.22 17:39, Matthew Knepley wrote:

On Fri, Feb 4, 2022 at 11:35 AM Anton Popov mailto:po...@uni-mainz.de>> wrote:
Hi Satish,

I just discovered that PETSc 3.16.4 fails to link against the latest
AMD
BLIS and LibFLAME libraries on a Linux box.

---
You set a value for --with-blas-lib= and --with-lapack-lib=,
but ['/opt/amd/amd-blis-3.1.0/lib/lp64/libblis.a'] and
['/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.a'] cannot be used
***

My previous experience with 3.9.4 on the same system was fully
successful. Looking in the configure logs (attached) reveals small
difference in the linking compared to 3.9.4

Could you please make a guess what went wrong?

Down in the log I see:

/usr/bin/ld: warning: libgfortran.so.5, needed by
/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so, not found (try using
-rpath or -rpath-link):

Thanks Matt, I'll try.


Did the gfortran library move or get upgraded?

Not at all. I have configured 3.9.4 just now to make a test, and it
perfectly finds all the libraries. So there must be something that
3.16.4
does differently.

Best,

Anton


Thanks,

   Matt

Best regards,

Anton


--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbe

Re: [petsc-users] 3.16.4 amd blis libflame linking problem

[Anton Popov]

Am 04.02.2022 um 18:54 schrieb Satish Balay:

On Fri, 4 Feb 2022, Anton Popov wrote:


Thanks Matt, Barry and Satish, for  your suggestions.

The problem was indeed in mismatch between the gfortran library version
(system has libgfortran.so.4 libflame needs libgfortran.so.5)

3.9.4 did not detect this during configure, but only gave error later during
test.

3.16.4 detected this immediately.

After installing libgfortran.so.5, both PETSc versions install just fine,
however I get the warning mentioned by Satish.

Hm - you could install libgfortran.so.5 without the corresponding compiler?

Yes, apparently it is possible on Ubuntu 18.04.



/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so

Ah - this is a binary download - not something compiled locally?

Yes it is the latest binary from AMD website.


I wonder if the compiler gave both an error [return code] and a warning 
previously [with the missing libgfortran.so.5] - as Barry was suggesting.

It was an error at runtime, configure was just fine.



As you say - you still get that warning [which is expected - as you are using a 
different version of gfortran than what libflame.so was built with]


Now since libgfortran.so.5 coexists with libgfortran.so.4 it links 
against the proper one, but gives a warning that both versions are 
available.


I think I will just compile BLIS and LibFLAME from sources on my system 
to avoid these problems altogether.


Best,

Anton




Satish


Maybe it is indeed worth upgrading everything to the compatible versions.

Best,

Anton


On 04.02.22 18:33, Barry Smith wrote:

On Feb 4, 2022, at 12:27 PM, Satish Balay  wrote:

Probably best if you can use the same version of gfortran to build both
petsc and libflame/blis


/usr/bin/ld: warning: libgfortran.so.5, needed by
/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so, not found (try using
-rpath or -rpath-link):

This is probably an ignore-able warning - but configure defaults to -Werror
mode here.

Hmm, if it needs libgfortran.so.5 then it needs it and it cannot be
ignored since a link cannot succeed. Flame presumably contains a lot of
old Fortran code from Lapack so would normally need the fortran
libraries.


Wrt forcing link with static libraries - you can try:

LIBS="/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.a
/opt/amd/amd-blis-3.1.0/lib/lp64/libblis.a"

[instead of --with-blas-lib= --with-lapack-lib= options].

Satish


On Fri, 4 Feb 2022, Barry Smith wrote:


Please do

   ldd -O /opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so

You may need to list the gfortran library directory of libgfortran.so.5 it
needs to use in LDFLAGS passed to PETSc configure

Barry

Note: Even though you explicitly listed a static library of libflame to
use our configure is goofy and loses that information and wants to link
with the shared version



On Feb 4, 2022, at 12:00 PM, Anton Popov  wrote:



On 04.02.22 17:39, Matthew Knepley wrote:

On Fri, Feb 4, 2022 at 11:35 AM Anton Popov mailto:po...@uni-mainz.de>> wrote:
Hi Satish,

I just discovered that PETSc 3.16.4 fails to link against the latest AMD
BLIS and LibFLAME libraries on a Linux box.

---
You set a value for --with-blas-lib= and --with-lapack-lib=,
but ['/opt/amd/amd-blis-3.1.0/lib/lp64/libblis.a'] and
['/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.a'] cannot be used
***

My previous experience with 3.9.4 on the same system was fully
successful. Looking in the configure logs (attached) reveals small
difference in the linking compared to 3.9.4

Could you please make a guess what went wrong?

Down in the log I see:

/usr/bin/ld: warning: libgfortran.so.5, needed by
/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so, not found (try using
-rpath or -rpath-link):

Thanks Matt, I'll try.


Did the gfortran library move or get upgraded?

Not at all. I have configured 3.9.4 just now to make a test, and it
perfectly finds all the libraries. So there must be something that 3.16.4
does differently.

Best,

Anton


   Thanks,

  Matt

Best regards,

Anton


--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/
<http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] 3.16.4 amd blis libflame linking problem

[Anton Popov]

Thanks Matt, Barry and Satish, for  your suggestions.

The problem was indeed in mismatch between the gfortran library version 
(system has libgfortran.so.4 libflame needs libgfortran.so.5)


3.9.4 did not detect this during configure, but only gave error later 
during test.


3.16.4 detected this immediately.

After installing libgfortran.so.5, both PETSc versions install just 
fine, however I get the warning mentioned by Satish.


Maybe it is indeed worth upgrading everything to the compatible versions.

Best,

Anton


On 04.02.22 18:33, Barry Smith wrote:



On Feb 4, 2022, at 12:27 PM, Satish Balay  wrote:

Probably best if you can use the same version of gfortran to build both petsc 
and libflame/blis


/usr/bin/ld: warning: libgfortran.so.5, needed by 
/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so, not found (try using -rpath 
or -rpath-link):

This is probably an ignore-able warning - but configure defaults to -Werror 
mode here.

   Hmm, if it needs libgfortran.so.5 then it needs it and it cannot be ignored 
since a link cannot succeed. Flame presumably contains a lot of old Fortran 
code from Lapack so would normally need the fortran libraries.


Wrt forcing link with static libraries - you can try:

LIBS="/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.a 
/opt/amd/amd-blis-3.1.0/lib/lp64/libblis.a"

[instead of --with-blas-lib= --with-lapack-lib= options].

Satish


On Fri, 4 Feb 2022, Barry Smith wrote:


Please do

  ldd -O /opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so

You may need to list the gfortran library directory of libgfortran.so.5 it 
needs to use in LDFLAGS passed to PETSc configure

Barry

Note: Even though you explicitly listed a static library of libflame to use our 
configure is goofy and loses that information and wants to link with the shared 
version



On Feb 4, 2022, at 12:00 PM, Anton Popov  wrote:



On 04.02.22 17:39, Matthew Knepley wrote:

On Fri, Feb 4, 2022 at 11:35 AM Anton Popov mailto:po...@uni-mainz.de>> wrote:
Hi Satish,

I just discovered that PETSc 3.16.4 fails to link against the latest AMD
BLIS and LibFLAME libraries on a Linux box.

---
You set a value for --with-blas-lib= and --with-lapack-lib=,
but ['/opt/amd/amd-blis-3.1.0/lib/lp64/libblis.a'] and
['/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.a'] cannot be used
***

My previous experience with 3.9.4 on the same system was fully
successful. Looking in the configure logs (attached) reveals small
difference in the linking compared to 3.9.4

Could you please make a guess what went wrong?

Down in the log I see:

/usr/bin/ld: warning: libgfortran.so.5, needed by 
/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so, not found (try using -rpath 
or -rpath-link):

Thanks Matt, I'll try.


Did the gfortran library move or get upgraded?

Not at all. I have configured 3.9.4 just now to make a test, and it perfectly 
finds all the libraries. So there must be something that 3.16.4 does 
differently.

Best,

Anton


  Thanks,

 Matt

Best regards,

Anton


--
What most experimenters take for granted before they begin their experiments is 
infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] 3.16.4 amd blis libflame linking problem

[Anton Popov]

On 04.02.22 17:39, Matthew Knepley wrote:

On Fri, Feb 4, 2022 at 11:35 AM Anton Popov  wrote:

Hi Satish,

I just discovered that PETSc 3.16.4 fails to link against the
latest AMD
BLIS and LibFLAME libraries on a Linux box.


---
You set a value for --with-blas-lib= and
--with-lapack-lib=,
but ['/opt/amd/amd-blis-3.1.0/lib/lp64/libblis.a'] and
['/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.a'] cannot be used

***

My previous experience with 3.9.4 on the same system was fully
successful. Looking in the configure logs (attached) reveals small
difference in the linking compared to 3.9.4

Could you please make a guess what went wrong?


Down in the log I see:

/usr/bin/ld: warning: libgfortran.so.5, needed by 
/opt/amd/amd-libflame-3.1.0/lib/lp64/libflame.so, not found (try using 
-rpath or -rpath-link):


Thanks Matt, I'll try.



Did the gfortran library move or get upgraded?


Not at all. I have configured 3.9.4 just now to make a test, and it 
perfectly finds all the libraries. So there must be something that 
3.16.4 does differently.


Best,

Anton



  Thanks,

     Matt

Best regards,

Anton



--
What most experimenters take for granted before they begin their 
experiments is infinitely more interesting than any results to which 
their experiments lead.

-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 
<http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Change Amat in FormJacobian

[Anton Popov]

On 14.06.21 15:04, Dave May wrote:


Hi Anton,

Hi Dave,


On Mon, 14 Jun 2021 at 14:47, Anton Popov <mailto:po...@uni-mainz.de>> wrote:


Hi Barry & Matt,

thanks for your quick response. These options were exactly what I
needed and expected:

-pc_mg_galerkin pmat
-pc_use_amat false

I just assumed that it’s a default behavior of the PC object.

So to clarify my case, I don't use nonlinear multigrid. Galerkin
is expected to deal with Pmat only, and it's enough if Amat
implements a matrix-vector product for the Krylov accelerator.

Matt, the reason for switching Amat during the iteration is a
quite common Picard-Newton combination. Jacobian matrix gives
accurate updates close to the solution, but is rather unstable far
form the solution. Picard matrix (approximate Jacobian) is quite
the opposite – it’s kind of stable, but slow. So the idea is to
begin the iteration with Picard matrix, and switch to the Jacobian
later.

If the assembled matrices are used, then the standard SNES
interface is just perfect. I can decide how to fill the matrices.
But I don’t bother with  Jacobian assembly and want to use a
built-in MFFD approximation instead. I did quite a few tests
previously and figured out that MFFD is practically the same as
closed-from matrix-free Jacobian for the later stages of the
iteration. The Picard matrix still does a good job as a
preconditioner. But it is important to start the iteration with
Picard and only change to MFFD later.

Is my workaround with the shell matrix acceptable, or there is a
better solution?


Given what you write, it sounds like you already have a good heuristic 
for when to switch from Picard to Newton,
Hence I think the simplest option is just to use two seperate SNES 
objects - one for performing Picard and one for Newton.



Yes, I considered this option initially. Sometimes it is necessary to 
switch back and forth between the methods, so it becomes a bit messy to 
organize this in the code.


But maybe if Newton fails after Picard, I should just reduce the time 
step and restart, instead of switching back to Picard. Thanks, Dave.


Thanks,

Anton


The stopping condition for the Picard object would encode your 
heuristic to abort earlier when the solution was deemed sufficiently 
accurate.


Thanks,
Dave


Thanks,
Anton

On 13.06.21 20:52, Barry Smith wrote:


  Anton,

  -pc_mg_galerkin pmat

  Though it seems simple, there is some subtly in swapping out
matrices with SNES.

  When using multigrid with SNES there are at least five distinct
uses of the Jacobian operator.

  * Perform matrix-vector product in line search to check
Wolf sufficient decrease convergence criteria
  * Perform the matrix-vector product for the Krylov
accelerator of the system
  * Perform smoothing on the finest level of MG
  * Perform the matrix-vector product needed on the finest
level of MG to compute the residual that will be
restricted to the coarser level of multigrid
  * When using Galerkin products to compute the coarser grid
operator performing the Galerkin matrix triple product


when one swaps out the mat, which of these do they wish to
change? The first two seem to naturally go together as do the
last three. In your case I presume you want to swap for the first
two, but always use pmat for the last three? To achieve this you
will also need -pc_use_amat  false

If you run with -info and -snes_view it will print out some of
the information about which operator it is using for each case,
but probably not all of them.

Note: if the pmat is actually an accurate computation of the
Jacobian then it is likely best not to ever use a matrix-free
product. It is only when pmat is approximated in some specific
way that using the matrix-free product would be useful to insure
the "Newton" method actually computes a Newton step.

Barry




On Jun 13, 2021, at 11:21 AM, Matthew Knepley mailto:knep...@gmail.com>> wrote:

On Sun, Jun 13, 2021 at 10:55 AM Anton Popov mailto:po...@uni-mainz.de>> wrote:

Hi,

I want a simple(?) thing. I want to decide and be able to
assign the
Jacobian matrix (Amat) on the fly within the FormJacobian
function (i.e.
during Newton iteration) to one of the following options:

1) built-in MFFD approximation
2) assembled preconditioner matrix (Pmat)

I have not found this scenario demonstrated in any example,
therefore
I’m asking how to do that.

Currently I do the following:

1) setup Amat as a shell matrix with a MATOP_MULT operation
that simply
retrieves a matrix object form its context and calls MatMult
on it

Re: [petsc-users] Change Amat in FormJacobian

[Anton Popov]

On 14.06.21 15:02, Matthew Knepley wrote:
On Mon, Jun 14, 2021 at 8:45 AM Anton Popov <mailto:po...@uni-mainz.de>> wrote:


Hi Barry & Matt,

thanks for your quick response. These options were exactly what I
needed and expected:

-pc_mg_galerkin pmat
-pc_use_amat false

I just assumed that it’s a default behavior of the PC object.

So to clarify my case, I don't use nonlinear multigrid. Galerkin
is expected to deal with Pmat only, and it's enough if Amat
implements a matrix-vector product for the Krylov accelerator.

Matt, the reason for switching Amat during the iteration is a
quite common Picard-Newton combination. Jacobian matrix gives
accurate updates close to the solution, but is rather unstable far
form the solution. Picard matrix (approximate Jacobian) is quite
the opposite – it’s kind of stable, but slow. So the idea is to
begin the iteration with Picard matrix, and switch to the Jacobian
later.

If the assembled matrices are used, then the standard SNES
interface is just perfect. I can decide how to fill the matrices.
But I don’t bother with  Jacobian assembly and want to use a
built-in MFFD approximation instead. I did quite a few tests
previously and figured out that MFFD is practically the same as
closed-from matrix-free Jacobian for the later stages of the
iteration. The Picard matrix still does a good job as a
preconditioner. But it is important to start the iteration with
Picard and only change to MFFD later.

Is my workaround with the shell matrix acceptable, or there is a
better solution?

It is fine. However, I wonder if you have time to try out a slightly 
different solution. I experimented with this, and it worked as well 
for me as switching between
Picard and Newton. I made a SNESCOMPOSITE with Picard and Newton as 
the two components. I made Newton take a single step, and then 
adjusted the
Picard steps a little.The additiveoptimal variant always converged for 
me, and took the same or less time. I also used left preconditioning 
of Picard by Newton,
but that is more involved, and for my simple problem was not a lot 
better. The advantage I see is that both solves are not plain vanilla, 
and you do not have any

switching logic.

hmmm, this sounds really interesting. I will definitely look into this 
possibility. Thanks Matt.


Thanks,

Anton



  Thanks,

    Matt

Thanks,
Anton

On 13.06.21 20:52, Barry Smith wrote:


  Anton,

  -pc_mg_galerkin pmat

  Though it seems simple, there is some subtly in swapping out
matrices with SNES.

  When using multigrid with SNES there are at least five distinct
uses of the Jacobian operator.

  * Perform matrix-vector product in line search to check
Wolf sufficient decrease convergence criteria
  * Perform the matrix-vector product for the Krylov
accelerator of the system
  * Perform smoothing on the finest level of MG
  * Perform the matrix-vector product needed on the finest
level of MG to compute the residual that will be
restricted to the coarser level of multigrid
  * When using Galerkin products to compute the coarser grid
operator performing the Galerkin matrix triple product


when one swaps out the mat, which of these do they wish to
change? The first two seem to naturally go together as do the
last three. In your case I presume you want to swap for the first
two, but always use pmat for the last three? To achieve this you
will also need -pc_use_amat  false

If you run with -info and -snes_view it will print out some of
the information about which operator it is using for each case,
but probably not all of them.

Note: if the pmat is actually an accurate computation of the
Jacobian then it is likely best not to ever use a matrix-free
product. It is only when pmat is approximated in some specific
way that using the matrix-free product would be useful to insure
the "Newton" method actually computes a Newton step.

Barry




On Jun 13, 2021, at 11:21 AM, Matthew Knepley mailto:knep...@gmail.com>> wrote:

On Sun, Jun 13, 2021 at 10:55 AM Anton Popov mailto:po...@uni-mainz.de>> wrote:

Hi,

I want a simple(?) thing. I want to decide and be able to
assign the
Jacobian matrix (Amat) on the fly within the FormJacobian
function (i.e.
during Newton iteration) to one of the following options:

1) built-in MFFD approximation
2) assembled preconditioner matrix (Pmat)

I have not found this scenario demonstrated in any example,
therefore
I’m asking how to do that.

Currently I do the following:

1) setup Amat as a shell matrix with a MATOP_MULT operation
that simply
retrieves a matrix object

Re: [petsc-users] Change Amat in FormJacobian

[Anton Popov]

Hi Barry & Matt,

thanks for your quick response. These options were exactly what I needed 
and expected:


-pc_mg_galerkin pmat
-pc_use_amat false

I just assumed that it’s a default behavior of the PC object.

So to clarify my case, I don't use nonlinear multigrid. Galerkin is 
expected to deal with Pmat only, and it's enough if Amat implements a 
matrix-vector product for the Krylov accelerator.


Matt, the reason for switching Amat during the iteration is a quite 
common Picard-Newton combination. Jacobian matrix gives accurate updates 
close to the solution, but is rather unstable far form the solution. 
Picard matrix (approximate Jacobian) is quite the opposite – it’s kind 
of stable, but slow. So the idea is to begin the iteration with Picard 
matrix, and switch to the Jacobian later.


If the assembled matrices are used, then the standard SNES interface is 
just perfect. I can decide how to fill the matrices. But I don’t bother 
with  Jacobian assembly and want to use a built-in MFFD approximation 
instead. I did quite a few tests previously and figured out that MFFD is 
practically the same as closed-from matrix-free Jacobian for the later 
stages of the iteration. The Picard matrix still does a good job as a 
preconditioner. But it is important to start the iteration with Picard 
and only change to MFFD later.


Is my workaround with the shell matrix acceptable, or there is a better 
solution?


Thanks,
Anton

On 13.06.21 20:52, Barry Smith wrote:


  Anton,

  -pc_mg_galerkin pmat

  Though it seems simple, there is some subtly in swapping out 
matrices with SNES.


  When using multigrid with SNES there are at least five distinct uses 
of the Jacobian operator.


  * Perform matrix-vector product in line search to check Wolf
sufficient decrease convergence criteria
  * Perform the matrix-vector product for the Krylov accelerator
of the system
  * Perform smoothing on the finest level of MG
  * Perform the matrix-vector product needed on the finest level
of MG to compute the residual that will be restricted to the
coarser level of multigrid
  * When using Galerkin products to compute the coarser grid
operator performing the Galerkin matrix triple product


when one swaps out the mat, which of these do they wish to change? The 
first two seem to naturally go together as do the last three. In your 
case I presume you want to swap for the first two, but always use pmat 
for the last three? To achieve this you will also need -pc_use_amat  false


If you run with -info and -snes_view it will print out some of the 
information about which operator it is using for each case, but 
probably not all of them.


Note: if the pmat is actually an accurate computation of the Jacobian 
then it is likely best not to ever use a matrix-free product. It is 
only when pmat is approximated in some specific way that using the 
matrix-free product would be useful to insure the "Newton" method 
actually computes a Newton step.


Barry



On Jun 13, 2021, at 11:21 AM, Matthew Knepley <mailto:knep...@gmail.com>> wrote:


On Sun, Jun 13, 2021 at 10:55 AM Anton Popov <mailto:po...@uni-mainz.de>> wrote:


Hi,

I want a simple(?) thing. I want to decide and be able to assign the
Jacobian matrix (Amat) on the fly within the FormJacobian
function (i.e.
during Newton iteration) to one of the following options:

1) built-in MFFD approximation
2) assembled preconditioner matrix (Pmat)

I have not found this scenario demonstrated in any example,
therefore
I’m asking how to do that.

Currently I do the following:

1) setup Amat as a shell matrix with a MATOP_MULT operation that
simply
retrieves a matrix object form its context and calls MatMult on it.

2) if I need MFFD, I put a matrix generated with MatCreateSNESMF
in the
Amat context (of course I also call MatMFFDComputeJacobian before
that).

3) if I need Pmat, I simply put Pmat in the Amat context.

4) call MatAssemblyBegin/End on Amat

So far so good.

However, shell Amat and assembled Pmat generate a problem if
Galerkin
multigrid is requested as a preconditioner (I just test on 1 CPU):

[0]PETSC ERROR: MatPtAP requires A, shell, to be compatible with P,
seqaij (Misses composed function MatPtAP_shell_seqaij_C)
[0]PETSC ERROR: #1 MatPtAP()
[0]PETSC ERROR: #2 MatGalerkin()
[0]PETSC ERROR: #3 PCSetUp_MG()
[0]PETSC ERROR: #4 PCSetUp()
[0]PETSC ERROR: #5 KSPSetUp()
[0]PETSC ERROR: #6 KSPSolve()
[0]PETSC ERROR: #7 SNESSolve_NEWTONLS()
[0]PETSC ERROR: #8 SNESSolve()

It seems like PETSc tries to apply Galerkin coarsening to the
shell Amat
matrix instead of the assembled Pmat. Why?

Pmat is internally generated by SNES using a DMDA attached to
SNES, so
it has everything to define Galerkin coarsening. And it actually
works
if 

[petsc-users] Change Amat in FormJacobian

[Anton Popov]

Hi,

I want a simple(?) thing. I want to decide and be able to assign the 
Jacobian matrix (Amat) on the fly within the FormJacobian function (i.e. 
during Newton iteration) to one of the following options:


1) built-in MFFD approximation
2) assembled preconditioner matrix (Pmat)

I have not found this scenario demonstrated in any example, therefore 
I’m asking how to do that.


Currently I do the following:

1) setup Amat as a shell matrix with a MATOP_MULT operation that simply 
retrieves a matrix object form its context and calls MatMult on it.


2) if I need MFFD, I put a matrix generated with MatCreateSNESMF in the 
Amat context (of course I also call MatMFFDComputeJacobian before that).


3) if I need Pmat, I simply put Pmat in the Amat context.

4) call MatAssemblyBegin/End on Amat

So far so good.

However, shell Amat and assembled Pmat generate a problem if Galerkin 
multigrid is requested as a preconditioner (I just test on 1 CPU):


[0]PETSC ERROR: MatPtAP requires A, shell, to be compatible with P, 
seqaij (Misses composed function MatPtAP_shell_seqaij_C)

[0]PETSC ERROR: #1 MatPtAP()
[0]PETSC ERROR: #2 MatGalerkin()
[0]PETSC ERROR: #3 PCSetUp_MG()
[0]PETSC ERROR: #4 PCSetUp()
[0]PETSC ERROR: #5 KSPSetUp()
[0]PETSC ERROR: #6 KSPSolve()
[0]PETSC ERROR: #7 SNESSolve_NEWTONLS()
[0]PETSC ERROR: #8 SNESSolve()

It seems like PETSc tries to apply Galerkin coarsening to the shell Amat 
matrix instead of the assembled Pmat. Why?


Pmat is internally generated by SNES using a DMDA attached to SNES, so 
it has everything to define Galerkin coarsening. And it actually works 
if I just get rid of the shell Amat.


I can get around this problem by wrapping a PC object in a shell matrix 
and pass it as Pmat. This is a rather ugly approach, so I wonder how to 
resolve the situation in a better way, if it is possible.


Thanks.
Anton

Re: [petsc-users] Fwd: Interpolation in staggered grid

[Anton Popov]

Hi Manuel,

you might consider useful taking a look at this code 
https://bitbucket.org/bkaus/lamem, which is a multi-DMDA staggered grid 
implementation.


Also, as Matt pointed out, there is a direct PETSc support for staggered 
grids (DMSTAG), recently contributed by Patrick Sanan.


Thanks,

Anton


On 15.10.2018 23:22, Manuel Valera wrote:


Thanks Matthew, i have made some progress but i am still unsure on how 
to proceed to make the DMDAs work as intended, i will try to lay out 
what i am trying now:


I was able to port the interpolation into the DMDA model but it works 
only in serial runs, and it becomes unstable for more than one 
processor running,


What i do is roughly this:

  * Calculate density at a cell-centered DMDA object, using a local
vector so i can access the ghost rows (indices -1 and max+1)
  * Interpolate into a face-centered DMDA object for velocities, using
also a local vector.

Doing this i get the right results using the same interpolation i used 
for my non-petsc implementation of the model, as long as i use only 
one processor but the doubts i have are:


  * How to use the local vectors properly: Is operating on them the
recommended course in this case?
  * How can i access the ghost indices in the global vector so i can
then communicate GlobalToLocal? would this be a better strategy?
  * I feel is wrong to interpolate a cell-centered based vector into a
face-centered based vector, using the indices of the latter, what
strategy would work best in this case?

I also tried opening a global vector with a different DA layout than 
the one created (opening global density on the velocities DMDA layout) 
and this introduced an error in the GlobalToLocal, LocalToLocal, 
LocalToGlobal communication, something didn't happen if i used the 
local versions of these vectors instead,


Thanks for your help,

Manuel





On Sat, Oct 6, 2018 at 4:45 AM Matthew Knepley > wrote:


On Fri, Oct 5, 2018 at 6:49 PM Manuel Valera mailto:mvaler...@sdsu.edu>> wrote:

Hello,

I'm trying to do a simple variable interpolation, from a cell
center to a face in a staggered grid, my model data management
is done with DMDAs, with two different DMs one for each cell
position,

I already did this task on a fortran only version of the model
using the 4 closest neighbors of the scalars (cell center) to
be interpolated at the velocity site (cell face), i did it
using a loop over the domain, somehow this easy task is not
translating into the DMDA framework,


Its not clear to me what problem you are having. We have done this
before. For example,


https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex30.c.html

I would note that the development version of PETSc now has DMStag
which supports staggered grid discretizations directly.

  Thanks,

    Matt

I'm not sure what I'm doing wrong or not aware of a easier way
Petsc may have for this task, if you could point out an easier
strategy or an example I would be grateful,

Thanks,






-- 
What most experimenters take for granted before they begin their

experiments is infinitely more interesting than any results to
which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

Re: [petsc-users] SuperLU_dist issue in 3.7.4

[Anton Popov]

Hong,

I checked out & compiled your new branch: 
hzhang/fix-superlu_dist-reuse-factornumeric. Unfortunately it did not 
solve the problem.


Sorry.

On 11/21/2016 04:43 AM, Hong wrote:

Anton,
I pushed a fix
https://bitbucket.org/petsc/petsc/commits/28865de08051eb99557d70672c208e14da23c8b1
in branch hzhang/fix-superlu_dist-reuse-factornumeric.
Can you give it a try to see if it works?
I do not have an example which produces your problem.

In your email, you asked "Setting Options.Fact = DOFACT for all 
factorizations is currently impossible via PETSc interface.

The user is expected to choose some kind of reuse model.
If you could add it, I (and other users probably too) would really 
appreciate that."


We do not allow user set superlu' Options.Fact = DOFACT. If user 
changes matrix structure, then user must call
KSPSetOperators() -> call symbolic matrix factorization again, in 
which we set Options.Fact = DOFACT.


I have a conceptual question. How can sparsity (column) permutation be 
reused if it's applied on top of the equilibration (row) permutation? 
Symbolic factorization should be repeated anyway. Does it run in some 
kind of faster update mode in this case? Please correct me if I 
misunderstand something.


I would still appreciate the full factorization even for the same 
pattern without destroying KSP/PC object (just as a custom option).




Hong


Thanks a lot,
Anton

Re: [petsc-users] SuperLU_dist issue in 3.7.4

[Anton Popov]

Thanks, Hong.

I will try as soon as possible and let you know.

Anton

On 11/21/2016 04:43 AM, Hong wrote:

Anton,
I pushed a fix
https://bitbucket.org/petsc/petsc/commits/28865de08051eb99557d70672c208e14da23c8b1
in branch hzhang/fix-superlu_dist-reuse-factornumeric.
Can you give it a try to see if it works?
I do not have an example which produces your problem.

In your email, you asked "Setting Options.Fact = DOFACT for all 
factorizations is currently impossible via PETSc interface.

The user is expected to choose some kind of reuse model.
If you could add it, I (and other users probably too) would really 
appreciate that."


We do not allow user set superlu' Options.Fact = DOFACT. If user 
changes matrix structure, then user must call
KSPSetOperators() -> call symbolic matrix factorization again, in 
which we set Options.Fact = DOFACT.


Hong

[petsc-users] KSP ex65 and DMShell context

[Anton Popov]

Hi all,

KSP ex65 fails with -pc_type mg -pc_mg_galerkin

[0]PETSC ERROR: Nonconforming object sizes
[0]PETSC ERROR: Matrix dimensions are incompatible, 129 != 65

This can be fixed either by adding MatTranspose after 
DMCreateInterpolation in CreateRestriction, or by commenting out 
DMShellSetCreateRestriction.


Replacing DMCreateInterpolation with DMCreateRestriction in 
CreateRestriction also fails with a segfault:


[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: [0] DMCreateRestriction line 1064 
/home/anton/LIB/petsc-3.7.4-debug/src/dm/interface/dm.c


This example uses DMDA objects as context variables, and calls destroy 
on them right after creating coarse DMShell. Due to PETSc reference 
counting mechanism they are not destroyed immediately, so everything 
works without access violations and memory leaks.


I wander how I should destroy my ordinary user contexts. One option is 
keeping a pointer to coarse context in the fine context, and destroying 
hierarchy recursively.


Is there anything better (e.g. something like DMShellSetDestroyContext)?

Thanks,
Anton

Re: [petsc-users] SuperLU_dist issue in 3.7.4

[Anton Popov]

s 1
 -->Test for matrix  168
...
Sherry may tell you why SamePattern_SameRowPerm cause the
difference here.
Best on the above experiments, I would set following as default
'-mat_superlu_diagpivotthresh 0.0' in petsc/superlu interface.
'-mat_superlu_dist_fact SamePattern' in petsc/superlu_dist
interface.

Hong

On Tue, Oct 25, 2016 at 10:38 AM, Hong <hzh...@mcs.anl.gov
<mailto:hzh...@mcs.anl.gov>> wrote:

Anton,
I guess, when you reuse matrix and its symbolic factor
with updated numerical values, superlu_dist requires this
option. I'm cc'ing Sherry to confirm it.

I'll check petsc/superlu-dist interface to set this flag
for this case.

    Hong


On Tue, Oct 25, 2016 at 8:20 AM, Anton Popov
<po...@uni-mainz.de <mailto:po...@uni-mainz.de>> wrote:

Hong,

I get all the problems gone and valgrind-clean output
if I specify this:

-mat_superlu_dist_fact SamePattern_SameRowPerm

What does SamePattern_SameRowPerm actually mean?
Row permutations are for large diagonal, column
permutations are for sparsity, right?
Will it skip subsequent matrix permutations for large
diagonal even if matrix values change significantly?

Surprisingly everything works even with:

-mat_superlu_dist_colperm PARMETIS
-mat_superlu_dist_parsymbfact TRUE

Thanks,
Anton

On 10/24/2016 09:06 PM, Hong wrote:

Anton:


If replacing superlu_dist with mumps, does your
code work?

yes

You may use mumps in your code, or tests different
options for superlu_dist:

-mat_superlu_dist_equil:  Equilibrate matrix (None)
-mat_superlu_dist_rowperm  Row permutation
(choose one of) LargeDiag NATURAL (None)
-mat_superlu_dist_colperm  Column
permutation (choose one of) NATURAL MMD_AT_PLUS_A
MMD_ATA METIS_AT_PLUS_A PARMETIS (None)
-mat_superlu_dist_replacetinypivot:  Replace
tiny pivots (None)
-mat_superlu_dist_parsymbfact:  Parallel
symbolic factorization (None)
-mat_superlu_dist_fact  Sparsity pattern
for repeated matrix factorization (choose one of)
SamePattern SamePattern_SameRowPerm (None)

The options inside <> are defaults. You may try
others. This might help narrow down the bug.

Hong



Hong

On 10/24/2016 05:47 PM, Hong wrote:

Barry,
Your change indeed fixed the error of his
testing code.
As Satish tested, on your branch, ex16 runs
smooth.

I do not understand why on maint or master
branch, ex16 creases inside superlu_dist,
but not with mumps.



I also confirm that ex16 runs fine with
latest fix, but unfortunately not my code.

This is something to be expected, since my
code preallocates once in the beginning. So
there is no way it can be affected by
multiple preallocations. Subsequently I only
do matrix assembly, that makes sure
structure doesn't change (set to get error
otherwise).

Summary: we don't have a simple test code to
debug superlu issue anymore.

Anton


Hong

On Mon, Oct 24, 2016 at 9:34 AM, Satish
Balay <ba...@mcs.anl.gov
<mailto:ba...@mcs.anl.gov>> wrote:

On Mon, 24 Oct 2016, Barry Smith wrote:

>
> > [Or perhaps Hong is using a
different test code and is observing bugs
> > with superlu_dist interface..]
>
>She states that her test does a
NEW MatCreate() for each matrix load (I
cut and pasted it in the email I just
sent). The bug I fixed was only related
to using the SAME matrix

Re: [petsc-users] SuperLU_dist issue in 3.7.4

[Anton Popov]

Hong,

I get all the problems gone and valgrind-clean output if I specify this:

-mat_superlu_dist_fact SamePattern_SameRowPerm

What does SamePattern_SameRowPerm actually mean?
Row permutations are for large diagonal, column permutations are for 
sparsity, right?
Will it skip subsequent matrix permutations for large diagonal even if 
matrix values change significantly?


Surprisingly everything works even with:

-mat_superlu_dist_colperm PARMETIS
-mat_superlu_dist_parsymbfact TRUE

Thanks,
Anton

On 10/24/2016 09:06 PM, Hong wrote:

Anton:


If replacing superlu_dist with mumps, does your code work?

yes

You may use mumps in your code, or tests different options for 
superlu_dist:


  -mat_superlu_dist_equil:  Equilibrate matrix (None)
  -mat_superlu_dist_rowperm  Row permutation (choose one 
of) LargeDiag NATURAL (None)
  -mat_superlu_dist_colperm  Column permutation 
(choose one of) NATURAL MMD_AT_PLUS_A MMD_ATA METIS_AT_PLUS_A PARMETIS 
(None)

  -mat_superlu_dist_replacetinypivot:  Replace tiny pivots (None)
  -mat_superlu_dist_parsymbfact:  Parallel symbolic 
factorization (None)
  -mat_superlu_dist_fact  Sparsity pattern for repeated 
matrix factorization (choose one of) SamePattern 
SamePattern_SameRowPerm (None)


The options inside <> are defaults. You may try others. This might 
help narrow down the bug.


Hong



Hong

On 10/24/2016 05:47 PM, Hong wrote:

Barry,
Your change indeed fixed the error of his testing code.
As Satish tested, on your branch, ex16 runs smooth.

I do not understand why on maint or master branch, ex16
creases inside superlu_dist, but not with mumps.



I also confirm that ex16 runs fine with latest fix, but
unfortunately not my code.

This is something to be expected, since my code preallocates
once in the beginning. So there is no way it can be affected
by multiple preallocations. Subsequently I only do matrix
assembly, that makes sure structure doesn't change (set to
get error otherwise).

Summary: we don't have a simple test code to debug superlu
issue anymore.

Anton


Hong

On Mon, Oct 24, 2016 at 9:34 AM, Satish Balay
> wrote:

On Mon, 24 Oct 2016, Barry Smith wrote:

>
> > [Or perhaps Hong is using a different test code and is
observing bugs
> > with superlu_dist interface..]
>
>She states that her test does a NEW MatCreate() for
each matrix load (I cut and pasted it in the email I
just sent). The bug I fixed was only related to using
the SAME matrix from one MatLoad() in another MatLoad().

Ah - ok.. Sorry - wasn't thinking clearly :(

Satish

Re: [petsc-users] SuperLU_dist issue in 3.7.4 failure of repeated calls to MatLoad() or MatMPIAIJSetPreallocation() with the same matrix

[Anton Popov]

On 10/25/2016 01:58 PM, Anton Popov wrote:



On 10/24/2016 10:32 PM, Barry Smith wrote:

Valgrind doesn't report any problems?



Valgrind hangs and never returns (waited hours for a 5 sec run) after 
entering factorization for the second time.


Before it happens it prints this (attached)

Anton






On Oct 24, 2016, at 12:09 PM, Anton Popov <po...@uni-mainz.de> wrote:



On 10/24/2016 05:47 PM, Hong wrote:

Barry,
Your change indeed fixed the error of his testing code.
As Satish tested, on your branch, ex16 runs smooth.

I do not understand why on maint or master branch, ex16 creases 
inside superlu_dist, but not with mumps.


I also confirm that ex16 runs fine with latest fix, but 
unfortunately not my code.


This is something to be expected, since my code preallocates once in 
the beginning. So there is no way it can be affected by multiple 
preallocations. Subsequently I only do matrix assembly, that makes 
sure structure doesn't change (set to get error otherwise).


Summary: we don't have a simple test code to debug superlu issue 
anymore.


Anton


Hong

On Mon, Oct 24, 2016 at 9:34 AM, Satish Balay <ba...@mcs.anl.gov> 
wrote:

On Mon, 24 Oct 2016, Barry Smith wrote:

[Or perhaps Hong is using a different test code and is observing 
bugs

with superlu_dist interface..]
She states that her test does a NEW MatCreate() for each 
matrix load (I cut and pasted it in the email I just sent). The 
bug I fixed was only related to using the SAME matrix from one 
MatLoad() in another MatLoad().

Ah - ok.. Sorry - wasn't thinking clearly :(

Satish





USING PICARD JACOBIAN for iteration 0, ||F||/||F0||=1.00e+00
==10744== Use of uninitialised value of size 8
==10744==at 0x18087A8: static_schedule (static_schedule.c:960)
==10744==by 0x17D42AB: pdgstrf (pdgstrf.c:572)
==10744==by 0x17B94B1: pdgssvx (pdgssvx.c:1124)
==10744==by 0xA9E777: MatLUFactorNumeric_SuperLU_DIST (superlu_dist.c:427)
==10744==by 0x6CAA90: MatLUFactorNumeric (matrix.c:3099)
==10744==by 0x137DFE9: PCSetUp_LU (lu.c:139)
==10744==by 0xECC779: PCSetUp (precon.c:968)
==10744==by 0x47AD01: PCStokesUserSetup (lsolve.c:602)
==10744==by 0x476EC3: PCStokesSetup (lsolve.c:173)
==10744==by 0x473BE4: FormJacobian (nlsolve.c:389)
==10744==by 0xF40C3D: SNESComputeJacobian (snes.c:2367)
==10745== Use of uninitialised value of size 8
==10745==at 0x18087A8: static_schedule (static_schedule.c:960)
==10745==by 0x17D42AB: pdgstrf (pdgstrf.c:572)
==10745==by 0x17B94B1: pdgssvx (pdgssvx.c:1124)
==10745==by 0xA9E777: MatLUFactorNumeric_SuperLU_DIST (superlu_dist.c:427)
==10744==by 0xFA5F1F: SNESSolve_KSPONLY (ksponly.c:38)
==10744==
==10745==by 0x6CAA90: MatLUFactorNumeric (matrix.c:3099)
==10745==by 0x137DFE9: PCSetUp_LU (lu.c:139)
==10745==by 0xECC779: PCSetUp (precon.c:968)
==10745==by 0x47AD01: PCStokesUserSetup (lsolve.c:602)
==10745==by 0x476EC3: PCStokesSetup (lsolve.c:173)
==10745==by 0x473BE4: FormJacobian (nlsolve.c:389)
==10745==by 0xF40C3D: SNESComputeJacobian (snes.c:2367)
==10745==by 0xFA5F1F: SNESSolve_KSPONLY (ksponly.c:38)
==10745==
==10745== Invalid write of size 4
==10745==at 0x18087A8: static_schedule (static_schedule.c:960)
==10745==by 0x17D42AB: pdgstrf (pdgstrf.c:572)
==10745==by 0x17B94B1: pdgssvx (pdgssvx.c:1124)
==10745==by 0xA9E777: MatLUFactorNumeric_SuperLU_DIST (superlu_dist.c:427)
==10745==by 0x6CAA90: MatLUFactorNumeric (matrix.c:3099)
==10745==by 0x137DFE9: PCSetUp_LU (lu.c:139)
==10745==by 0xECC779: PCSetUp (precon.c:968)
==10745==by 0x47AD01: PCStokesUserSetup (lsolve.c:602)
==10745==by 0x476EC3: PCStokesSetup (lsolve.c:173)
==10745==by 0x473BE4: FormJacobian (nlsolve.c:389)
==10745==by 0xF40C3D: SNESComputeJacobian (snes.c:2367)
==10745==by 0xFA5F1F: SNESSolve_KSPONLY (ksponly.c:38)
==10745==  Address 0xa077c48 is 200 bytes inside a block of size 13,936 free'd
==10745==at 0x4C2EDEB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10745==by 0x17A8A16: superlu_free_dist (memory.c:124)
==10745==by 0x18086A2: static_schedule (static_schedule.c:946)
==10745==by 0x17D42AB: pdgstrf (pdgstrf.c:572)
==10745==by 0x17B94B1: pdgssvx (pdgssvx.c:1124)
==10745==by 0xA9E777: MatLUFactorNumeric_SuperLU_DIST (superlu_dist.c:427)
==10745==by 0x6CAA90: MatLUFactorNumeric (matrix.c:3099)
==10745==by 0x137DFE9: PCSetUp_LU (lu.c:139)
==10745==by 0xECC779: PCSetUp (precon.c:968)
==10745==by 0x47AD01: PCStokesUserSetup (lsolve.c:602)
==10745==by 0x476EC3: PCStokesSetup (lsolve.c:173)
==10745==by 0x473BE4: FormJacobian (nlsolve.c:389)
==10745==  Block was alloc'd at
==10745==at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10745==by 0x17A89F4: superlu_malloc_dist (memory.c:118)
==10745==by 0x18051F5: static_schedule (static_schedule.c:274)
==10745==by 0x17D42AB: pdgstrf (p

Re: [petsc-users] SuperLU_dist issue in 3.7.4 failure of repeated calls to MatLoad() or MatMPIAIJSetPreallocation() with the same matrix

[Anton Popov]

On 10/24/2016 10:32 PM, Barry Smith wrote:

Valgrind doesn't report any problems?



Valgrind hangs and never returns (waited hours for a 5 sec run) after 
entering factorization for the second time.



On Oct 24, 2016, at 12:09 PM, Anton Popov <po...@uni-mainz.de> wrote:



On 10/24/2016 05:47 PM, Hong wrote:

Barry,
Your change indeed fixed the error of his testing code.
As Satish tested, on your branch, ex16 runs smooth.

I do not understand why on maint or master branch, ex16 creases inside 
superlu_dist, but not with mumps.


I also confirm that ex16 runs fine with latest fix, but unfortunately not my 
code.

This is something to be expected, since my code preallocates once in the 
beginning. So there is no way it can be affected by multiple preallocations. 
Subsequently I only do matrix assembly, that makes sure structure doesn't 
change (set to get error otherwise).

Summary: we don't have a simple test code to debug superlu issue anymore.

Anton


Hong

On Mon, Oct 24, 2016 at 9:34 AM, Satish Balay <ba...@mcs.anl.gov> wrote:
On Mon, 24 Oct 2016, Barry Smith wrote:


[Or perhaps Hong is using a different test code and is observing bugs
with superlu_dist interface..]

She states that her test does a NEW MatCreate() for each matrix load (I cut 
and pasted it in the email I just sent). The bug I fixed was only related to 
using the SAME matrix from one MatLoad() in another MatLoad().

Ah - ok.. Sorry - wasn't thinking clearly :(

Satish

Re: [petsc-users] SuperLU_dist issue in 3.7.4 failure of repeated calls to MatLoad() or MatMPIAIJSetPreallocation() with the same matrix

[Anton Popov]

On 10/24/2016 05:47 PM, Hong wrote:

Barry,
Your change indeed fixed the error of his testing code.
As Satish tested, on your branch, ex16 runs smooth.

I do not understand why on maint or master branch, ex16 creases inside 
superlu_dist, but not with mumps.




I also confirm that ex16 runs fine with latest fix, but unfortunately 
not my code.


This is something to be expected, since my code preallocates once in the 
beginning. So there is no way it can be affected by multiple 
preallocations. Subsequently I only do matrix assembly, that makes sure 
structure doesn't change (set to get error otherwise).


Summary: we don't have a simple test code to debug superlu issue anymore.

Anton


Hong

On Mon, Oct 24, 2016 at 9:34 AM, Satish Balay > wrote:


On Mon, 24 Oct 2016, Barry Smith wrote:

>
> > [Or perhaps Hong is using a different test code and is observing bugs
> > with superlu_dist interface..]
>
>She states that her test does a NEW MatCreate() for each
matrix load (I cut and pasted it in the email I just sent). The
bug I fixed was only related to using the SAME matrix from one
MatLoad() in another MatLoad().

Ah - ok.. Sorry - wasn't thinking clearly :(

Satish

Re: [petsc-users] SuperLU_dist issue in 3.7.4 failure of repeated calls to MatLoad() or MatMPIAIJSetPreallocation() with the same matrix

[Anton Popov]

Thank you Barry, Satish, Fande!

Is there a chance to get this fix in the maintenance release 3.7.5 
together with the latest SuperLU_DIST? Or next release is a more 
realistic option?


Anton

On 10/24/2016 01:58 AM, Satish Balay wrote:

The original testcode from Anton also works [i.e is valgrind clean] with this 
change..

Satish

On Sun, 23 Oct 2016, Barry Smith wrote:


Thanks Satish,

   I have fixed this in barry/fix-matmpixxxsetpreallocation-reentrant  (in 
next for testing)

 Fande,

 This will also make MatMPIAIJSetPreallocation() work properly with 
multiple calls (you will not need a MatReset()).

Barry



On Oct 21, 2016, at 6:48 PM, Satish Balay  wrote:

On Fri, 21 Oct 2016, Barry Smith wrote:


  valgrind first

balay@asterix /home/balay/download-pine/x/superlu_dist_test
$ mpiexec -n 2 $VG ./ex16 -f ~/datafiles/matrices/small
First MatLoad!
Mat Object: 2 MPI processes
  type: mpiaij
row 0: (0, 4.)  (1, -1.)  (6, -1.)
row 1: (0, -1.)  (1, 4.)  (2, -1.)  (7, -1.)
row 2: (1, -1.)  (2, 4.)  (3, -1.)  (8, -1.)
row 3: (2, -1.)  (3, 4.)  (4, -1.)  (9, -1.)
row 4: (3, -1.)  (4, 4.)  (5, -1.)  (10, -1.)
row 5: (4, -1.)  (5, 4.)  (11, -1.)
row 6: (0, -1.)  (6, 4.)  (7, -1.)  (12, -1.)
row 7: (1, -1.)  (6, -1.)  (7, 4.)  (8, -1.)  (13, -1.)
row 8: (2, -1.)  (7, -1.)  (8, 4.)  (9, -1.)  (14, -1.)
row 9: (3, -1.)  (8, -1.)  (9, 4.)  (10, -1.)  (15, -1.)
row 10: (4, -1.)  (9, -1.)  (10, 4.)  (11, -1.)  (16, -1.)
row 11: (5, -1.)  (10, -1.)  (11, 4.)  (17, -1.)
row 12: (6, -1.)  (12, 4.)  (13, -1.)  (18, -1.)
row 13: (7, -1.)  (12, -1.)  (13, 4.)  (14, -1.)  (19, -1.)
row 14: (8, -1.)  (13, -1.)  (14, 4.)  (15, -1.)  (20, -1.)
row 15: (9, -1.)  (14, -1.)  (15, 4.)  (16, -1.)  (21, -1.)
row 16: (10, -1.)  (15, -1.)  (16, 4.)  (17, -1.)  (22, -1.)
row 17: (11, -1.)  (16, -1.)  (17, 4.)  (23, -1.)
row 18: (12, -1.)  (18, 4.)  (19, -1.)  (24, -1.)
row 19: (13, -1.)  (18, -1.)  (19, 4.)  (20, -1.)  (25, -1.)
row 20: (14, -1.)  (19, -1.)  (20, 4.)  (21, -1.)  (26, -1.)
row 21: (15, -1.)  (20, -1.)  (21, 4.)  (22, -1.)  (27, -1.)
row 22: (16, -1.)  (21, -1.)  (22, 4.)  (23, -1.)  (28, -1.)
row 23: (17, -1.)  (22, -1.)  (23, 4.)  (29, -1.)
row 24: (18, -1.)  (24, 4.)  (25, -1.)  (30, -1.)
row 25: (19, -1.)  (24, -1.)  (25, 4.)  (26, -1.)  (31, -1.)
row 26: (20, -1.)  (25, -1.)  (26, 4.)  (27, -1.)  (32, -1.)
row 27: (21, -1.)  (26, -1.)  (27, 4.)  (28, -1.)  (33, -1.)
row 28: (22, -1.)  (27, -1.)  (28, 4.)  (29, -1.)  (34, -1.)
row 29: (23, -1.)  (28, -1.)  (29, 4.)  (35, -1.)
row 30: (24, -1.)  (30, 4.)  (31, -1.)
row 31: (25, -1.)  (30, -1.)  (31, 4.)  (32, -1.)
row 32: (26, -1.)  (31, -1.)  (32, 4.)  (33, -1.)
row 33: (27, -1.)  (32, -1.)  (33, 4.)  (34, -1.)
row 34: (28, -1.)  (33, -1.)  (34, 4.)  (35, -1.)
row 35: (29, -1.)  (34, -1.)  (35, 4.)
Second MatLoad!
Mat Object: 2 MPI processes
  type: mpiaij
==4592== Invalid read of size 4
==4592==at 0x5814014: MatView_MPIAIJ_ASCIIorDraworSocket (mpiaij.c:1402)
==4592==by 0x5814A75: MatView_MPIAIJ (mpiaij.c:1440)
==4592==by 0x53373D7: MatView (matrix.c:989)
==4592==by 0x40107E: main (ex16.c:30)
==4592==  Address 0xa47b460 is 20 bytes after a block of size 28 alloc'd
==4592==at 0x4C2FF83: memalign (vg_replace_malloc.c:858)
==4592==by 0x4FD121A: PetscMallocAlign (mal.c:28)
==4592==by 0x5842C70: MatSetUpMultiply_MPIAIJ (mmaij.c:41)
==4592==by 0x5809943: MatAssemblyEnd_MPIAIJ (mpiaij.c:747)
==4592==by 0x536B299: MatAssemblyEnd (matrix.c:5298)
==4592==by 0x5829C05: MatLoad_MPIAIJ (mpiaij.c:3032)
==4592==by 0x5337FEA: MatLoad (matrix.c:1101)
==4592==by 0x400D9F: main (ex16.c:22)
==4592==
==4591== Invalid read of size 4
==4591==at 0x5814014: MatView_MPIAIJ_ASCIIorDraworSocket (mpiaij.c:1402)
==4591==by 0x5814A75: MatView_MPIAIJ (mpiaij.c:1440)
==4591==by 0x53373D7: MatView (matrix.c:989)
==4591==by 0x40107E: main (ex16.c:30)
==4591==  Address 0xa482958 is 24 bytes before a block of size 7 alloc'd
==4591==at 0x4C2FF83: memalign (vg_replace_malloc.c:858)
==4591==by 0x4FD121A: PetscMallocAlign (mal.c:28)
==4591==by 0x4F31FB5: PetscStrallocpy (str.c:197)
==4591==by 0x4F0D3F5: PetscClassRegLogRegister (classlog.c:253)
==4591==by 0x4EF96E2: PetscClassIdRegister (plog.c:2053)
==4591==by 0x51FA018: VecInitializePackage (dlregisvec.c:165)
==4591==by 0x51F6DE9: VecCreate (veccreate.c:35)
==4591==by 0x51C49F0: VecCreateSeq (vseqcr.c:37)
==4591==by 0x5843191: MatSetUpMultiply_MPIAIJ (mmaij.c:104)
==4591==by 0x5809943: MatAssemblyEnd_MPIAIJ (mpiaij.c:747)
==4591==by 0x536B299: MatAssemblyEnd (matrix.c:5298)
==4591==by 0x5829C05: MatLoad_MPIAIJ (mpiaij.c:3032)
==4591==by 0x5337FEA: MatLoad (matrix.c:1101)
==4591==by 0x400D9F: main (ex16.c:22)
==4591==
[0]PETSC ERROR: - Error Message 
--
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: 

Re: [petsc-users] SuperLU_dist issue in 3.7.4

[Anton Popov]

Thank you Sherry for your efforts

but before I can setup an example that reproduces the problem, I have to 
ask PETSc related question.


When I pump matrix via MatView MatLoad it ignores its original partitioning.

Say originally I have 100 and 110 equations on two processors, after 
MatLoad I will have 105 and 105 also on two processors.


What do I do to pass partitioning info through MatView MatLoad?

I guess it's important for reproducing my setup exactly.

Thanks


On 10/19/2016 08:06 AM, Xiaoye S. Li wrote:
I looked at each valgrind-complained item in your email dated Oct. 
11.  Those reports are really superficial; I don't see anything  wrong 
with those lines (mostly uninitialized variables) singled out.  I did 
a few tests with the latest version in github,  all went fine.


Perhaps you can print your matrix that caused problem, I can run it 
using  your matrix.


Sherry


On Tue, Oct 11, 2016 at 2:18 PM, Anton <po...@uni-mainz.de 
<mailto:po...@uni-mainz.de>> wrote:




On 10/11/16 7:19 PM, Satish Balay wrote:

This log looks truncated. Are there any valgrind mesages
before this?
[like from your application code - or from MPI]

Yes it is indeed truncated. I only included relevant messages.


Perhaps you can send the complete log - with:
valgrind -q --tool=memcheck --leak-check=yes --num-callers=20
--track-origins=yes

[and if there were more valgrind messages from MPI - rebuild petsc

There are no messages originating from our code, just a few MPI
related ones (probably false positives) and from SuperLU_DIST
(most of them).

Thanks,
Anton

with --download-mpich - for a valgrind clean mpi]

Sherry,
Perhaps this log points to some issue in superlu_dist?

thanks,
Satish

On Tue, 11 Oct 2016, Anton Popov wrote:

Valgrind immediately detects interesting stuff:

==25673== Use of uninitialised value of size 8
==25673==at 0x178272C: static_schedule
(static_schedule.c:960)
==25674== Use of uninitialised value of size 8
==25674==at 0x178272C: static_schedule
(static_schedule.c:960)
==25674==by 0x174E74E: pdgstrf (pdgstrf.c:572)
==25674==by 0x1733954: pdgssvx (pdgssvx.c:1124)


==25673== Conditional jump or move depends on
uninitialised value(s)
==25673==at 0x1752143: pdgstrf (dlook_ahead_update.c:24)
==25673==by 0x1733954: pdgssvx (pdgssvx.c:1124)


==25673== Conditional jump or move depends on
uninitialised value(s)
==25673==at 0x5C83F43: PMPI_Recv (in
/opt/mpich3/lib/libmpi.so.12.1.0)
==25673==by 0x1755385: pdgstrf2_trsm (pdgstrf2.c:253)
==25673==by 0x1751E4F: pdgstrf (dlook_ahead_update.c:195)
==25673==by 0x1733954: pdgssvx (pdgssvx.c:1124)

==25674== Use of uninitialised value of size 8
==25674==at 0x62BF72B: _itoa_word (_itoa.c:179)
==25674==by 0x62C1289: printf_positional (vfprintf.c:2022)
==25674==by 0x62C2465: vfprintf (vfprintf.c:1677)
==25674==by 0x638AFD5: __vsnprintf_chk
(vsnprintf_chk.c:63)
==25674==by 0x638AF37: __snprintf_chk (snprintf_chk.c:34)
==25674==by 0x5CC6C08: MPIR_Err_create_code_valist (in
/opt/mpich3/lib/libmpi.so.12.1.0)
==25674==by 0x5CC7A9A: MPIR_Err_create_code (in
/opt/mpich3/lib/libmpi.so.12.1.0)
==25674==by 0x5C83FB1: PMPI_Recv (in
/opt/mpich3/lib/libmpi.so.12.1.0)
==25674==by 0x1755385: pdgstrf2_trsm (pdgstrf2.c:253)
==25674==by 0x1751E4F: pdgstrf (dlook_ahead_update.c:195)
==25674==by 0x1733954: pdgssvx (pdgssvx.c:1124)

==25674== Use of uninitialised value of size 8
==25674==at 0x1751E92: pdgstrf (dlook_ahead_update.c:205)
==25674==by 0x1733954: pdgssvx (pdgssvx.c:1124)

And it crashes after this:

==25674== Invalid write of size 4
==25674==at 0x1751F2F: pdgstrf (dlook_ahead_update.c:211)
==25674==by 0x1733954: pdgssvx (pdgssvx.c:1124)
==25674==by 0xAAEFAE: MatLUFactorNumeric_SuperLU_DIST
(superlu_dist.c:421)
==25674==  Address 0xa0 is not stack'd, malloc'd or
(recently) free'd
==25674==
[1]PETSC ERROR:


[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
Violation, probably
memory access out of range


On 10/11/2016 03:26 PM, Anton Popov wrote:

On 10/10/2016 07:11 PM, Satish

Re: [petsc-users] SuperLU_dist issue in 3.7.4

[Anton Popov]

Valgrind immediately detects interesting stuff:

==25673== Use of uninitialised value of size 8
==25673==at 0x178272C: static_schedule (static_schedule.c:960)
==25674== Use of uninitialised value of size 8
==25674==at 0x178272C: static_schedule (static_schedule.c:960)
==25674==by 0x174E74E: pdgstrf (pdgstrf.c:572)
==25674==by 0x1733954: pdgssvx (pdgssvx.c:1124)


==25673== Conditional jump or move depends on uninitialised value(s)
==25673==at 0x1752143: pdgstrf (dlook_ahead_update.c:24)
==25673==by 0x1733954: pdgssvx (pdgssvx.c:1124)


==25673== Conditional jump or move depends on uninitialised value(s)
==25673==at 0x5C83F43: PMPI_Recv (in /opt/mpich3/lib/libmpi.so.12.1.0)
==25673==by 0x1755385: pdgstrf2_trsm (pdgstrf2.c:253)
==25673==by 0x1751E4F: pdgstrf (dlook_ahead_update.c:195)
==25673==by 0x1733954: pdgssvx (pdgssvx.c:1124)

==25674== Use of uninitialised value of size 8
==25674==at 0x62BF72B: _itoa_word (_itoa.c:179)
==25674==by 0x62C1289: printf_positional (vfprintf.c:2022)
==25674==by 0x62C2465: vfprintf (vfprintf.c:1677)
==25674==by 0x638AFD5: __vsnprintf_chk (vsnprintf_chk.c:63)
==25674==by 0x638AF37: __snprintf_chk (snprintf_chk.c:34)
==25674==by 0x5CC6C08: MPIR_Err_create_code_valist (in 
/opt/mpich3/lib/libmpi.so.12.1.0)
==25674==by 0x5CC7A9A: MPIR_Err_create_code (in 
/opt/mpich3/lib/libmpi.so.12.1.0)

==25674==by 0x5C83FB1: PMPI_Recv (in /opt/mpich3/lib/libmpi.so.12.1.0)
==25674==by 0x1755385: pdgstrf2_trsm (pdgstrf2.c:253)
==25674==by 0x1751E4F: pdgstrf (dlook_ahead_update.c:195)
==25674==by 0x1733954: pdgssvx (pdgssvx.c:1124)

==25674== Use of uninitialised value of size 8
==25674==at 0x1751E92: pdgstrf (dlook_ahead_update.c:205)
==25674==by 0x1733954: pdgssvx (pdgssvx.c:1124)

And it crashes after this:

==25674== Invalid write of size 4
==25674==at 0x1751F2F: pdgstrf (dlook_ahead_update.c:211)
==25674==by 0x1733954: pdgssvx (pdgssvx.c:1124)
==25674==by 0xAAEFAE: MatLUFactorNumeric_SuperLU_DIST 
(superlu_dist.c:421)

==25674==  Address 0xa0 is not stack'd, malloc'd or (recently) free'd
==25674==
[1]PETSC ERROR: 

[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range



On 10/11/2016 03:26 PM, Anton Popov wrote:


On 10/10/2016 07:11 PM, Satish Balay wrote:

Thats from petsc-3.5

Anton - please post the stack trace you get with 
--download-superlu_dist-commit=origin/maint


I guess this is it:

[0]PETSC ERROR: [0] SuperLU_DIST:pdgssvx line 421 
/home/anton/LIB/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[0]PETSC ERROR: [0] MatLUFactorNumeric_SuperLU_DIST line 282 
/home/anton/LIB/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[0]PETSC ERROR: [0] MatLUFactorNumeric line 2985 
/home/anton/LIB/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: [0] PCSetUp_LU line 101 
/home/anton/LIB/petsc/src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: [0] PCSetUp line 930 
/home/anton/LIB/petsc/src/ksp/pc/interface/precon.c


According to the line numbers it crashes within 
MatLUFactorNumeric_SuperLU_DIST while calling pdgssvx.


Surprisingly this only happens on the second SNES iteration, but not 
on the first.


I'm trying to reproduce this behavior with PETSc KSP and SNES 
examples. However, everything I've tried up to now with SuperLU_DIST 
does just fine.


I'm also checking our code in Valgrind to make sure it's clean.

Anton


Satish


On Mon, 10 Oct 2016, Xiaoye S. Li wrote:

Which version of superlu_dist does this capture?   I looked at the 
original
error  log, it pointed to pdgssvx: line 161.  But that line is in 
comment

block, not the program.

Sherry


On Mon, Oct 10, 2016 at 7:27 AM, Anton Popov <po...@uni-mainz.de> 
wrote:




On 10/07/2016 05:23 PM, Satish Balay wrote:


On Fri, 7 Oct 2016, Kong, Fande wrote:

On Fri, Oct 7, 2016 at 9:04 AM, Satish Balay <ba...@mcs.anl.gov> 
wrote:

On Fri, 7 Oct 2016, Anton Popov wrote:

Hi guys,
are there any news about fixing buggy behavior of SuperLU_DIST, 
exactly



what


is described here:

https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.


mcs.anl.gov_pipermail_petsc-2Dusers_2015-2DAugust_026802.htm
l=CwIBAg=
54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00=DUUt3SRGI0_
JgtNaS3udV68GRkgV4ts7XKfj2opmiCY=RwruX6ckX0t9H89Z6LXKBfJBOAM2vG
1sQHw2tIsSQtA=bbB62oGLm582JebVs8xsUej_OX0eUwibAKsRRWKafos= ?

I'm using 3.7.4 and still get SEGV in pdgssvx routine. 
Everything works



fine


with 3.5.4.

Do I still have to stick to maint branch, and what are the 
chances for



these


fixes to be included in 3.7.5?


3.7.4. is off maint branch [as of a week ago]. So if you are seeing
issues with it - its best to debug and figure out the cause.

This bug is indeed inside of superlu_dist, and we started having 
this

issue
from PETSc-3.6.x. I think superlu_dist developers should have 
fixed this
bu

Re: [petsc-users] SuperLU_dist issue in 3.7.4

[Anton Popov]

On 10/10/2016 07:11 PM, Satish Balay wrote:

Thats from petsc-3.5

Anton - please post the stack trace you get with  
--download-superlu_dist-commit=origin/maint


I guess this is it:

[0]PETSC ERROR: [0] SuperLU_DIST:pdgssvx line 421 
/home/anton/LIB/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[0]PETSC ERROR: [0] MatLUFactorNumeric_SuperLU_DIST line 282 
/home/anton/LIB/petsc/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c
[0]PETSC ERROR: [0] MatLUFactorNumeric line 2985 
/home/anton/LIB/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: [0] PCSetUp_LU line 101 
/home/anton/LIB/petsc/src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: [0] PCSetUp line 930 
/home/anton/LIB/petsc/src/ksp/pc/interface/precon.c


According to the line numbers it crashes within 
MatLUFactorNumeric_SuperLU_DIST while calling pdgssvx.


Surprisingly this only happens on the second SNES iteration, but not on 
the first.


I'm trying to reproduce this behavior with PETSc KSP and SNES examples. 
However, everything I've tried up to now with SuperLU_DIST does just fine.


I'm also checking our code in Valgrind to make sure it's clean.

Anton


Satish


On Mon, 10 Oct 2016, Xiaoye S. Li wrote:


Which version of superlu_dist does this capture?   I looked at the original
error  log, it pointed to pdgssvx: line 161.  But that line is in comment
block, not the program.

Sherry


On Mon, Oct 10, 2016 at 7:27 AM, Anton Popov <po...@uni-mainz.de> wrote:



On 10/07/2016 05:23 PM, Satish Balay wrote:


On Fri, 7 Oct 2016, Kong, Fande wrote:

On Fri, Oct 7, 2016 at 9:04 AM, Satish Balay <ba...@mcs.anl.gov> wrote:

On Fri, 7 Oct 2016, Anton Popov wrote:

Hi guys,

are there any news about fixing buggy behavior of SuperLU_DIST, exactly


what


is described here:

https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.


mcs.anl.gov_pipermail_petsc-2Dusers_2015-2DAugust_026802.htm
l=CwIBAg=
54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00=DUUt3SRGI0_
JgtNaS3udV68GRkgV4ts7XKfj2opmiCY=RwruX6ckX0t9H89Z6LXKBfJBOAM2vG
1sQHw2tIsSQtA=bbB62oGLm582JebVs8xsUej_OX0eUwibAKsRRWKafos=  ?


I'm using 3.7.4 and still get SEGV in pdgssvx routine. Everything works


fine


with 3.5.4.

Do I still have to stick to maint branch, and what are the chances for


these


fixes to be included in 3.7.5?


3.7.4. is off maint branch [as of a week ago]. So if you are seeing
issues with it - its best to debug and figure out the cause.

This bug is indeed inside of superlu_dist, and we started having this

issue
from PETSc-3.6.x. I think superlu_dist developers should have fixed this
bug. We forgot to update superlu_dist??  This is not a thing users could
debug and fix.

I have many people in INL suffering from this issue, and they have to
stay
with PETSc-3.5.4 to use superlu_dist.


To verify if the bug is fixed in latest superlu_dist - you can try
[assuming you have git - either from petsc-3.7/maint/master]:

--download-superlu_dist --download-superlu_dist-commit=origin/maint


Satish

Hi Satish,

I did this:

git clone -b maint https://bitbucket.org/petsc/petsc.git petsc

--download-superlu_dist
--download-superlu_dist-commit=origin/maint (not sure this is needed,
since I'm already in maint)

The problem is still there.

Cheers,
Anton

Re: [petsc-users] SuperLU_dist issue in 3.7.4

[Anton Popov]

On 10/07/2016 05:23 PM, Satish Balay wrote:

On Fri, 7 Oct 2016, Kong, Fande wrote:


On Fri, Oct 7, 2016 at 9:04 AM, Satish Balay <ba...@mcs.anl.gov> wrote:


On Fri, 7 Oct 2016, Anton Popov wrote:


Hi guys,

are there any news about fixing buggy behavior of SuperLU_DIST, exactly

what

is described here:

https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.

mcs.anl.gov_pipermail_petsc-2Dusers_2015-2DAugust_026802.html=CwIBAg=
54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00=DUUt3SRGI0_
JgtNaS3udV68GRkgV4ts7XKfj2opmiCY=RwruX6ckX0t9H89Z6LXKBfJBOAM2vG
1sQHw2tIsSQtA=bbB62oGLm582JebVs8xsUej_OX0eUwibAKsRRWKafos=  ?

I'm using 3.7.4 and still get SEGV in pdgssvx routine. Everything works

fine

with 3.5.4.

Do I still have to stick to maint branch, and what are the chances for

these

fixes to be included in 3.7.5?

3.7.4. is off maint branch [as of a week ago]. So if you are seeing
issues with it - its best to debug and figure out the cause.


This bug is indeed inside of superlu_dist, and we started having this issue
from PETSc-3.6.x. I think superlu_dist developers should have fixed this
bug. We forgot to update superlu_dist??  This is not a thing users could
debug and fix.

I have many people in INL suffering from this issue, and they have to stay
with PETSc-3.5.4 to use superlu_dist.

To verify if the bug is fixed in latest superlu_dist - you can try
[assuming you have git - either from petsc-3.7/maint/master]:

--download-superlu_dist --download-superlu_dist-commit=origin/maint


Satish


Hi Satish,
I did this:

git clone -b maint https://bitbucket.org/petsc/petsc.git petsc

--download-superlu_dist
--download-superlu_dist-commit=origin/maint (not sure this is needed, 
since I'm already in maint)


The problem is still there.

Cheers,
Anton

[petsc-users] SuperLU_dist issue in 3.7.4

[Anton Popov]

Hi guys,

are there any news about fixing buggy behavior of SuperLU_DIST, exactly 
what is described here:


http://lists.mcs.anl.gov/pipermail/petsc-users/2015-August/026802.html ?

I'm using 3.7.4 and still get SEGV in pdgssvx routine. Everything works 
fine with 3.5.4.


Do I still have to stick to maint branch, and what are the chances for 
these fixes to be included in 3.7.5?


Thanks,

Anton

Re: [petsc-users] Scattering a sequential vector

[Anton Popov]

On 10/9/14 11:52 PM, Harshad Sahasrabudhe wrote:

Hi,

I need to scatter (distribute) a sequential vector on process 0 to all 
the processes according to a know set of global indices for each 
process. How can this be done using VecScatter?


Thanks,
Harshad
looks like this is appropriate: 
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreateToZero.html#VecScatterCreateToZero


Anton

Re: [petsc-users] Query regarding accessing data in a Petsc Vec object

[Anton Popov]

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetArray.html

On 9/28/14 1:15 PM, Parvathi M.K wrote:

Hey,
I am fairly new to PetSc. I was just curious as to how you can 
directly access the data stored in a Petsc object, say a vector 
without using VecGetValues().


I've gone through the header files, petscimpl.h and vecimpl.h which 
contain the definitions of the PetscObject and Vec structures. I 
cannot figure out which member of the struct holds the Vector data 
entered using VecSet(). Also, I could not find the definitions of the 
functions in VecOps. Could someone tell me where I can find these 
function definitions?


Thank you :)

Parvathi

Re: [petsc-users] Query regarding accessing data in a Petsc Vec object

[Anton Popov]

On 9/28/14 1:57 PM, Parvathi M.K wrote:
I want to access the data stored in the vec object directly without 
using any functions. Is this possible?


I tried using something like p-hdr._  in a printf statement (where p 
is a vector). I wish to know which element of the _p_Vec structure 
stores the data.


I think it's impossible for a good reason (or at least highly discouraged).
Why don't you want to use a prescribed interface? It's really light-weight.
But don't forget to call VecRestoreArray after directly manipulating the 
data.


Anton


On Sun, Sep 28, 2014 at 4:45 PM, Parvathi M.K phoenixaur...@gmail.com 
mailto:phoenixaur...@gmail.com wrote:


Hey,
I am fairly new to PetSc. I was just curious as to how you can
directly access the data stored in a Petsc object, say a vector
without using VecGetValues().

I've gone through the header files, petscimpl.h and vecimpl.h
which contain the definitions of the PetscObject and Vec
structures. I cannot figure out which member of the struct holds
the Vector data entered using VecSet(). Also, I could not find the
definitions of the functions in VecOps. Could someone tell me
where I can find these function definitions?

Thank you :)

Parvathi

Re: [petsc-users] Assembling a matrix for a DMComposite vector

[Anton Popov]

On 5/1/14 10:39 PM, Anush Krishnan wrote:

Hi Anton,

On 29 April 2014 05:14, Anton Popov po...@uni-mainz.de 
mailto:po...@uni-mainz.de wrote:



You can do the whole thing much easier (to my opinion).
Since you created two DMDA anyway, just do:

- find first index on every processor using MPI_Scan
- create two global vectors (no ghosts)
- put proper global indicies to global vectors
- create two local vectors (with ghosts) and set ALL entries to -1
(to have what you need in boundary ghosts)
- call global-to-local scatter

Done!


Won't the vectors contain floating point values? Are you storing your 
indices as real numbers?

YES, exactly. And then I cast them to PetscInt when I compose stencils.

Something like this:
idx[0] = (PetscInt) ivx[k][j][i];
idx[1] = (PetscInt) ivx[k][j][i+1];
idx[2] = (PetscInt) ivy[k][j][i];
... and so on, where ivx, ivy, ... are the index arrays in x, y .. 
directions


Then I insert (actually add) stencils using MatSetValues.

By the way, you can ideally preallocate in parallel with 
MatMPIAIJSetPreallocation. To count precisely number entries in the 
diagonal  off-diagonal blocks use the same mechanism to easily access 
global indices, and then compare them with the local row range, which is 
also known:

- within the range   - d_nnz[i]++;
- outside the range - o_nnz[i]++;

Anton



The advantage is that you can access global indices (including
ghosts) in every block using i-j-k indexing scheme.
I personally find this way quite easy to implement with PETSc

Anton


Thank you,
Anush

Re: [petsc-users] Use of DM to update ghost values

[Anton Popov]

On 4/5/14 1:41 AM, TAY wee-beng wrote:

Hi,

Supposed, I use DM to construct my grid. I would like to know if DM 
can be used to update the ghost values easily?


If I can use staggered grid, does it mean I have to create 3 DM grids 
(x,y,z)?
Yes this is how I do it currently. But this is not optimal, because you 
have to scatter to ghost points four times (vx, vy, vz, p separately). 
Just do it this way to start with, because by far it's the simplest 
implementation.


Alternatively you can create 2 distributed vectors, the one that 
contains non-overlapping partitioning of the grid points (to setup SNES, 
KSP, PC), and another overlapping one that contains all ghost points you 
need to compute your residual/Jacobian. Depending on the problem you 
solve this can be either star or box stencil (for each solution component).


Then setup scatter context.
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterCreate.html

You can reuse the same context for both accessing off-processor values 
and residual assembly, just need to change scatter mode SCATTER_FORWARD 
/ SCATTER_REVERSE and insert mode ADD_VALUES / INSERT_VALUES

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecScatterBegin.html

Anton


Also, how can I solve the momentum eqn involving u,v,w using KSP?

What examples can I follow for the above requirements?

Thank you.

Re: [petsc-users] Snes failled with reason -6

[Anton Popov]

On 12/20/13 3:40 PM, Que Cat wrote:

Barry,

It said that No support for this operation for this object type! 
Matrix format mpiaij does not have a built-in PETSc LU!.

Try with -pc_factor_mat_solver_package mumps (superlu_dist)

If you didn't install them reconfigure with:

--download-scalapack=1
--download-mumps=1

or:

--download-superlu_dist=1

Or just run sequentially

Anton


I run it with option |-snes_monitor -ksp_monitor_true_residual 
-snes_converged_reason -ksp_converged_reason|. and received :

0 SNES Function norm 1.414213647226e-01
0 KSP preconditioned resid norm 8.424084241999e+08 true resid norm 
1.414213647226e-01 ||r(i)||/||b|| 1.e+00
1 KSP preconditioned resid norm 1.354908290122e-02 true resid norm 
1.424694873074e-01 ||r(i)||/||b|| 1.007411345428e+00

Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 0

It looks like that the preconditioned residual converges, but the true 
residual does not.



On Fri, Dec 20, 2013 at 8:30 AM, Barry Smith bsm...@mcs.anl.gov 
mailto:bsm...@mcs.anl.gov wrote:



  Run with

   -snes_mf_operator -pc_type lu

   I’m sorry I sent the wrong instructions before.

   Barry

On Dec 20, 2013, at 7:54 AM, Que Cat quecat...@gmail.com
mailto:quecat...@gmail.com wrote:

 Hi Barry,

 Thanks for your exlanation. I used -snes_mf -pc_type lu and now
it failed with reason -3 (/* the linear solve failed */). So,
something is wrong with the function.


 On Thu, Dec 19, 2013 at 5:01 PM, Barry Smith bsm...@mcs.anl.gov
mailto:bsm...@mcs.anl.gov wrote:

   ratio is || A - Afd || /||A|| where  is the frobenus norm
and difference is || A - Afd ||.

   Try using -snes_mf -pc_type lu

This can often converge even if the A you provide is wrong,
so if this convergence sine then likely the Jacobian is wrong.

It is also possible that the function evaluation has a bug.

 http://www.mcs.anl.gov/petsc/documentation/faq.html#newton


Barry



 On Dec 19, 2013, at 4:26 PM, Que Cat quecat...@gmail.com
mailto:quecat...@gmail.com wrote:

  Hello,
 
  I use SNES to solve the non linear coupled equations. I have
checked the hand-coded jacobian with -snes_type test and received
the result:
  Norm of matrix ratio 1.39606e-23 difference 1.53372e-14
(user-defined state)
  Norm of matrix ratio 3.03157e-10 difference 0.33305 (constant
state -1.0)
  Norm of matrix ratio 3.03157e-10 difference 0.33305 (constant
state 1.0)
  It said that   if the ratio is O(1.e-8), the hand-coded
Jacobian is probably correct. What is the second paramenter
difference 1.53372e-14 ? Does it play any role in checking
the accuracy of hand-coded jacobian?
 
  The SNES solver for failed with the reason -6 the line search
failed. I tried with all other line search method stated in the
PETSC manual, but they all failed.
 
  I tried to figure out what is wrong with my code? Could you
please give me any recommendation to direct me how to check my
code? I appreciate your help.
 
  QUe
 

Re: [petsc-users] matnest

[Anton Popov]

On 10/14/13 11:47 AM, Dave May wrote:

Yes

Thanks Dave


On Monday, 14 October 2013, Anton Popov wrote:

Hi all,

do I need to call MatAssemblyBegin/End on a MATNEST matrix again
after the  submatrices get updated by MatSetValues +
MatAssemblyBegin/End?

Thanks,

Anton

[petsc-users] matnest

[Anton Popov]

Hi all,

do I need to call MatAssemblyBegin/End on a MATNEST matrix again after 
the submatrices get updated by MatSetValues + MatAssemblyBegin/End?


Thanks,

Anton

[petsc-users] MatCreateSchurComplement

[Anton Popov]

Dear all,

I guess MatCreateSchurComplement is somehow broken in PETSc 3.4.2

Attached is the modified ex23 from src/ksp/ksp/examples/tutorials

I just copied system matrix three times and created a Schur complement 
object.


However mpirun -n 4 ./ex23 gives: Unknown Mat type given: schurcomplement!

[0]PETSC ERROR: 

[0]PETSC ERROR: MatSetType() line 63 in 
/Users/anton/LIB/petsc-3.4.2/src/mat/interface/matreg.c
[0]PETSC ERROR: MatCreateSchurComplement() line 180 in 
/Users/anton/LIB/petsc-3.4.2/src/ksp/ksp/utils/schurm.c

[0]PETSC ERROR: main() line 121 in src/ksp/ksp/examples/tutorials/ex23.c
[1]PETSC ERROR: - Error Message 

[1]PETSC ERROR: Unknown type. Check for miss-spelling or missing 
external package needed for type:

see http://www.mcs.anl.gov/petsc/documentation/installation.html#external!
[1]PETSC ERROR: Unknown Mat type given: schurcomplement!
[1]PETSC ERROR: 


[1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR: 

[1]PETSC ERROR: ./ex23 on a arch-darwin-c-debug named 
mac11-005.geo.uni-mainz.de by anton Tue Oct  1 18:30:50 2013

[1]PETSC ERROR: Libraries linked from /opt/petsc/petsc-3.4.2-int32-debug/lib
[1]PETSC ERROR: Configure run at Mon Sep 30 12:17:59 2013
[1]PETSC ERROR: Configure options 
--prefix=/opt/petsc/petsc-3.4.2-int32-debug --download-f-blas-lapack=1 
--with-debugging=1 --with-valgrind=1 --with-valgrind-dir=/opt/valgrind 
--COPTFLAGS=-g -O0 --FOPTFLAGS=-g -O0 --CXXOPTFLAGS=-g -O0 
--with-shared-libraries=0 --with-cc=mpicc --with-cxx=mpicxx 
--with-fc=mpif90 --download-ml=1 --download-hypre=1 --download-blacs=1 
--download-scalapack=1 --download-metis=1 --download-parmetis=1 
--download-mumps=1 --download-superlu_dist=1


Similar error we have in our code if we use MatCreateSchurComplement 
function explicitely.


Any idea?

Thanks,

Anton

static char help[] = Solves a tridiagonal linear system.\n\n;

/*T
   Concepts: KSP^basic parallel example;
   Processors: n
T*/

/*
  Include petscksp.h so that we can use KSP solvers.  Note that this file
  automatically includes:
 petscsys.h   - base PETSc routines   petscvec.h - vectors
 petscmat.h - matrices
 petscis.h - index setspetscksp.h - Krylov subspace methods
 petscviewer.h - viewers   petscpc.h  - preconditioners

  Note:  The corresponding uniprocessor example is ex1.c
*/
#include petscksp.h

#undef __FUNCT__
#define __FUNCT__ main
int main(int argc,char **args)
{
  Vecx, b, u;  /* approx solution, RHS, exact solution */
  MatA;/* linear system matrix */
  MatL, D, U, S;
  KSPksp;  /* linear solver context */
  PC pc;   /* preconditioner context */
  PetscReal  norm,tol=1.e-11;  /* norm of solution error */
  PetscErrorCode ierr;
  PetscInt   i,n = 10,col[3],its,rstart,rend,nlocal;
  PetscScalarneg_one = -1.0,one = 1.0,value[3];

  PetscInitialize(argc,args,(char*)0,help);
  ierr = PetscOptionsGetInt(NULL,-n,n,NULL);CHKERRQ(ierr);

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Compute the matrix and right-hand-side vector that define
 the linear system, Ax = b.
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

  /*
 Create vectors.  Note that we form 1 vector from scratch and
 then duplicate as needed. For this simple case let PETSc decide how
 many elements of the vector are stored on each processor. The second
 argument to VecSetSizes() below causes PETSc to decide.
  */
  ierr = VecCreate(PETSC_COMM_WORLD,x);CHKERRQ(ierr);
  ierr = VecSetSizes(x,PETSC_DECIDE,n);CHKERRQ(ierr);
  ierr = VecSetFromOptions(x);CHKERRQ(ierr);
  ierr = VecDuplicate(x,b);CHKERRQ(ierr);
  ierr = VecDuplicate(x,u);CHKERRQ(ierr);

  /* Identify the starting and ending mesh points on each
 processor for the interior part of the mesh. We let PETSc decide
 above. */

  ierr = VecGetOwnershipRange(x,rstart,rend);CHKERRQ(ierr);
  ierr = VecGetLocalSize(x,nlocal);CHKERRQ(ierr);

  /*
 Create matrix.  When using MatCreate(), the matrix format can
 be specified at runtime.

 Performance tuning note:  For problems of substantial size,
 preallocation of matrix memory is crucial for attaining good
 performance. See the matrix chapter of the users manual for details.

 We pass in nlocal as the local size of the matrix to force it
 to have the same parallel layout as the vector created above.
  */
  ierr 

Re: [petsc-users] MatCreateSchurComplement

[Anton Popov]

On 10/1/13 9:45 PM, Jed Brown wrote:

Anton Popov po...@uni-mainz.de writes:


Dear all,

I guess MatCreateSchurComplement is somehow broken in PETSc 3.4.2

You can fix it by calling KSPMatRegisterAll() before
MatCreateSchurComplement,

Thanks, Jed. It solves the problem.

  and the issue is now fixed in 'maint' (it was
accidentally only fixed in 'master').

This problem came from simplifying dynamic loading and when the
backward-dependency was removed, we still considered KSPMatRegisterAll
to be collective on PETSC_COMM_WORLD, so it couldn't safely be called by
MatCreateSchurComplement.  Looks like you are the rare case below.


commit f550243c612070fecd53ff91e1d8ee65a2c6e25c
Author: Jed Brown j...@59a2.org
Date:   Wed Oct 17 16:16:08 2012 -0500

 Add KSPMatRegisterAll() to avoid dependency of libpetscmat on libpetscksp
 
 Register MATSCHURCOMPLEMENT there. User should explicitly call

 KSPMatRegisterAll() or KSPInitializePackage() if they want to make a
 MATSCHURCOMPLEMENT before calling KSPCreate. Such a use case should be
 very rare.
 
 Hg-commit: 3c58006af47185551c8aeaf6e1c32e68b79916fb


  include/petscksp.h |  1 +
  src/ksp/ksp/interface/dlregisksp.c |  2 ++
  src/ksp/ksp/utils/schurm.c | 25 +
  src/mat/interface/matregis.c   |  2 --
  4 files changed, 28 insertions(+), 2 deletions(-)

[petsc-users] PetscViewerBinaryMatlabOutputVecDA

[Anton Popov]

Hi guys,

just a quick question. I'm upgrading to 3.4.2 and it complains about 
PetscViewerBinaryMatlabOutputVecDA


What to use instead? I couldn't find it in Changes 3.4

Thanks,

Anton

[petsc-users] GlobalToLocal Begin/End

[Anton Popov]

Hi guys,

I need GlobalToLocal simultaneously about five vectors composed with 
different DMs.
What do you think will do better?

1) post all Begins first, followed by all Ends
2) post each Begin-End couple one after another

Many thanks,

Anton

[petsc-users] AMD interlagos optimization flags

[Anton Popov]

On 3/5/13 11:38 PM, Dharmendar Reddy wrote:
 Hello Anton,

 For me the problem is with gfortran. I use certain  object oriented 
 features available in Fortran 2003. When i last tested few months 
 agao, my code does not compile with gfortran. I guess it is 
 required/suggested to use the same compiler for petsc and the external 
 code which uses petsc. Am i right ? My code relies heavily on  zgemm 
 which i use via direct call to lapack or via petsc. ACML also has 
 ifort64_fma4 i was hoping they tweaked it to optimal performance on 
 the new AMD processors.

 Thanks
 Reddy
Dharmendar,

indeed, it's better to use the same compiler for everything. Which 
gfortran version are you using? What is the option that is still 
missing? According to this page 
http://fortranwiki.org/fortran/show/Fortran+2003+status the Fortran 2003 
standard support is fairly extensive. We use gfortran 4.7.2 and a few 
features of Fortran 2003 in one of the applications.

Also the -std=f2003 -ffree-form options should not be forgotten.


 On Tue, Mar 5, 2013 at 4:02 PM, Anton Popov popov at uni-mainz.de 
 mailto:popov at uni-mainz.de wrote:


 On 3/5/13 9:23 PM, Dharmendar Reddy wrote:
 Hello,
 Has any one installed petsc on the AMD 6200 series
 processor systems? I have a small cluster with quad socket AMD
 6274 (16Core)  processors. Should i use any special optimization
 flags or let petsc configure script decide ?

 I would like have two flavors of the instillation:

  1.  intel 13.2 compilers using intel mpi and intel MKL 11.0
  2.  intel 13.2 compilers using intel mpi and ACML 5.3 compile
 with intel. Now ACML has a weird organization of the
 libraries. It has same names for both the 64bit and 32 bit
 integer libraries but saved in different folders. Should i
 link to single thread or multithreaded acml ?

 Dharmendar,

 I use
 --COPTFLAGS=-mavx -mfma4 -O3
 --FOPTFLAGS=-mavx -mfma4 -O3

 and single-threaded ACML optimized for fma4 instruction set ( i.e.
 gfortran64_fma4/lib/libacml.a)
 Obviously, I use gcc and gfortran. All works fine. No need for
 Intel on this machine. If you think gcc is much slower, just try
 with optimization.

 1.




 -- 
 -
 Dharmendar Reddy Palle
 Graduate Student
 Microelectronics Research center,
 University of Texas at Austin,
 10100 Burnet Road, Bldg. 160
 MER 2.608F, TX 78758-4445
 e-mail: dharmareddy84 at gmail.com mailto:dharmareddy84 at gmail.com
 Phone: +1-512-350-9082 tel:%2B1-512-350-9082
 United States of America.
 Homepage: https://webspace.utexas.edu/~dpr342
 https://webspace.utexas.edu/%7Edpr342




 -- 
 -
 Dharmendar Reddy Palle
 Graduate Student
 Microelectronics Research center,
 University of Texas at Austin,
 10100 Burnet Road, Bldg. 160
 MER 2.608F, TX 78758-4445
 e-mail: dharmareddy84 at gmail.com mailto:dharmareddy84 at gmail.com
 Phone: +1-512-350-9082
 United States of America.
 Homepage: https://webspace.utexas.edu/~dpr342 
 https://webspace.utexas.edu/%7Edpr342

-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130306/f6c9ad23/attachment-0001.html

[petsc-users] AMD interlagos optimization flags

[Anton Popov]

On 3/5/13 9:23 PM, Dharmendar Reddy wrote:
 Hello,
 Has any one installed petsc on the AMD 6200 series processor 
 systems? I have a small cluster with quad socket AMD 6274 (16Core)  
 processors. Should i use any special optimization flags or let petsc 
 configure script decide ?

 I would like have two flavors of the instillation:

  1.  intel 13.2 compilers using intel mpi and intel MKL 11.0
  2.  intel 13.2 compilers using intel mpi and ACML 5.3 compile with
 intel. Now ACML has a weird organization of the libraries. It has
 same names for both the 64bit and 32 bit integer libraries but
 saved in different folders. Should i link to single thread or
 multithreaded acml ?

Dharmendar,

I use
--COPTFLAGS=-mavx -mfma4 -O3
--FOPTFLAGS=-mavx -mfma4 -O3

and single-threaded ACML optimized for fma4 instruction set ( i.e. 
gfortran64_fma4/lib/libacml.a)
Obviously, I use gcc and gfortran. All works fine. No need for Intel on 
this machine. If you think gcc is much slower, just try with optimization.

 1.



 -- 
 -
 Dharmendar Reddy Palle
 Graduate Student
 Microelectronics Research center,
 University of Texas at Austin,
 10100 Burnet Road, Bldg. 160
 MER 2.608F, TX 78758-4445
 e-mail: dharmareddy84 at gmail.com mailto:dharmareddy84 at gmail.com
 Phone: +1-512-350-9082
 United States of America.
 Homepage: https://webspace.utexas.edu/~dpr342 
 https://webspace.utexas.edu/%7Edpr342

-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130305/566a2190/attachment.html

[petsc-users] Which IDE you use to develop petsc application

[Anton Popov]

As promised, description files how to use Eclipse PTP for 
development/parallel debugging of PETSc applications are in the attachment.
I recommend installing debug versions of MPICH2, PETSc, parallel SDM 
debugger, and valgrind (if necessary) to /opt directory
Just follow the instructions.
Cheers,
Anton


On 11/18/12 4:43 PM, Fande Kong wrote:
 yeap, It sounds very interesting. Could you please give some 
 instructions on how to use eclipse link both Petsc and MPI?

 On Sat, Nov 17, 2012 at 3:08 AM, Anton Popov popov at uni-mainz.de 
 mailto:popov at uni-mainz.de wrote:

 On 11/16/12 2:03 AM, Barry Smith wrote:

 Because we are developing PETSc as a library that must be
 portable to many users most people who develop the PETSc
 libraries do not use a IDE.

 For someone who is writing AN APPLICATION that uses PETSc
 (especially an application with guis) it makes sense to use an
 IDL to do that development. What you chose should depend on
 what machine you develop on and what your are experienced with
 or comfortable with. In the users manual we try to provide
 some information on using PETSc from a variety of IDEs. We are
 interested in collecting information on using PETSc from IDEs
 and improving our documentation.

 I managed Eclipse PTP on Mac OS (probably the same on Linux).
 Provided that a debug (preferably valgrind-clean) version of
 MPICH2 library with MPD process manager can be installed, Eclipse
 Indigo can be easily configured for using PARALLEL SDM debugger
 with PETSc applications.

 If you're interested, I can send detailed configuration / setup
 instructions.

 Anton



 Barry


 On Nov 15, 2012, at 4:52 PM, Fande Kong fd.kong at siat.ac.cn
 mailto:fd.kong at siat.ac.cn wrote:

   Got it. Thanks.
 On Thu, Nov 15, 2012 at 3:49 PM, Matthew Knepley
 knepley at gmail.com mailto:knepley at gmail.com wrote:
 On Thu, Nov 15, 2012 at 5:41 PM, Fande Kong
 fande.kong at colorado.edu mailto:fande.kong at colorado.edu
 wrote:

 Hi all,

 In order to improve development the efficient of the
 petsc application
 development, which IDE are you using?

 Emacs+gdb.

Matt

 Regards,
 --
 Fande Kong
 Department of Computer Science
 University of Colorado at Boulder




 --
 What most experimenters take for granted before they begin
 their
 experiments is infinitely more interesting than any
 results to which
 their experiments lead.
 -- Norbert Wiener




 -- 
 Fande Kong
 ShenZhen Institutes of Advanced Technology
 Chinese Academy of Sciences






 -- 
 Fande Kong
 ShenZhen Institutes of Advanced Technology
 Chinese Academy of Sciences


-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121122/753ecdfa/attachment.html
-- next part --

# configure

./configure \
CC= ... \  set here C compiler command (e.g. gcc)
CXX= ... \ set here C++ compiler command (e.g. g++)
F77= ... \ set here FORTRAN 77 compiler command (e.g. gfortran)
FC= ... \  set here FORTRAN 90 compiler command (e.g. gfortran)
CPPFLAGS='-I...' \ full path to valgrind include directory (if you have 
valgrind) 
--prefix= ... \ set installation directory (I recommend /opt/mpich2-debug)
--enable-f77 \
--enable-fc \
--enable-cxx \
--enable-threads=runtime \
--enable-g=dbg,mem,meminit \
--enable-debuginfo \
--enable-shared \
--enable-sharedlibs=... \ osx-gcc for Mac OS, gcc for Linux
--enable-fast=none \
--with-thread-package=pthreads \
--with-pm=mpd

# build

make all

# install

sudo make install

# update the path in .bash_profile (.bashrc on Linux)

export PATH=$PATH:/opt/mpich2-debug/bin

# create file .mpd.conf in the home directory

# put the following string in this file

MPD_SECRETWORD=mr45-j9z

##
# IMPORTANT!!!
# when using debug version you should make sure that 
# mpiexec doesn't complain about mpd
# open a separate console and type the following command there

mpd 

# do not close the console while you're using mpiexec
# otherwise it cannot be guaranteed that mpiexec won't complain
##


# after installation check if everything works, type in the console:

mpicc -show
mpicxx -show
mpif90 -show 

# compile and run C and FORTRAN hello_world examples

mpicc mpi_hello.c
mpiexec -np 4 ./a.out

mpif90 mpi_hello.f
mpiexec -np 4 ./a.out

-- next part

[petsc-users] Which IDE you use to develop petsc application

[Anton Popov]

On 11/16/12 2:03 AM, Barry Smith wrote:
 Because we are developing PETSc as a library that must be portable to 
 many users most people who develop the PETSc libraries do not use a IDE.

 For someone who is writing AN APPLICATION that uses PETSc (especially an 
 application with guis) it makes sense to use an IDL to do that development. 
 What you chose should depend on what machine you develop on and what your are 
 experienced with or comfortable with. In the users manual we try to provide 
 some information on using PETSc from a variety of IDEs. We are interested in 
 collecting information on using PETSc from IDEs and improving our 
 documentation.
I managed Eclipse PTP on Mac OS (probably the same on Linux). Provided 
that a debug (preferably valgrind-clean) version of MPICH2 library with 
MPD process manager can be installed, Eclipse Indigo can be easily 
configured for using PARALLEL SDM debugger with PETSc applications.

If you're interested, I can send detailed configuration / setup 
instructions.

Anton


 Barry


 On Nov 15, 2012, at 4:52 PM, Fande Kong fd.kong at siat.ac.cn wrote:

   Got it. Thanks.
 On Thu, Nov 15, 2012 at 3:49 PM, Matthew Knepley knepley at gmail.com 
 wrote:
 On Thu, Nov 15, 2012 at 5:41 PM, Fande Kong fande.kong at colorado.edu 
 wrote:
 Hi all,

 In order to improve development the efficient of the petsc application
 development, which IDE are you using?
 Emacs+gdb.

Matt

 Regards,
 --
 Fande Kong
 Department of Computer Science
 University of Colorado at Boulder




 --
 What most experimenters take for granted before they begin their
 experiments is infinitely more interesting than any results to which
 their experiments lead.
 -- Norbert Wiener




 -- 
 Fande Kong
 ShenZhen Institutes of Advanced Technology
 Chinese Academy of Sciences

[petsc-users] Solving Ill Conditioned Problems

[Anton Popov]

Of, course you have a more sophisticated case, but if you just put the 
orders of magnitude of your matrix blocks
into a toy two-by-two matrix, i.e. (in matlab notation)
A = [1e9, 1; 1, 1e-9]
then it's immediately visible that A is singular (det(A) is zero)
Symmetric scaling, that Matt suggested, just makes it more obvious, 
after scaling the matrix A will be:
A = [1, 1; 1, 1] - clearly the rows are linearly dependent.
Scaling may help you, but of course, this is not an always working solution

Anton

On 10/18/12 8:38 PM, Nachiket Gokhale wrote:
 On Thu, Oct 18, 2012 at 2:28 PM, Matthew Knepley knepley at gmail.com wrote:
 On Thu, Oct 18, 2012 at 11:07 AM, Nachiket Gokhale gokhalen at gmail.com
 wrote:
 Hi,

 I am solving a Piezoelectric problem using Petsc. The structure is

 [ K_uuK_uv ]
 [K_uv ^T   -K_v,v ]

 More details about the formulation:  http://tinyurl.com/9hlbp4u

 K_uu  has elements O(1E9)  because the stiffnesses are in GPa,
 K_uv   has elements O(1) because piezoelectric coefficients are of that
 order
 K_v,v  has  elements O(1E-9) because the dielectric constants are of
 that order.

 I am using Petsc, with pc_type LU and MUMPS for the factorization,
 -ksp_type gmres. I am not sure if my solution is converging. A typical
 solve seems to be doing this:

 28 KSP preconditioned resid norm 5.642364260456e-06 true resid norm
 1.228976487745e-03 ||r(i)||/||b|| 3.317409023627e-14
   29 KSP preconditioned resid norm 5.540718271043e-06 true resid norm
 1.228453548651e-03 ||r(i)||/||b|| 3.315997440178e-14
   30 KSP preconditioned resid norm 1.973052106578e-03 true resid norm
 1.220399172500e-03 ||r(i)||/||b|| 3.294256047735e-14
   31 KSP preconditioned resid norm 1.155762663956e-17 true resid norm
 2.44763938e-04 ||r(i)||/||b|| 6.606955965570e-15

 Is there a right way to solve this set of equations? Is PCFieldSplit
 the recommended way?

 First, you should really non-dimensionalize. You can see what this would
 give
 you by symmetrically scaling your problem with [ 3e4 3e-4 ], namely
 everything
 will be O(1).

 Second, you might get something out of using FieldSplit, but its tough to
 tell
 without knowing more about the operators.

 Matt

 Thanks,

 -Nachiket
 Oh, right, thanks. I wasn't even thinking of it that way. I'll scale
 the variables and I'll give it a try.

 Cheers,

 -Nachiket





 --
 What most experimenters take for granted before they begin their experiments
 is infinitely more interesting than any results to which their experiments
 lead.
 -- Norbert Wiener

[petsc-users] block size

[Anton Popov]

Dear petsc team,

could you please tell me what's wrong with the attached example file?
I run it on 4 processors with petsc-3.3-p1.

What could error message Local size 1000 not compatible with block size 
3! mean?

I've another question related to this issue. What is the real purpose of 
PetscViewerBinarySkipInfo function?
I see no reason to skip creating info file, because the file produced 
by the attached example seems to be correct.

Moreover, similar block size error occurs in our code while reading file 
with multiple vectors, irrespective whether info file exists or not.

Thank you,

Anton

-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120718/cd65a159/attachment.html
-- next part --
//

#include petscvec.h
#include petscdmda.h   

//

#undef __FUNCT__
#define __FUNCT__ SaveVector

PetscErrorCode SaveVector(Vec V3, Vec V1)
{
PetscErrorCode ierr;
PetscViewerview_out;
char   SaveFileName[PETSC_MAX_PATH_LEN]=vec.dat;

ierr = PetscViewerCreate(PETSC_COMM_WORLD, view_out);  
CHKERRQ(ierr);
ierr = PetscViewerSetType(view_out, PETSCVIEWERBINARY); 
CHKERRQ(ierr);
ierr = PetscViewerFileSetMode(view_out, FILE_MODE_WRITE);   
CHKERRQ(ierr);
ierr = PetscViewerFileSetName(view_out, SaveFileName);  
CHKERRQ(ierr);

ierr = VecView(V1, view_out);   
CHKERRQ(ierr);
ierr = VecView(V3, view_out);   
CHKERRQ(ierr);

ierr = PetscViewerDestroy(view_out);   
CHKERRQ(ierr);

PetscFunctionReturn(0);
}

//

#undef __FUNCT__
#define __FUNCT__ ReadVector

PetscErrorCode ReadVector(Vec V3, Vec V1)
{
PetscErrorCode ierr;
PetscViewerview_in;
char   LoadFileName[PETSC_MAX_PATH_LEN]=vec.dat;

printf(Reading vector\n);

ierr = PetscViewerBinaryOpen(PETSC_COMM_WORLD, LoadFileName,
FILE_MODE_READ, view_in); CHKERRQ(ierr);


ierr = VecLoad(V1, view_in);CHKERRQ(ierr);
ierr = VecLoad(V3, view_in);CHKERRQ(ierr);


ierr = PetscViewerDestroy(view_in); CHKERRQ(ierr);

PetscFunctionReturn(0);
}

//

#undef __FUNCT__
#define __FUNCT__ main

int main(int argc,char **argv)
{

PetscErrorCode   ierr;
PetscInt dof, s = 1;
PetscInt M = 20, N = 20, P = 20;
PetscScalar  cf = 1.0;
DM   da3, da1;
Vec  V3w, V3r, V1w, V1r;

ierr = PetscInitialize(argc, argv, PETSC_NULL, PETSC_NULL); 
CHKERRQ(ierr);

dof = 3;
ierr = DMDACreate3d(PETSC_COMM_WORLD,
DMDA_BOUNDARY_NONE, DMDA_BOUNDARY_NONE, DMDA_BOUNDARY_NONE, 
DMDA_STENCIL_BOX,
M, N, P, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, dof, s,
PETSC_NULL, PETSC_NULL, PETSC_NULL, da3); CHKERRQ(ierr);

dof = 1;
ierr = DMDACreate3d(PETSC_COMM_WORLD,
DMDA_BOUNDARY_NONE, DMDA_BOUNDARY_NONE, DMDA_BOUNDARY_NONE, 
DMDA_STENCIL_BOX,
M, N, P, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, dof, s,
PETSC_NULL, PETSC_NULL, PETSC_NULL, da1); CHKERRQ(ierr);

 
ierr = DMCreateGlobalVector(da3, V3w); CHKERRQ(ierr);
ierr = DMCreateGlobalVector(da3, V3r); CHKERRQ(ierr);

ierr = DMCreateGlobalVector(da1, V1w); CHKERRQ(ierr);
ierr = DMCreateGlobalVector(da1, V1r); CHKERRQ(ierr);

ierr = VecSet(V3w, cf);  CHKERRQ(ierr);
ierr = VecSet(V1w, cf);  CHKERRQ(ierr);

ierr = SaveVector(V3w, V1w);  CHKERRQ(ierr);

ierr = ReadVector(V3r, V1r);  CHKERRQ(ierr);


ierr = DMDestroy(da3);  CHKERRQ(ierr);
ierr = DMDestroy(da1);  CHKERRQ(ierr);

ierr = VecDestroy(V3w); CHKERRQ(ierr);
ierr = VecDestroy(V3r); CHKERRQ(ierr);

ierr = VecDestroy(V1w); CHKERRQ(ierr);
ierr = VecDestroy(V1r); CHKERRQ(ierr);

ierr = PetscFinalize(); CHKERRQ(ierr);

return 0;

}
//

[petsc-users] block size

[Anton Popov]

Comment out one of vectors, your code works fine.
I wouldn't say fine because I need to store both vectors.

  So what is the workaround?

- I'm not allowed to mix vectors with different block size in the same file?
- I'm not allowed to write more then ONE vector in a file?


On 7/18/12 4:10 PM, Hong Zhang wrote:
 You use same datafile 'vec.dat' for writing two different vectors,
 V3 and V1:

 ierr = VecView(V1, view_out);
 ierr = VecView(V3, view_out);

 // here, vec.dat holds V3

 Then read it in the order
 ierr = VecLoad(V1, view_in);
 //crash here because reading V3 into V1

 ierr = VecLoad(V3, view_in);

 Comment out one of vectors, your code works fine.

 Hong




 On Wed, Jul 18, 2012 at 7:54 AM, Anton Popov popov at uni-mainz.de 
 mailto:popov at uni-mainz.de wrote:

 Dear petsc team,

 could you please tell me what's wrong with the attached example file?
 I run it on 4 processors with petsc-3.3-p1.

 What could error message Local size 1000 not compatible with
 block size 3! mean?

 I've another question related to this issue. What is the real
 purpose of PetscViewerBinarySkipInfo function?
 I see no reason to skip creating info file, because the file
 produced by the attached example seems to be correct.

 Moreover, similar block size error occurs in our code while
 reading file with multiple vectors, irrespective whether info
 file exists or not.

 Thank you,

 Anton




-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120718/5a9af5df/attachment.html

[petsc-users] fieldsplit recursive

[Anton Popov]

Thanks Barry,
We'll send the code to petsc-maint in a moment.
But we really don't want to bother you a lot.

Anton and Boris

On 6/28/12 8:55 PM, Barry Smith wrote:
Anton,

 This came about because we are now being much more pedantic about the 
 blocksizes of PETSc objects and not allowing them to be causally changed when 
 they shouldn't be.

 You can resolve this problem by editing the file 
 src/ksp/pc/impls/fieldsplit/fieldsplit.c locate the function

 #undef __FUNCT__
 #define __FUNCT__ PCApply_FieldSplit
 static PetscErrorCode PCApply_FieldSplit(PC pc,Vec x,Vec y)
 {
PC_FieldSplit *jac = (PC_FieldSplit*)pc-data;
PetscErrorCodeierr;
PC_FieldSplitLink ilink = jac-head;
PetscInt  cnt,bs;

PetscFunctionBegin;

 and add the two lines right here

x-map-bs = jac-bs;
y-map-bs = jac-bs;


 then run make cmake in that directory.

 To resolve this permanently we will need to figure out how to insure those 
 inner vectors are created with the correct block size. Are you willing to 
 share your code with petsc-maint at mcs.anl.gov so that we can reproduce the 
 problem and fix it properly for the long run? (The problem is in PETSc not in 
 your code).

 Barry


 
 On Jun 28, 2012, at 10:44 AM, Anton Popov wrote:

 Dear petsc team,

 I'm trying to use fieldsplit preconditioner for the velocity block in the 
 Stokes system which is also preconditioned by
 fieldsplit (kind of recursive).

 Running example 42 from src/ksp/ksp/examples/tutorials with petsc-3.2, as 
 follows:

 mpiexec -np 4 ./ex42 \
 -stokes_ksp_type gcr \
 -stokes_ksp_rtol 1.0e-6 \
 -stokes_pc_type fieldsplit \
 -stokes_pc_fieldsplit_type SCHUR \
 -stokes_pc_fieldsplit_schur_factorization_type UPPER \
 -stokes_fieldsplit_u_ksp_type fgmres \
 -stokes_fieldsplit_u_ksp_rtol 1e-3 \
 -stokes_fieldsplit_u_pc_type fieldsplit \
 -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \
 -stokes_fieldsplit_u_fieldsplit_pc_type ml \
 -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \
 -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \
 -stokes_fieldsplit_p_ksp_type preonly \
 -stokes_fieldsplit_p_pc_type jacobi \
 -stokes_ksp_monitor_blocks \
 -mx 16 \
 -model 3

 gives nicely looking output.

 But! Repeating the same exercise with petsc-3.3, like this (actually, there 
 is only one difference: factorization - fact):

 mpiexec -np 4 ./ex42 \
 -stokes_ksp_type gcr \
 -stokes_ksp_rtol 1.0e-6 \
 -stokes_pc_type fieldsplit \
 -stokes_pc_fieldsplit_type SCHUR \
 -stokes-pc_fieldsplit_schur_fact_type UPPER \
 -stokes_fieldsplit_u_ksp_type fgmres \
 -stokes_fieldsplit_u_ksp_rtol 1e-3 \
 -stokes_fieldsplit_u_pc_type fieldsplit \
 -stokes_fieldsplit_u_fieldsplit_ksp_type preonly \
 -stokes_fieldsplit_u_fieldsplit_pc_type ml \
 -stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \
 -stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \
 -stokes_fieldsplit_p_ksp_type preonly \
 -stokes_fieldsplit_p_pc_type jacobi \
 -stokes_ksp_monitor_blocks \
 -mx 16 \
 -model 3

 curses me hardly by claiming:

 [0]PETSC ERROR: - Error Message 
 
 [0]PETSC ERROR: Object is in wrong state!
 [0]PETSC ERROR: Blocksize of x vector 1 does not match fieldsplit blocksize 
 3!
 [0]PETSC ERROR: 
 
 [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun  5 14:20:42 
 CDT 2012
 [0]PETSC ERROR: See docs/changes/index.html for recent updates.
 [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
 [0]PETSC ERROR: See docs/index.html for manual pages.
 [0]PETSC ERROR: 
 
 [0]PETSC ERROR: ./ex42 on a int32-deb named mac11-005.geo.uni-mainz.de by 
 anton Thu Jun 28 17:06:53 2012
 [0]PETSC ERROR: Libraries linked from 
 /Users/anton/LIB/petsc-3.3-p0/int32-debug/lib
 [0]PETSC ERROR: Configure run at Tue Jun 12 15:32:21 2012
 [0]PETSC ERROR: Configure options PETSC_DIR=/Users/anton/LIB/petsc-3.3-p0 
 PETSC_ARCH=int32-debug --download-f-blas-lapack=1 --with-debugging=1 
 --COPTFLAGS=-g -O0 --FOPTFLAGS=-g -O0 --CXXOPTFLAGS=-g -O0 
 --with-c++-support=1 --with-fortran=1 --with-fortran-kernels=1 
 --with-large-file-io=1 --with-mpi-compilers=1 --with-cc=mpicc 
 --with-cxx=mpicxx --with-fc=mpif90 --download-ml=1 --download-hypre=1 
 --download-blacs=1 --download-scalapack=1 --download-metis=1 
 --download-parmetis=1 --download-mumps=1 --download-superlu_dist=1
 [0]PETSC ERROR: 
 
 [0]PETSC ERROR: PCApply_FieldSplit() line 726 in 
 /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c
 [0]PETSC ERROR: PCApply() line 384 in 
 /Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c
 [0]PETSC ERROR: KSPFGMRESCycle() line 169 in 
 /Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
 [0]PETSC ERROR: KSPSolve_FGMRES() line 294 in 
 /Users

[petsc-users] fieldsplit recursive

[Anton Popov]

Dear petsc team,

I'm trying to use fieldsplit preconditioner for the velocity block in 
the Stokes system which is also preconditioned by
fieldsplit (kind of recursive).

Running example 42 from src/ksp/ksp/examples/tutorials with petsc-3.2, 
as follows:

mpiexec -np 4 ./ex42 \
-stokes_ksp_type gcr \
-stokes_ksp_rtol 1.0e-6 \
-stokes_pc_type fieldsplit \
-stokes_pc_fieldsplit_type SCHUR \
-stokes_pc_fieldsplit_schur_factorization_type UPPER \
-stokes_fieldsplit_u_ksp_type fgmres \
-stokes_fieldsplit_u_ksp_rtol 1e-3 \
-stokes_fieldsplit_u_pc_type fieldsplit \
-stokes_fieldsplit_u_fieldsplit_ksp_type preonly \
-stokes_fieldsplit_u_fieldsplit_pc_type ml \
-stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \
-stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \
-stokes_fieldsplit_p_ksp_type preonly \
-stokes_fieldsplit_p_pc_type jacobi \
-stokes_ksp_monitor_blocks \
-mx 16 \
-model 3

gives nicely looking output.

But! Repeating the same exercise with petsc-3.3, like this (actually, 
there is only one difference: factorization - fact):

mpiexec -np 4 ./ex42 \
-stokes_ksp_type gcr \
-stokes_ksp_rtol 1.0e-6 \
-stokes_pc_type fieldsplit \
-stokes_pc_fieldsplit_type SCHUR \
-stokes-pc_fieldsplit_schur_fact_type UPPER \
-stokes_fieldsplit_u_ksp_type fgmres \
-stokes_fieldsplit_u_ksp_rtol 1e-3 \
-stokes_fieldsplit_u_pc_type fieldsplit \
-stokes_fieldsplit_u_fieldsplit_ksp_type preonly \
-stokes_fieldsplit_u_fieldsplit_pc_type ml \
-stokes_fieldsplit_u_pc_fieldsplit_block_size 3 \
-stokes_fieldsplit_u_pc_fieldsplit_type ADDITIVE \
-stokes_fieldsplit_p_ksp_type preonly \
-stokes_fieldsplit_p_pc_type jacobi \
-stokes_ksp_monitor_blocks \
-mx 16 \
-model 3

curses me hardly by claiming:

[0]PETSC ERROR: - Error Message 

[0]PETSC ERROR: Object is in wrong state!
[0]PETSC ERROR: Blocksize of x vector 1 does not match fieldsplit 
blocksize 3!
[0]PETSC ERROR: 

[0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 0, Tue Jun  5 
14:20:42 CDT 2012
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 

[0]PETSC ERROR: ./ex42 on a int32-deb named mac11-005.geo.uni-mainz.de 
by anton Thu Jun 28 17:06:53 2012
[0]PETSC ERROR: Libraries linked from 
/Users/anton/LIB/petsc-3.3-p0/int32-debug/lib
[0]PETSC ERROR: Configure run at Tue Jun 12 15:32:21 2012
[0]PETSC ERROR: Configure options 
PETSC_DIR=/Users/anton/LIB/petsc-3.3-p0 PETSC_ARCH=int32-debug 
--download-f-blas-lapack=1 --with-debugging=1 --COPTFLAGS=-g -O0 
--FOPTFLAGS=-g -O0 --CXXOPTFLAGS=-g -O0 --with-c++-support=1 
--with-fortran=1 --with-fortran-kernels=1 --with-large-file-io=1 
--with-mpi-compilers=1 --with-cc=mpicc --with-cxx=mpicxx 
--with-fc=mpif90 --download-ml=1 --download-hypre=1 --download-blacs=1 
--download-scalapack=1 --download-metis=1 --download-parmetis=1 
--download-mumps=1 --download-superlu_dist=1
[0]PETSC ERROR: 

[0]PETSC ERROR: PCApply_FieldSplit() line 726 in 
/Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c
[0]PETSC ERROR: PCApply() line 384 in 
/Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPFGMRESCycle() line 169 in 
/Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
[0]PETSC ERROR: KSPSolve_FGMRES() line 294 in 
/Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
[0]PETSC ERROR: KSPSolve() line 446 in 
/Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: PCApply_FieldSplit_Schur() line 693 in 
/Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/impls/fieldsplit/fieldsplit.c
[0]PETSC ERROR: PCApply() line 384 in 
/Users/anton/LIB/petsc-3.3-p0/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSPSolve_GCR_cycle() line 47 in 
/Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c
[0]PETSC ERROR: KSPSolve_GCR() line 117 in 
/Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/impls/gcr/gcr.c
[0]PETSC ERROR: KSPSolve() line 446 in 
/Users/anton/LIB/petsc-3.3-p0/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: solve_stokes_3d_coupled() line 2045 in 
src/ksp/ksp/examples/tutorials/ex42.c
[0]PETSC ERROR: main() line 2106 in src/ksp/ksp/examples/tutorials/ex42.c
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

Similar error appeared in our code after upgrading to petsc-3.3, and 
we're using similar functionality and options as I posted above.

Please explain this issue. An advice how to get rid of the error is also 
appreciated.

Thanks a lot

Anton

[petsc-users] stokes and fieldsplit

[Anton Popov]

Hi petsc team,
I have a question about using nested matrices and vectors for coupled 
solution of Stokes system with filedsplit preconditioner.

Is there a simple example explaining how I can create nested matrix and 
vector using
MatCreateNest and VecCreateNest functions, provided that I already 
have all constituting blocks correctly assembled?
I could not find these example online, and if I do it straightforwardly 
according to documentation, petsc complains that some vector blocks are 
not setup.

Do I need to define velocity and pressure splits using 
PCFieldSplitSetIS, because if I don't, petsc says Unhandled case, 
must have at least two fields, not 1!?

Do I need to explicitly specify the blocksize, if I want to use the 
filedsplit preconditioner for the velocity block?

A small example would be extremely helpful

Thanks a lot,

Anton

[petsc-users] MatSetOption bug?

[Anton Popov]

Thanks Matt, it's clear now

On 6/19/12 6:49 PM, Matthew Knepley wrote:
 On Tue, Jun 19, 2012 at 10:06 AM, Anton Popov popov at uni-mainz.de 
 mailto:popov at uni-mainz.de wrote:

 Hi petsc team!
 I've recently upgraded to 3.3, and currently experience problems.
 One of them follows.
 If I compile this simple code:


 In 3.3, we now require that MatSetUp() is called before messing with 
 the storage scheme.
 We should have been more specific in Changes. Moreover, this should be 
 a better error
 message. Will fix, and see below.

 #include petsc.h
 #undef __FUNCT__
 #define __FUNCT__ main
 int main( int argc,char **argv )
 {
Mat A;
PetscErrorCode ierr;
PetscInt n_rows_Local=100;
PetscInt n_cols_Local=100;
ierr = PetscInitialize(argc, argv, PETSC_NULL, PETSC_NULL);
 CHKERRQ(ierr);
ierr = MatCreate(PETSC_COMM_WORLD, A); CHKERRQ(ierr);
ierr = MatSetSizes(A, n_rows_Local, n_cols_Local,
 PETSC_DETERMINE, PETSC_DETERMINE); CHKERRQ(ierr);
ierr = MatSetType(A, MATAIJ); CHKERRQ(ierr);


 ierr = MatSetUp(A);CHKERRQ(ierr);

 Matt

ierr = MatSetOption(A, MAT_KEEP_NONZERO_PATTERN, PETSC_TRUE);
 CHKERRQ(ierr);
ierr = PetscFinalize(); CHKERRQ(ierr);
PetscFunctionReturn(0);
 }

 and run it in parallel (mpiexec -np 2 ./example) it gives me
 errors like this:

 [0]PETSC ERROR: - Error Message
 
 [0]PETSC ERROR: Null argument, when expecting valid pointer!
 [0]PETSC ERROR: Null Object: Parameter # 1!
 [0]PETSC ERROR:
 

 Commenting the line with MatSetOption removes the error, but
 doesn't solve the problem, because I actually need to
 KEEP_NONZERO_PATTERN

 Running the code sequentially, also vanishes the error.

 Please check whether it's a bug in petsc-3.3, or I should set this
 option in a different way.

 Thanks,

 Anton Popov




 -- 
 What most experimenters take for granted before they begin their 
 experiments is infinitely more interesting than any results to which 
 their experiments lead.
 -- Norbert Wiener

-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120620/451dc046/attachment-0001.html

[petsc-users] MatSetOption bug?

[Anton Popov]

Hi petsc team!
I've recently upgraded to 3.3, and currently experience problems. One of 
them follows.
If I compile this simple code:

#include petsc.h
#undef __FUNCT__
#define __FUNCT__ main
int main( int argc,char **argv )
{
 Mat A;
 PetscErrorCode ierr;
 PetscInt n_rows_Local=100;
 PetscInt n_cols_Local=100;
 ierr = PetscInitialize(argc, argv, PETSC_NULL, PETSC_NULL); 
CHKERRQ(ierr);
 ierr = MatCreate(PETSC_COMM_WORLD, A); CHKERRQ(ierr);
 ierr = MatSetSizes(A, n_rows_Local, n_cols_Local, PETSC_DETERMINE, 
PETSC_DETERMINE); CHKERRQ(ierr);
 ierr = MatSetType(A, MATAIJ); CHKERRQ(ierr);
 ierr = MatSetOption(A, MAT_KEEP_NONZERO_PATTERN, PETSC_TRUE); 
CHKERRQ(ierr);
 ierr = PetscFinalize(); CHKERRQ(ierr);
 PetscFunctionReturn(0);
}

and run it in parallel (mpiexec -np 2 ./example) it gives me errors like 
this:

[0]PETSC ERROR: - Error Message 

[0]PETSC ERROR: Null argument, when expecting valid pointer!
[0]PETSC ERROR: Null Object: Parameter # 1!
[0]PETSC ERROR: 


Commenting the line with MatSetOption removes the error, but doesn't 
solve the problem, because I actually need to KEEP_NONZERO_PATTERN

Running the code sequentially, also vanishes the error.

Please check whether it's a bug in petsc-3.3, or I should set this 
option in a different way.

Thanks,

Anton Popov

[petsc-users] Using direct solvers in parallel

[Anton Popov]

On 5/15/12 9:25 AM, Thomas Witkowski wrote:
 I made some comparisons of using umfpack, superlu, superlu_dist and 
 mumps to solve systems with sparse matrices arising from finite 
 element method. The size of the matrices range from around 5 to 
 more than 3 million unknowns. I used 1, 2, 4, 8 and 16 nodes to make 
 the benchmark. Now, I wonder that in all cases the sequential umfpack 
 was the fastest one. So even with 16 cores, superlu_dist and mumps are 
 slower. Can anybody of you confirm this observation? Are there any 
 other parallel direct solvers around which are more efficient?

 Thomas
You should definitely try PARDISO. On shared memory machine is 
unbeatable by any kind of MPI-based solver, and offers functionality for 
both symmetric and non-symmetric matrices. There is a version available 
from MKL library.

If you're fine enough with symmetric matrices, then this thing - HSL 
MA87 - is even better than PARDISO. It's a commercial solver, but it's 
free for academic use.

The only problem I see here is that both of them are difficult to use 
together with PETSc, since they're OMP-based. But difficult probably 
doesn't mean impossible.

Cheers,

Anton

[petsc-users] Debugger question

[Anton Popov]

I support 100% what Barry said. Just get the work done. Cray and IBM 
Linux systems do not support ALL the systems calls that PETSc uses. So 
it's always kind of problem to purge manually petscconf.h in between of 
configure and make on their machines. I wander how you could install 
any PETSc without modifying petscconf.h. If you just don't care, usually 
you get segfaults right at PetscInitialize() step. Literally it means, 
there is no way you can debug anything, they should reinstall PETSc, 
keeping in mind the exact list of system calls they support, and PETSc 
requirements.

By the way, the times when GNU compilers were order of magnitude 
slower than vendor compilers have passed long ago. Just give it a try, 
compile some simple computationally intensive code with gcc and 
something from vendor with aggressive optimization, and check 
execution time on a large data set. I'm sure you'll be surprised.

Cheers,
Anton

On 4/3/12 3:57 AM, Barry Smith wrote:
 On Apr 2, 2012, at 8:10 PM, Tabrez Ali wrote:

 Hello

 I am trying to debug a program using the switch '-on_error_attach_debugger' 
 but  the vendor/sysadmin built PETSc 3.2.00 is unable to start the debugger 
 in xterm (see text below). But xterm is installed. What am I doing wrong?

 Btw the segfault happens during a call to MatMult but only with 
 vendor/sysadmin supplied PETSc 3.2 with PGI and Intel compilers only and 
 _not_ with CRAY or GNU compilers.
 My advice, blow off the vendor/sysadmin supplied PETSc 3.2 and just 
 built it yourself so you can get real work done instead of trying to debug 
 their mess.   I promise the vendor one is not like a billion times faster or 
 anything.

 Barry



 I also dont get the segfault if I build PETSc 3.2-p7 myself with PGI/Intel 
 compilers.

 Any ideas on how to diagnose the problem? Unfortunately I cannot seem to run 
 valgrind on this particular machine.

 Thanks in advance.

 Tabrez

 ---

 stali at krakenpf1:~/meshes  which xterm
 /usr/bin/xterm
 stali at krakenpf1:~/meshes  aprun -n 1 ./defmod -f 
 2d_point_load_dyn_abc.inp -on_error_attach_debugger
 ...
 ...
 ...
 [0]PETSC ERROR: 
 
 [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
 probably memory access out of range
 [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
 [0]PETSC ERROR: or see 
 http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind[0]PETSC
  ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find 
 memory corruption errors
 [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and 
 run
 [0]PETSC ERROR: to get more information on the crash.
 [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown 
 file
 [0]PETSC ERROR: PETSC: Attaching gdb to ./defmod of pid 32384 on display 
 localhost:20.0 on machine nid10649
 Unable to start debugger in xterm: No such file or directory
 aborting job:
 application called MPI_Abort(MPI_COMM_WORLD, 0) - process 0
 _pmii_daemon(SIGCHLD): [NID 10649] [c23-3c0s6n1] [Mon Apr  2 13:06:48 2012] 
 PE 0 exit signal Aborted
 Application 133198 exit codes: 134
 Application 133198 resources: utime ~1s, stime ~0s

-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120403/0433a79d/attachment.htm

[petsc-users] errors in compiling ~\petsc-3.2-p7\src\ksp\ksp\examples\tutorials\ex2f.F

[Anton Popov]

Try adding -lgfortran to LFLAGS variable

On 3/20/12 3:18 AM, Roc Wang wrote:
 Hello,

 I am trying to compile the example source code of 
 ~\petsc-3.2-p7\src\ksp\ksp\examples\tutorials\ex2f.F. However the 
 error information shows:


 ex2f.o: In function `MAIN__':
 ex2f.F90:(.text+0x1b): undefined reference to `_gfortran_set_std'
 ex2f.F90:(.text+0x6f1): undefined reference to `_gfortran_st_write'
 ex2f.F90:(.text+0x706): undefined reference to `_gfortran_transfer_real'
 ex2f.F90:(.text+0x71b): undefined reference to 
 `_gfortran_transfer_integer'
 ex2f.F90:(.text+0x727): undefined reference to `_gfortran_st_write_done'
 ex2f.F90:(.text+0x773): undefined reference to `_gfortran_st_write'
 ex2f.F90:(.text+0x788): undefined reference to 
 `_gfortran_transfer_integer'
 ex2f.F90:(.text+0x794): undefined reference to `_gfortran_st_write_done'
 ex2f.o: In function `mykspmonitor_':
 ex2f.F90:(.text+0x896): undefined reference to `_gfortran_st_write'
 ex2f.F90:(.text+0x8ae): undefined reference to 
 `_gfortran_transfer_integer'
 ex2f.F90:(.text+0x8ba): undefined reference to `_gfortran_st_write_done'
 ex2f.F90:(.text+0x924): undefined reference to `_gfortran_st_write'
 ex2f.F90:(.text+0x93c): undefined reference to 
 `_gfortran_transfer_integer'
 ex2f.F90:(.text+0x954): undefined reference to `_gfortran_transfer_real'
 ex2f.F90:(.text+0x960): undefined reference to `_gfortran_st_write_done'
 make: *** [ex2f] Error 1

 My makefile is like this:

 
 PETSC_DIR =/usr/global/petsc/3.1-p8
 PETSC_ARCH =linux-intel11-debug
 FFLAGS  = -I${PETSC_DIR}/include -I${PETSC_DIR}/${PETSC_ARCH}/include
 LFLAGS = -L${PETSC_DIR}/${PETSC_ARCH}/lib -lpetsc\
 -L/usr/global/intel/mkl/10.3.1.107/mkl/lib/intel64\
 -Wl,-R/usr/global/intel/mkl/10.3.1.107/mkl/lib/intel64\
 -lmkl_solver_lp64_sequential\
 -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core 
 -Wl,--end-group\
 -lX11


 FC  = mpif90
 BIN = ex2f
 OBJS= ex2f.o


 ${BIN}:  ${OBJS}
 ${FC} -o ex2f ${OBJS} ${LFLAGS}

 ex2f.o:ex2f.F90
 ${FC} -c ${FFLAGS} ex2f.F90

 clean:
 rm -f ex2f *.o


 Should there any more libraries be included in the makefile?

-- next part --
An HTML attachment was scrubbed...
URL: 
http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120320/766d03e4/attachment-0001.htm

[petsc-users] multigrid_repartitioning

[Anton Popov]

Dear PETSc developers/users,

Does PETSc support repartitioning of coarse grid operators onto a 
smaller subset of processors during coarsening? Or the coarsest grid 
must be represented on the same amount of processors as the finest? It 
becomes a problem when a lot of processors are employed (say 4096 or 
more). If it is supported, how can I use it?

Thank you,

Anton Popov

[petsc-users] [petsc-maint #103068] problems compiling

[Anton Popov]

Hi Dominik,

It really depends on which Cray machine you are installing. If you like 
I can describe my recent installation experience on Cray XE6 Rosa in 
Zurich (it uses Linux).
First make sure which compiler you are using. For me, GNU compilers did 
the job well. I also prefer using ACML BLAS/LAPACK Library, because XE6 
uses AMD processors.

So just go to your .bashrc and add something like:

module load PrgEnv-gnu
module load acml

# PETSC 3.X
export PETSC_DIR=$HOME/LIB/petsc-3.X-pY
export PETSC_ARCH=PETSC_CRAY_XE6_GNU_OPT
export PATH=$PATH:$PETSC_DIR/bin/

Don't forget to logout/login for changes to take effect. Also, you 
should only have one PrgEnv module in your .bashrc.

Depending on the packages you need, you can modify the configuration 
command below:

./config/configure.py \
--with-batch=1 \
--known-mpi-shared=0 \
--known-memcmp-ok \
--with-blas-lapack-lib=$ACML_DIR/gfortran64/lib/libacml.a \
--COPTFLAGS=-O3 \
--FOPTFLAGS=-O3 \
--CXXOPTFLAGS=-O3 \
--with-x=0 \
--with-debugging=0 \
--with-clib-autodetect=0 \
--with-cxxlib-autodetect=0 \
--with-fortranlib-autodetect=0 \
--with-shared=0 \
--with-dynamic=0 \
--with-mpi-compilers=1 \
--with-cc=cc \
--with-cxx=CC \
--with-fc=ftn \
--download-blacs=1 \
--download-scalapack=1 \
--download-mumps=1 \
--download-superlu_dist=1 \
--download-parmetis=1 \
--download-ml=1

If you don't need ACML, just make PETSc download and install BLAS/LAPACK 
for you.

What is important here is to always keep the following:
--with-batch=1 \
--known-mpi-shared=0 \
--with-clib-autodetect=0 \
--with-cxxlib-autodetect=0 \
--with-fortranlib-autodetect=0 \

NOTE: on CRAY machine you have to submit conftest executable on single 
processor via batch system (Rosa uses slurm) and run reconfigure.py to 
finalize configuration.

After you proceed with this, you have to MANUALLY DELETE the following 
keys in $PETSC_DIR/$PETSC_ARCH/include/petscconf.h

PETSC_HAVE_SYS_PROCFS_H
PETSC_HAVE_DLFCN_H
PETSC_HAVE_SYS_SOCKET_H
PETSC_HAVE_SYS_UTSNAME_H
PETSC_HAVE_PWD_H
PETSC_HAVE_GETWD
PETSC_HAVE_UNAME
PETSC_HAVE_GETHOSTNAME
PETSC_HAVE_GETDOMAINNAME
PETSC_HAVE_NETINET_IN_H
PETSC_HAVE_NETDB_H
PETSC_USE_SOCKET_VIEWER
PETSC_HAVE_GETPWUID

Otherwise nothing will run, and you'll get segfaults in the very 
beginning. The reason behind it is that Cray version of Linux only 
supports reduced set of system calls.

NOW you can do make all

Hope it'll be helpful,

Anton



On 1/20/12 4:45 PM, Matthew Knepley wrote:
 On Fri, Jan 20, 2012 at 9:43 AM, Dominik Szczerba 
 dominik at itis.ethz.ch mailto:dominik at itis.ethz.ch wrote:

 I am building on a Cray system with crayftn fortran compiler. I
 specify --with-fc=crayftn and --FFLAGS=-em. Despite the latter, I am
 still getting the below cited error. Interestingly, the mod file IS
 THERE:

  find /tmp/petsc-0Bnd1w -iname \*.mod
 /tmp/petsc-0Bnd1w/config.compilers/CONFIGTEST.mod

 How do I go on from here?


 The problem is not that is doesn't create a module file, but that it 
 will not put it
 where we ask it to. Please send entire configure.log (don't Cc users).

Matt

 Thanks
 Dominik



 
 
 TEST checkFortranModuleInclude from
 
 config.compilers(/users/dsz/pack/petsc-3.2-p5/config/BuildSystem/config/compilers.py:1155)
 TESTING: checkFortranModuleInclude from
 config.compilers(config/BuildSystem/config/compilers.py:1155)
  Figures out what flag is used to specify the include path for
 Fortran modules
Pushing language FC
 sh: crayftn -c -o /tmp/petsc-0Bnd1w/config.compilers/conftest.o
 -I/tmp/petsc-0Bnd1w/config.compilers
 -I/tmp/petsc-0Bnd1w/config.setCompilers -em -O3
 /tmp/petsc-0Bnd1w/config.compilers/conftest.F
 Executing: crayftn -c -o /tmp/petsc-0Bnd1w/config.compilers/conftest.o
 -I/tmp/petsc-0Bnd1w/config.compilers
 -I/tmp/petsc-0Bnd1w/config.setCompilers -em -O3
 /tmp/petsc-0Bnd1w/config.compilers/conftest.F
 sh:
 Successful compile:
 Source:
  module configtest
  integer testint
  parameter (testint = 42)
  end module configtest
 
 ***
 UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log
 for details):
 
 ---
 Fortran module was not created during the compile.
 configtest.mod/CONFIGTEST.mod not found
 
 ***
  File /users/dsz/pack/petsc-3.2-p5/config/configure.py, line 283,
 in petsc_configure
framework.configure(out = sys.stdout)
  File
 /users/dsz/pack/petsc-3.2-p5/config/BuildSystem/config/framework.py,
 line 925, in configure
child.configure()
  File
 

[petsc-users] [petsc-maint #103068] problems compiling

[Anton Popov]

Well, I simply gave up on Cray compilers, as well as on the other 
compilers installed on this machine (Intel, Portland, Pathscale). After 
many attempts only GNU compilers were functional with PETSc.

On 1/20/12 6:12 PM, Dominik Szczerba wrote:
 Hi Anton,

 Many thanks for your hints. I am on the same machine. The key
 difference here is that I want to use the cray environment to compare
 performance with gnu. And I have no issues with gnu and my experience
 is very similar to yours.

 Many thanks and best regards,
 Dominik

 On Fri, Jan 20, 2012 at 6:08 PM, Anton Popovpopov at uni-mainz.de  wrote:
 Hi Dominik,

 It really depends on which Cray machine you are installing. If you like I
 can describe my recent installation experience on Cray XE6 Rosa in Zurich
 (it uses Linux).
 First make sure which compiler you are using. For me, GNU compilers did the
 job well. I also prefer using ACML BLAS/LAPACK Library, because XE6 uses AMD
 processors.

 So just go to your .bashrc and add something like:

 module load PrgEnv-gnu
 module load acml

 # PETSC 3.X
 export PETSC_DIR=$HOME/LIB/petsc-3.X-pY
 export PETSC_ARCH=PETSC_CRAY_XE6_GNU_OPT
 export PATH=$PATH:$PETSC_DIR/bin/

 Don't forget to logout/login for changes to take effect. Also, you should
 only have one PrgEnv module in your .bashrc.

 Depending on the packages you need, you can modify the configuration command
 below:

 ./config/configure.py \
 --with-batch=1 \
 --known-mpi-shared=0 \
 --known-memcmp-ok \
 --with-blas-lapack-lib=$ACML_DIR/gfortran64/lib/libacml.a \
 --COPTFLAGS=-O3 \
 --FOPTFLAGS=-O3 \
 --CXXOPTFLAGS=-O3 \
 --with-x=0 \
 --with-debugging=0 \
 --with-clib-autodetect=0 \
 --with-cxxlib-autodetect=0 \
 --with-fortranlib-autodetect=0 \
 --with-shared=0 \
 --with-dynamic=0 \
 --with-mpi-compilers=1 \
 --with-cc=cc \
 --with-cxx=CC \
 --with-fc=ftn \
 --download-blacs=1 \
 --download-scalapack=1 \
 --download-mumps=1 \
 --download-superlu_dist=1 \
 --download-parmetis=1 \
 --download-ml=1

 If you don't need ACML, just make PETSc download and install BLAS/LAPACK for
 you.

 What is important here is to always keep the following:
 --with-batch=1 \
 --known-mpi-shared=0 \
 --with-clib-autodetect=0 \
 --with-cxxlib-autodetect=0 \
 --with-fortranlib-autodetect=0 \

 NOTE: on CRAY machine you have to submit conftest executable on single
 processor via batch system (Rosa uses slurm) and run reconfigure.py to
 finalize configuration.

 After you proceed with this, you have to MANUALLY DELETE the following keys
 in $PETSC_DIR/$PETSC_ARCH/include/petscconf.h

 PETSC_HAVE_SYS_PROCFS_H
 PETSC_HAVE_DLFCN_H
 PETSC_HAVE_SYS_SOCKET_H
 PETSC_HAVE_SYS_UTSNAME_H
 PETSC_HAVE_PWD_H
 PETSC_HAVE_GETWD
 PETSC_HAVE_UNAME
 PETSC_HAVE_GETHOSTNAME
 PETSC_HAVE_GETDOMAINNAME
 PETSC_HAVE_NETINET_IN_H
 PETSC_HAVE_NETDB_H
 PETSC_USE_SOCKET_VIEWER
 PETSC_HAVE_GETPWUID

 Otherwise nothing will run, and you'll get segfaults in the very beginning.
 The reason behind it is that Cray version of Linux only supports reduced set
 of system calls.

 NOW you can do make all

 Hope it'll be helpful,

 Anton

 


 On 1/20/12 4:45 PM, Matthew Knepley wrote:

 On Fri, Jan 20, 2012 at 9:43 AM, Dominik Szczerbadominik at itis.ethz.ch
 wrote:
 I am building on a Cray system with crayftn fortran compiler. I
 specify --with-fc=crayftn and --FFLAGS=-em. Despite the latter, I am
 still getting the below cited error. Interestingly, the mod file IS
 THERE:

 find /tmp/petsc-0Bnd1w -iname \*.mod
 /tmp/petsc-0Bnd1w/config.compilers/CONFIGTEST.mod

 How do I go on from here?

 The problem is not that is doesn't create a module file, but that it will
 not put it
 where we ask it to. Please send entire configure.log (don't Cc users).

 Matt

 Thanks
 Dominik




 
 TEST checkFortranModuleInclude from

 config.compilers(/users/dsz/pack/petsc-3.2-p5/config/BuildSystem/config/compilers.py:1155)
 TESTING: checkFortranModuleInclude from
 config.compilers(config/BuildSystem/config/compilers.py:1155)
   Figures out what flag is used to specify the include path for Fortran
 modules
 Pushing language FC
 sh: crayftn -c -o /tmp/petsc-0Bnd1w/config.compilers/conftest.o
 -I/tmp/petsc-0Bnd1w/config.compilers
 -I/tmp/petsc-0Bnd1w/config.setCompilers -em -O3
 /tmp/petsc-0Bnd1w/config.compilers/conftest.F
 Executing: crayftn -c -o /tmp/petsc-0Bnd1w/config.compilers/conftest.o
 -I/tmp/petsc-0Bnd1w/config.compilers
 -I/tmp/petsc-0Bnd1w/config.setCompilers -em -O3
 /tmp/petsc-0Bnd1w/config.compilers/conftest.F
 sh:
 Successful compile:
 Source:
   module configtest
   integer testint
   parameter (testint = 42)
   end module configtest

 ***
  UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log
 for details):

 ---
 Fortran