[petsc-users] Including constrained dofs in a matrix/vector created with DMCreateXXXX()

[Jordan Wagner via petsc-users]

Hi,

Over the past few months, I have implemented dmplex/section in my 
preexisting finite element code. We already had our own implementations 
for things like the finite element space, boundary conditions, 
projections, etc, so I am mostly using dmplex/section at its lowest 
level; no DT/DS stuff for the moment. Everything is working fine at this 
level, and I am overall very happy with how it ties into the code that 
we already have.


I am currently doing my own preallocation for the system Jacobian but am 
wanting to use DMCreateMatrix() instead, as it seems pointless to keep 
using my own crappy function when this one is just sitting there 
waiting. However, I dislike how I am currently implementing this and was 
hoping to get some suggestions or some advice. My problem is handling 
nicely the Dirichlet dofs. My code takes the approach of first 
assembling the entire Jacobian matrix and load vector as if no 
constraints are imposed. We then have a function that applies the 
essential conditions after assembly and extracts a submatrix and 
subvector similar to the way ex49 
 
is doing.


Since DMCreateMatrix() automatically partitions out the constraints 
specified in the section, I find myself having to create two nearly 
identical sections: one that has constraints and one that does not, and 
then I clone the DM and set the default section of each respectively. I 
can then use DMCreateMatrix() on the one dm/section with no constraints 
to preallocate the larger matrix that I am assembling. Then I use the 
dm/section with constraints to do all of my boundary condition and 
projection operations on the previously created matrix and vector. 
Essentially, I am using an unconstrained section only for allocation and 
a constrained section for most everything else.


This process actually works fine for me, but its pretty ugly, and I am 
sure there is a better way. I am wondering if I am missing something 
that could keep me from having to go through this process of creating 
two sections that differ only in the constraints. It seems to me if 
there were an option for DMCreateMatrix()/DMCreateVector() to either 
include or not include the dofs, that would completely solve my problem. 
That way, I can use the same section for both creation and allocation.


So, the main question I have is just whether there is a function or 
option that I am missing like DMCreate() but with the ability to 
retain the constrained rows and columns rather than compressing them out.


Thanks a lot, as always.

Re: [petsc-users] PetscBinaryRead fails

[Yuyun Yang via petsc-users]

Oh I see, thanks for confirming it works with an 8-byte file. I didn't generate 
it from a code but just put a random number in a blank file. I thought it would 
automatically be the right size but that isn't the case.


Get Outlook for iOS

From: Smith, Barry F. 
Sent: Monday, February 18, 2019 8:06:16 PM
To: Yuyun Yang
Cc: petsc-users@mcs.anl.gov
Subject: Re: [petsc-users] PetscBinaryRead fails


  I don't know how you got your _test file but it is the wrong size

$ ls -l _test
-rw-r--r--@ 1 barrysmith  staff  6 Feb 18 21:25 _test
~/Src/petsc/test-dir (maint=) arch-basic

it only has 6 bytes but to hold a PETSC_SCALAR (when PetscScalar is real) it 
should have 8 bytes. For example the code

  PetscViewer viewer;
  PetscScalar v;
  int fd;

  PetscViewerBinaryOpen(PETSC_COMM_WORLD, "joe", FILE_MODE_WRITE, );
PetscViewerBinaryGetDescriptor(viewer, );
PetscBinaryWrite(fd, , 1, PETSC_SCALAR,PETSC_FALSE);
PetscViewerDestroy();

produces a file with 8 bytes in it

$ ls -l joe
-rw-r--r--  1 barrysmith  staff  8 Feb 18 22:03 joe

which can be read in by your program.

> On Feb 18, 2019, at 9:23 PM, Yuyun Yang  wrote:
>
> I used FILE_MODE_READ. Please see attached file (the function is part of 
> another file therefore all the other include statements; I've extracted it 
> here). The test file only contains one number. My directory is hard-coded 
> into the main function.
>
> Thanks a lot!
> Yuyun
>
> -Original Message-
> From: Smith, Barry F. 
> Sent: Monday, February 18, 2019 6:41 PM
> To: Yuyun Yang 
> Cc: petsc-users@mcs.anl.gov
> Subject: Re: [petsc-users] PetscBinaryRead fails
>
>
>  Send program and data file if you cannot get it to work.
>
>
>> On Feb 18, 2019, at 8:08 PM, Yuyun Yang via petsc-users 
>>  wrote:
>>
>> Hello,
>>
>> I am writing a very simple function just to read in a single value from an 
>> input file (which also just contains one single scalar value). Using 
>> PetscBinaryRead(fd, , 1, PETSC_SCALAR) gives me the following error. 
>> Could you help me figure out why?
>>
>> [0]PETSC ERROR: - Error Message
>> --
>> [0]PETSC ERROR: Read from file failed
>> [0]PETSC ERROR: Read past end of file
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
>> trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown [0]PETSC ERROR:
>> ./output on a arch-linux2-c-debug named DESKTOP-02K82L1.stanford.edu
>> by yyy910805 Mon Feb 18 17:58:30 2019 [0]PETSC ERROR: Configure options 
>> --with-cc=gcc --with-cxx=g++ --with-shared-libraries=1 --download-hdf5 
>> --download-metis --download-parmetis --download-hypre --download-superlu 
>> --download-mumps --download-scalapack --download-mpich 
>> --download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
>> [0]PETSC ERROR: #1 PetscBinaryRead() line 276 in
>> /home/yyy910805/petsc/src/sys/fileio/sysio.c
>> 0.00
>>
>> Thanks a lot!
>> Yuyun
>
> <_test>

Re: [petsc-users] PetscBinaryRead fails

[Smith, Barry F. via petsc-users]

  I don't know how you got your _test file but it is the wrong size

$ ls -l _test
-rw-r--r--@ 1 barrysmith  staff  6 Feb 18 21:25 _test
~/Src/petsc/test-dir (maint=) arch-basic

it only has 6 bytes but to hold a PETSC_SCALAR (when PetscScalar is real) it 
should have 8 bytes. For example the code

  PetscViewer viewer;
  PetscScalar v;
  int fd;
  
  PetscViewerBinaryOpen(PETSC_COMM_WORLD, "joe", FILE_MODE_WRITE, );
PetscViewerBinaryGetDescriptor(viewer, );
PetscBinaryWrite(fd, , 1, PETSC_SCALAR,PETSC_FALSE);
PetscViewerDestroy();

produces a file with 8 bytes in it

$ ls -l joe
-rw-r--r--  1 barrysmith  staff  8 Feb 18 22:03 joe

which can be read in by your program.

> On Feb 18, 2019, at 9:23 PM, Yuyun Yang  wrote:
> 
> I used FILE_MODE_READ. Please see attached file (the function is part of 
> another file therefore all the other include statements; I've extracted it 
> here). The test file only contains one number. My directory is hard-coded 
> into the main function.
> 
> Thanks a lot!
> Yuyun
> 
> -Original Message-
> From: Smith, Barry F.  
> Sent: Monday, February 18, 2019 6:41 PM
> To: Yuyun Yang 
> Cc: petsc-users@mcs.anl.gov
> Subject: Re: [petsc-users] PetscBinaryRead fails
> 
> 
>  Send program and data file if you cannot get it to work.
> 
> 
>> On Feb 18, 2019, at 8:08 PM, Yuyun Yang via petsc-users 
>>  wrote:
>> 
>> Hello,
>> 
>> I am writing a very simple function just to read in a single value from an 
>> input file (which also just contains one single scalar value). Using 
>> PetscBinaryRead(fd, , 1, PETSC_SCALAR) gives me the following error. 
>> Could you help me figure out why?
>> 
>> [0]PETSC ERROR: - Error Message 
>> --
>> [0]PETSC ERROR: Read from file failed
>> [0]PETSC ERROR: Read past end of file
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
>> trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown [0]PETSC ERROR: 
>> ./output on a arch-linux2-c-debug named DESKTOP-02K82L1.stanford.edu 
>> by yyy910805 Mon Feb 18 17:58:30 2019 [0]PETSC ERROR: Configure options 
>> --with-cc=gcc --with-cxx=g++ --with-shared-libraries=1 --download-hdf5 
>> --download-metis --download-parmetis --download-hypre --download-superlu 
>> --download-mumps --download-scalapack --download-mpich 
>> --download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
>> [0]PETSC ERROR: #1 PetscBinaryRead() line 276 in 
>> /home/yyy910805/petsc/src/sys/fileio/sysio.c
>> 0.00
>> 
>> Thanks a lot!
>> Yuyun
> 
> <_test>

Re: [petsc-users] PetscBinaryRead fails

[Yuyun Yang via petsc-users]

I used FILE_MODE_READ. Please see attached file (the function is part of 
another file therefore all the other include statements; I've extracted it 
here). The test file only contains one number. My directory is hard-coded into 
the main function.

Thanks a lot!
Yuyun

-Original Message-
From: Smith, Barry F.  
Sent: Monday, February 18, 2019 6:41 PM
To: Yuyun Yang 
Cc: petsc-users@mcs.anl.gov
Subject: Re: [petsc-users] PetscBinaryRead fails


  Send program and data file if you cannot get it to work.


> On Feb 18, 2019, at 8:08 PM, Yuyun Yang via petsc-users 
>  wrote:
> 
> Hello,
>  
> I am writing a very simple function just to read in a single value from an 
> input file (which also just contains one single scalar value). Using 
> PetscBinaryRead(fd, , 1, PETSC_SCALAR) gives me the following error. 
> Could you help me figure out why?
>  
> [0]PETSC ERROR: - Error Message 
> --
> [0]PETSC ERROR: Read from file failed
> [0]PETSC ERROR: Read past end of file
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
> trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown [0]PETSC ERROR: 
> ./output on a arch-linux2-c-debug named DESKTOP-02K82L1.stanford.edu 
> by yyy910805 Mon Feb 18 17:58:30 2019 [0]PETSC ERROR: Configure options 
> --with-cc=gcc --with-cxx=g++ --with-shared-libraries=1 --download-hdf5 
> --download-metis --download-parmetis --download-hypre --download-superlu 
> --download-mumps --download-scalapack --download-mpich 
> --download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
> [0]PETSC ERROR: #1 PetscBinaryRead() line 276 in 
> /home/yyy910805/petsc/src/sys/fileio/sysio.c
> 0.00
>  
> Thanks a lot!
> Yuyun

//#include "genFuncs.hpp"

#include 
#include 
#include 
#include 
#include 
#include 
#include 

#include 
#include 
#include 

using namespace std;

PetscErrorCode loadValueFromCheckpoint(const string outputDir, const string 
filename, PetscScalar ) {
  PetscErrorCode ierr = 0;
  #if VERBOSE > 1
  ierr = PetscPrinf(PETSC_COMM_WORLD, "Starting loadValueFromCheckpoint in 
genFuncs.cpp.\n"); CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD, "Attempting to load %s%s\n", 
outputDir.c_str(), filename.c_str()); CHKERRQ(ierr);
  #endif

  string checkpointFile = outputDir + filename;
  bool fileExists = doesFileExist(checkpointFile);
  if (fileExists) {
PetscPrintf(PETSC_COMM_WORLD, "Loading %s\n", filename.c_str());
PetscViewer viewer;
int fd;
PetscViewerBinaryOpen(PETSC_COMM_WORLD, checkpointFile.c_str(), 
FILE_MODE_READ, );
PetscViewerBinaryGetDescriptor(viewer, );
PetscBinaryRead(fd, , 1, PETSC_SCALAR);
PetscViewerDestroy();
  }
  else {
PetscPrintf(PETSC_COMM_WORLD, "Warning: %s not found.\n", 
checkpointFile.c_str());
  }

  #if VERBOSE > 1
  ierr = PetscPrintf(PETSC_COMM_WORLD, "Ending loadValueFromCheckpoint in 
%s.\n", filename.c_str()); CHKERRQ(ierr);
  #endif

  return ierr;
}


int main(int argc, char **args) {
  PetscErrorCode ierr = 0;

  ierr = PetscInitialize(, , NULL, NULL); CHKERRQ(ierr);

  // directory and filename
  const string outputDir = "/home/yyy910805/";
  const string filename = "_test";
  PetscScalar value = 0;

  loadValueFromCheckpoint(outputDir, filename, value);
  ierr = PetscPrintf(PETSC_COMM_WORLD, " %f\n", value); CHKERRQ(ierr);

  ierr = PetscFinalize(); CHKERRQ(ierr);
  
  return 0;
}


_test
Description: _test

Re: [petsc-users] PetscBinaryRead fails

[Smith, Barry F. via petsc-users]

  Send program and data file if you cannot get it to work.


> On Feb 18, 2019, at 8:08 PM, Yuyun Yang via petsc-users 
>  wrote:
> 
> Hello,
>  
> I am writing a very simple function just to read in a single value from an 
> input file (which also just contains one single scalar value). Using 
> PetscBinaryRead(fd, , 1, PETSC_SCALAR) gives me the following error. 
> Could you help me figure out why?
>  
> [0]PETSC ERROR: - Error Message 
> --
> [0]PETSC ERROR: Read from file failed
> [0]PETSC ERROR: Read past end of file
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
> trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, unknown
> [0]PETSC ERROR: ./output on a arch-linux2-c-debug named 
> DESKTOP-02K82L1.stanford.edu by yyy910805 Mon Feb 18 17:58:30 2019
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ 
> --with-shared-libraries=1 --download-hdf5 --download-metis 
> --download-parmetis --download-hypre --download-superlu --download-mumps 
> --download-scalapack --download-mpich 
> --download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
> [0]PETSC ERROR: #1 PetscBinaryRead() line 276 in 
> /home/yyy910805/petsc/src/sys/fileio/sysio.c
> 0.00
>  
> Thanks a lot!
> Yuyun

[petsc-users] PetscBinaryRead fails

[Yuyun Yang via petsc-users]

Hello,

I am writing a very simple function just to read in a single value from an 
input file (which also just contains one single scalar value). Using 
PetscBinaryRead(fd, , 1, PETSC_SCALAR) gives me the following error. 
Could you help me figure out why?

[0]PETSC ERROR: - Error Message 
--
[0]PETSC ERROR: Read from file failed
[0]PETSC ERROR: Read past end of file
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, unknown
[0]PETSC ERROR: ./output on a arch-linux2-c-debug named 
DESKTOP-02K82L1.stanford.edu by yyy910805 Mon Feb 18 17:58:30 2019
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ 
--with-shared-libraries=1 --download-hdf5 --download-metis --download-parmetis 
--download-hypre --download-superlu --download-mumps --download-scalapack 
--download-mpich 
--download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
[0]PETSC ERROR: #1 PetscBinaryRead() line 276 in 
/home/yyy910805/petsc/src/sys/fileio/sysio.c
0.00

Thanks a lot!
Yuyun

Re: [petsc-users] saving results

[Smith, Barry F. via petsc-users]

   I'm guessing you mange your own time stepping (and nonlinear solver if there 
is one)? 

   You can save the solution with a call to VecView() and then reload the 
solution with a VecLoad() but you need to manage any other restart data that 
you may need, like the value of the current time etc. 

   As Jed says it doesn't make sense to save "partial" solutions within the 
linear solver.  Just save it every several time-steps.

   Barry

   Also the code should not be "crashing" at seemingly long times (after hours) 
with a Segmentation fault. Send us the full error message and we'll see if 
there is some way we can help you fix this problem.


> On Feb 18, 2019, at 9:47 AM, Sal Am via petsc-users  
> wrote:
> 
> Is there a function/command line option to save the solution as it is solving 
> (and read in the file from where it crashed and keep iterating from there 
> perhaps)?
> Had a seg fault and all the results until that point seems to have been lost. 
> 
> 
> 
> 

Re: [petsc-users] saving results

[Jed Brown via petsc-users]

What kind of problems are you solving?  Running for a Krylov method for
tens of thousands of iterations is very rarely recommended.

Regarding storage, it's significantly more expensive to store the Krylov
basis (even when it's a recurrence) than the current approximation.
Some methods require some work to compute the current approximation.
Anyway, you can use a KSPMonitor to write checkpoints at whatever
interval you like.  In case of a crash, use VecLoad() to read in the
checkpoint file and use KSPSetInitialGuessNonzero() to make it be used
as an initial guess.

Sal Am  writes:

> This is how I run it:
>
> -ksp_type bcgs -pc_type gamg -mattransposematmult_via scalable
> -build_twosided allreduce -ksp_monitor_true_residual
> -ksp_monitor_if_not_converged -log_view -ksp_max_it 10 -ksp_rtol 1.0e-7
>
> so BCGS with GAMG as preconditioner. I am guessing writing at every
> timestep would be expensive, maybe every hour? I am not sure what would be
> a good number here if the simulations lasts more than a day.
>
>
>
> On Mon, Feb 18, 2019 at 4:10 PM Jed Brown  wrote:
>
>> What kind of solver are you using and how often do you want to write?
>>
>> Sal Am via petsc-users  writes:
>>
>> > Is there a function/command line option to save the solution as it is
>> > solving (and read in the file from where it crashed and keep iterating
>> from
>> > there perhaps)?
>> > Had a seg fault and all the results until that point seems to have been
>> > lost.
>>

Re: [petsc-users] saving results

[Jed Brown via petsc-users]

What kind of solver are you using and how often do you want to write?

Sal Am via petsc-users  writes:

> Is there a function/command line option to save the solution as it is
> solving (and read in the file from where it crashed and keep iterating from
> there perhaps)?
> Had a seg fault and all the results until that point seems to have been
> lost.