Re: [petsc-dev] block size information in gamg
FYI, I pushed a fix for snes/ex56 into knepley/feature-plex-bdfunc so this should fix next. On Fri, Oct 14, 2016 at 12:25 PM, Mark Adams wrote: > Note, I have a pull request in (for weeks) that adds a convergence study > to snes/ex56 so this might conflict Stefano is doing. > > Also, next is broken because of what I think was an API change from Matt > et.al. We are getting a bit tied up here. > > On Fri, Oct 14, 2016 at 11:05 AM, Mark Adams wrote: > >> Stefano, There is a bug, I think, that was discussed on an earlier thread >> and it did not get resolved AFAIK. If you feel like folding this into this >> project (if you page it back in) then that would be nice. Here is a quote: >> >> My pull request for snes/ex56 has: >> >> ierr = VecNorm(xx,NORM_INFINITY,&mdisp[iter]);CHKERRQ(ierr); >> >> When I tried to use VecStrideNorm I got the error that block size was >> not set. You can't set it manually so I was stuck (not a big deal, but >> there is either a bug in Plex or a bug in the way that I am using it). >> >> Mark >> >> On Tue, Oct 11, 2016 at 8:54 AM, Stefano Zampini < >> stefano.zamp...@gmail.com> wrote: >> >>> Here is a first attempt: stefano_zampini/allow-late-matsetblocksizes >>> >>> The new block sizes are not yet propagated to the diagonal and >>> off-diagonal part. >>> ex56 seems to produce the same AMG hierarchy. >>> >>> 2016-10-11 9:54 GMT+03:00 Stefano Zampini : >>> > > So it doesn't want to change the block size if map->range has been set > (i.e. PetscLayoutSetUp()) was called. But if the matrix is AIJ so long as > !(map->n % bs) one can actually set the block size at any time. So how > about if I change the Mat code to allow setting the block size late for > AIJ? Maybe we can also get rid of PCFieldSplitSetBlockSize() and a > PCSPAISetBlockSize()? > > This is precisely what I was referring to as hacking. For sure MatSetBlockSize (and PetscLayoutSetBlockSize) can be patched if the bs value passed in is valid (i.e. !(map->n % bs) == 1). > One issue that comes up with this change is: should setting the block > size later automatically reset the block size of the two inner AIJ > matrices? Or should they remain one? I could have it update the inner > block > sizes at the expense of a tiny bit more code. But one complication is that > MatSetUpMultiply_MPIAIJ() always sets the column block size of the > off-diagaonl matrix to 1; so this needs to be respected. > > I think the row block size of the inner matrices should be modified, as well as the column block size of the diagonal part. We should write a comprehensive test that sets the block sizes after matrix setup to really understand what needs to be fixed. I can take care of it if you wish. The PETSc design policy of "there is one way to do something" dictates > that it is best not to have PCSetBlockSize() unless absolutely necessary > so > I am trying to see if it is absolutely necessary. > > I was not asking for a general PCSetBlockSize(); just for a way to customize GAMG when needed (something like HYPRE_BoomerAMGSetNumFunc tions) > Barry > > > > > > On Oct 10, 2016, at 10:18 AM, Stefano Zampini < > stefano.zamp...@gmail.com> wrote: > > > > Mark, > > > > I was wondering if you could provide an API/command line > customization for PCGAMG to explicitly provide the "block size" of the > problem to the gamg solver, instead of using the block size of the matrix. > The matrix block size could be always used as a fallback if this new > information as not been provided. > > > > I'm interfacing a complicated finite element package to PETSc , > where you cannot do any assumption on the blocksize of the problem during > matrix setup, and (AFAIK) I cannot change the block size with a call to > MatSetBlockSize after the matrix has been finalized without hacking (that > I > would like to avoid). > > > > Thanks > > -- > > Stefano > > -- Stefano >>> >>> >>> >>> -- >>> Stefano >>> >> >> >
Re: [petsc-dev] block size information in gamg
Note, I have a pull request in (for weeks) that adds a convergence study to snes/ex56 so this might conflict Stefano is doing. Also, next is broken because of what I think was an API change from Matt et.al. We are getting a bit tied up here. On Fri, Oct 14, 2016 at 11:05 AM, Mark Adams wrote: > Stefano, There is a bug, I think, that was discussed on an earlier thread > and it did not get resolved AFAIK. If you feel like folding this into this > project (if you page it back in) then that would be nice. Here is a quote: > > My pull request for snes/ex56 has: > > ierr = VecNorm(xx,NORM_INFINITY,&mdisp[iter]);CHKERRQ(ierr); > > When I tried to use VecStrideNorm I got the error that block size was not > set. You can't set it manually so I was stuck (not a big deal, but there is > either a bug in Plex or a bug in the way that I am using it). > > Mark > > On Tue, Oct 11, 2016 at 8:54 AM, Stefano Zampini < > stefano.zamp...@gmail.com> wrote: > >> Here is a first attempt: stefano_zampini/allow-late-matsetblocksizes >> >> The new block sizes are not yet propagated to the diagonal and >> off-diagonal part. >> ex56 seems to produce the same AMG hierarchy. >> >> 2016-10-11 9:54 GMT+03:00 Stefano Zampini : >> >>> >>> >>> So it doesn't want to change the block size if map->range has been set (i.e. PetscLayoutSetUp()) was called. But if the matrix is AIJ so long as !(map->n % bs) one can actually set the block size at any time. So how about if I change the Mat code to allow setting the block size late for AIJ? Maybe we can also get rid of PCFieldSplitSetBlockSize() and a PCSPAISetBlockSize()? >>> This is precisely what I was referring to as hacking. For sure >>> MatSetBlockSize (and PetscLayoutSetBlockSize) can be patched if the bs >>> value passed in is valid (i.e. !(map->n % bs) == 1). >>> >>> One issue that comes up with this change is: should setting the block size later automatically reset the block size of the two inner AIJ matrices? Or should they remain one? I could have it update the inner block sizes at the expense of a tiny bit more code. But one complication is that MatSetUpMultiply_MPIAIJ() always sets the column block size of the off-diagaonl matrix to 1; so this needs to be respected. >>> I think the row block size of the inner matrices should be modified, as >>> well as the column block size of the diagonal part. >>> We should write a comprehensive test that sets the block sizes after >>> matrix setup to really understand what needs to be fixed. >>> I can take care of it if you wish. >>> >>> The PETSc design policy of "there is one way to do something" dictates that it is best not to have PCSetBlockSize() unless absolutely necessary so I am trying to see if it is absolutely necessary. >>> I was not asking for a general PCSetBlockSize(); just for a way to >>> customize GAMG when needed (something like HYPRE_BoomerAMGSetNumFunc >>> tions) >>> >>> >>> Barry > On Oct 10, 2016, at 10:18 AM, Stefano Zampini < stefano.zamp...@gmail.com> wrote: > > Mark, > > I was wondering if you could provide an API/command line customization for PCGAMG to explicitly provide the "block size" of the problem to the gamg solver, instead of using the block size of the matrix. The matrix block size could be always used as a fallback if this new information as not been provided. > > I'm interfacing a complicated finite element package to PETSc , where you cannot do any assumption on the blocksize of the problem during matrix setup, and (AFAIK) I cannot change the block size with a call to MatSetBlockSize after the matrix has been finalized without hacking (that I would like to avoid). > > Thanks > -- > Stefano >>> >>> >>> -- >>> Stefano >>> >> >> >> >> -- >> Stefano >> > >
Re: [petsc-dev] block size information in gamg
Stefano, There is a bug, I think, that was discussed on an earlier thread and it did not get resolved AFAIK. If you feel like folding this into this project (if you page it back in) then that would be nice. Here is a quote: My pull request for snes/ex56 has: ierr = VecNorm(xx,NORM_INFINITY,&mdisp[iter]);CHKERRQ(ierr); When I tried to use VecStrideNorm I got the error that block size was not set. You can't set it manually so I was stuck (not a big deal, but there is either a bug in Plex or a bug in the way that I am using it). Mark On Tue, Oct 11, 2016 at 8:54 AM, Stefano Zampini wrote: > Here is a first attempt: stefano_zampini/allow-late-matsetblocksizes > > The new block sizes are not yet propagated to the diagonal and > off-diagonal part. > ex56 seems to produce the same AMG hierarchy. > > 2016-10-11 9:54 GMT+03:00 Stefano Zampini : > >> >> >> >>> >>> So it doesn't want to change the block size if map->range has been set >>> (i.e. PetscLayoutSetUp()) was called. But if the matrix is AIJ so long as >>> !(map->n % bs) one can actually set the block size at any time. So how >>> about if I change the Mat code to allow setting the block size late for >>> AIJ? Maybe we can also get rid of PCFieldSplitSetBlockSize() and a >>> PCSPAISetBlockSize()? >>> >>> >> This is precisely what I was referring to as hacking. For sure >> MatSetBlockSize (and PetscLayoutSetBlockSize) can be patched if the bs >> value passed in is valid (i.e. !(map->n % bs) == 1). >> >> >>> One issue that comes up with this change is: should setting the block >>> size later automatically reset the block size of the two inner AIJ >>> matrices? Or should they remain one? I could have it update the inner block >>> sizes at the expense of a tiny bit more code. But one complication is that >>> MatSetUpMultiply_MPIAIJ() always sets the column block size of the >>> off-diagaonl matrix to 1; so this needs to be respected. >>> >>> >> I think the row block size of the inner matrices should be modified, as >> well as the column block size of the diagonal part. >> We should write a comprehensive test that sets the block sizes after >> matrix setup to really understand what needs to be fixed. >> I can take care of it if you wish. >> >> The PETSc design policy of "there is one way to do something" dictates >>> that it is best not to have PCSetBlockSize() unless absolutely necessary so >>> I am trying to see if it is absolutely necessary. >>> >>> >> I was not asking for a general PCSetBlockSize(); just for a way to >> customize GAMG when needed (something like HYPRE_BoomerAMGSetNumFunc >> tions) >> >> >> >>> Barry >>> >>> >>> >>> >>> > On Oct 10, 2016, at 10:18 AM, Stefano Zampini < >>> stefano.zamp...@gmail.com> wrote: >>> > >>> > Mark, >>> > >>> > I was wondering if you could provide an API/command line customization >>> for PCGAMG to explicitly provide the "block size" of the problem to the >>> gamg solver, instead of using the block size of the matrix. The matrix >>> block size could be always used as a fallback if this new information as >>> not been provided. >>> > >>> > I'm interfacing a complicated finite element package to PETSc , where >>> you cannot do any assumption on the blocksize of the problem during matrix >>> setup, and (AFAIK) I cannot change the block size with a call to >>> MatSetBlockSize after the matrix has been finalized without hacking (that I >>> would like to avoid). >>> > >>> > Thanks >>> > -- >>> > Stefano >>> >>> >> >> >> -- >> Stefano >> > > > > -- > Stefano >
Re: [petsc-dev] block size information in gamg
Here is a first attempt: stefano_zampini/allow-late-matsetblocksizes The new block sizes are not yet propagated to the diagonal and off-diagonal part. ex56 seems to produce the same AMG hierarchy. 2016-10-11 9:54 GMT+03:00 Stefano Zampini : > > > >> >> So it doesn't want to change the block size if map->range has been set >> (i.e. PetscLayoutSetUp()) was called. But if the matrix is AIJ so long as >> !(map->n % bs) one can actually set the block size at any time. So how >> about if I change the Mat code to allow setting the block size late for >> AIJ? Maybe we can also get rid of PCFieldSplitSetBlockSize() and a >> PCSPAISetBlockSize()? >> >> > This is precisely what I was referring to as hacking. For sure > MatSetBlockSize (and PetscLayoutSetBlockSize) can be patched if the bs > value passed in is valid (i.e. !(map->n % bs) == 1). > > >> One issue that comes up with this change is: should setting the block >> size later automatically reset the block size of the two inner AIJ >> matrices? Or should they remain one? I could have it update the inner block >> sizes at the expense of a tiny bit more code. But one complication is that >> MatSetUpMultiply_MPIAIJ() always sets the column block size of the >> off-diagaonl matrix to 1; so this needs to be respected. >> >> > I think the row block size of the inner matrices should be modified, as > well as the column block size of the diagonal part. > We should write a comprehensive test that sets the block sizes after > matrix setup to really understand what needs to be fixed. > I can take care of it if you wish. > > The PETSc design policy of "there is one way to do something" dictates >> that it is best not to have PCSetBlockSize() unless absolutely necessary so >> I am trying to see if it is absolutely necessary. >> >> > I was not asking for a general PCSetBlockSize(); just for a way to > customize GAMG when needed (something like HYPRE_BoomerAMGSetNumFunctions) > > > >> Barry >> >> >> >> >> > On Oct 10, 2016, at 10:18 AM, Stefano Zampini < >> stefano.zamp...@gmail.com> wrote: >> > >> > Mark, >> > >> > I was wondering if you could provide an API/command line customization >> for PCGAMG to explicitly provide the "block size" of the problem to the >> gamg solver, instead of using the block size of the matrix. The matrix >> block size could be always used as a fallback if this new information as >> not been provided. >> > >> > I'm interfacing a complicated finite element package to PETSc , where >> you cannot do any assumption on the blocksize of the problem during matrix >> setup, and (AFAIK) I cannot change the block size with a call to >> MatSetBlockSize after the matrix has been finalized without hacking (that I >> would like to avoid). >> > >> > Thanks >> > -- >> > Stefano >> >> > > > -- > Stefano > -- Stefano
Re: [petsc-dev] block size information in gamg
> > So it doesn't want to change the block size if map->range has been set > (i.e. PetscLayoutSetUp()) was called. But if the matrix is AIJ so long as > !(map->n % bs) one can actually set the block size at any time. So how > about if I change the Mat code to allow setting the block size late for > AIJ? Maybe we can also get rid of PCFieldSplitSetBlockSize() and a > PCSPAISetBlockSize()? > > This is precisely what I was referring to as hacking. For sure MatSetBlockSize (and PetscLayoutSetBlockSize) can be patched if the bs value passed in is valid (i.e. !(map->n % bs) == 1). > One issue that comes up with this change is: should setting the block size > later automatically reset the block size of the two inner AIJ matrices? Or > should they remain one? I could have it update the inner block sizes at the > expense of a tiny bit more code. But one complication is that > MatSetUpMultiply_MPIAIJ() always sets the column block size of the > off-diagaonl matrix to 1; so this needs to be respected. > > I think the row block size of the inner matrices should be modified, as well as the column block size of the diagonal part. We should write a comprehensive test that sets the block sizes after matrix setup to really understand what needs to be fixed. I can take care of it if you wish. The PETSc design policy of "there is one way to do something" dictates that > it is best not to have PCSetBlockSize() unless absolutely necessary so I am > trying to see if it is absolutely necessary. > > I was not asking for a general PCSetBlockSize(); just for a way to customize GAMG when needed (something like HYPRE_BoomerAMGSetNumFunctions) > Barry > > > > > > On Oct 10, 2016, at 10:18 AM, Stefano Zampini > wrote: > > > > Mark, > > > > I was wondering if you could provide an API/command line customization > for PCGAMG to explicitly provide the "block size" of the problem to the > gamg solver, instead of using the block size of the matrix. The matrix > block size could be always used as a fallback if this new information as > not been provided. > > > > I'm interfacing a complicated finite element package to PETSc , where > you cannot do any assumption on the blocksize of the problem during matrix > setup, and (AFAIK) I cannot change the block size with a call to > MatSetBlockSize after the matrix has been finalized without hacking (that I > would like to avoid). > > > > Thanks > > -- > > Stefano > > -- Stefano
Re: [petsc-dev] block size information in gamg
Stefano, There is currently a PCFieldSplitSetBlockSize() and a PCSPAISetBlockSize() PetscErrorCode MatSetBlockSize(Mat mat,PetscInt bs) { PetscErrorCode ierr; PetscFunctionBegin; PetscValidHeaderSpecific(mat,MAT_CLASSID,1); PetscValidLogicalCollectiveInt(mat,bs,2); ierr = PetscLayoutSetBlockSize(mat->rmap,bs);CHKERRQ(ierr); ierr = PetscLayoutSetBlockSize(mat->cmap,bs);CHKERRQ(ierr); PetscErrorCode PetscLayoutSetBlockSize(PetscLayout map,PetscInt bs) { PetscFunctionBegin; if (bs < 0) PetscFunctionReturn(0); if (map->n > 0 && map->n % bs) SETERRQ2(map->comm,PETSC_ERR_ARG_INCOMP,"Local size %D not compatible with block size %D",map->n,bs); if (map->range && map->bs > 0 && map->bs != bs) SETERRQ2(map->comm,PETSC_ERR_ARG_INCOMP,"Cannot change block size %D to %D",map->bs,bs); if (map->mapping) { PetscInt lbs; PetscErrorCode ierr; ierr = ISLocalToGlobalMappingGetBlockSize(map->mapping,&lbs);CHKERRQ(ierr); if (lbs != bs) SETERRQ2(map->comm,PETSC_ERR_PLIB,"Blocksize of localtoglobalmapping %D must match that of layout %D",lbs,bs); } map->bs = bs; PetscFunctionReturn(0); So it doesn't want to change the block size if map->range has been set (i.e. PetscLayoutSetUp()) was called. But if the matrix is AIJ so long as !(map->n % bs) one can actually set the block size at any time. So how about if I change the Mat code to allow setting the block size late for AIJ? Maybe we can also get rid of PCFieldSplitSetBlockSize() and a PCSPAISetBlockSize()? One issue that comes up with this change is: should setting the block size later automatically reset the block size of the two inner AIJ matrices? Or should they remain one? I could have it update the inner block sizes at the expense of a tiny bit more code. But one complication is that MatSetUpMultiply_MPIAIJ() always sets the column block size of the off-diagaonl matrix to 1; so this needs to be respected. The PETSc design policy of "there is one way to do something" dictates that it is best not to have PCSetBlockSize() unless absolutely necessary so I am trying to see if it is absolutely necessary. Barry > On Oct 10, 2016, at 10:18 AM, Stefano Zampini > wrote: > > Mark, > > I was wondering if you could provide an API/command line customization for > PCGAMG to explicitly provide the "block size" of the problem to the gamg > solver, instead of using the block size of the matrix. The matrix block size > could be always used as a fallback if this new information as not been > provided. > > I'm interfacing a complicated finite element package to PETSc , where you > cannot do any assumption on the blocksize of the problem during matrix setup, > and (AFAIK) I cannot change the block size with a call to MatSetBlockSize > after the matrix has been finalized without hacking (that I would like to > avoid). > > Thanks > -- > Stefano