Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Vaclav Hapla]

OK, now I got you. So I think it's actually not that important when they are created but whether and in which order they are registered on different ranks (see attached edited MWE). Yes, that's a relevant concern I must admit, mainly for things working with subcomms.ThanksVaclav[0] event1 registered as 3
[0] event2 registered as 4
[2] event1 registered as 3
[2] event2 registered as 4
[4] event1 registered as 3
[4] event2 registered as 4
[5] event2 registered as 3
[5] event1 registered as 4
[7] event2 registered as 3
[7] event1 registered as 4
[1] event2 registered as 3
[3] event2 registered as 3
[3] event1 registered as 4
[1] event1 registered as 4
[6] event1 registered as 3
[6] event2 registered as 4

*** WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r 
-fCourier9' to print this document***


-- PETSc Performance Summary: 
--



  ##
  ##
  #  WARNING!!!#
  ##
  #   This code was compiled with a debugging option,  #
  #   To get timing results run ./configure#
  #   using --with-debugging=no, the performance will  #
  #   be generally two or three times faster.  #
  ##
  ##


./ex on a arch-mac-mpich-matlab-hdf5 named vhapla-macbook-pro with 8 
processors, by vhapla Mon Jul  2 18:36:13 2018
Using Petsc Development GIT revision: v3.9.2-600-g2e10669da5  GIT Date: 
2018-06-15 17:49:51 +0200

 Max   Max/MinAvg  Total 
Time (sec):   8.012e+00  1.00011   8.011e+00
Objects:  1.000e+00  1.0   1.000e+00
Flop: 0.000e+00  0.0   0.000e+00  0.000e+00
Flop/sec:0.000e+00  0.0   0.000e+00  0.000e+00
Memory:   3.821e+04  1.0  3.057e+05
MPI Messages: 0.000e+00  0.0   0.000e+00  0.000e+00
MPI Message Lengths:  0.000e+00  0.0   0.000e+00  0.000e+00
MPI Reductions:   1.200e+01  1.0

Flop counting convention: 1 flop = 1 real number operation of type 
(multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N 
flop
and VecAXPY() for complex vectors of length N --> 
8N flop

Summary of Stages:   - Time --  - Flop -  --- Messages ---  -- 
Message Lengths --  -- Reductions --
Avg %Total Avg %Total   counts   %Total 
Avg %Total   counts   %Total 
 0:  Main Stage: 8.0109e+00 100.0%  0.e+00   0.0%  0.000e+00   0.0%  
0.000e+000.0%  4.000e+00  33.3% 


See the 'Profiling' chapter of the users' manual for details on interpreting 
output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flop: Max - maximum over all processors
   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and 
PetscLogStagePop().
  %T - percent time in this phase %F - percent flop in this phase
  %M - percent messages in this phase %L - percent message lengths in 
this phase
  %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all 
processors)



  ##
  ##
  #  WARNING!!!#
  ##
  #   This code was compiled with a debugging option,  #
  #   To get timing results run ./configure#
  #   using --with-debugging=no, the performance will  #
  #   be generally two or three times faster.  #
  ##
  ##


Event  

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Stefano Zampini]

The intent was not to show you a deadlock; was to provide you evidence that
the PetscLogEvent model is meant to have the same event created
collectively (hopefully at class registration) on all processes. See also
PetscLogViewDefault

2018-07-02 18:45 GMT+03:00 Vaclav Hapla :

> Thanks for the MWE. But it behaves exactly as I would anticipate, see log.
> I think that naming the event differently for each rank is not good but
> fortunately PETSc just ignores this for rank > 0. Important is the event id
> which is the same. I correctly get that the time spent in this event is
> slightly more than 8 s and the ratio between the shortest and the longest
> time among all ranks is 8.0.
>
> Where's the deadlock? Note that PetscLogEventBegin/End and
> PetscStageLogPush/Pop are not collective!
>
> And thanks for another argument for better specificity of the log view. If
> the same event would be nested at several levels, the obtained total time
> wouldn't say anything useful.
>
> Thanks
>
> Vaclav
>
>
>
> 2. 7. 2018 v 17:20, Stefano Zampini :
>
>
>
>
>> I don't see why this should lead to deadlock? With current class-wise
>> events you can already have many simultaneous instances of the same event
>> at once.
>>
>>
> Attached a MVE to better explain myself. If you run with p processes, you
> will see what I mean
>
>
>
> --
> Stefano
> 
>
>
>
>


-- 
Stefano

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Vaclav Hapla]

Thanks for the MWE. But it behaves exactly as I would anticipate, see log. I think that naming the event differently for each rank is not good but fortunately PETSc just ignores this for rank > 0. Important is the event id which is the same. I correctly get that the time spent in this event is slightly more than 8 s and the ratio between the shortest and the longest time among all ranks is 8.0.Where's the deadlock? Note that PetscLogEventBegin/End and PetscStageLogPush/Pop are not collective!And thanks for another argument for better specificity of the log view. If the same event would be nested at several levels, the obtained total time wouldn't say anything useful.ThanksVaclav
*** WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r 
-fCourier9' to print this document***


-- PETSc Performance Summary: 
--



  ##
  ##
  #  WARNING!!!#
  ##
  #   This code was compiled with a debugging option,  #
  #   To get timing results run ./configure#
  #   using --with-debugging=no, the performance will  #
  #   be generally two or three times faster.  #
  ##
  ##


./ex on a arch-mac-mpich-matlab-hdf5 named vhapla-macbook-pro with 8 
processors, by vhapla Mon Jul  2 17:27:41 2018
Using Petsc Development GIT revision: v3.9.2-600-g2e10669da5  GIT Date: 
2018-06-15 17:49:51 +0200

 Max   Max/MinAvg  Total 
Time (sec):   8.012e+00  1.6   8.012e+00
Objects:  1.000e+00  1.0   1.000e+00
Flop: 0.000e+00  0.0   0.000e+00  0.000e+00
Flop/sec:0.000e+00  0.0   0.000e+00  0.000e+00
Memory:   3.819e+04  1.0  3.055e+05
MPI Messages: 0.000e+00  0.0   0.000e+00  0.000e+00
MPI Message Lengths:  0.000e+00  0.0   0.000e+00  0.000e+00
MPI Reductions:   1.200e+01  1.0

Flop counting convention: 1 flop = 1 real number operation of type 
(multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N 
flop
and VecAXPY() for complex vectors of length N --> 
8N flop

Summary of Stages:   - Time --  - Flop -  --- Messages ---  -- 
Message Lengths --  -- Reductions --
Avg %Total Avg %Total   counts   %Total 
Avg %Total   counts   %Total 
 0:  Main Stage: 8.0115e+00 100.0%  0.e+00   0.0%  0.000e+00   0.0%  
0.000e+000.0%  4.000e+00  33.3% 


See the 'Profiling' chapter of the users' manual for details on interpreting 
output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flop: Max - maximum over all processors
   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and 
PetscLogStagePop().
  %T - percent time in this phase %F - percent flop in this phase
  %M - percent messages in this phase %L - percent message lengths in 
this phase
  %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all 
processors)



  ##
  ##
  #  WARNING!!!#
  ##
  #   This code was compiled with a debugging option,  #
  #   To get timing results run ./configure#
  #   using --with-debugging=no, the performance will  #
  #   be generally two or three times faster.  #
  ##
  ##


EventCount  

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Matthew Knepley]

On Mon, Jul 2, 2018 at 10:19 AM Vaclav Hapla 
wrote:

> 2. 7. 2018 v 16:38, Matthew Knepley :
>
> On Mon, Jul 2, 2018 at 9:33 AM Vaclav Hapla 
> wrote:
>
>>
>>
>> 2. 7. 2018 v 15:50, Matthew Knepley :
>>
>> On Mon, Jul 2, 2018 at 8:28 AM Vaclav Hapla 
>> wrote:
>>
>>> 2. 7. 2018 v 15:05, Matthew Knepley :
>>>
>>> On Mon, Jul 2, 2018 at 7:54 AM Vaclav Hapla 
>>> wrote:
>>>


 2. 7. 2018 v 14:48, Matthew Knepley :

 On Mon, Jul 2, 2018 at 3:48 AM Vaclav Hapla 
 wrote:

> Barry wrote:
>
>   This could get ugly real fast, for example, for vector operations,
> there may be dozens of named vectors and each one gets its own logging?
> You'd have to make sure that only the objects you care about get named, is
> that possible?
>
>I don't know if there is a good solution within the PETSc logging
> infrastructure to get what you want but maybe what you propose is the best
> possible.
>
>
> As I suggest, this behavior would be only triggered by a specific
> option.
>
> I think there are actually 4 strings which could be used as an event
> name suffix in log view:
> 1) name
> 2) prefix
> 3) type
> 4) custom string (set by something like PetscObjectSetLogViewSuffix)
> I think the best would be to let user choose by offering
> -log_view_by_{name,prefix,type,suffix}.
>
> For example, with -log_view_by_prefix, you could readily distinguish
> PCTelescope outer and inner apply, because you would see a separate
> "PCApply (telescope_)" event.
> With -log_view_by_type, you would see PCApply (telescope).
>
> I think this would be useful because the current class-wide events
> like MatMult or PCApply aggregate very different operations from which 
> some
> are for free and some form hotspots.
>
>
> Stefano wrote:
>
> The issue with this sort of “dynamic” logging is that now PETSc
> requires PetscLogEvent created during the registration of the class, so
> that all the ranks in PETSC_COMM_WORLD have the same events registered.
> What you propose is not generally supported for this specific reason.
>
> Your “log_name” may work if users register their own classes (with
> their own LogEvents created properly), and currently we don’t have support
> (maybe I’m wrong) to add an “InitializePackage” method for the users’
> registered classes.
>
>
> I don't agree. What I suggest is basically an ability to allow
> automatically created object-wise events, so it _can't_ be managed during
> the class registration. In presence of respective option, the event would
> be created during PetscLogEventBegin by taking the class-wide event's 
> name,
> concatenating the suffix and registering a new event. The event id would 
> be
> stored in the PetscObject structure.
>
>
> Matt wrote:
>
> As people have pointed out, this would not work well for Events.
> However, this is exactly what stages are for.
> Use separate stages for the different types of MatMult. I did this,
> for example, when looking at performance
> on different MG levels.
>
>
> Yes, performance on different MG levels is a nice use case. I don't
> understand how you inject stages into MatMults. To me it's exactly the 
> same
> problem as with events - you have to define MatMult_custom where you take
> the original mult and wrap into PetscStageLogPush/Pop and then use
> MatSetOperation to redefine MatMult. Or do you mean something more 
> elegant?
>

 You could do that, but usually I think of stages as being structural. I
 think for your example I would push/pop the stage
 inside your Mat operation wrapper (I don't see why you need another
 one), and this behavior could be controlled with
 another option so you could turn it off.


 I meant hierarchies of typically Mats or PCs, where you don't define
 any custom operations but compose together existing types (which should be
 promoted I believe). So no "my" wrapper. As I wrote below:

  Think e.g. of having additive MATCOMPOSITE wrapping multiplicative
> MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want to measure
> this MATAIJ instance's MatMult separately but you surely don't want to
> rewrite implementation of MatMult_Transpose or force yourself to use
> MATSHELL just to hang the events on MatMult*.
>
>
>>> Its not enough to make separate stages for additive MC, multiplicative
>>> MC, and MT? If you want stages for every single
>>> combination created dynamically, you can push another stage when each of
>>> these combinations is created using GetTag()
>>> or something like that. You could switch between these behaviors with an
>>> option.
>>>
>>>
>>> I'm not sure I understand. Do you mean registering and pushing/popping

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Stefano Zampini]

> I don't see why this should lead to deadlock? With current class-wise
> events you can already have many simultaneous instances of the same event
> at once.
>
>
Attached a MVE to better explain myself. If you run with p processes, you
will see what I mean



-- 
Stefano
#include "petsc.h"

int main(int argc, char *argv[])
{
  PetscErrorCode ierr;
  char name[64];
  PetscLogEvent e;
  ierr = PetscInitialize(,,NULL,0);CHKERRQ(ierr);

  ierr = PetscSNPrintf(name,sizeof(name),"Event rank %d",PetscGlobalRank);CHKERRQ(ierr);
  ierr = PetscLogEventRegister(name,PETSC_OBJECT_CLASSID,);CHKERRQ(ierr);
  ierr = PetscLogEventBegin(e,NULL,NULL,NULL,NULL);CHKERRQ(ierr);
  ierr = PetscSleep(PetscGlobalRank+1);CHKERRQ(ierr);
  ierr = PetscLogEventEnd(e,NULL,NULL,NULL,NULL);CHKERRQ(ierr);
  ierr = PetscFinalize();
  return ierr;
}

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Vaclav Hapla]

> 2. 7. 2018 v 16:38, Matthew Knepley :
> 
> On Mon, Jul 2, 2018 at 9:33 AM Vaclav Hapla  > wrote:
> 
> 
>> 2. 7. 2018 v 15:50, Matthew Knepley > >:
>> 
>> On Mon, Jul 2, 2018 at 8:28 AM Vaclav Hapla > > wrote:
>>> 2. 7. 2018 v 15:05, Matthew Knepley >> >:
>>> 
>>> On Mon, Jul 2, 2018 at 7:54 AM Vaclav Hapla >> > wrote:
>>> 
>>> 
 2. 7. 2018 v 14:48, Matthew Knepley >>> >:
 
 On Mon, Jul 2, 2018 at 3:48 AM Vaclav Hapla >>> > wrote:
 Barry wrote:
>   This could get ugly real fast, for example, for vector operations, 
> there may be dozens of named vectors and each one gets its own logging? 
> You'd have to make sure that only the objects you care about get named, 
> is that possible?
> 
>I don't know if there is a good solution within the PETSc logging 
> infrastructure to get what you want but maybe what you propose is the 
> best possible.
 
 As I suggest, this behavior would be only triggered by a specific option.
 
 I think there are actually 4 strings which could be used as an event name 
 suffix in log view:
 1) name
 2) prefix
 3) type
 4) custom string (set by something like PetscObjectSetLogViewSuffix)
 I think the best would be to let user choose by offering 
 -log_view_by_{name,prefix,type,suffix}.
 
 For example, with -log_view_by_prefix, you could readily distinguish 
 PCTelescope outer and inner apply, because you would see a separate 
 "PCApply (telescope_)" event.
 With -log_view_by_type, you would see PCApply (telescope).
 
 I think this would be useful because the current class-wide events like 
 MatMult or PCApply aggregate very different operations from which some are 
 for free and some form hotspots.
 
 
 Stefano wrote:
> The issue with this sort of “dynamic” logging is that now PETSc requires 
> PetscLogEvent created during the registration of the class, so that all 
> the ranks in PETSC_COMM_WORLD have the same events registered.
> What you propose is not generally supported for this specific reason.
> 
> Your “log_name” may work if users register their own classes (with their 
> own LogEvents created properly), and currently we don’t have support 
> (maybe I’m wrong) to add an “InitializePackage” method for the users’ 
> registered classes.
 
 
 I don't agree. What I suggest is basically an ability to allow 
 automatically created object-wise events, so it _can't_ be managed during 
 the class registration. In presence of respective option, the event would 
 be created during PetscLogEventBegin by taking the class-wide event's 
 name, concatenating the suffix and registering a new event. The event id 
 would be stored in the PetscObject structure.
 
 
 Matt wrote:
> As people have pointed out, this would not work well for Events. However, 
> this is exactly what stages are for.
> Use separate stages for the different types of MatMult. I did this, for 
> example, when looking at performance
> on different MG levels.
 
 Yes, performance on different MG levels is a nice use case. I don't 
 understand how you inject stages into MatMults. To me it's exactly the 
 same problem as with events - you have to define MatMult_custom where you 
 take the original mult and wrap into PetscStageLogPush/Pop and then use 
 MatSetOperation to redefine MatMult. Or do you mean something more elegant?
 
 You could do that, but usually I think of stages as being structural. I 
 think for your example I would push/pop the stage
 inside your Mat operation wrapper (I don't see why you need another one), 
 and this behavior could be controlled with
 another option so you could turn it off.
>>> 
>>> I meant hierarchies of typically Mats or PCs, where you don't define any 
>>> custom operations but compose together existing types (which should be 
>>> promoted I believe). So no "my" wrapper. As I wrote below:
>>> 
>>  Think e.g. of having additive MATCOMPOSITE wrapping multiplicative 
>> MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want to measure 
>> this MATAIJ instance's MatMult separately but you surely don't want to 
>> rewrite implementation of MatMult_Transpose or force yourself to use 
>> MATSHELL just to hang the events on MatMult*.
>>> 
>>> 
>>> Its not enough to make separate stages for additive MC, multiplicative MC, 
>>> and MT? If you want stages for every single
>>> combination created dynamically, you can push another stage when each of 
>>> these combinations is created using GetTag()
>>> or something like that. You could switch between these behaviors with an 
>>> 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Matthew Knepley]

On Mon, Jul 2, 2018 at 9:33 AM Vaclav Hapla 
wrote:

>
>
> 2. 7. 2018 v 15:50, Matthew Knepley :
>
> On Mon, Jul 2, 2018 at 8:28 AM Vaclav Hapla 
> wrote:
>
>> 2. 7. 2018 v 15:05, Matthew Knepley :
>>
>> On Mon, Jul 2, 2018 at 7:54 AM Vaclav Hapla 
>> wrote:
>>
>>>
>>>
>>> 2. 7. 2018 v 14:48, Matthew Knepley :
>>>
>>> On Mon, Jul 2, 2018 at 3:48 AM Vaclav Hapla 
>>> wrote:
>>>
 Barry wrote:

   This could get ugly real fast, for example, for vector operations,
 there may be dozens of named vectors and each one gets its own logging?
 You'd have to make sure that only the objects you care about get named, is
 that possible?

I don't know if there is a good solution within the PETSc logging
 infrastructure to get what you want but maybe what you propose is the best
 possible.


 As I suggest, this behavior would be only triggered by a specific
 option.

 I think there are actually 4 strings which could be used as an event
 name suffix in log view:
 1) name
 2) prefix
 3) type
 4) custom string (set by something like PetscObjectSetLogViewSuffix)
 I think the best would be to let user choose by offering
 -log_view_by_{name,prefix,type,suffix}.

 For example, with -log_view_by_prefix, you could readily distinguish
 PCTelescope outer and inner apply, because you would see a separate
 "PCApply (telescope_)" event.
 With -log_view_by_type, you would see PCApply (telescope).

 I think this would be useful because the current class-wide events like
 MatMult or PCApply aggregate very different operations from which some are
 for free and some form hotspots.


 Stefano wrote:

 The issue with this sort of “dynamic” logging is that now PETSc
 requires PetscLogEvent created during the registration of the class, so
 that all the ranks in PETSC_COMM_WORLD have the same events registered.
 What you propose is not generally supported for this specific reason.

 Your “log_name” may work if users register their own classes (with
 their own LogEvents created properly), and currently we don’t have support
 (maybe I’m wrong) to add an “InitializePackage” method for the users’
 registered classes.


 I don't agree. What I suggest is basically an ability to allow
 automatically created object-wise events, so it _can't_ be managed during
 the class registration. In presence of respective option, the event would
 be created during PetscLogEventBegin by taking the class-wide event's name,
 concatenating the suffix and registering a new event. The event id would be
 stored in the PetscObject structure.


 Matt wrote:

 As people have pointed out, this would not work well for Events.
 However, this is exactly what stages are for.
 Use separate stages for the different types of MatMult. I did this, for
 example, when looking at performance
 on different MG levels.


 Yes, performance on different MG levels is a nice use case. I don't
 understand how you inject stages into MatMults. To me it's exactly the same
 problem as with events - you have to define MatMult_custom where you take
 the original mult and wrap into PetscStageLogPush/Pop and then use
 MatSetOperation to redefine MatMult. Or do you mean something more elegant?

>>>
>>> You could do that, but usually I think of stages as being structural. I
>>> think for your example I would push/pop the stage
>>> inside your Mat operation wrapper (I don't see why you need another
>>> one), and this behavior could be controlled with
>>> another option so you could turn it off.
>>>
>>>
>>> I meant hierarchies of typically Mats or PCs, where you don't define any
>>> custom operations but compose together existing types (which should be
>>> promoted I believe). So no "my" wrapper. As I wrote below:
>>>
>>>  Think e.g. of having additive MATCOMPOSITE wrapping multiplicative
 MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want to measure
 this MATAIJ instance's MatMult separately but you surely don't want to
 rewrite implementation of MatMult_Transpose or force yourself to use
 MATSHELL just to hang the events on MatMult*.


>> Its not enough to make separate stages for additive MC, multiplicative
>> MC, and MT? If you want stages for every single
>> combination created dynamically, you can push another stage when each of
>> these combinations is created using GetTag()
>> or something like that. You could switch between these behaviors with an
>> option.
>>
>>
>> I'm not sure I understand. Do you mean registering and pushing/popping
>> these stages in the user's code? You can surely call PetscStageLogRegister
>> somewhere after PetscInitialize, but where do you place your
>> PetscStageLogPush/Pop calls?
>>
>
> No. You would create a stage when the MATCOMPOSITE is created (or once
> 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Vaclav Hapla]

> 2. 7. 2018 v 15:50, Matthew Knepley :
> 
> On Mon, Jul 2, 2018 at 8:28 AM Vaclav Hapla  > wrote:
>> 2. 7. 2018 v 15:05, Matthew Knepley > >:
>> 
>> On Mon, Jul 2, 2018 at 7:54 AM Vaclav Hapla > > wrote:
>> 
>> 
>>> 2. 7. 2018 v 14:48, Matthew Knepley >> >:
>>> 
>>> On Mon, Jul 2, 2018 at 3:48 AM Vaclav Hapla >> > wrote:
>>> Barry wrote:
   This could get ugly real fast, for example, for vector operations, there 
 may be dozens of named vectors and each one gets its own logging? You'd 
 have to make sure that only the objects you care about get named, is that 
 possible?
 
I don't know if there is a good solution within the PETSc logging 
 infrastructure to get what you want but maybe what you propose is the best 
 possible.
>>> 
>>> As I suggest, this behavior would be only triggered by a specific option.
>>> 
>>> I think there are actually 4 strings which could be used as an event name 
>>> suffix in log view:
>>> 1) name
>>> 2) prefix
>>> 3) type
>>> 4) custom string (set by something like PetscObjectSetLogViewSuffix)
>>> I think the best would be to let user choose by offering 
>>> -log_view_by_{name,prefix,type,suffix}.
>>> 
>>> For example, with -log_view_by_prefix, you could readily distinguish 
>>> PCTelescope outer and inner apply, because you would see a separate 
>>> "PCApply (telescope_)" event.
>>> With -log_view_by_type, you would see PCApply (telescope).
>>> 
>>> I think this would be useful because the current class-wide events like 
>>> MatMult or PCApply aggregate very different operations from which some are 
>>> for free and some form hotspots.
>>> 
>>> 
>>> Stefano wrote:
 The issue with this sort of “dynamic” logging is that now PETSc requires 
 PetscLogEvent created during the registration of the class, so that all 
 the ranks in PETSC_COMM_WORLD have the same events registered.
 What you propose is not generally supported for this specific reason.
 
 Your “log_name” may work if users register their own classes (with their 
 own LogEvents created properly), and currently we don’t have support 
 (maybe I’m wrong) to add an “InitializePackage” method for the users’ 
 registered classes.
>>> 
>>> 
>>> I don't agree. What I suggest is basically an ability to allow 
>>> automatically created object-wise events, so it _can't_ be managed during 
>>> the class registration. In presence of respective option, the event would 
>>> be created during PetscLogEventBegin by taking the class-wide event's name, 
>>> concatenating the suffix and registering a new event. The event id would be 
>>> stored in the PetscObject structure.
>>> 
>>> 
>>> Matt wrote:
 As people have pointed out, this would not work well for Events. However, 
 this is exactly what stages are for.
 Use separate stages for the different types of MatMult. I did this, for 
 example, when looking at performance
 on different MG levels.
>>> 
>>> Yes, performance on different MG levels is a nice use case. I don't 
>>> understand how you inject stages into MatMults. To me it's exactly the same 
>>> problem as with events - you have to define MatMult_custom where you take 
>>> the original mult and wrap into PetscStageLogPush/Pop and then use 
>>> MatSetOperation to redefine MatMult. Or do you mean something more elegant?
>>> 
>>> You could do that, but usually I think of stages as being structural. I 
>>> think for your example I would push/pop the stage
>>> inside your Mat operation wrapper (I don't see why you need another one), 
>>> and this behavior could be controlled with
>>> another option so you could turn it off.
>> 
>> I meant hierarchies of typically Mats or PCs, where you don't define any 
>> custom operations but compose together existing types (which should be 
>> promoted I believe). So no "my" wrapper. As I wrote below:
>> 
>  Think e.g. of having additive MATCOMPOSITE wrapping multiplicative 
> MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want to measure 
> this MATAIJ instance's MatMult separately but you surely don't want to 
> rewrite implementation of MatMult_Transpose or force yourself to use 
> MATSHELL just to hang the events on MatMult*.
>> 
>> 
>> Its not enough to make separate stages for additive MC, multiplicative MC, 
>> and MT? If you want stages for every single
>> combination created dynamically, you can push another stage when each of 
>> these combinations is created using GetTag()
>> or something like that. You could switch between these behaviors with an 
>> option.
> 
> I'm not sure I understand. Do you mean registering and pushing/popping these 
> stages in the user's code? You can surely call PetscStageLogRegister 
> somewhere after PetscInitialize, but where do you place your 
> PetscStageLogPush/Pop calls?
> 
> No. You 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Matthew Knepley]

On Mon, Jul 2, 2018 at 8:28 AM Vaclav Hapla 
wrote:

> 2. 7. 2018 v 15:05, Matthew Knepley :
>
> On Mon, Jul 2, 2018 at 7:54 AM Vaclav Hapla 
> wrote:
>
>>
>>
>> 2. 7. 2018 v 14:48, Matthew Knepley :
>>
>> On Mon, Jul 2, 2018 at 3:48 AM Vaclav Hapla 
>> wrote:
>>
>>> Barry wrote:
>>>
>>>   This could get ugly real fast, for example, for vector operations,
>>> there may be dozens of named vectors and each one gets its own logging?
>>> You'd have to make sure that only the objects you care about get named, is
>>> that possible?
>>>
>>>I don't know if there is a good solution within the PETSc logging
>>> infrastructure to get what you want but maybe what you propose is the best
>>> possible.
>>>
>>>
>>> As I suggest, this behavior would be only triggered by a specific option.
>>>
>>> I think there are actually 4 strings which could be used as an event
>>> name suffix in log view:
>>> 1) name
>>> 2) prefix
>>> 3) type
>>> 4) custom string (set by something like PetscObjectSetLogViewSuffix)
>>> I think the best would be to let user choose by offering
>>> -log_view_by_{name,prefix,type,suffix}.
>>>
>>> For example, with -log_view_by_prefix, you could readily distinguish
>>> PCTelescope outer and inner apply, because you would see a separate
>>> "PCApply (telescope_)" event.
>>> With -log_view_by_type, you would see PCApply (telescope).
>>>
>>> I think this would be useful because the current class-wide events like
>>> MatMult or PCApply aggregate very different operations from which some are
>>> for free and some form hotspots.
>>>
>>>
>>> Stefano wrote:
>>>
>>> The issue with this sort of “dynamic” logging is that now PETSc requires
>>> PetscLogEvent created during the registration of the class, so that all the
>>> ranks in PETSC_COMM_WORLD have the same events registered.
>>> What you propose is not generally supported for this specific reason.
>>>
>>> Your “log_name” may work if users register their own classes (with their
>>> own LogEvents created properly), and currently we don’t have support (maybe
>>> I’m wrong) to add an “InitializePackage” method for the users’ registered
>>> classes.
>>>
>>>
>>> I don't agree. What I suggest is basically an ability to allow
>>> automatically created object-wise events, so it _can't_ be managed during
>>> the class registration. In presence of respective option, the event would
>>> be created during PetscLogEventBegin by taking the class-wide event's name,
>>> concatenating the suffix and registering a new event. The event id would be
>>> stored in the PetscObject structure.
>>>
>>>
>>> Matt wrote:
>>>
>>> As people have pointed out, this would not work well for Events.
>>> However, this is exactly what stages are for.
>>> Use separate stages for the different types of MatMult. I did this, for
>>> example, when looking at performance
>>> on different MG levels.
>>>
>>>
>>> Yes, performance on different MG levels is a nice use case. I don't
>>> understand how you inject stages into MatMults. To me it's exactly the same
>>> problem as with events - you have to define MatMult_custom where you take
>>> the original mult and wrap into PetscStageLogPush/Pop and then use
>>> MatSetOperation to redefine MatMult. Or do you mean something more elegant?
>>>
>>
>> You could do that, but usually I think of stages as being structural. I
>> think for your example I would push/pop the stage
>> inside your Mat operation wrapper (I don't see why you need another one),
>> and this behavior could be controlled with
>> another option so you could turn it off.
>>
>>
>> I meant hierarchies of typically Mats or PCs, where you don't define any
>> custom operations but compose together existing types (which should be
>> promoted I believe). So no "my" wrapper. As I wrote below:
>>
>>  Think e.g. of having additive MATCOMPOSITE wrapping multiplicative
>>> MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want to measure
>>> this MATAIJ instance's MatMult separately but you surely don't want to
>>> rewrite implementation of MatMult_Transpose or force yourself to use
>>> MATSHELL just to hang the events on MatMult*.
>>>
>>>
> Its not enough to make separate stages for additive MC, multiplicative MC,
> and MT? If you want stages for every single
> combination created dynamically, you can push another stage when each of
> these combinations is created using GetTag()
> or something like that. You could switch between these behaviors with an
> option.
>
>
> I'm not sure I understand. Do you mean registering and pushing/popping
> these stages in the user's code? You can surely call PetscStageLogRegister
> somewhere after PetscInitialize, but where do you place your
> PetscStageLogPush/Pop calls?
>

No. You would create a stage when the MATCOMPOSITE is created (or once when
any MATCOMPOSITE is created), and push/pop
on application.

  Matt


> Thanks
>
> Vaclav
>
>
> The reason I think this is preferable is that we do not mess with any
> logging infrastructure, we just use stages inside of 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Vaclav Hapla]

> 2. 7. 2018 v 15:20, Stefano Zampini :
> 
> 
> 
> 2018-07-02 11:48 GMT+03:00 Vaclav Hapla  >:
> Barry wrote:
>>   This could get ugly real fast, for example, for vector operations, there 
>> may be dozens of named vectors and each one gets its own logging? You'd have 
>> to make sure that only the objects you care about get named, is that 
>> possible?
>> 
>>I don't know if there is a good solution within the PETSc logging 
>> infrastructure to get what you want but maybe what you propose is the best 
>> possible.
> 
> As I suggest, this behavior would be only triggered by a specific option.
> 
> I think there are actually 4 strings which could be used as an event name 
> suffix in log view:
> 1) name
> 2) prefix
> 3) type
> 4) custom string (set by something like PetscObjectSetLogViewSuffix)
> I think the best would be to let user choose by offering 
> -log_view_by_{name,prefix,type,suffix}.
> 
> For example, with -log_view_by_prefix, you could readily distinguish 
> PCTelescope outer and inner apply, because you would see a separate "PCApply 
> (telescope_)" event.
> With -log_view_by_type, you would see PCApply (telescope).
> 
> I think this would be useful because the current class-wide events like 
> MatMult or PCApply aggregate very different operations from which some are 
> for free and some form hotspots.
> 
> 
> Stefano wrote:
>> The issue with this sort of “dynamic” logging is that now PETSc requires 
>> PetscLogEvent created during the registration of the class, so that all the 
>> ranks in PETSC_COMM_WORLD have the same events registered.
>> What you propose is not generally supported for this specific reason.
>> 
>> Your “log_name” may work if users register their own classes (with their own 
>> LogEvents created properly), and currently we don’t have support (maybe I’m 
>> wrong) to add an “InitializePackage” method for the users’ registered 
>> classes.
> 
> 
> I don't agree. What I suggest is basically an ability to allow automatically 
> created object-wise events, so it _can't_ be managed during the class 
> registration. In presence of respective option, the event would be created 
> during PetscLogEventBegin by taking the class-wide event's name, 
> concatenating the suffix and registering a new event. The event id would be 
> stored in the PetscObject structure.
> 
> 
> As I said before, now PETSc event model assumes the events are created at 
> class registration.

I don't think there's such assumption. You can register an event where you 
want. There's nothing special about the class-wise events other that they are 
registered during *InitializePackage (which can be however called much later 
than PetscInitialize). They are otherwise by no means tied to that class.

There are even examples which do that:
http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Profiling/PetscLogEventRegister.html
 


> I'm not saying your dynamic thing cannot be done; I'm saying it will likely 
> deadlock

I don't see why this should lead to deadlock? With current class-wise events 
you can already have many simultaneous instances of the same event at once.

Thanks

Vaclav

>  
> 
> Matt wrote:
>> As people have pointed out, this would not work well for Events. However, 
>> this is exactly what stages are for.
>> Use separate stages for the different types of MatMult. I did this, for 
>> example, when looking at performance
>> on different MG levels.
> 
> Yes, performance on different MG levels is a nice use case. I don't 
> understand how you inject stages into MatMults. To me it's exactly the same 
> problem as with events - you have to define MatMult_custom where you take the 
> original mult and wrap into PetscStageLogPush/Pop and then use 
> MatSetOperation to redefine MatMult. Or do you mean something more elegant?
> 
> Thanks
> 
> Vaclav
> 
> 
> 
>> 29. 6. 2018 v 22:42, Smith, Barry F. > >:
>> 
>> 
>> 
>>> On Jun 29, 2018, at 9:33 AM, Vaclav Hapla >> > wrote:
>>> 
>>> 
>>> 
 22. 6. 2018 v 17:47, Smith, Barry F. >>> >:
 
 
 
> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet  > wrote:
> 
> Hello,
> I’m solving a system using a MATSHELL and PCGAMG.
> The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited 
> from the MATSHELL) I’d like to exploit during the solution phase when the 
> smoother on the finest level is doing MatMults.
> 
> Is there some way to:
> 1) decouple in -log_view the time spent in the MATSHELL MatMult and in 
> the smoothers MatMult
 
 You can register a new event and then inside your MATSHELL MatMult() call 
 PetscLogEventBegin/End on your new event.
 
  Note that the MatMult() like will still contain the time for your 
 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Vaclav Hapla]

> 2. 7. 2018 v 15:05, Matthew Knepley :
> 
> On Mon, Jul 2, 2018 at 7:54 AM Vaclav Hapla  > wrote:
> 
> 
>> 2. 7. 2018 v 14:48, Matthew Knepley > >:
>> 
>> On Mon, Jul 2, 2018 at 3:48 AM Vaclav Hapla > > wrote:
>> Barry wrote:
>>>   This could get ugly real fast, for example, for vector operations, there 
>>> may be dozens of named vectors and each one gets its own logging? You'd 
>>> have to make sure that only the objects you care about get named, is that 
>>> possible?
>>> 
>>>I don't know if there is a good solution within the PETSc logging 
>>> infrastructure to get what you want but maybe what you propose is the best 
>>> possible.
>> 
>> As I suggest, this behavior would be only triggered by a specific option.
>> 
>> I think there are actually 4 strings which could be used as an event name 
>> suffix in log view:
>> 1) name
>> 2) prefix
>> 3) type
>> 4) custom string (set by something like PetscObjectSetLogViewSuffix)
>> I think the best would be to let user choose by offering 
>> -log_view_by_{name,prefix,type,suffix}.
>> 
>> For example, with -log_view_by_prefix, you could readily distinguish 
>> PCTelescope outer and inner apply, because you would see a separate "PCApply 
>> (telescope_)" event.
>> With -log_view_by_type, you would see PCApply (telescope).
>> 
>> I think this would be useful because the current class-wide events like 
>> MatMult or PCApply aggregate very different operations from which some are 
>> for free and some form hotspots.
>> 
>> 
>> Stefano wrote:
>>> The issue with this sort of “dynamic” logging is that now PETSc requires 
>>> PetscLogEvent created during the registration of the class, so that all the 
>>> ranks in PETSC_COMM_WORLD have the same events registered.
>>> What you propose is not generally supported for this specific reason.
>>> 
>>> Your “log_name” may work if users register their own classes (with their 
>>> own LogEvents created properly), and currently we don’t have support (maybe 
>>> I’m wrong) to add an “InitializePackage” method for the users’ registered 
>>> classes.
>> 
>> 
>> I don't agree. What I suggest is basically an ability to allow automatically 
>> created object-wise events, so it _can't_ be managed during the class 
>> registration. In presence of respective option, the event would be created 
>> during PetscLogEventBegin by taking the class-wide event's name, 
>> concatenating the suffix and registering a new event. The event id would be 
>> stored in the PetscObject structure.
>> 
>> 
>> Matt wrote:
>>> As people have pointed out, this would not work well for Events. However, 
>>> this is exactly what stages are for.
>>> Use separate stages for the different types of MatMult. I did this, for 
>>> example, when looking at performance
>>> on different MG levels.
>> 
>> Yes, performance on different MG levels is a nice use case. I don't 
>> understand how you inject stages into MatMults. To me it's exactly the same 
>> problem as with events - you have to define MatMult_custom where you take 
>> the original mult and wrap into PetscStageLogPush/Pop and then use 
>> MatSetOperation to redefine MatMult. Or do you mean something more elegant?
>> 
>> You could do that, but usually I think of stages as being structural. I 
>> think for your example I would push/pop the stage
>> inside your Mat operation wrapper (I don't see why you need another one), 
>> and this behavior could be controlled with
>> another option so you could turn it off.
> 
> I meant hierarchies of typically Mats or PCs, where you don't define any 
> custom operations but compose together existing types (which should be 
> promoted I believe). So no "my" wrapper. As I wrote below:
> 
  Think e.g. of having additive MATCOMPOSITE wrapping multiplicative 
 MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want to measure 
 this MATAIJ instance's MatMult separately but you surely don't want to 
 rewrite implementation of MatMult_Transpose or force yourself to use 
 MATSHELL just to hang the events on MatMult*.
> 
> 
> Its not enough to make separate stages for additive MC, multiplicative MC, 
> and MT? If you want stages for every single
> combination created dynamically, you can push another stage when each of 
> these combinations is created using GetTag()
> or something like that. You could switch between these behaviors with an 
> option.

I'm not sure I understand. Do you mean registering and pushing/popping these 
stages in the user's code? You can surely call PetscStageLogRegister somewhere 
after PetscInitialize, but where do you place your PetscStageLogPush/Pop calls?

Thanks

Vaclav

> 
> The reason I think this is preferable is that we do not mess with any logging 
> infrastructure, we just use stages inside of other objects.
> 
>   Thanks,
> 
>  Matt
>  
> Thanks
> 
> Vaclav
> 
> 
>> 
>>   Matt
>>  
>> Thanks
>> 
>> Vaclav
>> 
>> 
>> 
>>> 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Stefano Zampini]

2018-07-02 11:48 GMT+03:00 Vaclav Hapla :

> Barry wrote:
>
>   This could get ugly real fast, for example, for vector operations, there
> may be dozens of named vectors and each one gets its own logging? You'd
> have to make sure that only the objects you care about get named, is that
> possible?
>
>I don't know if there is a good solution within the PETSc logging
> infrastructure to get what you want but maybe what you propose is the best
> possible.
>
>
> As I suggest, this behavior would be only triggered by a specific option.
>
> I think there are actually 4 strings which could be used as an event name
> suffix in log view:
> 1) name
> 2) prefix
> 3) type
> 4) custom string (set by something like PetscObjectSetLogViewSuffix)
> I think the best would be to let user choose by offering
> -log_view_by_{name,prefix,type,suffix}.
>
> For example, with -log_view_by_prefix, you could readily distinguish
> PCTelescope outer and inner apply, because you would see a separate
> "PCApply (telescope_)" event.
> With -log_view_by_type, you would see PCApply (telescope).
>
> I think this would be useful because the current class-wide events like
> MatMult or PCApply aggregate very different operations from which some are
> for free and some form hotspots.
>
>
> Stefano wrote:
>
> The issue with this sort of “dynamic” logging is that now PETSc requires
> PetscLogEvent created during the registration of the class, so that all the
> ranks in PETSC_COMM_WORLD have the same events registered.
> What you propose is not generally supported for this specific reason.
>
> Your “log_name” may work if users register their own classes (with their
> own LogEvents created properly), and currently we don’t have support (maybe
> I’m wrong) to add an “InitializePackage” method for the users’ registered
> classes.
>
>
> I don't agree. What I suggest is basically an ability to allow
> automatically created object-wise events, so it _can't_ be managed during
> the class registration. In presence of respective option, the event would
> be created during PetscLogEventBegin by taking the class-wide event's name,
> concatenating the suffix and registering a new event. The event id would be
> stored in the PetscObject structure.
>
>
As I said before, now PETSc event model assumes the events are created at
class registration. I'm not saying your dynamic thing cannot be done; I'm
saying it will likely deadlock


>
> Matt wrote:
>
> As people have pointed out, this would not work well for Events. However,
> this is exactly what stages are for.
> Use separate stages for the different types of MatMult. I did this, for
> example, when looking at performance
> on different MG levels.
>
>
> Yes, performance on different MG levels is a nice use case. I don't
> understand how you inject stages into MatMults. To me it's exactly the same
> problem as with events - you have to define MatMult_custom where you take
> the original mult and wrap into PetscStageLogPush/Pop and then use
> MatSetOperation to redefine MatMult. Or do you mean something more elegant?
>
> Thanks
>
> Vaclav
>
>
>
> 29. 6. 2018 v 22:42, Smith, Barry F. :
>
>
>
> On Jun 29, 2018, at 9:33 AM, Vaclav Hapla 
> wrote:
>
>
>
> 22. 6. 2018 v 17:47, Smith, Barry F. :
>
>
>
> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet 
> wrote:
>
> Hello,
> I’m solving a system using a MATSHELL and PCGAMG.
> The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited from
> the MATSHELL) I’d like to exploit during the solution phase when the
> smoother on the finest level is doing MatMults.
>
> Is there some way to:
> 1) decouple in -log_view the time spent in the MATSHELL MatMult and in the
> smoothers MatMult
>
>
> You can register a new event and then inside your MATSHELL MatMult() call
> PetscLogEventBegin/End on your new event.
>
>  Note that the MatMult() like will still contain the time for your
> MatShell mult so you will need to subtract it off to get the time for your
> non-shell matmults.
>
>
> In PERMON, we sometimes have quite complicated hierarchy of wrapped
> matrices and want to measure MatMult{,Transpose,Add,TransposeAdd}
> separately for particular ones. Think e.g. of having additive MATCOMPOSITE
> wrapping multiplicative MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ.
> You want to measure this MATAIJ instance's MatMult separately but you
> surely don't want to rewrite implementation of MatMult_Transpose or force
> yourself to use MATSHELL just to hang the events on MatMult*.
>
> We had a special wrapper type just adding some prefix to the events for
> the given object but this is not nice. What about adding a functionality to
> PetscLogEventBegin/End that would distinguish based on the first
> PetscObject's name or option prefix? Of course optionally not to break guys
> relying on current behavior - e.g. under something like -log_view_by_name.
> To me it's quite an elegant solution working for any PetscObject and any
> event.
>
>
>   This could get ugly real fast, for 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Matthew Knepley]

On Mon, Jul 2, 2018 at 7:54 AM Vaclav Hapla 
wrote:

>
>
> 2. 7. 2018 v 14:48, Matthew Knepley :
>
> On Mon, Jul 2, 2018 at 3:48 AM Vaclav Hapla 
> wrote:
>
>> Barry wrote:
>>
>>   This could get ugly real fast, for example, for vector operations,
>> there may be dozens of named vectors and each one gets its own logging?
>> You'd have to make sure that only the objects you care about get named, is
>> that possible?
>>
>>I don't know if there is a good solution within the PETSc logging
>> infrastructure to get what you want but maybe what you propose is the best
>> possible.
>>
>>
>> As I suggest, this behavior would be only triggered by a specific option.
>>
>> I think there are actually 4 strings which could be used as an event name
>> suffix in log view:
>> 1) name
>> 2) prefix
>> 3) type
>> 4) custom string (set by something like PetscObjectSetLogViewSuffix)
>> I think the best would be to let user choose by offering
>> -log_view_by_{name,prefix,type,suffix}.
>>
>> For example, with -log_view_by_prefix, you could readily distinguish
>> PCTelescope outer and inner apply, because you would see a separate
>> "PCApply (telescope_)" event.
>> With -log_view_by_type, you would see PCApply (telescope).
>>
>> I think this would be useful because the current class-wide events like
>> MatMult or PCApply aggregate very different operations from which some are
>> for free and some form hotspots.
>>
>>
>> Stefano wrote:
>>
>> The issue with this sort of “dynamic” logging is that now PETSc requires
>> PetscLogEvent created during the registration of the class, so that all the
>> ranks in PETSC_COMM_WORLD have the same events registered.
>> What you propose is not generally supported for this specific reason.
>>
>> Your “log_name” may work if users register their own classes (with their
>> own LogEvents created properly), and currently we don’t have support (maybe
>> I’m wrong) to add an “InitializePackage” method for the users’ registered
>> classes.
>>
>>
>> I don't agree. What I suggest is basically an ability to allow
>> automatically created object-wise events, so it _can't_ be managed during
>> the class registration. In presence of respective option, the event would
>> be created during PetscLogEventBegin by taking the class-wide event's name,
>> concatenating the suffix and registering a new event. The event id would be
>> stored in the PetscObject structure.
>>
>>
>> Matt wrote:
>>
>> As people have pointed out, this would not work well for Events. However,
>> this is exactly what stages are for.
>> Use separate stages for the different types of MatMult. I did this, for
>> example, when looking at performance
>> on different MG levels.
>>
>>
>> Yes, performance on different MG levels is a nice use case. I don't
>> understand how you inject stages into MatMults. To me it's exactly the same
>> problem as with events - you have to define MatMult_custom where you take
>> the original mult and wrap into PetscStageLogPush/Pop and then use
>> MatSetOperation to redefine MatMult. Or do you mean something more elegant?
>>
>
> You could do that, but usually I think of stages as being structural. I
> think for your example I would push/pop the stage
> inside your Mat operation wrapper (I don't see why you need another one),
> and this behavior could be controlled with
> another option so you could turn it off.
>
>
> I meant hierarchies of typically Mats or PCs, where you don't define any
> custom operations but compose together existing types (which should be
> promoted I believe). So no "my" wrapper. As I wrote below:
>
>  Think e.g. of having additive MATCOMPOSITE wrapping multiplicative
>> MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want to measure
>> this MATAIJ instance's MatMult separately but you surely don't want to
>> rewrite implementation of MatMult_Transpose or force yourself to use
>> MATSHELL just to hang the events on MatMult*.
>>
>>
Its not enough to make separate stages for additive MC, multiplicative MC,
and MT? If you want stages for every single
combination created dynamically, you can push another stage when each of
these combinations is created using GetTag()
or something like that. You could switch between these behaviors with an
option.

The reason I think this is preferable is that we do not mess with any
logging infrastructure, we just use stages inside of other objects.

  Thanks,

 Matt


> Thanks
>
> Vaclav
>
>
>
>   Matt
>
>
>> Thanks
>>
>> Vaclav
>>
>>
>>
>> 29. 6. 2018 v 22:42, Smith, Barry F. :
>>
>>
>>
>> On Jun 29, 2018, at 9:33 AM, Vaclav Hapla 
>> wrote:
>>
>>
>>
>> 22. 6. 2018 v 17:47, Smith, Barry F. :
>>
>>
>>
>> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet 
>> wrote:
>>
>> Hello,
>> I’m solving a system using a MATSHELL and PCGAMG.
>> The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited
>> from the MATSHELL) I’d like to exploit during the solution phase when the
>> smoother on the finest level is doing MatMults.
>>
>> Is there some way to:
>> 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Vaclav Hapla]

> 2. 7. 2018 v 14:48, Matthew Knepley :
> 
> On Mon, Jul 2, 2018 at 3:48 AM Vaclav Hapla  > wrote:
> Barry wrote:
>>   This could get ugly real fast, for example, for vector operations, there 
>> may be dozens of named vectors and each one gets its own logging? You'd have 
>> to make sure that only the objects you care about get named, is that 
>> possible?
>> 
>>I don't know if there is a good solution within the PETSc logging 
>> infrastructure to get what you want but maybe what you propose is the best 
>> possible.
> 
> As I suggest, this behavior would be only triggered by a specific option.
> 
> I think there are actually 4 strings which could be used as an event name 
> suffix in log view:
> 1) name
> 2) prefix
> 3) type
> 4) custom string (set by something like PetscObjectSetLogViewSuffix)
> I think the best would be to let user choose by offering 
> -log_view_by_{name,prefix,type,suffix}.
> 
> For example, with -log_view_by_prefix, you could readily distinguish 
> PCTelescope outer and inner apply, because you would see a separate "PCApply 
> (telescope_)" event.
> With -log_view_by_type, you would see PCApply (telescope).
> 
> I think this would be useful because the current class-wide events like 
> MatMult or PCApply aggregate very different operations from which some are 
> for free and some form hotspots.
> 
> 
> Stefano wrote:
>> The issue with this sort of “dynamic” logging is that now PETSc requires 
>> PetscLogEvent created during the registration of the class, so that all the 
>> ranks in PETSC_COMM_WORLD have the same events registered.
>> What you propose is not generally supported for this specific reason.
>> 
>> Your “log_name” may work if users register their own classes (with their own 
>> LogEvents created properly), and currently we don’t have support (maybe I’m 
>> wrong) to add an “InitializePackage” method for the users’ registered 
>> classes.
> 
> 
> I don't agree. What I suggest is basically an ability to allow automatically 
> created object-wise events, so it _can't_ be managed during the class 
> registration. In presence of respective option, the event would be created 
> during PetscLogEventBegin by taking the class-wide event's name, 
> concatenating the suffix and registering a new event. The event id would be 
> stored in the PetscObject structure.
> 
> 
> Matt wrote:
>> As people have pointed out, this would not work well for Events. However, 
>> this is exactly what stages are for.
>> Use separate stages for the different types of MatMult. I did this, for 
>> example, when looking at performance
>> on different MG levels.
> 
> Yes, performance on different MG levels is a nice use case. I don't 
> understand how you inject stages into MatMults. To me it's exactly the same 
> problem as with events - you have to define MatMult_custom where you take the 
> original mult and wrap into PetscStageLogPush/Pop and then use 
> MatSetOperation to redefine MatMult. Or do you mean something more elegant?
> 
> You could do that, but usually I think of stages as being structural. I think 
> for your example I would push/pop the stage
> inside your Mat operation wrapper (I don't see why you need another one), and 
> this behavior could be controlled with
> another option so you could turn it off.

I meant hierarchies of typically Mats or PCs, where you don't define any custom 
operations but compose together existing types (which should be promoted I 
believe). So no "my" wrapper. As I wrote below:

>>>  Think e.g. of having additive MATCOMPOSITE wrapping multiplicative 
>>> MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want to measure 
>>> this MATAIJ instance's MatMult separately but you surely don't want to 
>>> rewrite implementation of MatMult_Transpose or force yourself to use 
>>> MATSHELL just to hang the events on MatMult*.

Thanks

Vaclav


> 
>   Matt
>  
> Thanks
> 
> Vaclav
> 
> 
> 
>> 29. 6. 2018 v 22:42, Smith, Barry F. > >:
>> 
>> 
>> 
>>> On Jun 29, 2018, at 9:33 AM, Vaclav Hapla >> > wrote:
>>> 
>>> 
>>> 
 22. 6. 2018 v 17:47, Smith, Barry F. >>> >:
 
 
 
> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet  > wrote:
> 
> Hello,
> I’m solving a system using a MATSHELL and PCGAMG.
> The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited 
> from the MATSHELL) I’d like to exploit during the solution phase when the 
> smoother on the finest level is doing MatMults.
> 
> Is there some way to:
> 1) decouple in -log_view the time spent in the MATSHELL MatMult and in 
> the smoothers MatMult
 
 You can register a new event and then inside your MATSHELL MatMult() call 
 PetscLogEventBegin/End on your new event.
 
  Note that the MatMult() like will still contain the time for your 
 MatShell mult so you 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Matthew Knepley]

On Mon, Jul 2, 2018 at 3:48 AM Vaclav Hapla 
wrote:

> Barry wrote:
>
>   This could get ugly real fast, for example, for vector operations, there
> may be dozens of named vectors and each one gets its own logging? You'd
> have to make sure that only the objects you care about get named, is that
> possible?
>
>I don't know if there is a good solution within the PETSc logging
> infrastructure to get what you want but maybe what you propose is the best
> possible.
>
>
> As I suggest, this behavior would be only triggered by a specific option.
>
> I think there are actually 4 strings which could be used as an event name
> suffix in log view:
> 1) name
> 2) prefix
> 3) type
> 4) custom string (set by something like PetscObjectSetLogViewSuffix)
> I think the best would be to let user choose by offering
> -log_view_by_{name,prefix,type,suffix}.
>
> For example, with -log_view_by_prefix, you could readily distinguish
> PCTelescope outer and inner apply, because you would see a separate
> "PCApply (telescope_)" event.
> With -log_view_by_type, you would see PCApply (telescope).
>
> I think this would be useful because the current class-wide events like
> MatMult or PCApply aggregate very different operations from which some are
> for free and some form hotspots.
>
>
> Stefano wrote:
>
> The issue with this sort of “dynamic” logging is that now PETSc requires
> PetscLogEvent created during the registration of the class, so that all the
> ranks in PETSC_COMM_WORLD have the same events registered.
> What you propose is not generally supported for this specific reason.
>
> Your “log_name” may work if users register their own classes (with their
> own LogEvents created properly), and currently we don’t have support (maybe
> I’m wrong) to add an “InitializePackage” method for the users’ registered
> classes.
>
>
> I don't agree. What I suggest is basically an ability to allow
> automatically created object-wise events, so it _can't_ be managed during
> the class registration. In presence of respective option, the event would
> be created during PetscLogEventBegin by taking the class-wide event's name,
> concatenating the suffix and registering a new event. The event id would be
> stored in the PetscObject structure.
>
>
> Matt wrote:
>
> As people have pointed out, this would not work well for Events. However,
> this is exactly what stages are for.
> Use separate stages for the different types of MatMult. I did this, for
> example, when looking at performance
> on different MG levels.
>
>
> Yes, performance on different MG levels is a nice use case. I don't
> understand how you inject stages into MatMults. To me it's exactly the same
> problem as with events - you have to define MatMult_custom where you take
> the original mult and wrap into PetscStageLogPush/Pop and then use
> MatSetOperation to redefine MatMult. Or do you mean something more elegant?
>

You could do that, but usually I think of stages as being structural. I
think for your example I would push/pop the stage
inside your Mat operation wrapper (I don't see why you need another one),
and this behavior could be controlled with
another option so you could turn it off.

  Matt


> Thanks
>
> Vaclav
>
>
>
> 29. 6. 2018 v 22:42, Smith, Barry F. :
>
>
>
> On Jun 29, 2018, at 9:33 AM, Vaclav Hapla 
> wrote:
>
>
>
> 22. 6. 2018 v 17:47, Smith, Barry F. :
>
>
>
> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet 
> wrote:
>
> Hello,
> I’m solving a system using a MATSHELL and PCGAMG.
> The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited from
> the MATSHELL) I’d like to exploit during the solution phase when the
> smoother on the finest level is doing MatMults.
>
> Is there some way to:
> 1) decouple in -log_view the time spent in the MATSHELL MatMult and in the
> smoothers MatMult
>
>
> You can register a new event and then inside your MATSHELL MatMult() call
> PetscLogEventBegin/End on your new event.
>
>  Note that the MatMult() like will still contain the time for your
> MatShell mult so you will need to subtract it off to get the time for your
> non-shell matmults.
>
>
> In PERMON, we sometimes have quite complicated hierarchy of wrapped
> matrices and want to measure MatMult{,Transpose,Add,TransposeAdd}
> separately for particular ones. Think e.g. of having additive MATCOMPOSITE
> wrapping multiplicative MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ.
> You want to measure this MATAIJ instance's MatMult separately but you
> surely don't want to rewrite implementation of MatMult_Transpose or force
> yourself to use MATSHELL just to hang the events on MatMult*.
>
> We had a special wrapper type just adding some prefix to the events for
> the given object but this is not nice. What about adding a functionality to
> PetscLogEventBegin/End that would distinguish based on the first
> PetscObject's name or option prefix? Of course optionally not to break guys
> relying on current behavior - e.g. under something like -log_view_by_name.
> To me 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Vaclav Hapla]

Barry wrote:
>   This could get ugly real fast, for example, for vector operations, there 
> may be dozens of named vectors and each one gets its own logging? You'd have 
> to make sure that only the objects you care about get named, is that possible?
> 
>I don't know if there is a good solution within the PETSc logging 
> infrastructure to get what you want but maybe what you propose is the best 
> possible.

As I suggest, this behavior would be only triggered by a specific option.

I think there are actually 4 strings which could be used as an event name 
suffix in log view:
1) name
2) prefix
3) type
4) custom string (set by something like PetscObjectSetLogViewSuffix)
I think the best would be to let user choose by offering 
-log_view_by_{name,prefix,type,suffix}.

For example, with -log_view_by_prefix, you could readily distinguish 
PCTelescope outer and inner apply, because you would see a separate "PCApply 
(telescope_)" event.
With -log_view_by_type, you would see PCApply (telescope).

I think this would be useful because the current class-wide events like MatMult 
or PCApply aggregate very different operations from which some are for free and 
some form hotspots.


Stefano wrote:
> The issue with this sort of “dynamic” logging is that now PETSc requires 
> PetscLogEvent created during the registration of the class, so that all the 
> ranks in PETSC_COMM_WORLD have the same events registered.
> What you propose is not generally supported for this specific reason.
> 
> Your “log_name” may work if users register their own classes (with their own 
> LogEvents created properly), and currently we don’t have support (maybe I’m 
> wrong) to add an “InitializePackage” method for the users’ registered classes.


I don't agree. What I suggest is basically an ability to allow automatically 
created object-wise events, so it _can't_ be managed during the class 
registration. In presence of respective option, the event would be created 
during PetscLogEventBegin by taking the class-wide event's name, concatenating 
the suffix and registering a new event. The event id would be stored in the 
PetscObject structure.


Matt wrote:
> As people have pointed out, this would not work well for Events. However, 
> this is exactly what stages are for.
> Use separate stages for the different types of MatMult. I did this, for 
> example, when looking at performance
> on different MG levels.

Yes, performance on different MG levels is a nice use case. I don't understand 
how you inject stages into MatMults. To me it's exactly the same problem as 
with events - you have to define MatMult_custom where you take the original 
mult and wrap into PetscStageLogPush/Pop and then use MatSetOperation to 
redefine MatMult. Or do you mean something more elegant?

Thanks

Vaclav



> 29. 6. 2018 v 22:42, Smith, Barry F. :
> 
> 
> 
>> On Jun 29, 2018, at 9:33 AM, Vaclav Hapla  wrote:
>> 
>> 
>> 
>>> 22. 6. 2018 v 17:47, Smith, Barry F. :
>>> 
>>> 
>>> 
 On Jun 22, 2018, at 5:43 AM, Pierre Jolivet  
 wrote:
 
 Hello,
 I’m solving a system using a MATSHELL and PCGAMG.
 The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited from 
 the MATSHELL) I’d like to exploit during the solution phase when the 
 smoother on the finest level is doing MatMults.
 
 Is there some way to:
 1) decouple in -log_view the time spent in the MATSHELL MatMult and in the 
 smoothers MatMult
>>> 
>>> You can register a new event and then inside your MATSHELL MatMult() call 
>>> PetscLogEventBegin/End on your new event.
>>> 
>>>  Note that the MatMult() like will still contain the time for your MatShell 
>>> mult so you will need to subtract it off to get the time for your non-shell 
>>> matmults.
>> 
>> In PERMON, we sometimes have quite complicated hierarchy of wrapped matrices 
>> and want to measure MatMult{,Transpose,Add,TransposeAdd} separately for 
>> particular ones. Think e.g. of having additive MATCOMPOSITE wrapping 
>> multiplicative MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want 
>> to measure this MATAIJ instance's MatMult separately but you surely don't 
>> want to rewrite implementation of MatMult_Transpose or force yourself to use 
>> MATSHELL just to hang the events on MatMult*.
>> 
>> We had a special wrapper type just adding some prefix to the events for the 
>> given object but this is not nice. What about adding a functionality to 
>> PetscLogEventBegin/End that would distinguish based on the first 
>> PetscObject's name or option prefix? Of course optionally not to break guys 
>> relying on current behavior - e.g. under something like -log_view_by_name. 
>> To me it's quite an elegant solution working for any PetscObject and any 
>> event.
> 
>   This could get ugly real fast, for example, for vector operations, there 
> may be dozens of named vectors and each one gets its own logging? You'd have 
> to make sure that only the objects you care about get named, is that 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Matthew Knepley]

On Fri, Jun 29, 2018 at 9:34 AM Vaclav Hapla 
wrote:

>
>
> > 22. 6. 2018 v 17:47, Smith, Barry F. :
> >
> >
> >
> >> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet 
> wrote:
> >>
> >> Hello,
> >> I’m solving a system using a MATSHELL and PCGAMG.
> >> The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited
> from the MATSHELL) I’d like to exploit during the solution phase when the
> smoother on the finest level is doing MatMults.
> >>
> >> Is there some way to:
> >> 1) decouple in -log_view the time spent in the MATSHELL MatMult and in
> the smoothers MatMult
> >
> >   You can register a new event and then inside your MATSHELL MatMult()
> call PetscLogEventBegin/End on your new event.
> >
> >Note that the MatMult() like will still contain the time for your
> MatShell mult so you will need to subtract it off to get the time for your
> non-shell matmults.
>
> In PERMON, we sometimes have quite complicated hierarchy of wrapped
> matrices and want to measure MatMult{,Transpose,Add,TransposeAdd}
> separately for particular ones. Think e.g. of having additive MATCOMPOSITE
> wrapping multiplicative MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ.
> You want to measure this MATAIJ instance's MatMult separately but you
> surely don't want to rewrite implementation of MatMult_Transpose or force
> yourself to use MATSHELL just to hang the events on MatMult*.
>
> We had a special wrapper type just adding some prefix to the events for
> the given object but this is not nice. What about adding a functionality to
> PetscLogEventBegin/End that would distinguish based on the first
> PetscObject's name or option prefix? Of course optionally not to break guys
> relying on current behavior - e.g. under something like -log_view_by_name.
> To me it's quite an elegant solution working for any PetscObject and any
> event.
>

As people have pointed out, this would not work well for Events. However,
this is exactly what stages are for.
Use separate stages for the different types of MatMult. I did this, for
example, when looking at performance
on different MG levels.

  Matt


> I can do that if I get some upvotes.
>
> Vaclav
>
> >
> >> 2) hardwire a specific MatMult implementation for the smoother on the
> finest level
> >
> >   In the latest release you do MatSetOperation() to override the normal
> matrix vector product with anything else you want.
> >
> >>
> >> Thanks in advance,
> >> Pierre
> >>
> >> PS : here is what I have right now,
> >> MatMult  118 1.0 1.0740e+02 1.6 1.04e+13 1.6 1.7e+06
> 6.1e+05 0.0e+00 47100 90 98  0  47100 90 98  0 81953703
> >> […]
> >> PCSetUp2 1.0 8.6513e+00 1.0 1.01e+09 1.7 2.6e+05
> 4.0e+05 1.8e+02  5  0 14 10 66   5  0 14 10 68 94598
> >> PCApply   14 1.0 8.0373e+01 1.1 9.06e+12 1.6 1.3e+06
> 6.0e+05 2.1e+01 45 87 72 78  8  45 87 72 78  8 95365211 // I’m guessing a
> lot of time here is being wasted in doing inefficient MatMults on the
> finest level but this is only speculation
> >>
> >> Same code with -pc_type none -ksp_max_it 13,
> >> MatMult   14 1.0 1.2936e+01 1.7 1.35e+12 1.6 2.0e+05
> 6.1e+05 0.0e+00 15100 78 93  0  15100 78 93  0 88202079
> >>
> >> The grid itself is rather simple (two levels, extremely aggressive
> coarsening),
> >>   type is MULTIPLICATIVE, levels=2 cycles=v
> >>   KSP Object: (mg_coarse_) 1024 MPI processes
> >> linear system matrix = precond matrix:
> >> Mat Object: 1024 MPI processes
> >>   type: mpiaij
> >>   rows=775, cols=775
> >>   total: nonzeros=1793, allocated nonzeros=1793
> >>
> >> linear system matrix followed by preconditioner matrix:
> >> Mat Object: 1024 MPI processes
> >>   type: shell
> >>   rows=1369307136, cols=1369307136
> >> Mat Object: 1024 MPI processes
> >>   type: mpiaij
> >>   rows=1369307136, cols=1369307136
> >>   total: nonzeros=19896719360, allocated nonzeros=19896719360
> >
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Stefano Zampini]

> On Jun 29, 2018, at 5:33 PM, Vaclav Hapla  wrote:
> 
> 
> 
>> 22. 6. 2018 v 17:47, Smith, Barry F. :
>> 
>> 
>> 
>>> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet  
>>> wrote:
>>> 
>>> Hello,
>>> I’m solving a system using a MATSHELL and PCGAMG.
>>> The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited from 
>>> the MATSHELL) I’d like to exploit during the solution phase when the 
>>> smoother on the finest level is doing MatMults.
>>> 
>>> Is there some way to:
>>> 1) decouple in -log_view the time spent in the MATSHELL MatMult and in the 
>>> smoothers MatMult
>> 
>>  You can register a new event and then inside your MATSHELL MatMult() call 
>> PetscLogEventBegin/End on your new event.
>> 
>>   Note that the MatMult() like will still contain the time for your MatShell 
>> mult so you will need to subtract it off to get the time for your non-shell 
>> matmults.
> 
> In PERMON, we sometimes have quite complicated hierarchy of wrapped matrices 
> and want to measure MatMult{,Transpose,Add,TransposeAdd} separately for 
> particular ones. Think e.g. of having additive MATCOMPOSITE wrapping 
> multiplicative MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want 
> to measure this MATAIJ instance's MatMult separately but you surely don't 
> want to rewrite implementation of MatMult_Transpose or force yourself to use 
> MATSHELL just to hang the events on MatMult*.
> 
> We had a special wrapper type just adding some prefix to the events for the 
> given object but this is not nice. What about adding a functionality to 
> PetscLogEventBegin/End that would distinguish based on the first 
> PetscObject's name or option prefix? Of course optionally not to break guys 
> relying on current behavior - e.g. under something like -log_view_by_name. To 
> me it's quite an elegant solution working for any PetscObject and any event.

The issue with this sort of “dynamic” logging is that now PETSc requires 
PetscLogEvent created during the registration of the class, so that all the 
ranks in PETSC_COMM_WORLD have the same events registered.
What you propose is not generally supported for this specific reason.

Your “log_name” may work if users register their own classes (with their own 
LogEvents created properly), and currently we don’t have support (maybe I’m 
wrong) to add an “InitializePackage” method for the users’ registered classes.

> 
> I can do that if I get some upvotes.
> 
> Vaclav
> 
>> 
>>> 2) hardwire a specific MatMult implementation for the smoother on the 
>>> finest level
>> 
>>  In the latest release you do MatSetOperation() to override the normal 
>> matrix vector product with anything else you want. 
>> 
>>> 
>>> Thanks in advance,
>>> Pierre
>>> 
>>> PS : here is what I have right now,
>>> MatMult  118 1.0 1.0740e+02 1.6 1.04e+13 1.6 1.7e+06 6.1e+05 
>>> 0.0e+00 47100 90 98  0  47100 90 98 0 81953703
>>> […]
>>> PCSetUp2 1.0 8.6513e+00 1.0 1.01e+09 1.7 2.6e+05 4.0e+05 
>>> 1.8e+02  5  0 14 10 66   5  0 14 10 68 94598
>>> PCApply   14 1.0 8.0373e+01 1.1 9.06e+12 1.6 1.3e+06 6.0e+05 
>>> 2.1e+01 45 87 72 78  8  45 87 72 78  8 95365211 // I’m guessing a lot of 
>>> time here is being wasted in doing inefficient MatMults on the finest level 
>>> but this is only speculation
>>> 
>>> Same code with -pc_type none -ksp_max_it 13,
>>> MatMult   14 1.0 1.2936e+01 1.7 1.35e+12 1.6 2.0e+05 6.1e+05 
>>> 0.0e+00 15100 78 93  0  15100 78 93 0 88202079
>>> 
>>> The grid itself is rather simple (two levels, extremely aggressive 
>>> coarsening),
>>>  type is MULTIPLICATIVE, levels=2 cycles=v
>>>  KSP Object: (mg_coarse_) 1024 MPI processes
>>> linear system matrix = precond matrix:
>>>Mat Object: 1024 MPI processes
>>>  type: mpiaij
>>>  rows=775, cols=775
>>>  total: nonzeros=1793, allocated nonzeros=1793
>>> 
>>> linear system matrix followed by preconditioner matrix:
>>> Mat Object: 1024 MPI processes
>>>  type: shell
>>>  rows=1369307136, cols=1369307136
>>> Mat Object: 1024 MPI processes
>>>  type: mpiaij
>>>  rows=1369307136, cols=1369307136
>>>  total: nonzeros=19896719360, allocated nonzeros=19896719360

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Smith, Barry F.]

> On Jun 29, 2018, at 9:33 AM, Vaclav Hapla  wrote:
> 
> 
> 
>> 22. 6. 2018 v 17:47, Smith, Barry F. :
>> 
>> 
>> 
>>> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet  
>>> wrote:
>>> 
>>> Hello,
>>> I’m solving a system using a MATSHELL and PCGAMG.
>>> The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited from 
>>> the MATSHELL) I’d like to exploit during the solution phase when the 
>>> smoother on the finest level is doing MatMults.
>>> 
>>> Is there some way to:
>>> 1) decouple in -log_view the time spent in the MATSHELL MatMult and in the 
>>> smoothers MatMult
>> 
>>  You can register a new event and then inside your MATSHELL MatMult() call 
>> PetscLogEventBegin/End on your new event.
>> 
>>   Note that the MatMult() like will still contain the time for your MatShell 
>> mult so you will need to subtract it off to get the time for your non-shell 
>> matmults.
> 
> In PERMON, we sometimes have quite complicated hierarchy of wrapped matrices 
> and want to measure MatMult{,Transpose,Add,TransposeAdd} separately for 
> particular ones. Think e.g. of having additive MATCOMPOSITE wrapping 
> multiplicative MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want 
> to measure this MATAIJ instance's MatMult separately but you surely don't 
> want to rewrite implementation of MatMult_Transpose or force yourself to use 
> MATSHELL just to hang the events on MatMult*.
> 
> We had a special wrapper type just adding some prefix to the events for the 
> given object but this is not nice. What about adding a functionality to 
> PetscLogEventBegin/End that would distinguish based on the first 
> PetscObject's name or option prefix? Of course optionally not to break guys 
> relying on current behavior - e.g. under something like -log_view_by_name. To 
> me it's quite an elegant solution working for any PetscObject and any event.

   This could get ugly real fast, for example, for vector operations, there may 
be dozens of named vectors and each one gets its own logging? You'd have to 
make sure that only the objects you care about get named, is that possible?

I don't know if there is a good solution within the PETSc logging 
infrastructure to get what you want but maybe what you propose is the best 
possible.

   Barry

> 
> I can do that if I get some upvotes.
> 
> Vaclav
> 
>> 
>>> 2) hardwire a specific MatMult implementation for the smoother on the 
>>> finest level
>> 
>>  In the latest release you do MatSetOperation() to override the normal 
>> matrix vector product with anything else you want. 
>> 
>>> 
>>> Thanks in advance,
>>> Pierre
>>> 
>>> PS : here is what I have right now,
>>> MatMult  118 1.0 1.0740e+02 1.6 1.04e+13 1.6 1.7e+06 6.1e+05 
>>> 0.0e+00 47100 90 98  0  47100 90 98  0 81953703
>>> […]
>>> PCSetUp2 1.0 8.6513e+00 1.0 1.01e+09 1.7 2.6e+05 4.0e+05 
>>> 1.8e+02  5  0 14 10 66   5  0 14 10 68 94598
>>> PCApply   14 1.0 8.0373e+01 1.1 9.06e+12 1.6 1.3e+06 6.0e+05 
>>> 2.1e+01 45 87 72 78  8  45 87 72 78  8 95365211 // I’m guessing a lot of 
>>> time here is being wasted in doing inefficient MatMults on the finest level 
>>> but this is only speculation
>>> 
>>> Same code with -pc_type none -ksp_max_it 13,
>>> MatMult   14 1.0 1.2936e+01 1.7 1.35e+12 1.6 2.0e+05 6.1e+05 
>>> 0.0e+00 15100 78 93  0  15100 78 93  0 88202079
>>> 
>>> The grid itself is rather simple (two levels, extremely aggressive 
>>> coarsening),
>>>  type is MULTIPLICATIVE, levels=2 cycles=v
>>>  KSP Object: (mg_coarse_) 1024 MPI processes
>>> linear system matrix = precond matrix:
>>>Mat Object: 1024 MPI processes
>>>  type: mpiaij
>>>  rows=775, cols=775
>>>  total: nonzeros=1793, allocated nonzeros=1793
>>> 
>>> linear system matrix followed by preconditioner matrix:
>>> Mat Object: 1024 MPI processes
>>>  type: shell
>>>  rows=1369307136, cols=1369307136
>>> Mat Object: 1024 MPI processes
>>>  type: mpiaij
>>>  rows=1369307136, cols=1369307136
>>>  total: nonzeros=19896719360, allocated nonzeros=19896719360

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Vaclav Hapla]

> 22. 6. 2018 v 17:47, Smith, Barry F. :
> 
> 
> 
>> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet  
>> wrote:
>> 
>> Hello,
>> I’m solving a system using a MATSHELL and PCGAMG.
>> The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited from 
>> the MATSHELL) I’d like to exploit during the solution phase when the 
>> smoother on the finest level is doing MatMults.
>> 
>> Is there some way to:
>> 1) decouple in -log_view the time spent in the MATSHELL MatMult and in the 
>> smoothers MatMult
> 
>   You can register a new event and then inside your MATSHELL MatMult() call 
> PetscLogEventBegin/End on your new event.
> 
>Note that the MatMult() like will still contain the time for your MatShell 
> mult so you will need to subtract it off to get the time for your non-shell 
> matmults.

In PERMON, we sometimes have quite complicated hierarchy of wrapped matrices 
and want to measure MatMult{,Transpose,Add,TransposeAdd} separately for 
particular ones. Think e.g. of having additive MATCOMPOSITE wrapping 
multiplicative MATCOMPOSITE wrapping MATTRANSPOSE wrapping MATAIJ. You want to 
measure this MATAIJ instance's MatMult separately but you surely don't want to 
rewrite implementation of MatMult_Transpose or force yourself to use MATSHELL 
just to hang the events on MatMult*.

We had a special wrapper type just adding some prefix to the events for the 
given object but this is not nice. What about adding a functionality to 
PetscLogEventBegin/End that would distinguish based on the first PetscObject's 
name or option prefix? Of course optionally not to break guys relying on 
current behavior - e.g. under something like -log_view_by_name. To me it's 
quite an elegant solution working for any PetscObject and any event.

I can do that if I get some upvotes.

Vaclav

> 
>> 2) hardwire a specific MatMult implementation for the smoother on the finest 
>> level
> 
>   In the latest release you do MatSetOperation() to override the normal 
> matrix vector product with anything else you want. 
> 
>> 
>> Thanks in advance,
>> Pierre
>> 
>> PS : here is what I have right now,
>> MatMult  118 1.0 1.0740e+02 1.6 1.04e+13 1.6 1.7e+06 6.1e+05 
>> 0.0e+00 47100 90 98  0  47100 90 98  0 81953703
>> […]
>> PCSetUp2 1.0 8.6513e+00 1.0 1.01e+09 1.7 2.6e+05 4.0e+05 
>> 1.8e+02  5  0 14 10 66   5  0 14 10 68 94598
>> PCApply   14 1.0 8.0373e+01 1.1 9.06e+12 1.6 1.3e+06 6.0e+05 
>> 2.1e+01 45 87 72 78  8  45 87 72 78  8 95365211 // I’m guessing a lot of 
>> time here is being wasted in doing inefficient MatMults on the finest level 
>> but this is only speculation
>> 
>> Same code with -pc_type none -ksp_max_it 13,
>> MatMult   14 1.0 1.2936e+01 1.7 1.35e+12 1.6 2.0e+05 6.1e+05 
>> 0.0e+00 15100 78 93  0  15100 78 93  0 88202079
>> 
>> The grid itself is rather simple (two levels, extremely aggressive 
>> coarsening),
>>   type is MULTIPLICATIVE, levels=2 cycles=v
>>   KSP Object: (mg_coarse_) 1024 MPI processes
>> linear system matrix = precond matrix:
>> Mat Object: 1024 MPI processes
>>   type: mpiaij
>>   rows=775, cols=775
>>   total: nonzeros=1793, allocated nonzeros=1793
>> 
>> linear system matrix followed by preconditioner matrix:
>> Mat Object: 1024 MPI processes
>>   type: shell
>>   rows=1369307136, cols=1369307136
>> Mat Object: 1024 MPI processes
>>   type: mpiaij
>>   rows=1369307136, cols=1369307136
>>   total: nonzeros=19896719360, allocated nonzeros=19896719360
> 

Re: [petsc-dev] GAMG and custom MatMults in smoothers

[Smith, Barry F.]

> On Jun 22, 2018, at 5:43 AM, Pierre Jolivet  
> wrote:
> 
> Hello,
> I’m solving a system using a MATSHELL and PCGAMG.
> The MPIAIJ Mat I’m giving to GAMG has a specific structure (inherited from 
> the MATSHELL) I’d like to exploit during the solution phase when the smoother 
> on the finest level is doing MatMults.
> 
> Is there some way to:
> 1) decouple in -log_view the time spent in the MATSHELL MatMult and in the 
> smoothers MatMult

   You can register a new event and then inside your MATSHELL MatMult() call 
PetscLogEventBegin/End on your new event.

Note that the MatMult() like will still contain the time for your MatShell 
mult so you will need to subtract it off to get the time for your non-shell 
matmults.

> 2) hardwire a specific MatMult implementation for the smoother on the finest 
> level

   In the latest release you do MatSetOperation() to override the normal matrix 
vector product with anything else you want. 

> 
> Thanks in advance,
> Pierre
> 
> PS : here is what I have right now,
> MatMult  118 1.0 1.0740e+02 1.6 1.04e+13 1.6 1.7e+06 6.1e+05 
> 0.0e+00 47100 90 98  0  47100 90 98  0 81953703
> […]
> PCSetUp2 1.0 8.6513e+00 1.0 1.01e+09 1.7 2.6e+05 4.0e+05 
> 1.8e+02  5  0 14 10 66   5  0 14 10 68 94598
> PCApply   14 1.0 8.0373e+01 1.1 9.06e+12 1.6 1.3e+06 6.0e+05 
> 2.1e+01 45 87 72 78  8  45 87 72 78  8 95365211 // I’m guessing a lot of time 
> here is being wasted in doing inefficient MatMults on the finest level but 
> this is only speculation
> 
> Same code with -pc_type none -ksp_max_it 13,
> MatMult   14 1.0 1.2936e+01 1.7 1.35e+12 1.6 2.0e+05 6.1e+05 
> 0.0e+00 15100 78 93  0  15100 78 93  0 88202079
> 
> The grid itself is rather simple (two levels, extremely aggressive 
> coarsening),
>type is MULTIPLICATIVE, levels=2 cycles=v
>KSP Object: (mg_coarse_) 1024 MPI processes
> linear system matrix = precond matrix:
>  Mat Object: 1024 MPI processes
>type: mpiaij
>rows=775, cols=775
>total: nonzeros=1793, allocated nonzeros=1793
> 
>  linear system matrix followed by preconditioner matrix:
>  Mat Object: 1024 MPI processes
>type: shell
>rows=1369307136, cols=1369307136
>  Mat Object: 1024 MPI processes
>type: mpiaij
>rows=1369307136, cols=1369307136
>total: nonzeros=19896719360, allocated nonzeros=19896719360