Re: [petsc-dev] odd log behavior

Python users including myself would love NaN since NaN is the default missing value marker for reasons of computational speed and convenience. For example, if you load these values into pandas, no extra code is needed to handle them. Other choices such as N/A would require some extra work for

Re: [petsc-dev] odd log behavior

Oh, this is not hard: /* /\* Put NaN into the time for all events that may not be time accurately since they may happen asynchronously on the GPU *\/ */ /* #if defined(PETSC_HAVE_DEVICE) */ /* if (!PetscLogGpuTimeFlag && petsc_gflops > 0) { */ /*

Re: [petsc-dev] odd log behavior

Ok thanks, And looking at the actual tests that I am doing more carefully I am not seeing much difference in the numbers that I care about with or without the GPU timers. And FWIW, from a user experience point of view, having -log_view present you with a sea of nans by default is a bit

Re: [petsc-dev] odd log behavior

Mark, Yes, Jed has already asked for ways to select certain operations at runtime that would get timed via the GPU timer. I am not sure of the ideal way to handle it. One possibility is PetscLogGpuTimeEvent(event number) which indicates events you want the time logged for. Another is

Re: [petsc-dev] odd log behavior

I am not sure I understand the logic, we print the ratio of max/min. I report max and look at the ratio to see if I might be catching some load imbalance or whatever. Is there a problem with that workflow? I assume there is or you would not have done this, so can I add a method that I think also

Re: [petsc-dev] odd log behavior

Only KSPSolve, SNESSolve, TSSolve will have legitimate values. "High-level" functions like PtAP can be asynchronous (meaning the GPU returns to the CPU to do more stuff) before they are complete, hence their timings would be incorrect and so they must be labeled with a special marker and not

Re: [petsc-dev] odd log behavior

On Tue, Apr 26, 2022 at 8:00 PM Barry Smith wrote: > > The current nan output has to be replaced to get the column alignment > correct, I just didn't feel like making that change also in the same MR. > > Something like Unknown or anything that fits in the column space would > be fine. It

Re: [petsc-dev] odd log behavior

> On Apr 26, 2022, at 6:40 PM, Matthew Knepley wrote: > > On Tue, Apr 26, 2022 at 6:12 PM Justin Chang > wrote: > I think N/A (not applicable) would be a better message than NaN. Prior to > this mail, even I thought I broke something with these NaN's > > We would

Re: [petsc-dev] odd log behavior

The current nan output has to be replaced to get the column alignment correct, I just didn't feel like making that change also in the same MR. Something like Unknown or anything that fits in the column space would be fine. It just means for the given run the timing numbers are not

Re: [petsc-dev] odd log behavior

On Tue, Apr 26, 2022 at 6:12 PM Justin Chang wrote: > I think N/A (not applicable) would be a better message than NaN. Prior to > this mail, even I thought I broke something with these NaN's > We would need some logic to check for NaN and change the format. Matt > On Tue, Apr 26, 2022 at

Re: [petsc-dev] odd log behavior

I think N/A (not applicable) would be a better message than NaN. Prior to this mail, even I thought I broke something with these NaN's On Tue, Apr 26, 2022 at 2:49 PM Matthew Knepley wrote: > On Tue, Apr 26, 2022 at 12:03 PM Mark Adams wrote: > >> Well, Nans are a clear sign that something is

Re: [petsc-dev] odd log behavior

On Tue, Apr 26, 2022 at 12:03 PM Mark Adams wrote: > Well, Nans are a clear sign that something is very wrong. > Barry chose them so that it could not be mistaken for an actual number. Matt > On Tue, Apr 26, 2022 at 11:52 AM Jacob Faibussowitsch > wrote: > >> There is an automatic

Re: [petsc-dev] odd log behavior

Well, Nans are a clear sign that something is very wrong. On Tue, Apr 26, 2022 at 11:52 AM Jacob Faibussowitsch wrote: > There is an automatic warning that shows when you do run with > `-log_view_gpu_time`, but perhaps there should also be an automatic warning > when *not* running with it. It

Re: [petsc-dev] odd log behavior

There is an automatic warning that shows when you do run with `-log_view_gpu_time`, but perhaps there should also be an automatic warning when *not* running with it. It is unfortunate that NaN is the value printed as this implies a bug but AFAIK it is unavoidable (Barry can say more on this

Re: [petsc-dev] odd log behavior

You have to add -log_view_gpu_time See https://gitlab.com/petsc/petsc/-/merge_requests/5056 Jose > El 26 abr 2022, a las 16:39, Mark Adams escribió: > > I'm seeing this on Perlmutter with Kokkos-CUDA. Nans in most log timing data > except the two 'Solve' lines. > Just cg/jacobi on snes/ex56.

[petsc-dev] odd log behavior

I'm seeing this on Perlmutter with Kokkos-CUDA. Nans in most log timing data except the two 'Solve' lines. Just cg/jacobi on snes/ex56. Any ideas? VecTDot2 1.0 nan nan 1.20e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan-nan 0 0.00e+000