Nothing is clear to me. One way to debug is put in just one stage at a time (comment out all others) and make sure each one generates consistent numbers by itself. Then add a second stage. You have to debug this just like any other bug...
Barry > On Apr 10, 2015, at 3:58 PM, Fande Kong <[email protected]> wrote: > > Thanks. I did add my several events. From the following cases, we possibly > can find an explanation. > > (1) > > Summary of Stages: ----- Time ------ ----- Flops ----- --- Messages --- > -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total > Avg %Total counts %Total > 0: Main Stage: 1.2115e+03 91.0% 8.6211e+12 2.3% 5.937e+07 3.2% > 7.314e+03 77.2% 2.344e+03 10.1% > 1: MG Apply: 1.1994e+02 9.0% 3.6819e+14 97.7% 1.780e+09 96.8% > 2.160e+03 22.8% 2.080e+04 89.9% > > (2) Summary of Stages: ----- Time ------ ----- Flops ----- --- > Messages --- -- Message Lengths -- -- Reductions -- > Avg %Total Avg %Total counts %Total > Avg %Total counts %Total > 0: Main Stage: 9.9736e+02 47.6% 2.4648e+13 0.8% 5.813e+07 1.0% > 1.013e+03 32.5% 3.151e+03 2.7% > 1: MG Apply: 1.0959e+03 52.4% 2.9005e+15 99.2% 5.734e+09 99.0% > 2.102e+03 67.5% 1.114e+05 97.2% > > > In the fist case, we do not have any warnings and the second case produces a > warning. I am using MG as a preconditioner. When the MG is cheap like case 1, > the Main Stage time is larger than the MG stage. We do not have any warnings. > If the MG is expensive, the Main Stage time is smaller than MG. We have a > warning now because SNESSolve time is larger than Main Stage. SNESSolve time > includes outer solver and MG preconditioner. > > Why we do not add MG time to the Main Stage. Should the Main Stage be the > total time? > > Fande, > > On Fri, Apr 10, 2015 at 2:30 PM, Barry Smith <[email protected]> wrote: > > Are you adding your own events or stages or just running with -log_summary? > > If you added your own events or stages this likely happened because you > changed between stages within an event so its start time is logged in one > stage but the end time correct gets put into a different stage. > > Barry > > > On Apr 10, 2015, at 3:22 PM, Fande Kong <[email protected]> wrote: > > > > Hi all, > > > > I am running an application with more than 4,000 cores. I get a warning in > > the -log_summary: Warning -- total time of even greater than time of entire > > stage -- something is wrong with the timer. > > > > I have no any clues for this warning. Any suggestions to remove it? > > > > Thanks, > > > > Fande, > >
