Hi Jed, > It's also a matter of the need for a 'yes/no'-testing. Running a > fixed test like > if (err > eps) fail_test(); > is probably too harsh and we instead use some kind of continuous > metric to judge the outcome of the test. > > > Agreed, but note that an arbitrarily small continuum difference can > force an additional iteration. Maybe our tests should be configured to > "always run exactly 10 iterations". I wonder if a Bayesian filter could > be trained to recognize real failures.
Testing for the same number of iterations is - as you mentioned - a terrible metric. I see this regularly on GPUs, where rounding modes differ slightly from CPUs. Running a fixed (low) number of iterations is certainly the better choice here, provided that the systems we use for the tests are neither too ill-conditioned nor too well-behaved so that we can eventually reuse the tests for some preconditioners. > Speaking of a HTML table, something that is spinning in my head for > a long time already is that one can easily draw diagrams > automatically showing the convergence history of the residual norm > obtained in a test run. Coloring the frame of the plot proportional > to the relative deviation from a reference convergence history gives > you a quick idea of how far a test is off the reference. It won't > work for all tests, but it gives you on idea about the sanity of the > implementation. > > > I spend years trying to convince people to use actual norms instead of > jpeg semi-norms and now you want to reverse that? Blasphemy! Haha :-D Did I just throw the first stone? ;-) > Actually, that's an interesting idea. I guess it could be presented in a > more compressed way than the numerical values, thus providing a better > summary. Yep. Actual norms should still be used for collecting 'hard facts' about the state, and a 'pretty picture layer' will drive up efficiency in interpreting the results. Best regards, Karli
