Hi Greg -- > 1. It seems that start_test -performance ignores program-specific > options in <file>.compopts and <file>.execopts, although the files > COMPOPTS and EXECOPTS are used. When not running in performance > mode, the program-specific files are read. Is this correct/expected?
I think this is currently "expected" (or at least, known) but regrettable and something we'd like to fix. Specifically, for consistency and clarity, I believe that our intention is to ignore both .compopts/COMPOPTS and .execops/EXECOPTS for performance testing, but that we haven't taken the time to sort through existing performance tests to convert them over to this approach (e.g., by creating PERFCOMPOPTS/PERFEXECOPTS files for them that duplicate the contents of COMPOPTS/EXECOPTS if that's what should be done). Others should correct my memory if I've got that wrong. Apologies for what's arguably a bad wart that we've learned to live with (or ignore) and haven't had a chance to fix yet... > 2. If the program generates variable output, like reporting the timing > of an internal step, you need a <file>.prediff script to remove the > lines for start_test. But this script is also run for the > performance tests, which can eliminate exactly the data you're > trying to gather. So in this case you don't want <file>.prediff to > run. Again, is this the expected behavior, and if so is there a > recommended way to have both ways of running start_test pass? I don't know whether we've discussed this case much in the past. I could easily imagine having a .perfprediff file for performance pre-diffing or some other way to override .prediff's for a performance test, but don't know that we've ever entertained this option. Typically what we do to filter timing information (or other run-to-run specific data) out of a test's output is to use a config param or const to control whether or not that information is printed out and set it one way for correctness testing, the other for performance. I don't know whether this approach makes sense in your scenario or not, but personally find it more elegant than a prediff (maybe just because I hate writing prediff scripts). -Brad ------------------------------------------------------------------------------ _______________________________________________ Chapel-bugs mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-bugs
