2009/12/13 Robert Miller <[email protected]>: > Originally, refinement_matrices etc. were set up to run tests for a > certain amount of time, but several machines on boxen which we run > tests on have really messed up clocks (10-30 seconds of real time > correspond to about 1 second of their clock time), so we couldn't rely > on that. > > So the decision was to set the number of tests to be run instead. I > added a line to set the random seed to something genuinely random, so > that the "randomized" tests would be different each time they were > run.
Personally, I'm against disagree with tests being run this way in the Sage doctest test suite. I think such randomized testing should be done totally separately, e.g., as is done in the modular/modsym/tests.py file. I think there should be a huge range of randomized testing that gets run in parallel with available resources somewhere... but tend to view this as the thing to do next after doctest coverage is at 100%. William > > Regarding the output you saw printed, it is quite likely that the > keyboard interrupt caused this, since (as you can see) the handling of > keyboardinterrupt errors in partn_ref needs some work. I am verifying > that example now. > > I'll take a look at these doctest timings and see if I can't come up > with something more reasonable. I'll also see about improving the > keyboard interrupt handling. > > On Sun, Dec 13, 2009 at 4:48 PM, Alex Ghitza <[email protected]> wrote: >> >> Dear Robert, >> >> I was trying to see if I can shorten the randomized doctests in >> refinement_*.pyx, and I noticed two things that seem odd (to me, since >> I have no knowledge of the algorithms or implementation details for >> these files). >> >> 1. The time that the random tests take (with the same values of the >> arguments) can vary quite wildly. For instance (this is with >> sage-4.3.rc0 on sage.math): >> >> sage: time >> sage.groups.perm_gps.partn_ref.refinement_matrices.random_tests(5, 100, 200, >> 40) >> All passed: 50 random tests on 10 matrices. >> CPU times: user 56.00 s, sys: 0.01 s, total: 56.01 s >> Wall time: 56.01 s >> sage: time >> sage.groups.perm_gps.partn_ref.refinement_matrices.random_tests(5, 100, 200, >> 40) >> All passed: 50 random tests on 10 matrices. >> CPU times: user 18.71 s, sys: 0.00 s, total: 18.71 s >> Wall time: 18.72 s >> sage: time >> sage.groups.perm_gps.partn_ref.refinement_matrices.random_tests(5, 100, 200, >> 40) >> Exception KeyboardInterrupt: KeyboardInterrupt() in >> 'sage.groups.perm_gps.partn_ref.refinement_matrices.compare_matrices' ignored >> Exception KeyboardInterrupt: KeyboardInterrupt() in >> 'sage.groups.perm_gps.partn_ref.refinement_matrices.compare_matrices' ignored >> Exception KeyboardInterrupt: KeyboardInterrupt() in >> 'sage.groups.perm_gps.partn_ref.refinement_matrices.compare_matrices' ignored >> Exception KeyboardInterrupt: KeyboardInterrupt() in >> 'sage.groups.perm_gps.partn_ref.refinement_matrices.compare_matrices' ignored >> All passed: 50 random tests on 10 matrices. >> CPU times: user 1140.61 s, sys: 0.04 s, total: 1140.65 s >> Wall time: 1140.66 s >> >> >> For the last run, I was becoming convinced that sage was stuck, so I >> tried to interrupt with Ctrl-C a few times. I don't know whether the >> fact that this eventually ended not too long after that is actually >> related to pressing Ctrl-C (see also below). >> >> >> 2. In another instance, I again got impatient and pressed Ctrl-C. >> Here is the outcome: >> >> sage: time >> sage.groups.perm_gps.partn_ref.refinement_matrices.random_tests(5, 100, 200, >> 40) >> All passed: 50 random tests on 10 matrices. >> CPU times: user 35.64 s, sys: 0.00 s, total: 35.64 s >> Wall time: 35.64 s >> sage: time >> sage.groups.perm_gps.partn_ref.refinement_matrices.random_tests(5, 100, 200, >> 40) >> Exception KeyboardInterrupt: KeyboardInterrupt() in >> 'sage.groups.perm_gps.partn_ref.refinement_matrices.compare_matrices' ignored >> Exception KeyboardInterrupt: KeyboardInterrupt() in >> 'sage.groups.perm_gps.partn_ref.refinement_matrices.compare_matrices' ignored >> Exception KeyboardInterrupt: KeyboardInterrupt() in >> 'sage.groups.perm_gps.partn_ref.refinement_matrices.compare_matrices' ignored >> Exception KeyboardInterrupt: KeyboardInterrupt() in >> 'sage.groups.perm_gps.partn_ref.refinement_matrices.compare_matrices' ignored >> Exception KeyboardInterrupt: KeyboardInterrupt() in >> 'sage.groups.perm_gps.partn_ref.refinement_matrices.compare_matrices' ignored >> Exception KeyboardInterrupt: KeyboardInterrupt() in >> 'sage.groups.perm_gps.partn_ref.refinement_matrices.compare_matrices' ignored >> M: >> [11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30 0 6 0 0 0 0 0 0 2 >> 0 0 0 0 0 1 27 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 0 >> 27 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 21 0 0 13 0 0 0 0 0 0 0 0 0 0 14 0 0 >> 12 28 0 0 0 24 0 0 0 0 17 0 0 0 0 0 0 0 0 20 0 6 0 0 19 >> 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 12 0 0 29 0 0 0 0 0 8 >> 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0] >> [ 0 0 0 0 0 0 0 0 0 0 0 9 5 0 0 0 0 4 0 0 0 0 0 0 0 >> 0 8 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 26 0 0 29 0 0 >> 0 0 0 0 0 28 0 0 16 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 11 0 24 12 0 0 0 0 0 0 8 0 0 0 2 0 28 0 0 0 0 0 0 0 >> 0 0 12 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 22 21 0 >> 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 30 0 0 0 27 0 0 0 0 >> 0 0 0 8 0 0 0 0 0 0 0 22 6 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0] >> [ 0 27 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 >> 0 0 0 24 0 0 22 0 0 0 8 0 0 0 0 0 0 0 0 0 12 0 21 0 0 0 >> 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 27 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 0 5 >> 0 0 30 0 0 0 23 23 0 13 0 0 0 0 22 0 26 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 6 0 20 0 19 0 0 0 0 18 0 6 0 0 16 0 0 0 >> 0 0 0 0 0 0 0 22 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 18 >> 0 0 0] >> [21 0 0 0 0 0 0 17 0 0 3 0 0 0 0 0 0 0 0 0 26 0 0 0 0 >> 0 0 0 0 0 0 0 0 13 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 22 >> 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 22 0 0 0 >> 0 0 0 0 0 0 0 25 0 0 7 0 21 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 25 0 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0 0 >> 0 0 0 9 0 0 0 0 0 0 0 0 0 0 0 4 0 11 0 0 0 0 0 0 0 10 >> 0 0 0 0 15 0 30 0 0 0 0 0 25 0 0 0 0 15 0 0 0 0 0 0 0 >> 0 0 5] >> [ 0 0 0 0 0 0 0 0 0 0 0 0 0 10 4 0 0 0 22 0 15 0 0 0 0 >> 0 17 0 26 0 0 21 0 0 0 0 0 0 0 0 0 0 28 0 0 0 0 0 0 0 0 >> 0 0 0 0 10 0 0 0 0 0 0 0 9 0 2 0 0 0 0 0 17 26 0 2 20 >> 12 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 >> 0 0 0 24 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 11 0 0 26 0 0 12 0 0 0 0 0 0 0 0 0 30 23 23 0 0 0 0 0 >> 0 0 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 >> 0 0 0] >> [ 0 1 0 0 0 5 0 0 0 0 0 0 20 0 0 0 0 0 4 5 0 0 29 0 0 >> 0 23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 8 >> 0 0 0 3 0 0 0 0 0 0 0 0 16 0 29 0 0 0 0 0 0 0 18 0 0 >> 0 0 0 17 0 16 0 8 17 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 6 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 29 0 0 0 0 >> 0 0 0 0 0 0 16 18 0 0 0 0 0 0 0 0 0 0 0 0 27 0 0 0 0 0 >> 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 29 5 0 0 0 0 0 0 >> 0 0 0] >> [ 0 0 0 0 0 0 0 0 0 0 0 29 0 0 23 15 0 0 0 0 0 0 0 0 0 >> 12 1 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 28 0 12 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 25 0 0 0 >> 0 0 24 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 10 0 0 0 >> 0 8 0 0 0 15 0 0 23 8 0 27 0 0 0 0 0 0 0 0 0 0 13 8 0 0 >> 0 15 0 0 25 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 19 0 0 0 15 0 0 0 0 0 0 >> 26 0 28] >> [15 8 0 19 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 0 24 0 >> 0 0 0 23 0 0 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 21 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 6 0 23 0 0 0 0 4 0 0 0 0 0 0 0 >> 3 0 0 0 0 0 22 0 0 0 0 0 0 9 0 24 0 0 0 0 0 0 0 0 0 0 >> 0 9 0 6 0 0 0 0 0 23 7 0 0 0 0 0 0 0 21 0 0 0 0 0 0 >> 0 4 0 0 0 0 0 0 0 0 0 0 0 0 12 0 0 0 20 0 0 0 0 0 30 0 >> 0 0 0 0 0 0 0 0 0 0 0 2 0 3 0 28 2 0 0 0 0 0 0 0 0 >> 0 0 0] >> [ 0 0 0 0 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 7 0 0 14 0 0 0 12 0 0 0 0 6 0 0 0 0 0 >> 0 0 0 0 0 13 0 15 18 0 0 0 0 4 0 0 0 0 0 0 16 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 28 0 23 13 0 0 0 >> 0 0 0 0 19 0 0 0 0 10 0 0 19 0 9 0 0 11 28 0 0 0 0 0 0 >> 1 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 2 0 >> 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 26 0 0] >> [ 0 0 29 0 0 3 0 0 23 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 >> 0 18 0 0 9 0 0 23 0 0 0 0 0 29 0 0 0 0 0 0 0 0 0 0 0 0 >> 20 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 2 0 >> 0 0 0 7 0 0 0 27 0 0 3 0 0 0 0 0 23 0 0 0 0 0 0 0 0 0 >> 23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 6 0 0 19 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 >> 0 2 0 0 0 0 0 0 0 0 0 26 0 0 0 0 0 0 0 0 22 23 0 0 0 >> 0 0 0] >> [ 0 29 14 0 15 0 0 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 5 0 0 29 0 0 0 0 0 0 24 3 0 0 0 29 0 0 11 0 0 0 0 0 0 0 >> 0 19 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 21 0 0 0 0 18 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 21 0 0 0 0 0 0 0 0 0 0 0 9 0 0 0 >> 0 0 0 0 20 0 0 0 0 0 1 0 0 0 18 4 0 0 0 0 0 29 0 12 0 30 >> 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 >> 29 0 0] >> [ 0 0 0 7 0 9 0 0 0 0 0 0 0 0 0 0 30 0 23 0 0 0 0 0 0 >> 0 24 30 0 0 0 19 0 0 24 0 0 0 0 0 0 0 0 8 4 0 0 0 0 0 0 >> 0 0 0 0 0 14 0 5 0 0 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 11 0 0 15 0 0 30 0 0 0 0 0 9 0 0 0 20 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 4 0 0 0 6 0 0 0 23 0 0 0 0 0 25 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 23 0 0 0 0 0 12 11 0 0 10 0 >> 0 0 0] >> perm: >> [145, 172, 27, 82, 64, 148, 70, 3, 162, 84, 168, 7, 170, 9, 79, 34, 13, 152, >> 1, 178, 39, 42, 140, 175, 43, 154, 136, 77, 144, 176, 24, 97, 169, 104, 135, >> 133, 45, 143, 132, 134, 94, 81, 121, 14, 51, 89, 36, 53, 35, 54, 106, 165, >> 37, 83, 139, 10, 69, 16, 101, 117, 73, 151, 17, 74, 6, 109, 128, 72, 56, >> 120, 119, 60, 142, 149, 155, 156, 38, 173, 46, 122, 26, 99, 127, 21, 63, >> 138, 62, 108, 80, 111, 92, 124, 65, 150, 86, 130, 103, 102, 52, 48, 66, 67, >> 75, 112, 137, 25, 164, 105, 4, 59, 55, 153, 19, 15, 180, 28, 141, 177, 0, >> 87, 85, 166, 44, 160, 129, 30, 50, 20, 8, 131, 47, 98, 2, 18, 171, 147, 174, >> 110, 114, 90, 93, 40, 91, 61, 118, 95, 76, 113, 158, 107, 161, 163, 167, 88, >> 116, 123, 159, 32, 33, 22, 115, 57, 179, 29, 126, 146, 157, 31, 71, 41, 100, >> 58, 23, 49, 96, 12, 5, 68, 11, 125, 78] >> CPU times: user 2691.73 s, sys: 0.05 s, total: 2691.78 s >> Wall time: 2691.80 s >> >> >> This appears to indicate that a bug was found -- but I don't know >> whether my pressing Ctrl-C did actually eventually interrupt >> something, and maybe that's why the computation didn't work out >> anymore. Again the whole thing was over not long after my pressing >> Ctrl-C, and note that in all it took almost 45 minutes. >> >> >> >> So what's the point of all this? First, can you (or I, given further >> instructions) check whether the above is really a bug? Second, if the >> computation does get stuck on some random inputs, would there be a way >> to detect this in random_tests and report it? Otherwise we would >> never know about this, except for people sometimes reporting timeouts >> in running long doctests. Note also that I'm only writing about >> refinement_matrices.pyx here, but these issues might also affect >> refinement_binary.pyx and refinement_graphs.pyx (I haven't played with >> these enough to encounter these problems). >> >> >> >> This goes without saying: a perfectly acceptable response from you is >> "you have to wait until my thesis is done" :) I just wanted to send >> this out so I don't forget about it. >> >> >> Best, >> Alex >> >> >> -- >> >> Alex Ghitza -- Lecturer in Mathematics -- The University of Melbourne >> -- Australia -- http://www.ms.unimelb.edu.au/~aghitza/ >> > > > > -- > Robert L. Miller > http://www.rlmiller.org/ > > -- > To post to this group, send an email to [email protected] > To unsubscribe from this group, send an email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/sage-devel > URL: http://www.sagemath.org > -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org -- To post to this group, send an email to [email protected] To unsubscribe from this group, send an email to [email protected] For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
