Re: Problem with running lots of tests (I think)
Guys, Okay, just to follow-up in case anyone cared what the resolution on this one was, changing the loop full of ok()s to one giant pass() or fail() per loop fixed _everything_. Plus it runs a lot faster now. I know I've seen test suites that do thousands and thousands of tests, but they must be doing it some other way that I'm not privvy to or somethin'. Anyway, thanx everyone for the help; I've finally gotten a non-developer release of Data::Random out tonight. -- Buddy
Re: Problem with running lots of tests (I think)
On 2011.11.15 1:14 AM, Buddy Burden wrote: Okay, just to follow-up in case anyone cared what the resolution on this one was, changing the loop full of ok()s to one giant pass() or fail() per loop fixed _everything_. Plus it runs a lot faster now. I know I've seen test suites that do thousands and thousands of tests, but they must be doing it some other way that I'm not privvy to or somethin'. There is a 3rd option. The perl core maintains a separate implementation of Test::More in t/test.pl. You can drop that into your project and use it for your enormous tests. It does not record test results, that I know, and should not consume memory. I use it to test Test::More and am considering spinning it out into its own project. -- 94. Crucifixes do not ward off officers, and I should not test that. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Problem with running lots of tests (I think)
On Mon, Oct 31, 2011 at 06:17:37PM -0700, Buddy Burden wrote: Leon, *** Signal 9 That one is obvious, it has been SIGKILLed. Probably the tester thought the tests were hanging. Well, that's probably the most common error ... surely there can't be _that_ many CPAN Testers folks hanging around actually _watching_ the tests run and killing them when they take too long. No, but there are testers who have watchdog processes to kill off anything that runs for an unfeasibly long time. Typically CPAN Testers testing is done in an automated fashion, no? And, anyway, if you _were_ going to run the tests manually, you wouldn't turn on $AUTOMATED_TESTING, would you? I start my test runs manually, and then just leave them running in the background. Every so often I check on them, and if one of them appears to have got stuck I give it a kick. -- David Cantrell | London Perl Mongers Deputy Chief Heretic Every normal man must be tempted at times to spit on his hands, hoist the black flag, and begin slitting throats. -- H. L. Mencken
Re: Problem with running lots of tests (I think)
David, Well, that's probably the most common error ... surely there can't be _that_ many CPAN Testers folks hanging around actually _watching_ the tests run and killing them when they take too long. No, but there are testers who have watchdog processes to kill off anything that runs for an unfeasibly long time. Okay. I guess the next question then is: what constitutes an unfeasibly long time? Looks like the full version of this test file takes around 200 seconds ... on my machine. Other people's machines ... who can say? I guess I'm not sure what to do here. What do other folks advise? Just give up and don't run so many tests? That seems sorta contradictory to the idea that more tests is better. Or is there a better solution that I'm not seeing? Typically CPAN Testers testing is done in an automated fashion, no? And, anyway, if you _were_ going to run the tests manually, you wouldn't turn on $AUTOMATED_TESTING, would you? I start my test runs manually, and then just leave them running in the background. Every so often I check on them, and if one of them appears to have got stuck I give it a kick. H well, a standard make test would indeed appear to be stuck, in that test files don't have any output until either a test fails or they're all done. OTOH, the test itself is spitting out oks at what is most likely a furious rate ... it's just that Test::Harness is eating them all. And what, dare I ask, constitutes a kick? :-D Okay, this is addressing the signal 9 ones. And I'm pursuing the out of memory ones. Does anyone have any ideas about the no plan in output ones? Remembering that this is using the latest versions of Test::More and Test::Harness? -- Buddy
Re: Problem with running lots of tests (I think)
On Tue, Nov 01, 2011 at 10:48:43AM -0700, Buddy Burden wrote: David, Well, that's probably the most common error ... surely there can't be _that_ many CPAN Testers folks hanging around actually _watching_ the tests run and killing them when they take too long. No, but there are testers who have watchdog processes to kill off anything that runs for an unfeasibly long time. Okay. I guess the next question then is: what constitutes an unfeasibly long time? Looks like the full version of this test file takes around 200 seconds ... on my machine. Other people's machines ... who can say? I guess I'm not sure what to do here. What do other folks advise? Contact the individual testers, I guess. I start my test runs manually, and then just leave them running in the background. Every so often I check on them, and if one of them appears to have got stuck I give it a kick. H well, a standard make test would indeed appear to be stuck, in that test files don't have any output until either a test fails or they're all done. This seems to vary from one place to another - I'm not sure exactly what the differences are that cause it, but in some places some of my long-running tests *do* output some kind of count of how far they've got, and in some they don't. It's probably something to do with output buffering. OTOH, the test itself is spitting out oks at what is most likely a furious rate ... it's just that Test::Harness is eating them all. And what, dare I ask, constitutes a kick? :-D Control-C. But I don't just do it if nothing appears to be happening. But if there's no change in several minutes then I might do it. -- David Cantrell | London Perl Mongers Deputy Chief Heretic If you can read this, thank a teacher. If you're reading it in English, thank Chaucer.
Re: Problem with running lots of tests (I think)
David, I guess I'm not sure what to do here. What do other folks advise? Contact the individual testers, I guess. I'm not sure what to say though ... hey, dude, your automated testing is being rude to my tests, so go fix that? I mean, I wouldn't put it that way, obviously, but i can't help but feel like it would come out sounding that way ... This seems to vary from one place to another - I'm not sure exactly what the differences are that cause it, but in some places some of my long-running tests *do* output some kind of count of how far they've got, and in some they don't. It's probably something to do with output buffering. Hunh. Well, I suppose I could specifically put something (like a diag call) in there to spit out something every now and again, only if $AUTOMATED_TESTING. Maybe that would keep people's scripts from assuming the thing is hung? No, wait: a diag gets eaten by Test::Harness too, doesn't it? What could I do to get the output through Test::Harness without making it think there's a failure? -- Buddy
Re: Problem with running lots of tests (I think)
On Tue, Nov 1, 2011 at 12:56 AM, Buddy Burden barefootco...@gmail.com wrote: t/rand_time.t (Wstat: 9 Tests: 1764021 Failed: 0) Non-zero wait status: 9 *** Signal 9 That one is obvious, it has been SIGKILLed. Probably the tester thought the tests were hanging. t/rand_time.t (Wstat: 139 Tests: 996881 Failed: 0) Non-zero wait status: 139 Parse errors: No plan found in TAP output If you're using done_testing, you should explicitly have a test/build requirement on Test::More 0.88. Out of memory! Segmentation Fault - core dumped make: *** [test_dynamic] Error 139 Yeah, Test::Builder maintains some state for every test result. See https://metacpan.org/module/Test::Builder#MEMORY for some suggestions on how to handle that. Leon
Re: Problem with running lots of tests (I think)
Leon, *** Signal 9 That one is obvious, it has been SIGKILLed. Probably the tester thought the tests were hanging. Well, that's probably the most common error ... surely there can't be _that_ many CPAN Testers folks hanging around actually _watching_ the tests run and killing them when they take too long. Typically CPAN Testers testing is done in an automated fashion, no? And, anyway, if you _were_ going to run the tests manually, you wouldn't turn on $AUTOMATED _TESTING, would you? t/rand_time.t (Wstat: 139 Tests: 996881 Failed: 0) Non-zero wait status: 139 Parse errors: No plan found in TAP output If you're using done_testing, you should explicitly have a test/build requirement on Test::More 0.88. Ah, yes, I should have mentioned that: I did indeed forget to do that the first time, but the latest dev version has that corrected. In fact, the CPAN Testers giving me the above error all have the same version of Test::More that I have (0.98), and a version of Test::Harness that was actually _newer_ than mine (3.23), although I upgraded and retested on my side just to make sure that wasn't it. Out of memory! Segmentation Fault - core dumped make: *** [test_dynamic] Error 139 Yeah, Test::Builder maintains some state for every test result. See https://metacpan.org/module/Test::Builder#MEMORY for some suggestions on how to handle that. Cool, I'll check that out. -- Buddy
