Re: Stateful prove
On 29 Nov 2007, at 14:56, Andy Armstrong wrote: I'd like # Run all tests $ prove --state=save -rb t/*.t # Re-run failures from above $ prove --state=fail -rbv # Re-run failures and remember failures $ prove --state=save,fail -rbv Using the third form repeatedly would run only the test programs that failed previously - so as you fixed things the list of tests to run would shrink. We like? Kinda :-) One I'd like even more would be to run tests in order of most- recently-failed. It's something I've hacked together several times at various companies with T::H::Straps. Adrian
Stateful prove
I'd like # Run all tests $ prove --state=save -rb t/*.t # Re-run failures from above $ prove --state=fail -rbv # Re-run failures and remember failures $ prove --state=save,fail -rbv Using the third form repeatedly would run only the test programs that failed previously - so as you fixed things the list of tests to run would shrink. We like? -- Andy Armstrong, Hexten
Re: Stateful prove
On 29 Nov 2007, at 15:04, Adrian Howard wrote: On 29 Nov 2007, at 14:56, Andy Armstrong wrote: I'd like # Run all tests $ prove --state=save -rb t/*.t # Re-run failures from above $ prove --state=fail -rbv # Re-run failures and remember failures $ prove --state=save,fail -rbv Using the third form repeatedly would run only the test programs that failed previously - so as you fixed things the list of tests to run would shrink. We like? Kinda :-) One I'd like even more would be to run tests in order of most- recently-failed. It's something I've hacked together several times at various companies with T::H::Straps. $ prove --state=fail,all,save ? (option order matters, 'fail' adds the failed tests to the run queue, 'all' adds everything, list is de-duped so the failures run first, saved for next time) -- Andy Armstrong, Hexten
Re: Stateful prove
On 29 Nov 2007, at 15:08, Andy Armstrong wrote: $ prove --state=fail,all,save ? (option order matters, 'fail' adds the failed tests to the run queue, 'all' adds everything, list is de-duped so the failures run first, saved for next time) And, of course, $ prove --state=pass to make sure you didn't break anything new. And any other sensible options we can think of. -- Andy Armstrong, Hexten
Re: Stateful prove
On 29 Nov 2007, at 15:08, Andy Armstrong wrote: [snip] One I'd like even more would be to run tests in order of most- recently-failed. It's something I've hacked together several times at various companies with T::H::Straps. $ prove --state=fail,all,save ? (option order matters, 'fail' adds the failed tests to the run queue, 'all' adds everything, list is de-duped so the failures run first, saved for next time) I think that's subtly different from most-recently-failed order. -state=fail,all,save gives me failing tests first. If I'm reading you right it doesn't apply ordering within the groups of tests. Running all tests in most-recently-failed order gives me (roughly) the tests that a more likely to fail first - even (and this is the nice bit) when they happen to be passing at the moment. So, with most-recently-failed, as soon as I see an ok on the list I know that all my previously failing tests pass. As soon as I see the first few oks go past I can feel fairly confident that the things that are likely to break haven't. I can then go back to coding while the rest of the suite runs. If anything else fails then I'll see it at the end, and it will pop to the front of the list. I like this coz my preferred way of working is to run all of the tests all of the time, rather than just the particular test script that I'm working on at the moment. That way if I have some dumb code that breaks something else in the system I get the feedback earlier. Make vague sense? Cheers, Adrian
Re: Stateful prove
On 29 Nov 2007, at 16:05, Adrian Howard wrote: I think that's subtly different from most-recently-failed order. -state=fail,all,save gives me failing tests first. If I'm reading you right it doesn't apply ordering within the groups of tests. Running all tests in most-recently-failed order gives me (roughly) the tests that a more likely to fail first - even (and this is the nice bit) when they happen to be passing at the moment. Aha. Right. Clarity. So, with most-recently-failed, as soon as I see an ok on the list I know that all my previously failing tests pass. As soon as I see the first few oks go past I can feel fairly confident that the things that are likely to break haven't. I can then go back to coding while the rest of the suite runs. If anything else fails then I'll see it at the end, and it will pop to the front of the list. I like this coz my preferred way of working is to run all of the tests all of the time, rather than just the particular test script that I'm working on at the moment. That way if I have some dumb code that breaks something else in the system I get the feedback earlier. Make vague sense? Yup. I'm working on it now so I'll incorporate something like that. Thanks :) -- Andy Armstrong, Hexten
Re: Stateful prove
--- Andy Armstrong [EMAIL PROTECTED] wrote: Yup. I'm working on it now so I'll incorporate something like that. Thanks :) While you're at it, we can eliminate this bug (this is related to a stateful prove): http://rt.cpan.org/Public/Bug/Display.html?id=2101 Someone wants to be able to shuffle tests but if they fail, use the same playback order. This can be done by choosing a random seed for srand and then passing it in for a deterministic shuffle: prove --shuffle=$seed Or perhaps App::Prove can cache the last random seed used and use that if someone wants to repeat the last shuffle: prove --last-shuffle Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: Stateful prove
On 29 Nov 2007, at 16:18, Ovid wrote: Yup. I'm working on it now so I'll incorporate something like that. Thanks :) While you're at it, we can eliminate this bug (this is related to a stateful prove): http://rt.cpan.org/Public/Bug/Display.html?id=2101 Someone wants to be able to shuffle tests but if they fail, use the same playback order. This can be done by choosing a random seed for srand and then passing it in for a deterministic shuffle: prove --shuffle=$seed Or perhaps App::Prove can cache the last random seed used and use that if someone wants to repeat the last shuffle: prove --last-shuffle Good call. I'll do that too then. -- Andy Armstrong, Hexten
Re: Stateful prove
# from Andy Armstrong # on Thursday 29 November 2007 06:56: # Run all tests $ prove --state=save -rb t/*.t I would rather just '--save-state', why do the =foo,bar thing ? Just always save? Then --only-failed will always work without needing to re-run with the save flag? I guess an env var or alias could take care of that though. --Eric -- Turns out the optimal technique is to put it in reverse and gun it. --Steven Squyres (on challenges in interplanetary robot navigation) --- http://scratchcomputing.com ---
Re: Stateful prove
On Nov 29, 2007, at 10:22 AM, Andy Armstrong wrote: prove --shuffle=$seed Or perhaps App::Prove can cache the last random seed used and use that if someone wants to repeat the last shuffle: prove --last-shuffle Good call. I'll do that too then. I really really really don't like the idea of basing it on a given seed. I want to be able to know state about the tests that have run, the order they were in, etc, without having to go through prove to generate the sequence for me. -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: Stateful prove
On 29 Nov 2007, at 17:28, Andy Lester wrote: Good call. I'll do that too then. I really really really don't like the idea of basing it on a given seed. I want to be able to know state about the tests that have run, the order they were in, etc, without having to go through prove to generate the sequence for me. My plan is to record the order in which the tests were most recently run and to be able to use that as the order this time too. That OK? -- Andy Armstrong, Hexten
Re: Stateful prove
On Nov 29, 2007, at 11:30 AM, Andy Armstrong wrote: I really really really don't like the idea of basing it on a given seed. I want to be able to know state about the tests that have run, the order they were in, etc, without having to go through prove to generate the sequence for me. My plan is to record the order in which the tests were most recently run and to be able to use that as the order this time too. That OK? Yes, thank you. -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: Stateful prove
# from Andy Armstrong # on Thursday 29 November 2007 09:08: On 29 Nov 2007, at 17:10, Eric Wilhelm wrote: # Run all tests $ prove --state=save -rb t/*.t I would rather just '--save-state', why do the =foo,bar thing ? Because it allows you to express what you'd like to have happen with the saved state. You can express it in many ways. My question is: why choose this one? If the option parser preserved the order of the flags, you wouldn't have to cram that all into an '=foo,bar' opterand. --save-state --no-save-state --failed-only --failed-first (Yes, needs more thought in the abbreviation and short uniqueness than that (so --failed would mean failed-only (--failures-first?))) I think it is more a matter of expressing what you would like to have happen -- the user doesn't need to be thinking about the saved state so much. --Eric -- If you only know how to use a hammer, every problem begins to look like a nail. --Richard B. Johnson --- http://scratchcomputing.com ---
Re: Stateful prove
On 29 Nov 2007, at 17:10, Eric Wilhelm wrote: # Run all tests $ prove --state=save -rb t/*.t I would rather just '--save-state', why do the =foo,bar thing ? Because it allows you to express what you'd like to have happen with the saved state. -- Andy Armstrong, Hexten
Re: Stateful prove
On 29 Nov 2007, at 17:42, Eric Wilhelm wrote: If the option parser preserved the order of the flags, you wouldn't have to cram that all into an '=foo,bar' opterand. --save-state --no-save-state --failed-only --failed-first (Yes, needs more thought in the abbreviation and short uniqueness than that (so --failed would mean failed-only (--failures-first?))) Possibly. I liked the idea of grouping all the stateful stuff in one option. I'm working on the state engine now. Once that's done we can review the interface. -- Andy Armstrong, Hexten
Re: Stateful prove
# from Andy Armstrong # on Thursday 29 November 2007 12:26: Eric: I *really* like have all the state selectors in a single switch. Please have a look at it and tell me if you find it awful. How does an alias play-along with that? Can I say --state=save in the alias and --state=hot on the command-line? That is, how do I set my environment to always save the state and always run only the failed tests (unless all the previously failing tests pass, in which case continue running the rest to make sure.) $ prove # ten tests fail, fix some code... $ prove # ran only the failing tests -- 5 still fail, fix some code... $ prove # all the failing tests now pass and all the rest still pass Seems like a typical work-cycle, no? If so, it should be doable via alias or env var. --Eric -- Issues of control, repair, improvement, cost, or just plain understandability all come down strongly in favor of open source solutions to complex problems of any sort. --Robert G. Brown --- http://scratchcomputing.com ---
Re: Stateful prove
* Andy Armstrong [EMAIL PROTECTED] [2007-11-29 16:10]: $ prove --state=fail,all,save ? (option order matters, 'fail' adds the failed tests to the run queue, 'all' adds everything, list is de-duped so the failures run first, saved for next time) Why this mental overhead? Do they really have to be ordered? Is there any way that any other order would do anything remotely useful? Would `save,fail,all` be useful? Or `all,save,fail`? Every order I can think of besides the one you gave is pretty useless in an order-sensitive scheme. So I think the order of actions should be predetermined, in which case I’m with Eric – I think these should be regular options. Plus, I don’t think it’s very descriptive to use `state` in the names of the options at all. So that would be something like: --save-fails --only-fails --fails-first For grouping purposes, just give them single-letter aliases, eg. `-S`, `-O`, and `-F`. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Stateful prove
On 29 Nov 2007, at 22:10, Eric Wilhelm wrote: How does an alias play-along with that? Can I say --state=save in the alias and --state=hot on the command-line? You can as of r887 :) -- Andy Armstrong, Hexten
Re: Stateful prove
On 29 Nov 2007, at 22:09, A. Pagaltzis wrote: (option order matters, 'fail' adds the failed tests to the run queue, 'all' adds everything, list is de-duped so the failures run first, saved for next time) Why this mental overhead? Do they really have to be ordered? Is there any way that any other order would do anything remotely useful? Would `save,fail,all` be useful? Or `all,save,fail`? Every order I can think of besides the one you gave is pretty useless in an order-sensitive scheme. Sure - in that case. But failed,todo,all would do something sensible and distinct from todo,failed,all -- Andy Armstrong, Hexten
Re: Stateful prove
* Andy Armstrong [EMAIL PROTECTED] [2007-11-29 23:25]: On 29 Nov 2007, at 22:09, A. Pagaltzis wrote: Why this mental overhead? Do they really have to be ordered? Is there any way that any other order would do anything remotely useful? Would `save,fail,all` be useful? Or `all,save,fail`? Every order I can think of besides the one you gave is pretty useless in an order-sensitive scheme. Sure - in that case. But failed,todo,all would do something sensible and distinct from todo,failed,all Aha. I see; those were examples that hadn’t come up before. So ordering is useful to avoid an explosion of options. At the same time, there is a bunch of aspects where order sensitivity clearly does not make sense: does save,todo,failed do something useful? Or todo,save,failed? Or all,failed,todo or all,todo,failed? There is also Eric’s inquiry about how composed commands line, eg. from aliases, would be handled. Hm. I think this needs more thought… Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Stateful prove
On 29 Nov 2007, at 22:34, A. Pagaltzis wrote: Aha. I see; those were examples that hadn’t come up before. So ordering is useful to avoid an explosion of options. At the same time, there is a bunch of aspects where order sensitivity clearly does not make sense: does save,todo,failed do something useful? Or todo,save,failed? Or all,failed,todo or all,todo,failed? save isn't positional - it's just a switch. all selects everything - so if it appears before other options they will have no effect. There is also Eric’s inquiry about how composed commands line, eg. from aliases, would be handled. Fixed - although I have test failures on our smoke boxes at the moment. -- Andy Armstrong, Hexten
Re: Stateful prove
On 29 Nov 2007, at 22:38, Andy Armstrong wrote: Fixed - although I have test failures on our smoke boxes at the moment. Fixed also. -- Andy Armstrong, Hexten
Re: Stateful prove
Andy Armstrong wrote: I'd like # Run all tests $ prove --state=save -rb t/*.t # Re-run failures from above $ prove --state=fail -rbv # Re-run failures and remember failures $ prove --state=save,fail -rbv Using the third form repeatedly would run only the test programs that failed previously - so as you fixed things the list of tests to run would shrink. We like? When does state get cleared? -- I have a date with some giant cartoon robots and booze.
Re: Stateful prove
# from A. Pagaltzis # on Thursday 29 November 2007 14:34: failed,todo,all would do something sensible and distinct from todo,failed,all Aha. I see; those were examples that hadn’t come up before. So ordering is useful to avoid an explosion of options. Or ... WE COULD HAVE AN ORDERING OPTION PARSER! Sorry for the yelling, this one just happens to tickle my yelly-bone. --Eric -- The first rule about Debian is you don't talk about Debian --- http://scratchcomputing.com ---
Re: Stateful prove
On 29 Nov 2007, at 23:36, Michael G Schwern wrote: When does state get cleared? When you save a new state over the top of it. In practice you don't need to explicitly clear it I think. I could add an erase option - or you could just rm .prove. -- Andy Armstrong, Hexten
Re: Stateful prove
On 30 Nov 2007, at 00:25, Eric Wilhelm wrote: Aha. I see; those were examples that hadn’t come up before. So ordering is useful to avoid an explosion of options. Or ... WE COULD HAVE AN ORDERING OPTION PARSER! SPEAK UP. We could, indeed, have an ordering option parser. But I suspect ordered options are less easy to grok than a string that is explicitly a query the ordering of whose elements are significant. I've just added slow and fast options - so you can run the tests slowest to fastest or fastest to slowest. That's useful in conjunction with -j because it ensures parallel runs kick off with the slowest test avoiding the situation where you slow test happens to run last and you lose some of the benefit of parallel testing. So would you really have me add two more switches to prove's top level interface? The state engine has a fairly general mechanism for building queries from lists of terms - it seems wrong to couple that tightly to prove's UI by exposing each element of its syntax as a top level switch. Sorry for the yelling, this one just happens to tickle my yelly-bone. Well quite. But I see it as pretty much orthogonal to the question at hand. If we decided to go with top level switches to control the state engine we'd cross the option ordering bridge. But I simply don't think it's a good idea. As I write there are nine different options that can be passed to --state. Already that would be a lot of pollution in prove's already fairly crowded UI. How is it not better to keep all of those options in the --state namespace? It makes the documentation clearer, the UI easier to remember, the code easier to maintain. Have you tried it? -- Andy Armstrong, Hexten
