Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
I'd prefer a run-tests.php option that sets the timeout limit in seconds. Nice idea, but I'm not sure it's achievable under CLI. - Steph -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
Hi Travis, All it does is adds another option, -x, to run-tests.php. This sets an environmental variable which can then be checked for in the SKIPIF section of very slow-running tests. How do you specify that test A is slow? Is there a certain skipif message you include, or...? Yep. The two that bug me most are actually labelled 'slow test', so having a skipif condition for those wouldn't be too obscure. Note that it's only intended for when you're repeatedly running the same group of tests... I like Chris's idea better, but can't see a way to implement it. There's probably something really obvious I'm missing, let me know if. - Steph -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
Hey Stas, +$environment['EXEMPT_SLOW_TESTS'] = 1; Maybe SKIP_SLOW_TESTS? If it's checked in skip section... :) Otherwise - good idea! You can tell I'm reading my mail backwards today... I used 'EXEMPT' because the option is 'x' and I wanted it to be easy to remember. 's' wasn't available. - Steph -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
I'd prefer a run-tests.php option that sets the timeout limit in seconds.
Nice idea, but I'm not sure it's achievable under CLI.
Yes, it is. Check the system_with_timeout() function in the run-tests.php
script.
There you've the timeout hardcoded ('$leak_check ? 300 : 60'). You would
just need to make it configurable by some environment var.
Nuno
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
Yes, it is. Check the system_with_timeout() function in the run-tests.php
script.
There you've the timeout hardcoded ('$leak_check ? 300 : 60'). You would
just need to make it configurable by some environment var.
I already tried hard-coding both tv_sec and tv_usec to 0 and it makes no
difference here.
- Steph
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
On Jun 19, 2008, at 7:07 AM, Steph Fox wrote:
Yes, it is. Check the system_with_timeout() function in the run-
tests.php script.
There you've the timeout hardcoded ('$leak_check ? 300 : 60'). You
would just need to make it configurable by some environment var.
I already tried hard-coding both tv_sec and tv_usec to 0 and it
makes no difference here.
I can add this to PHPT - it uses a timeout based on reading the open
proc. The only problem is that it is currently treated as an error.
I could maybe add a new reporting level of timeout, though I do like
the idea of having some sort of meta-data to conditionally skip tests.
Maybe a better solution is to add an --exclude pattern and ask
people to either place potentially slow tests in tests/slow/, or name
then test case.slow.phpt? Being able to exclude a pattern of test
names definitely has more use than just setting a timeout.
Thoughts?
-T
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
Yes, it is. Check the system_with_timeout() function in the run-tests.php
script.
There you've the timeout hardcoded ('$leak_check ? 300 : 60'). You would
just need to make it configurable by some environment var.
I already tried hard-coding both tv_sec and tv_usec to 0 and it makes no
difference here.
uhm, file a bug report then. It was supposed to work..
Nuno
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
uhm, file a bug report then. It was supposed to work.. There are some open on this already. Also - Windows is the only environment where we actually have control over whether it works or not. Everything else relies on a system call. - Steph -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
There's nothing wrong with that approach. I'm trying to find something that addresses the problem (i.e., how can I not run tests that are going to take a long time to run?), while providing enough flexibility to answer other problems (i.e., how can I skip X tests that I don't care about?). If there's a way to address the problem without making it so specific, I'm all for it. I mostly agree - I'm just looking at 'here and now' rather than 'when the nice new test suite stuff is done'. 'Here and now', there isn't a reliable way to set this up and skipif looks like the cleanest option. I think the naming thing would be a better idea for something that's just starting out. It's a less good idea for a test suite that already exists IMHO. Agreed. If we were to go this route, I'd actually consider adding a run-tests.ini or some such that allows you to configure things like excluded file patterns. That might not be a bad idea. - Steph -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
Steph Fox wrote: So what was wrong with the simple skipif and env var approach again? The problem is it only skips the test! I'd like my slow tests to run (this generally occurs when I use a very remote DB). I end up manually increasing the timeout in stream_select() in run-tests.sh. This works for me on Linux. If we make the timeout value adjustable, you can set it to a low value so your slow tests are quickly aborted, and I can set it to a high value so my tests are run. Chris -- Christopher Jones, Oracle Email: [EMAIL PROTECTED]Tel: +1 650 506 8630 Blog: http://blogs.oracle.com/opal/ Free PHP Book: http://tinyurl.com/f8jad -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
Hi Chris, If we make the timeout value adjustable, you can set it to a low value so your slow tests are quickly aborted, and I can set it to a high value so my tests are run. It's easily enough done, it's just a separate issue. I can easily put a workaround in the Windows version of select.c to make a 0-second timeout work, but that's as far as it goes. The problem in run-tests (if anyone's interested - Travis seems to have a better solution) is that the actual read is always only 8192 bytes, and the actual API call never times out because stream_select() is reset at every iteration and called afresh. We'd need to be able to set a timeout limit in the run-tests script rather than in the call to stream_select(). And if we do that the tests of course fail, which isn't a good outcome. So 'skipif' suits my needs better, but not yours. I'll add both. - Steph Chris -- Christopher Jones, Oracle Email: [EMAIL PROTECTED]Tel: +1 650 506 8630 Blog: http://blogs.oracle.com/opal/ Free PHP Book: http://tinyurl.com/f8jad -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
On Jun 19, 2008, at 9:03 AM, Steph Fox wrote: There's nothing wrong with that approach. I'm trying to find something that addresses the problem (i.e., how can I not run tests that are going to take a long time to run?), while providing enough flexibility to answer other problems (i.e., how can I skip X tests that I don't care about?). If there's a way to address the problem without making it so specific, I'm all for it. I mostly agree - I'm just looking at 'here and now' rather than 'when the nice new test suite stuff is done'. 'Here and now', there isn't a reliable way to set this up and skipif looks like the cleanest option. The only problem with that is everything you add I've got to add to my GSoC project so PHPT :-) That might not be a bad idea. I think the long-term goal should be the ability to force skip files based on an --exclude parameter, an ini conf file (looks for -- ini-file file or cwd/tests.ini), and an ENV variable. The first and last would just be separated by the PATH_SEPARATOR for regex patterns. In addition, we can definitely make the time-out something that's settable via the command line and conf, but as you noted in your next email, that is a separate issue. I've created a few tickets on these so we can make sure to track these issues: * add exclude: http://phpt.info/ticket/69 * add timeout: http://phpt.info/ticket/70 -T -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
Steph Fox wrote: So 'skipif' suits my needs better, but not yours. I'll add both. Thanks Steph. Chris -- Christopher Jones, Oracle Email: [EMAIL PROTECTED]Tel: +1 650 506 8630 Blog: http://blogs.oracle.com/opal/ Free PHP Book: http://tinyurl.com/f8jad -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
Hi! +$environment['EXEMPT_SLOW_TESTS'] = 1; Maybe SKIP_SLOW_TESTS? If it's checked in skip section... :) Otherwise - good idea! -- Stanislav Malyshev, Zend Software Architect [EMAIL PROTECTED] http://www.zend.com/ (408)253-8829 MSN: [EMAIL PROTECTED] -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
Hey Steph; On Jun 18, 2008, at 3:37 PM, Steph Fox wrote: I'm using this locally because two of our tests take over 10 minutes each to run on my laptop, and I run the relevant bits of test suite every time I make a change. All it does is adds another option, -x, to run-tests.php. This sets an environmental variable which can then be checked for in the SKIPIF section of very slow-running tests. How do you specify that test A is slow? Is there a certain skipif message you include, or...? -T -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [PATCH] make it possible to skip very slow tests
Steph Fox wrote: Hi again, I'm using this locally because two of our tests take over 10 minutes each to run on my laptop, and I run the relevant bits of test suite every time I make a change. All it does is adds another option, -x, to run-tests.php. This sets an environmental variable which can then be checked for in the SKIPIF section of very slow-running tests. Any objections if I commit it in 5_3/HEAD? I'd prefer a run-tests.php option that sets the timeout limit in seconds. Chris -- Christopher Jones, Oracle Email: [EMAIL PROTECTED]Tel: +1 650 506 8630 Blog: http://blogs.oracle.com/opal/ Free PHP Book: http://tinyurl.com/f8jad -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
