Re: [PHP-DEV] Tests repository
On Fri, 12 Mar 2010, Daniel Convissor wrote: On Fri, Mar 12, 2010 at 06:53:49PM -0500, Eric Stewart wrote: If I'm given the choice of those differences being in one file or in three different files (5.2, 5.3, 6.0)? I choose one file. Even if that files is more complex. It's still one file. One commit. One place to look for problems. And you get the bonus of being able to see the actual differences of the same functionality in different versions of PHP all in one place. Agreed. It seems we'll need not only a ==VERSION== section of some sort, but a ==VERSION_OUTPUT== type of thing as well. For example, when things get deprecated. The test suite needs to ensure that the function works normally in 5.2 but throws a deprecated warning in 5.3. I'd rather see two test cases for this then. Derick -- http://derickrethans.nl | http://xdebug.org Like Xdebug? Consider a donation: http://xdebug.org/donate.php twitter: @derickr and @xdebug -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On 12 Mar 2010, at 23:33, Jani Taskinen wrote: 13.3.2010 0:18, Stanislav Malyshev wrote: Hi! There are going to be some technical challenges. Some (maybe a lot) of test will need updates or rewriting. run-tests.php may need more improvements than what is already planned. Knowing this, I would still rather update run-tests.php and fix the tests, then continue to applying tests to different branches of the code base. I still have yet to hear *how* these tests are supposed to be updated, to work everywhere. It's very easy to say oh, we'll just fix them - but how exactly you're going to fix them if the test is supposed to do one thing in 5.2 and another in 5.3? Are you going to have 2 versions of What tests are you really talking about here? I thought we have regression tests in there which test that stuff does not change between versions. Such test AFAICT help us to keep stuff to work like it worked before and after some change somewhere in the related code. So there should not be any need for any updates given the tests aren't for some reason different between branches in which case they aren't really the same test anymore. Short version: if test works in 5.2 it also has to (!) work in 5.3. Otherwise the test is pointless. Can you define the case you're referring to here or are we actually talking about totally different thing? I recently wrote a patch for DOMDocument::saveHTML which made a test from 5.3 start failing: it expected there was no first argument, triggering an error when one was provided. I, in changing the function to mirror DOMDocument::saveXML, made this fail by adding a node argument: if we cannot even add functionality to where we currently trigger errors, then we can do very little to the language. For the subset of tests which are /not/ testing errors are thrown, I entirely agree, we should just change behaviour of code that does not currently throw errors. But we have tests for more than just that. We have tests that check errors are thrown in cases, and those tests we should be able to change at will. -- Geoffrey Sneddon http://gsnedders.com/ -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On 12 Mar 2010, at 20:23, Pierre Joye wrote: On Fri, Mar 12, 2010 at 8:10 PM, Alexey Zakhlestin indey...@gmail.com wrote: On 12.03.2010, at 22:06, Pierre Joye wrote: Many tests fail because they are written for a given platform, or even worst (from a portability point of view), only for a given configuration (library version, system version,etc.). And that's not about windows vs other, tests can work on a Debian/ubuntu version and fail on another. Well, these tests should just be removed/rewritten. Php-tests should test php, not libraries I would be interested to know how can we test the file API without testing libc, for example. But yes, that's the idea. Redesign the tests in a way that we output something when the results of the test is not what was expected, instead of testing random outputs from random libraries (in various versions and flavours). No big change in run-tests (none?) but a lot of necessary changes in existing tests. To prefix the following comment: I've spent the previous six months working full-time on Opera's new ECMAScript (i.e., JavaScript) engine, and have spent a fair bit of time looking at ECMAScript test suites. I'm somewhat doubtful about how we can ever test PHP that well: by and large PHP-the-language is defined by PHP-the-interpreter; as long as this is the case, how can a test have a pass/fail condition: the interpreter is by definition correct as it is the definition of the language you are testing? To take an example from ECMAScript, Math.sin is defined as, Returns an implementation-dependent approximation to the sine of x. The argument is expressed in radians. This is followed by a definition which defines exact return values for the case of the argument being ±0, ±Inf, or NaN. The intention of the ECMAScript spec saying such a thing is /exactly/ that of using system libraries: it is designed so that you can use whatever the sin function of the maths library you are using returns. In the PHP manual, sin is defined as, sin() returns the sine of the arg parameter. The arg parameter is in radians. This is similar, but contains one vital difference: the PHP function says it returns the _exact_ sine (or, at the very least, does not permit any variation from it). If the PHP interpreter is to actually follow the manual in this case, it is pretty much impossible for it to use any system library, as they could easily vary from the value of sine, which the manual does not permit. If nothing else, I think PHP testing would become a lot easier if we had some clear specification of _what_ PHP was actually expected to do. At the moment, all there is to go by is what is in the manual (almost always far too vague to actually exhaustively test) and what the implementation itself does (and it's never a good situation to be working out what the implementation /should/ do from what it actually does). The first thing that needs to be done with a lot of tests that fail on certain platforms currently within PHP is work out what result is expected, and that is currently almost impossible for the above reasons. I don't see how we can make any large gains in testing until we have some clear way of knowing what behaviour is expected. I'm certainly willing to help spec out more of PHP to the extent necessary to be able to decently test it, if there is interest in doing so. -- Geoffrey Sneddon http://gsnedders.com/ -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On 12 Mar 2010, at 12:37, Jani Taskinen wrote:
On 03/12/2010 12:29 PM, Hannes Magnusson wrote:
On Fri, Mar 12, 2010 at 11:18, Jani Taskinenjani.taski...@iki.fi
wrote:
Having tests in multiple branches is PITA. Hasn't anyone
considered that the
best way would be to move all tests into their own repository
(directory..whatever :) in SVN..? Considering they are supposed to
be used
for testing against regressions and BC breaks, they should always be
runnable using any PHP version?
Thats actually a fairly good idea.
Some tests however are not supposed to work in earlier releases, so
we
need to either add a new
==SKIP-VERSION==
5.2, 5.1, 5.0
Perhaps something like required min version is better.
Any ideas who has been working on the improved test stuff? Or was
that just a dream?
I'd rather just check for whether a feature exists or not, rather than
hard-coding version numbers in tests. For example, hat happens if the
PHP 5.3/6 case repeats itself, with more and more being backported? Do
we need to go through and change skip-version metadata on hundreds of
tests? Could we not just check that, e.g., get_called_class exists?
With such feature testing (in skipif?) we could just do something
along the lines of:
---FILE---
?php
if (version_compare(PHP_VERSION, '5.3', '=')
function_exists('get_called_class'))
echo 'PASS';
elseif (version_compare(PHP_VERSION, '5.3', '') !
function_exists('get_called_class'))
echo 'PASS';
else
echo 'FAIL';
?
--EXPECTF--
PASS
Then, through the magic of logic, we would be guaranteed to have the
right results for each version if all tests pass, and only one place
to change when backporting/removing a feature.
-g
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository (was: Re: [PHP-DEV] PHP 6)
On Fri, Mar 12, 2010 at 11:18, Jani Taskinen jani.taski...@iki.fi wrote: Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be used for testing against regressions and BC breaks, they should always be runnable using any PHP version? Thats actually a fairly good idea. Some tests however are not supposed to work in earlier releases, so we need to either add a new ==SKIP-VERSION== 5.2, 5.1, 5.0 or add more things to skipif. -Hannes -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository (was: Re: [PHP-DEV] PHP 6)
On Fri, Mar 12, 2010 at 11:18 AM, Jani Taskinen jani.taski...@iki.fi wrote: Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be used for testing against regressions and BC breaks, they should always be runnable using any PHP version? BC tests and easiness are two of the reasons why we have a donwloadable phpt packages on the windows download pages. Some changes / improvements are of course needed for the run-tests.php but IIRC, someone was rewriting it already? Can we use this svn link/external somehow? Having one module for the tests but you get them when checking out a php-src branch. Gwynne can certainly answer this question :) Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
Hannes Magnusson wrote: On Fri, Mar 12, 2010 at 11:18, Jani Taskinenjani.taski...@iki.fi wrote: Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be used for testing against regressions and BC breaks, they should always be runnable using any PHP version? Thats actually a fairly good idea. Some tests however are not supposed to work in earlier releases, so we need to either add a new ==SKIP-VERSION== 5.2, 5.1, 5.0 or add more things to skipif. Further - that might also offer a path for identifying better what features are added with a version? I'm still having fun with some aspects of making code compatible to 5.3 while still working with PHP4. Ideally it would have been nice to freeze our last 5.2 builds as the final PHP4 compatible versions, but 5.3 crept out in the field and so we have a bit of a mess with people complaining about 'bugs' which are simply warning messages. On one hand telling them how to change PHP settings to ignore them is probably the correct action, but you all know what users are like ;) -- Lester Caine - G8HFL - Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk// Firebird - http://www.firebirdsql.org/index.php -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On 03/12/2010 12:29 PM, Hannes Magnusson wrote: On Fri, Mar 12, 2010 at 11:18, Jani Taskinenjani.taski...@iki.fi wrote: Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be used for testing against regressions and BC breaks, they should always be runnable using any PHP version? Thats actually a fairly good idea. Some tests however are not supposed to work in earlier releases, so we need to either add a new ==SKIP-VERSION== 5.2, 5.1, 5.0 Perhaps something like required min version is better. Any ideas who has been working on the improved test stuff? Or was that just a dream? --Jani -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
Jani Taskinen schrieb: Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be used for testing against regressions and BC breaks, they should always be runnable using any PHP version? It is doable, likely desired but it is going to be a bit of work. What you save is the work of having to update multiple branches manually. What you risk is that not each and every test is prepared for being run with every version - although, maybe, in theory it should be. This is certainly something that can be solved by adding some more magic to run-tests.php, e.g. ==SKIP-VERSION== as suggested by Hannes - practicalities. Also, you may end up shipping the same huge set of tests for every PHP version regardless if all tests you ship are compatible with that version. Again, some magic should solve that - practicalities. There is one thing I fear, although it is desired to do. Many extensions link external libraries. If you throw all tests in one place and run all tests with all PHP versions, I strongly assume you'll get more reports on test failures. That is because the likeliness of someone out there running new tests designed for the latest version of an external library against an old library will increase. Yes, you want to know about any such incompatibility. Yes, it helps you making your software better. But it may mean significantly more work in particular if you follow the common paragidma of test ample must always be green at any time on every system. What may cure the extra work problem is to be more strict on compatible versions. I am aware how much people love BC. But there's a point where keeping BC should be left as an exercise to those asking for BC. All in all, I like the idea but I fear extra work. Ulf -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On 12.03.2010 12:37, Jani Taskinen wrote: Some tests however are not supposed to work in earlier releases, so we need to either add a new ==SKIP-VERSION== 5.2, 5.1, 5.0 Perhaps something like required min version is better. Imo you need both V.E.R, V.E.R for version stuff, since you have to test for feature that disappeared or are scheduled for removal, and others that are not there yet. But any case skipping versions is not a good approach since you'd have to update it every time a new major release is made for old tests. Cheers, Jordi signature.asc Description: OpenPGP digital signature
Re: [PHP-DEV] Tests repository
On 03/12/2010 01:48 PM, Ulf Wendel wrote: Jani Taskinen schrieb: Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be used for testing against regressions and BC breaks, they should always be runnable using any PHP version? What you save is the work of having to update multiple branches manually. Not only update, don't forget the maintaining of them. What you risk is that not each and every test is prepared for being run with every version - although, maybe, in theory it should be. This is It should not be theory for regression tests? If new release does not pass the old tests but the old versions still do, then it's quite likely the new version is buggy? Now we have different versions of same tests in each branch (in the worst cases) and thus the behaviour might really be different between them when it should not be. Also, you may end up shipping the same huge set of tests for every PHP version regardless if all tests you ship are compatible with that version. Again, some magic should solve that - practicalities. It's still one package for all. Thus less work, less space..? :) There is one thing I fear, although it is desired to do. Many extensions link external libraries. If you throw all tests in one place and run all tests with all PHP versions, I strongly assume you'll get more reports on test failures. That is because the likeliness of someone out there running new tests designed for the latest version of an external library against an old library will increase. This part I didn't quite understand..isn't this same issue with the current situation as well? versions. I am aware how much people love BC. But there's a point where keeping BC should be left as an exercise to those asking for BC. The BC is our holy grail. And having been bitten by this myself sometimes in last 3 years, I rather be a bit anal about BC now than I was before. :) --Jani -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
Jani Taskinen schrieb: What you risk is that not each and every test is prepared for being run with every version - although, maybe, in theory it should be. This is It should not be theory for regression tests? If new release does not pass the old tests but the old versions still do, then it's quite likely the new version is buggy? Now we have different versions of same tests in each branch (in the worst cases) and thus the behaviour might really be different between them when it should not be. I am with you on the idea of a central repository and the arguments behind your two questions. Though, you know, I'm lazy and I fear you really find issues that need to be fixed :-) . For a transition period there's likely to be more work and the number of test failures is likely to go up. That is nothing to really worry about as long as you manage to educate users that it is not a quality defect of PHP as such but a temporary matter of an different and improved testing approach. There is one thing I fear, although it is desired to do. Many extensions link external libraries. If you throw all tests in one place and run all tests with all PHP versions, I strongly assume you'll get more reports on test failures. That is because the likeliness of someone out there running new tests designed for the latest version of an external library against an old library will increase. This part I didn't quite understand..isn't this same issue with the current situation as well? Its only a diffuse feeling of mine, I could well be wrong. In any case I am not trying to argue against your proposal. Just the opposite. Let's assume the worst case status quo of different versions of the same test in each branch. The one in an old branch has been written with the versions of the external library in mind that has been current when the branch was current. The one in a current branch has been written against the latest version of the external library. I continue to assume that users of the old PHP branch run rather old systems whereas users of a current PHP branch run on current systems. Therefore only few people may have tried to run the old test against the latest library or the other way around. It is just a feeling that your proposal will implicitly cause some combinations of PHP version x test x external library version to be tested that have not been checked before. Your one test for all proposal is likely to unveil some different versions of the same test and external library is not BC issues. That's good. But its gonna be work... Ulf -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository (was: Re: [PHP-DEV] PHP 6)
On Mar 12, 2010, at 5:33 AM, Pierre Joye wrote: Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be used for testing against regressions and BC breaks, they should always be runnable using any PHP version? BC tests and easiness are two of the reasons why we have a donwloadable phpt packages on the windows download pages. Some changes / improvements are of course needed for the run-tests.php but IIRC, someone was rewriting it already? Can we use this svn link/external somehow? Having one module for the tests but you get them when checking out a php-src branch. Gwynne can certainly answer this question :) Yes, this is quite possible. It's not quite as clean and smooth as CVS' modules support was, which is a continuing source of annoyance to me about SVN, but it's not too terribly painful either. Just the usual svn:externals properly on the php-src branches pointing to whereever the tests end up will do nicely. For the record, my standing: PHP 5.4: -1 PHP 6: -1 PHP 7 with new Unicode stuff: +1 -- Gwynne -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
Hi! Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be Yes, but: some tests are version-dependant, some are not. And since we have this output match testing paradigm, it would probably keep being this way. If you're going to have branches in that SVN, how it would be different from what we have now? -- Stanislav Malyshev, Zend Software Architect s...@zend.com http://www.zend.com/ (408)253-8829 MSN: s...@zend.com -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On 03/12/2010 08:08 PM, Stanislav Malyshev wrote: Hi! Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be Yes, but: some tests are version-dependant, some are not. And since we That's why we'd need to add some section to select the minimum version required to run the test. have this output match testing paradigm, it would probably keep being this way. If you're going to have branches in that SVN, how it would be different from what we have now? Considering what I said above about version check..why would you need branches in it? And what/how does it matter what the tests output if it needs to be same in any PHP version anyway..? The word repository was bad choice. The idea is just to have shared tests. Perhaps that makes it more clear? --Jani -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On 03/12/2010 04:15 PM, Ulf Wendel wrote: For a transition period there's likely to be more work and the number of test failures is likely to go up. That is nothing to really worry about as long as you manage to educate users that it is not a quality defect of PHP as such but a temporary matter of an different and improved testing approach. I'd say it's more work since some tests do not exists in some branch. As we might soon have quite closely related 3 branches, such differences are quite small. As trunk would have exactly same tests as PHP_5_3 has, the real differences would be only with PHP_5_2. So if we want such common tests, I think the time to do it is about now. Let's assume the worst case status quo of different versions of the same test in each branch. The one in an old branch has been written with the versions of the external library in mind that has been current when the branch was current. The one in a current branch has been written against the latest version of the external library. I continue to assume that users of the old PHP branch run rather old systems whereas users of a current PHP branch run on current systems. Therefore only few people may have tried to run the old test against the latest library or the other way around. It is just a feeling that your proposal will implicitly cause some combinations of PHP version x test x external library version to be tested that have not been checked before. So you actually fear that you'd have to fix some things in older branches too? Well, I think (hope) such cases are rare and such problems actually already exist too. Of course we can always stay with the status quo, but is that really good for PHP in general? Your one test for all proposal is likely to unveil some different versions of the same test and external library is not BC issues. That's good. But its gonna be work... IMO, such work is never a waste. --Jani -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
Hi! That's why we'd need to add some section to select the minimum version required to run the test. How minimum version would help? If 5.2 requires one test, 5.3 another and trunk another - you either have to have 3 tests or declare that only thing you're testing is trunk (which is obviously a very bad idea). Considering what I said above about version check..why would you need branches in it? And what/how does it matter what the tests output if it needs to be same in any PHP version anyway..? It doesn't need to be the same now. Actually, for a lot of tests it isn't, for one reason or another (mostly error messages and stuff like that now, but there's more). -- Stanislav Malyshev, Zend Software Architect s...@zend.com http://www.zend.com/ (408)253-8829 MSN: s...@zend.com -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On Fri, Mar 12, 2010 at 7:14 PM, Jani Taskinen jani.taski...@iki.fi wrote: On 03/12/2010 08:08 PM, Stanislav Malyshev wrote: Hi! Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be Yes, but: some tests are version-dependant, some are not. And since we That's why we'd need to add some section to select the minimum version required to run the test. have this output match testing paradigm, it would probably keep being this way. If you're going to have branches in that SVN, how it would be different from what we have now? Considering what I said above about version check..why would you need branches in it? And what/how does it matter what the tests output if it needs to be same in any PHP version anyway..? What you describe may work for PHP core features without external dependencies but will fail for anything with external dependencies. Many tests fail because they are written for a given platform, or even worst (from a portability point of view), only for a given configuration (library version, system version,etc.). And that's not about windows vs other, tests can work on a Debian/ubuntu version and fail on another. However the idea of shared tests is what I've been adovocating for some time already, to improve portability and BC checks. It requires to write true unit tests, and huge test including dozen of cases being tested in one test. It will also require to rewite many tests to output something when it fails instead of testing standard outpout with system dependent messages (think about localization too, which can affect behaviors as well). All in all I lilke the idea and will support it as much as I can. We have to keep in mind that it will need a couple of testfests to realize it(read: huge work, more like doing it from scratch for 40-60% of the existing test, rough estimation). Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On 12.03.2010, at 22:06, Pierre Joye wrote: Many tests fail because they are written for a given platform, or even worst (from a portability point of view), only for a given configuration (library version, system version,etc.). And that's not about windows vs other, tests can work on a Debian/ubuntu version and fail on another. Well, these tests should just be removed/rewritten. Php-tests should test php, not libraries -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On Fri, Mar 12, 2010 at 8:10 PM, Alexey Zakhlestin indey...@gmail.com wrote: On 12.03.2010, at 22:06, Pierre Joye wrote: Many tests fail because they are written for a given platform, or even worst (from a portability point of view), only for a given configuration (library version, system version,etc.). And that's not about windows vs other, tests can work on a Debian/ubuntu version and fail on another. Well, these tests should just be removed/rewritten. Php-tests should test php, not libraries I would be interested to know how can we test the file API without testing libc, for example. But yes, that's the idea. Redesign the tests in a way that we output something when the results of the test is not what was expected, instead of testing random outputs from random libraries (in various versions and flavours). No big change in run-tests (none?) but a lot of necessary changes in existing tests. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On 12.03.2010, at 22:20, Stanislav Malyshev wrote: Hi! Well, these tests should just be removed/rewritten. Php-tests should test php, not libraries That's easy to say - but how exactly you're going to test functionality of, say. ext/intl without testing the underlying ICU library? Well, here's the way I see it: Extensions (including ext/intl) declare their API and tests are made againt this API. No more and no less. If, for example, some error-text comes directly from underlying library and extension doesn't have control over it it shouldn't be part of API. In this case we can test that some error-text was returned but we shouldn't test for exact text-match -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On 12.03.2010, at 22:23, Pierre Joye wrote: On Fri, Mar 12, 2010 at 8:10 PM, Alexey Zakhlestin indey...@gmail.com wrote: On 12.03.2010, at 22:06, Pierre Joye wrote: Many tests fail because they are written for a given platform, or even worst (from a portability point of view), only for a given configuration (library version, system version,etc.). And that's not about windows vs other, tests can work on a Debian/ubuntu version and fail on another. Well, these tests should just be removed/rewritten. Php-tests should test php, not libraries I would be interested to know how can we test the file API without testing libc, for example. see my reply to Stas. The idea is to test things which we can guarantee. If our documentation says, that function does this or that, then we should check for it and wrap system-calls in such way, that our file API always does these things or fails in strict predictable manner. And if we can't guarantee some behaviour then we should add corresponding note to documentation and we shouldn't add tests. -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On Fri, Mar 12, 2010 at 8:39 PM, Alexey Zakhlestin indey...@gmail.com wrote: On 12.03.2010, at 22:23, Pierre Joye wrote: On Fri, Mar 12, 2010 at 8:10 PM, Alexey Zakhlestin indey...@gmail.com wrote: On 12.03.2010, at 22:06, Pierre Joye wrote: Many tests fail because they are written for a given platform, or even worst (from a portability point of view), only for a given configuration (library version, system version,etc.). And that's not about windows vs other, tests can work on a Debian/ubuntu version and fail on another. Well, these tests should just be removed/rewritten. Php-tests should test php, not libraries I would be interested to know how can we test the file API without testing libc, for example. see my reply to Stas. The idea is to test things which we can guarantee. If our documentation says, that function does this or that, then we should check for it and wrap system-calls in such way, that our file API always does these things or fails in strict predictable manner. And if we can't guarantee some behaviour then we should add corresponding note to documentation and we shouldn't add tests. That's exactly (for part of the needs) what I describe in the 2nd part of my reply. -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository (run-tests.php)
On Fri, Mar 12, 2010 at 6:37 AM, Jani Taskinen jani.taski...@iki.fi wrote: On 03/12/2010 12:29 PM, Hannes Magnusson wrote: On Fri, Mar 12, 2010 at 11:18, Jani Taskinenjani.taski...@iki.fi wrote: Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be used for testing against regressions and BC breaks, they should always be runnable using any PHP version? Thats actually a fairly good idea. Some tests however are not supposed to work in earlier releases, so we need to either add a new ==SKIP-VERSION== 5.2, 5.1, 5.0 Perhaps something like required min version is better. Any ideas who has been working on the improved test stuff? Or was that just a dream? First of all, I apologize for the subject line change, but this really is a different issue than the test repository itself. It was started by George during last years GSOC. Zoe and Stefan were also working on it after GSOC ended. Both Zoe and Stefan ran out of time and it went dormant for a bit. I contacted Zoe about the status and offered to pick it up, but I also offered to organize TestFest this year. --Jani -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php I really do want to do both but I can't do both at the same time as they both will be big time sinks for a part time volunteer. Solutions: 1. I continue to work on TestFest and then work on run-tests.php as soon as the TestFest load lightens up enough to allow it. 2. Someone else picks up run-tests.php and I focus solely on TestFest. 3. Someone else picks up TestFest and I focus solely on run-tests.php. My vote is option 1 as I really like both projects. But it wouldn't be fair for me to slow either down if the need for both is pressing. Eric Lee Stewart
Re: [PHP-DEV] Tests repository
Hi! That's easy to say - but how exactly you're going to test functionality of, say. ext/intl without testing the underlying ICU library? Well, here's the way I see it: Extensions (including ext/intl) declare their API and tests are made againt this API. No more and no less. It's all sounds very nice in theory, except that in practice it doesn't work this way. Different versions of the library return different data, different errors, have different conditions, etc. You can just declare half of the tests to be not API, but in fact it is, that's what people use and that's what they want to be sure that it works - just different versions of it act differently. If, for example, some error-text comes directly from underlying library and extension doesn't have control over it it shouldn't be part of API. In this case we can test that some error-text was returned but we shouldn't test for exact text-match How do you know that some text is the error and that it's the correct one and not just generic wrong library call one which is produced on any bad call? And you can't just declare that number formatting routine returns some text - it would be useless. You want very specific text. But what if some detail of that text changes for some of the test cases? -- Stanislav Malyshev, Zend Software Architect s...@zend.com http://www.zend.com/ (408)253-8829 MSN: s...@zend.com -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On Fri, Mar 12, 2010 at 4:55 PM, Stanislav Malyshev s...@zend.com wrote: Hi! That's easy to say - but how exactly you're going to test functionality of, say. ext/intl without testing the underlying ICU library? Well, here's the way I see it: Extensions (including ext/intl) declare their API and tests are made againt this API. No more and no less. It's all sounds very nice in theory, except that in practice it doesn't work this way. Different versions of the library return different data, different errors, have different conditions, etc. You can just declare half of the tests to be not API, but in fact it is, that's what people use and that's what they want to be sure that it works - just different versions of it act differently. If, for example, some error-text comes directly from underlying library and extension doesn't have control over it it shouldn't be part of API. In this case we can test that some error-text was returned but we shouldn't test for exact text-match How do you know that some text is the error and that it's the correct one and not just generic wrong library call one which is produced on any bad call? And you can't just declare that number formatting routine returns some text - it would be useless. You want very specific text. But what if some detail of that text changes for some of the test cases? -- Stanislav Malyshev, Zend Software Architect s...@zend.com http://www.zend.com/ (408)253-8829 MSN: s...@zend.com -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php There are going to be some technical challenges. Some (maybe a lot) of test will need updates or rewriting. run-tests.php may need more improvements than what is already planned. Knowing this, I would still rather update run-tests.php and fix the tests, then continue to applying tests to different branches of the code base. +1 to centralize tests. Eric Lee Stewart
Re: [PHP-DEV] Tests repository
Hi! There are going to be some technical challenges. Some (maybe a lot) of test will need updates or rewriting. run-tests.php may need more improvements than what is already planned. Knowing this, I would still rather update run-tests.php and fix the tests, then continue to applying tests to different branches of the code base. I still have yet to hear *how* these tests are supposed to be updated, to work everywhere. It's very easy to say oh, we'll just fix them - but how exactly you're going to fix them if the test is supposed to do one thing in 5.2 and another in 5.3? Are you going to have 2 versions of the file? if(version) wrapping the file? Two sets of output? Check for versions before outputting data? I think it's irresponsible to take such kind of decision without actually seeing what you're going to deal with and how. -- Stanislav Malyshev, Zend Software Architect s...@zend.com http://www.zend.com/ (408)253-8829 MSN: s...@zend.com -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
13.3.2010 0:18, Stanislav Malyshev wrote: Hi! There are going to be some technical challenges. Some (maybe a lot) of test will need updates or rewriting. run-tests.php may need more improvements than what is already planned. Knowing this, I would still rather update run-tests.php and fix the tests, then continue to applying tests to different branches of the code base. I still have yet to hear *how* these tests are supposed to be updated, to work everywhere. It's very easy to say oh, we'll just fix them - but how exactly you're going to fix them if the test is supposed to do one thing in 5.2 and another in 5.3? Are you going to have 2 versions of What tests are you really talking about here? I thought we have regression tests in there which test that stuff does not change between versions. Such test AFAICT help us to keep stuff to work like it worked before and after some change somewhere in the related code. So there should not be any need for any updates given the tests aren't for some reason different between branches in which case they aren't really the same test anymore. Short version: if test works in 5.2 it also has to (!) work in 5.3. Otherwise the test is pointless. Can you define the case you're referring to here or are we actually talking about totally different thing? --Jani -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On Fri, Mar 12, 2010 at 11:33 PM, Jani Taskinen jani.taski...@sci.fi wrote: 13.3.2010 0:18, Stanislav Malyshev wrote: Hi! There are going to be some technical challenges. Some (maybe a lot) of test will need updates or rewriting. run-tests.php may need more improvements than what is already planned. Knowing this, I would still rather update run-tests.php and fix the tests, then continue to applying tests to different branches of the code base. I still have yet to hear *how* these tests are supposed to be updated, to work everywhere. It's very easy to say oh, we'll just fix them - but how exactly you're going to fix them if the test is supposed to do one thing in 5.2 and another in 5.3? Are you going to have 2 versions of What tests are you really talking about here? I thought we have regression tests in there which test that stuff does not change between versions. Such test AFAICT help us to keep stuff to work like it worked before and after some change somewhere in the related code. So there should not be any need for any updates given the tests aren't for some reason different between branches in which case they aren't really the same test anymore. Short version: if test works in 5.2 it also has to (!) work in 5.3. Otherwise the test is pointless. Can you define the case you're referring to here or are we actually talking about totally different thing? Did you read our replies? I cited two kind of tests failing even on the same version. There are also many new errors or behaviors changes between 5.2 and 5.3, some errors are based on system error messages or using the underlying libraries errors. As I said already, it is possible but the way we test things have to be rewamped. Especially the variationxyz kind of tests with dozen of cases in one single phpt, including all kind of errors, system, php, etc. That's not a small move, but a huge task which will take some months to be done. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On Fri, Mar 12, 2010 at 5:39 PM, Pierre Joye pierre@gmail.com wrote: On Fri, Mar 12, 2010 at 11:33 PM, Jani Taskinen jani.taski...@sci.fi wrote: 13.3.2010 0:18, Stanislav Malyshev wrote: Hi! There are going to be some technical challenges. Some (maybe a lot) of test will need updates or rewriting. run-tests.php may need more improvements than what is already planned. Knowing this, I would still rather update run-tests.php and fix the tests, then continue to applying tests to different branches of the code base. I still have yet to hear *how* these tests are supposed to be updated, to work everywhere. It's very easy to say oh, we'll just fix them - but how exactly you're going to fix them if the test is supposed to do one thing in 5.2 and another in 5.3? Are you going to have 2 versions of What tests are you really talking about here? I thought we have regression tests in there which test that stuff does not change between versions. Such test AFAICT help us to keep stuff to work like it worked before and after some change somewhere in the related code. So there should not be any need for any updates given the tests aren't for some reason different between branches in which case they aren't really the same test anymore. Short version: if test works in 5.2 it also has to (!) work in 5.3. Otherwise the test is pointless. Can you define the case you're referring to here or are we actually talking about totally different thing? Did you read our replies? I cited two kind of tests failing even on the same version. There are also many new errors or behaviors changes between 5.2 and 5.3, some errors are based on system error messages or using the underlying libraries errors. As I said already, it is possible but the way we test things have to be rewamped. Especially the variationxyz kind of tests with dozen of cases in one single phpt, including all kind of errors, system, php, etc. That's not a small move, but a huge task which will take some months to be done. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php I think specific examples (ie: a link to a 5.2 test and the corresponding 5.3 test) which are currently impossible to merge would be most beneficial at this point. I'll search for some this evening. If anyone knows of one or two off the top of their head, please provide links. Eric Lee Stewart
Re: [PHP-DEV] Tests repository
Hi! What tests are you really talking about here? I thought we have regression tests in there which test that stuff does not change between Yes, we have those. But we also have other tests, which are different between versions. should not be any need for any updates given the tests aren't for some reason different between branches in which case they aren't really the same test anymore. I'm not into paying semantic games about the meaning of the word is. Define them as different tests if you want - how that solves the problem, if you have different tests in the next version? You still have two of them, not one. Short version: if test works in 5.2 it also has to (!) work in 5.3. Otherwise the test is pointless. It's not what happens with real .phpt tests, and they are not pointless. We can spend weeks discussing how the fine theory it should be, but in practice if 5.2 outputs one thing and 5.3 another then the tests differ. Just take Zend engine tests directory - out of 511 tests 62 are different in 5.3 - more than 10%. Most of them are messages, but not only. What you're going to do with them? -- Stanislav Malyshev, Zend Software Architect s...@zend.com http://www.zend.com/ (408)253-8829 MSN: s...@zend.com -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
Hi! I think specific examples (ie: a link to a 5.2 test and the corresponding 5.3 test) which are currently impossible to merge would be most beneficial at this point. Do a diff between two test dirs (you can use engine tests, php main tests, etc.), get a list of different files and diff them between versions. You'll see what kind of stuff is being done in tests now and how differences arise. There are many ways to get test that is different between 5.2 and 5.3. -- Stanislav Malyshev, Zend Software Architect s...@zend.com http://www.zend.com/ (408)253-8829 MSN: s...@zend.com -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On Sat, Mar 13, 2010 at 12:04 AM, Eric Stewart ericleestew...@gmail.com wrote: I think specific examples (ie: a link to a 5.2 test and the corresponding 5.3 test) which are currently impossible to merge would be most beneficial at this point. I'll search for some this evening. If anyone knows of one or two off the top of their head, please provide links. It is always possible to write a test for both versions (except namespace). However it requires a radical change on how we test things. Instead of testing the output of given script, we will have to add logics in a test, something similar to the classic xUnit frameworks. For example, we will have to do the error msg check inside the test instead of in run-tests which will only check the output as it does now, or even better, only OK or FAILED. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
Hi!
It is always possible to write a test for both versions (except
namespace). However it requires a radical change on how we test
things.
Instead of testing the output of given script, we will have to add
logics in a test, something similar to the classic xUnit frameworks.
For example, we will have to do the error msg check inside the test
It is possible, but that means instead of 2 trees of files you'd have
these trees distributed inside test files in a myriad of small if()s:
if(version == X) { test this error message }
elseif(version == Y) { test that error message }
else { test that another error message }
and the same for different pieces of function, etc. Why is it better?
--
Stanislav Malyshev, Zend Software Architect
s...@zend.com http://www.zend.com/
(408)253-8829 MSN: s...@zend.com
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On Fri, Mar 12, 2010 at 6:27 PM, Stanislav Malyshev s...@zend.com wrote:
Hi!
It is always possible to write a test for both versions (except
namespace). However it requires a radical change on how we test
things.
Instead of testing the output of given script, we will have to add
logics in a test, something similar to the classic xUnit frameworks.
For example, we will have to do the error msg check inside the test
It is possible, but that means instead of 2 trees of files you'd have these
trees distributed inside test files in a myriad of small if()s:
if(version == X) { test this error message }
elseif(version == Y) { test that error message }
else { test that another error message }
and the same for different pieces of function, etc. Why is it better?
--
Stanislav Malyshev, Zend Software Architect
s...@zend.com http://www.zend.com/
(408)253-8829 MSN: s...@zend.com
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
There are going to be differences, that's a given and I don't dispute that.
If I'm given the choice of those differences being in one file or in three
different files (5.2, 5.3, 6.0)? I choose one file. Even if that files is
more complex. It's still one file. One commit. One place to look for
problems. And you get the bonus of being able to see the actual differences
of the same functionality in different versions of PHP all in one place.
So yes, I do think that is better.
Eric Lee Stewart
Re: [PHP-DEV] Tests repository
On Sat, Mar 13, 2010 at 12:27 AM, Stanislav Malyshev s...@zend.com wrote:
Hi!
It is always possible to write a test for both versions (except
namespace). However it requires a radical change on how we test
things.
Instead of testing the output of given script, we will have to add
logics in a test, something similar to the classic xUnit frameworks.
For example, we will have to do the error msg check inside the test
It is possible, but that means instead of 2 trees of files you'd have these
trees distributed inside test files in a myriad of small if()s:
if(version == X) { test this error message }
elseif(version == Y) { test that error message }
else { test that another error message }
and the same for different pieces of function, etc. Why is it better?
I'm not saying it is better to write test, but it is better to
maintain one set of tests for all branches and keep them uptodate. But
that's definitivelly more work, and not only for the 1st shot. That's
why I would rather triple check if we really want to go down this way.
Cheers,
--
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On Fri, Mar 12, 2010 at 06:53:49PM -0500, Eric Stewart wrote: If I'm given the choice of those differences being in one file or in three different files (5.2, 5.3, 6.0)? I choose one file. Even if that files is more complex. It's still one file. One commit. One place to look for problems. And you get the bonus of being able to see the actual differences of the same functionality in different versions of PHP all in one place. Agreed. It seems we'll need not only a ==VERSION== section of some sort, but a ==VERSION_OUTPUT== type of thing as well. For example, when things get deprecated. The test suite needs to ensure that the function works normally in 5.2 but throws a deprecated warning in 5.3. Thanks, --Dan -- T H E A N A L Y S I S A N D S O L U T I O N S C O M P A N Y data intensive web and database programming http://www.AnalysisAndSolutions.com/ 4015 7th Ave #4, Brooklyn NY 11232 v: 718-854-0335 f: 718-854-0409 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository (run-tests.php)
On Fri, Mar 12, 2010 at 22:18, Eric Stewart ericleestew...@gmail.com wrote: On Fri, Mar 12, 2010 at 6:37 AM, Jani Taskinen jani.taski...@iki.fi wrote: On 03/12/2010 12:29 PM, Hannes Magnusson wrote: On Fri, Mar 12, 2010 at 11:18, Jani Taskinenjani.taski...@iki.fi wrote: Having tests in multiple branches is PITA. Hasn't anyone considered that the best way would be to move all tests into their own repository (directory..whatever :) in SVN..? Considering they are supposed to be used for testing against regressions and BC breaks, they should always be runnable using any PHP version? Thats actually a fairly good idea. Some tests however are not supposed to work in earlier releases, so we need to either add a new ==SKIP-VERSION== 5.2, 5.1, 5.0 Perhaps something like required min version is better. Any ideas who has been working on the improved test stuff? Or was that just a dream? First of all, I apologize for the subject line change, but this really is a different issue than the test repository itself. It was started by George during last years GSOC. Zoe and Stefan were also working on it after GSOC ended. Both Zoe and Stefan ran out of time and it went dormant for a bit. I contacted Zoe about the status and offered to pick it up, but I also offered to organize TestFest this year. Write up a description and post it again on http://wiki.php.net/gsoc ? -Hannes -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Tests repository
On Fri, Mar 12, 2010 at 9:52 PM, Pierre Joye pierre@gmail.com wrote
On Sat, Mar 13, 2010 at 12:27 AM, Stanislav Malyshev s...@zend.com wrote:
Hi!
It is possible, but that means instead of 2 trees of files you'd have these
trees distributed inside test files in a myriad of small if()s:
if(version == X) { test this error message }
elseif(version == Y) { test that error message }
else { test that another error message }
and the same for different pieces of function, etc. Why is it better?
I'm not saying it is better to write test, but it is better to
maintain one set of tests for all branches and keep them uptodate. But
that's definitivelly more work, and not only for the 1st shot. That's
why I would rather triple check if we really want to go down this way.
One thing to notice is change in output between versions.
During testfest we wrote various tests and tried to make them as
generic as possible to run in 5.2/5.3/6.
It worked in mosts tests but some suttle differences would make it a
pain to write a unified test.
For example: When writing the tests for trunk string became unicode
Whatever path PHP6 does go down, we need to rememebr small changes
like this can come along in other aspects
So just a version check for the test is not all that's needed, we need
something to deal with these differences as well.
--
Rafael Dohms
PHP Evangelist and Community Leader
http://www.rafaeldohms.com.br
http://www.phpsp.org.br
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
