Re: RFC: Test::JSON
Chromatic wrote: On Mon, 2005-11-14 at 12:45 -0800, Ovid wrote: Yes, I can see that. I could actually have dropped Test::Differences "eq_or_diff" and just used the "is_deeply" function from Test::More, but when working with large data structures, there's just no comparison between the two. I suppose I could make Test::Differences optional and fall back on is_deeply if they don't have T:D installed. At some point, people install modules for the convenience of not writing that code themselves. I think they'll survive if you require modules for the convenience of not having to write that code yourself, at least if you don't go crazy with it. The question is, what level of deps is "crazy" for something that they don't actually need on their computer permanently but only need for 2 seconds to install something of yours. Adam K
Re: CPANTS: released_while_burning_midnight_oil
A. Pagaltzis wrote: * Ian Langworth <[EMAIL PROTECTED]> [2005-11-14 18:15]: PS. If you feel that sarcasm and satire are not best reflected in email, I cordially suggest that you eat a helicopter. What wine is more appropriate with helicopters, though, white or red? If they're UN Stormtrooper Black Helicopters, it has to be a Pinot Noir. -- David Cantrell
Re: RFC: Test::JSON
-BEGIN PGP SIGNED MESSAGE- Moin, On Monday 14 November 2005 23:22, Adam Kennedy wrote: > Chromatic wrote: > > On Mon, 2005-11-14 at 12:45 -0800, Ovid wrote: > >>Yes, I can see that. I could actually have dropped Test::Differences > >>"eq_or_diff" and just used the "is_deeply" function from Test::More, > >>but when working with large data structures, there's just no > >> comparison between the two. I suppose I could make > >> Test::Differences optional and fall back on is_deeply if they don't > >> have T:D installed. > > > > At some point, people install modules for the convenience of not > > writing that code themselves. I think they'll survive if you require > > modules for the convenience of not having to write that code > > yourself, at least if you don't go crazy with it. > > The question is, what level of deps is "crazy" for something that they > don't actually need on their computer permanently but only need for 2 > seconds to install something of yours. This reminds me of my mini-rant about testing module dependencies. As I said, enjoy re-suing code, but sometimes less is more. Finding the middle ground is hard, but the extremes can be spotted easy. :) Thats als why I find it really good that perl contains so many modules in a standard distribution, your users don't need to install tons of stuff just to use Foo::BAR. Best wishes, Tels - -- Signed on Tue Nov 15 11:13:46 2005 with key 0x93B84C15. Visit my photo gallery at http://bloodgate.com/photos/ PGP key on http://bloodgate.com/tels.asc or per email. "Yeah, baby, yeah!" -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.4 (GNU/Linux) iQEVAwUBQ3m1PncLPEOTuEwVAQEPSAf+LZzx2MUmYC4+4ixX17oQkHRAQyZ/tiG6 XywrcLBJ/nuyxFc0f9sBfIJgiD/pLUadT4Ac20F69FM0I7uUkaTQOuhzqxOAOkVR 7P3EU3i3TjKbSwg35mqmXKvwF1WT3YBJB1YQ8hzUneDnu+O8fJrMnLq+24RRgely MkgX7hpIGmLjfPD2uhpcINi28IvzA/zlPwqydx7WiJTPF5oIdZg2pn3JS5UGlYoG qSgzW+ZeHeo3pLGMftC7pGrEBfJpGQ5sEo4GAAbMQx/sh1yGGoNXuOXAmNL4V/fH Kp0gNYqVMnD93LFSIqjlrRlVopFrDGupjldFQKBBZQP6b0eoadSTVA== =arzN -END PGP SIGNATURE-
Re: Private tests
-BEGIN PGP SIGNED MESSAGE- Moin, On Monday 14 November 2005 18:21, Chris Dolan wrote: > Hello all, > > I've just published an article about public vs. private regression > tests. I've defined private tests as t/*.t files that are for the > author only and don't go in MANIFEST. Naturally, those don't get as > much publicity as tests included in CPAN distributions. > > In the article I advocate that some tests should be private. In > particular, >1) those that test non-critical aspects of a module (like > documentation and coding style) >2) those that are too expensive to run often >3) those that require special software or customization > In my conclusion I describe a possible system where authors publish > the results of private tests with their distributions as a trust- > based kwalitee system. That is, authors assert kwalitee rather than > be judged for it. > >http://www.chrisdolan.net/talk/index.php/2005/11/14/private- > regression-tests/ > > Both positive and negative feedback is very welcome! Private tests will only be run by the author, meaning they will be only run on a very small subset of all systems the modules can be used on. This limits their usefullness quite a bit. Case ein point: I can test my modules on linux, 32 bit, unthreaded, under unicode, and under perl 5.8.x. Thats about it, everything else gets really really complicated for me to set up and maintain/test. So, no win32, no mac, now solaris, no irix, no perl 5.6.x, no iso-something, no EBDIC (or however it is spelled), no threading, no 64 bit, no SMP system. As for 1), these things should matter (the "broken window analogy") and you would be surprised to know how these tests can pass on your system, and still fail on other systesm (forget to include the .pod file in MANIFEST is the most obvious one). As for 2), random testing should be employed (Math::BigInt does this, it runs 256 or so tests with random number patterns (and thus known results like "2 * A - A == A"). The tests are quite fast, but they cover only a small subset of potential values. However, since each system and user runs a new, different random set, you end up with a really huge testing number being run. (Yes, this has found some bugs) And for 3), this might be the only point I can think that private tests are usefull (I have a private testset for Graph::Easy that I use from time to time, it is not public mainly because it fails/hangs/takes forever and is work-in-progress). However, I have to actually read your article to find out what your proposal solves (compared to me just running thetest once in a while :) Hope that was usefull :) Best wishes, Tels - -- Signed on Tue Nov 15 11:04:21 2005 with key 0x93B84C15. Visit my photo gallery at http://bloodgate.com/photos/ PGP key on http://bloodgate.com/tels.asc or per email. "Now, _you_ behave!" -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.4 (GNU/Linux) iQEVAwUBQ3m0pHcLPEOTuEwVAQF6lQf8CtubDMQjLdCBcEGNczxZj2Y1kTVhOU7Z bvweeJe4RWFfmd2JMw7dwiu3Sjb57hNlVkl4LwN+7vx3tm3DsnhRUoMHvkDtCddC 8bfxpLcOi8WMlySAud+unKnpZVwlwn2rZ/enu2Dd01QKOgOQkBr1HWTJUguPv4No eWT3UiEZhV4hU764gtF7a8raHjbvxpTJcNk22KHnRmyTKX+SugCyI0qkmIQrFntl cQWXyA9GV7V+5bK5/Sp2eWv2MXX3fhNDxtZywkmqun+6/YhPgSDJQp3FcKThZFYy WxPXsrXVIXFJbbtkSs+PGe18VdXqEFCHOI29H1+9gyiC3FW3N6AARw== =GA3y -END PGP SIGNATURE-
Re: CPANTS: released_while_burning_midnight_oil
David Cantrell wrote: A. Pagaltzis wrote: * Ian Langworth <[EMAIL PROTECTED]> [2005-11-14 18:15]: PS. If you feel that sarcasm and satire are not best reflected in email, I cordially suggest that you eat a helicopter. What wine is more appropriate with helicopters, though, white or red? If they're UN Stormtrooper Black Helicopters, it has to be a Pinot Noir. Of course, if the module is under the Text:: hierarchy, then a glass of Chardonnay is recommended. David -- "It's overkill of course, but you can never have too much overkill."
Re: RFC: Test::JSON
On Tue, 2005-11-15 at 09:22 +1100, Adam Kennedy wrote: > The question is, what level of deps is "crazy" for something that they > don't actually need on their computer permanently but only need for 2 > seconds to install something of yours. I sleep pretty well at night refusing to support people who don't run the tests that ship with my modules. There's my crazy. If you're (not you Adam, but the generic you) not willing to give up maybe a couple of megabytes of disk space for all of the testing modules on the CPAN altogether, I feel little obligation to spend my free time to help you debug a problem the hard way. -- c
Re: Private tests
Tels, I believe you have misunderstood my intentions. I was not advocating that any algorithmic tests be non-public. The only tests that should be private are ones that satisfy one or more of the following restrictions: 1. require special additional software that’s difficult or expensive to acquire, 2. require special configuration to run properly, 3. don’t affect the quality of the final software, or 4. take too long to run. That is, tests like spell-checking, pod-checking, Perl::Critic and ones that take tens of minutes to run. Naturally, any test whose results may vary from machine to machine should be public wherever feasible. After reading some of the insightful comments posted on my blog, I've been convinced that the private tests should be included in the CPAN distribution, but disabled in some way (perhaps via a file extension other than .t?). That way, resourceful or interested users can run the tests but average users don't need to. The example I included at the end of the article stating a failed test under Windows was only tangentially related and perhaps was a distraction. I was saying that perhaps it would be useful if authors could assert things about their personal testing experience that would be machine readable and useful to others. In that example I was suggesting that the author could announce that the hypothetical module was known to fail under Windows and, as a TODO test, was an implicit request for help from other developers. Chris On Nov 15, 2005, at 4:12 AM, Tels wrote: -BEGIN PGP SIGNED MESSAGE- Moin, On Monday 14 November 2005 18:21, Chris Dolan wrote: Hello all, I've just published an article about public vs. private regression tests. I've defined private tests as t/*.t files that are for the author only and don't go in MANIFEST. Naturally, those don't get as much publicity as tests included in CPAN distributions. In the article I advocate that some tests should be private. In particular, 1) those that test non-critical aspects of a module (like documentation and coding style) 2) those that are too expensive to run often 3) those that require special software or customization In my conclusion I describe a possible system where authors publish the results of private tests with their distributions as a trust- based kwalitee system. That is, authors assert kwalitee rather than be judged for it. http://www.chrisdolan.net/talk/index.php/2005/11/14/private- regression-tests/ Both positive and negative feedback is very welcome! Private tests will only be run by the author, meaning they will be only run on a very small subset of all systems the modules can be used on. This limits their usefullness quite a bit. Case ein point: I can test my modules on linux, 32 bit, unthreaded, under unicode, and under perl 5.8.x. Thats about it, everything else gets really really complicated for me to set up and maintain/test. So, no win32, no mac, now solaris, no irix, no perl 5.6.x, no iso-something, no EBDIC (or however it is spelled), no threading, no 64 bit, no SMP system. As for 1), these things should matter (the "broken window analogy") and you would be surprised to know how these tests can pass on your system, and still fail on other systesm (forget to include the .pod file in MANIFEST is the most obvious one). As for 2), random testing should be employed (Math::BigInt does this, it runs 256 or so tests with random number patterns (and thus known results like "2 * A - A == A"). The tests are quite fast, but they cover only a small subset of potential values. However, since each system and user runs a new, different random set, you end up with a really huge testing number being run. (Yes, this has found some bugs) And for 3), this might be the only point I can think that private tests are usefull (I have a private testset for Graph::Easy that I use from time to time, it is not public mainly because it fails/hangs/takes forever and is work-in-progress). However, I have to actually read your article to find out what your proposal solves (compared to me just running thetest once in a while :) Hope that was usefull :) Best wishes, Tels - -- Signed on Tue Nov 15 11:04:21 2005 with key 0x93B84C15. Visit my photo gallery at http://bloodgate.com/photos/ PGP key on http://bloodgate.com/tels.asc or per email. "Now, _you_ behave!" -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.4 (GNU/Linux) iQEVAwUBQ3m0pHcLPEOTuEwVAQF6lQf8CtubDMQjLdCBcEGNczxZj2Y1kTVhOU7Z bvweeJe4RWFfmd2JMw7dwiu3Sjb57hNlVkl4LwN+7vx3tm3DsnhRUoMHvkDtCddC 8bfxpLcOi8WMlySAud+unKnpZVwlwn2rZ/enu2Dd01QKOgOQkBr1HWTJUguPv4No eWT3UiEZhV4hU764gtF7a8raHjbvxpTJcNk22KHnRmyTKX+SugCyI0qkmIQrFntl cQWXyA9GV7V+5bK5/Sp2eWv2MXX3fhNDxtZywkmqun+6/YhPgSDJQp3FcKThZFYy WxPXsrXVIXFJbbtkSs+PGe18VdXqEFCHOI29H1+9gyiC3FW3N6AARw== =GA3y -END PGP SIGNATURE- -- Chris Dolan, Software Developer, Clotho Advanced
Re: Private tests
On Tue, 2005-11-15 at 15:23 -0600, Chris Dolan wrote: > After reading some of the insightful comments posted on my blog, I've > been convinced that the private tests should be included in the CPAN > distribution, but disabled in some way (perhaps via a file extension > other than .t?). That way, resourceful or interested users can run > the tests but average users don't need to. I posted a small Module::Build subclass that shows one way to do this to Perl Monks: http://perlmonks.org/?node_id=508160 Perhaps a better approach is to store these tests in a subdirectory of t/. -- c
Re: Private tests
Le mardi 15 novembre 2005 à 15:23, Chris Dolan écrivait: > > After reading some of the insightful comments posted on my blog, I've > been convinced that the private tests should be included in the CPAN > distribution, but disabled in some way (perhaps via a file extension > other than .t?). That way, resourceful or interested users can run > the tests but average users don't need to. What about a special environment variable, like RUN_PRIVATE_TESTS? -- Philippe "BooK" Bruhat When you do not think for yourself, no one thinks for you... (Moral from Groo The Wanderer #61 (Epic))
Re: Private tests
Philippe 'BooK' Bruhat wrote: Le mardi 15 novembre 2005 à 15:23, Chris Dolan écrivait: After reading some of the insightful comments posted on my blog, I've been convinced that the private tests should be included in the CPAN distribution, but disabled in some way (perhaps via a file extension other than .t?). That way, resourceful or interested users can run the tests but average users don't need to. What about a special environment variable, like RUN_PRIVATE_TESTS? I've been working on a concept of taggable tests on some of my larger commercial stuff, integrating with the Test::More skip() function, and some form of environment variables does indeed seem the best way to do such a thing. For example, with applications that talk to the database, a lot of the time I want to be able to just test all the code that doesn't need a live production database, or need a database with the production schema, or need a schema that it is allowed to modify, and so on and so forth. So I have a bunch of environment variables like TEST_TAG_DB_RO TEST_TAG_DB_RW, TEST_TAG_DB_SCHEMA and so on and so forth. It seems to be working OK, I'm sure there's be a Test::Tags module in a year or something. Adam K
Re: Private tests
On Nov 15, 2005, at 3:38 PM, chromatic wrote: I posted a small Module::Build subclass that shows one way to do this to Perl Monks: http://perlmonks.org/?node_id=508160 Yeah, I saw that one. Perhaps a better approach is to store these tests in a subdirectory of t/. Beware that M::B has a recursive mode for finding tests. It's set by the author, so you should be safe in this case, but it's a point worth remembering. *light bulb* And in fact, that could be the run-time trigger. Hmmm Chris -- Chris Dolan, Software Developer, Clotho Advanced Media Inc. 608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703 Clotho Advanced Media, Inc. - Creators of MediaLandscape Software (http://www.media-landscape.com/) and partners in the revolutionary Croquet project (http://www.opencroquet.org/)
Re: Private tests
Adam Kennedy wrote: What about a special environment variable, like RUN_PRIVATE_TESTS? I've been working on a concept of taggable tests on some of my larger commercial stuff, integrating with the Test::More skip() function, and some form of environment variables does indeed seem the best way to do such a thing. Isn't Test::Manifest sort of geared to do this kind of thing? (Or could be extended to do so?)
Re: Private tests
David Golden wrote:
Adam Kennedy wrote:
What about a special environment variable, like RUN_PRIVATE_TESTS?
I've been working on a concept of taggable tests on some of my larger
commercial stuff, integrating with the Test::More skip() function, and
some form of environment variables does indeed seem the best way to do
such a thing.
Isn't Test::Manifest sort of geared to do this kind of thing? (Or could
be extended to do so?)
No. The manifest solution seems to work very well for specifying order,
but I wouldn't want to use it for finding testing subsets.
For starters, the tags need to be orthogonal. So if I use the tags
NEEDS_DNS and NEEDS_DATABASE and NEEDS_APACHE then you need to be able
to enable or disable the tags/flags individually.
Also, doing it at the manifest level means you can only deal with things
at a per-file level, which what is much more useful is to do it at a
sub-test level.
If you are testing something, you can have both it's normal tests and
just wrap the tests that need the database or DNS or whatever in a skip.
Here's an example of how the output ends up. Excuse the failures, I'm
working on something :)
[EMAIL PROTECTED]:~/GeoSol/trunk/cgi-bin/test$ wprove -rl
Password:
00_legacy/20050200...ok
00_legacy/20050300...ok
00_legacy/20050900...ok
./01_compile.ok
./02_integrity...ok
./03_database_integrity..ok
456/689 skipped: Skipping: TEST_TAG_DB_RO is disabled
./04_library_apisok
./06_code_symbolsNOK 421
# Failed test 'GeoSol::Interface::Role::File->can('DataToView')'
# in ./06_code_symbols.t at line 176.
# GeoSol::Interface::Role::File->can('DataToView') failed
# Looks like you failed 1 test of 798.
./06_code_symbolsdubious
Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 421
Failed 1/798 tests, 99.87% okay
./07_templates...ok
./08_images..ok
auto/geosol__entity..ok
auto/geosol__viewok
auto/geosol_base.ok
auto/geosol_cachemanager.ok
auto/geosol_config...ok
auto/geosol_data__simple.ok
12/14 skipped: Skipping: TEST_TAG_DB_RO is disabled
auto/geosol_data__string.ok
22/25 skipped: Skipping: TEST_TAG_DB_RO is disabled
auto/geosol_data_boolean.ok
auto/geosol_data_datetimeok
auto/geosol_data_integer.ok
auto/geosol_data_listok
auto/geosol_data_longstring..ok
auto/geosol_data_shortstring.ok
auto/geosol_db...ok
13/16 skipped: Skipping: TEST_TAG_DB_RO is disabled
auto/geosol_db_configok
auto/geosol_db_schemaok
9/12 skipped: Skipping: TEST_TAG_DB_RO is disabled
auto/geosol_emailok
auto/geosol_entity_application...ok
auto/geosol_entity_message...ok
auto/geosol_entity_reportok
auto/geosol_entity_resumeok
auto/geosol_entity_session...ok
auto/geosol_entity_user..ok
auto/geosol_fieldok
auto/geosol_html.ok
etc etc...
Adam K
Re: Private tests
On Tue, 2005-11-15 at 22:33 -0600, Chris Dolan wrote: > Beware that M::B has a recursive mode for finding tests. It's set by > the author, so you should be safe in this case, but it's a point > worth remembering. I haven't looked at the code again just now, but wouldn't overriding find_test_files() and filtering out specific subdirectories based on active environment variables do the trick? -- c
