Re: [Distutils] RFC: Standard Declaration of tests in eggs
Phillip J. Eby wrote: At 01:11 PM 1/5/2007 -0500, Jim Fulton wrote: Phillip J. Eby wrote: ... Why can't an entry point invoke a test loader itself? This seems much simpler and more straightforward to me. Because that requires you to write code for something that can adequately be expressed through an existing configuration mechanism. *And* you have to write that code in every project. It's probably a couple of lines. Import a loader and call it. That doesn't seem like a big deal to me. Nice and explicit too. And it's enormously repetitive if you have a lot of projects. Besides, I expect most projects will have a single test suite (function) that could just be named directly. You expect wrongly. :) I can only speak based on my own experience. Projects like Zope and ZODB certainly want some sort of tool to find and assemble tests. Eggs are allowing and encouraging us to move away from such large projects. As we break up Zope into individual packages, I'm finding that the individual packages have single test suites. 'nose' and 'py.test' are popular *precisely because* they do NOT require this. Setuptools changed to emulate them by adding ScanningLoader, even though it originally did the test suite function thing you're proposing. See, I originally wrote all my test suites the way you're proposing, but I changed because this is one of those convention beats configuration situations. It's easier to just conform to a policy and have a tool that applies the policy. Different projects may have different policies, so providing an option for the loader, allows them to follow it. (And it made my life easier, too, as I began doing more small projects instead of a few large ones.) And, if you want your approach to be widely adopted, it would be best to allow projects to follow their own policies without requiring them to add any code, even if it's a one-liner. They'll be adding code one way or another. Either they add it to their test modules, or they add it to setup.py. The advantage of the former is that it provides more flexibility without requiring any more work. It is also more explicit. So, *technically*, there may be no reason to do this, but from a usability, friendliness, compatiblity, marketing, adoption, etc. format I don't think the standard will be successful without allowing this out. People often have heavy psychological investment in existing tools, and it is hundreds of times easier to rationalize adding a line of configuration to setup() than it is to rationalize adding two or three lines of code. One is merely a packaging change, the other is a programming change. We disagree. I don't have any problem with having tools that can be used to find and assemble tests into a test suite. I find the unittest loader framework to be baroque. I'd rather let people use whatever framework they want to do this and to control this from Python. I withdraw the proposal. I wouldn't object to the proposal you want, if you want to write it. I don't have the time or interest to write it myself. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] RFC: Standard Declaration of tests in eggs
Phillip J. Eby wrote: At 10:14 AM 1/22/2007 -0500, Jim Fulton wrote: I'd rather let people use whatever framework they want to do this and to control this from Python. The approach I suggested certainly allows that to happen, I know. That's why I wouldn't object to it. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] RFC: Standard Declaration of tests in eggs
Hello all. I'm the primary author of nose. Michal Kwiatkowski was kind enough to point me to this thread, and I wanted to chime in with a few questions and answers. Apologies for pulling things from various parts of the thread together, I didn't want to send multiple replies all covering the same ground. Nose works fine with setuptools in source packages. It provides nose.collector for use as a test_suite with the test command, and a nosetests command that uses nose's own testrunner and supports loading options from setup.cfg. So for today's use I think it works pretty well. That's the good news. The bad: PJE wrote: (I'm not sure if nose really works with eggs, though; ScanningLoader will discover tests in eggs as long as the source is included, but nose may be strictly file-based for all I know. Changing it probably wouldn't be too difficult, however.) There's no explicit egg support and, unlike ScanningLoader, nose uses naive os and os.path functions to access the filesystem, so I think I'll need to supply a 2nd loader than uses pkg_resources. The loader must support a .loadTestFromName(name, obj) call, where name will be an empty string, and obj will be the object specified by a test entry point. The return value must be a unittest.TestSuite. And it must be possible to call the loader multiple times with different 'obj' values. This is broken in nose 0.9.x -- I have the logic of loadTestsFromName backwards, in that it only uses the module argument to determine the package context of the name to be loaded; if the name is blank and the module is 'foo' it will try to load tests from 'foo.' Not so good. It also currently is a generator that yields tests as they are loaded, rather finding them all and returning a TestSuite -- I don't know whether that will also be a problem. And nose currently uses imp.find_module and imp.load_module to import test modules, since that seemed to be the easiest way to handle the very common case of multiple test modules with the same name in different (non-package) directories. I don't know whether those will work correctly when importing from an egg. I'm hoping to fix all of these loader problems in the 0.10 series. I'll make whatever changes will be needed to support loading from eggs and the final form of this proposal (so long as that doesn't break anything that works now), but as you can see I'm going to need some guidance about what I need to do. JP ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] RFC: Standard Declaration of tests in eggs
At 06:05 PM 1/8/2007 +, John J Lee wrote: On Mon, 8 Jan 2007, David Fraser wrote: [...] So I need to go and code a test enumerator for py.test to discover tests within eggs (which is possible) [...] Or (for new code) use nose, which AIUI is intended to be pretty much py.test implemented within the unittest framework (plus whatever bits and pieces the nose authors wanted to change, it's true). I'm not sure whether nose handles test discovery in eggs, but I believe that, like py.test, it has pluggable enumeration capabilities. What would be nice is if we could get some folks working on py.test and nose to chime in on the proposal, to make sure that it's sound. At the moment, the proposal (or at least my suggested version of it) allows for: * specification of one loader for all tests in an egg * specification of zero or more entry points for locating tests The only way to provide any configuration to the loader is via attributes of the entry points. For example, if the entry point is a module, you could set your options location via attributes of the module. The loader must support a .loadTestFromName(name, obj) call, where name will be an empty string, and obj will be the object specified by a test entry point. The return value must be a unittest.TestSuite. And it must be possible to call the loader multiple times with different 'obj' values. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] RFC: Standard Declaration of tests in eggs
Here is a rough draft proposal for declaring tests in eggs: Introduction Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests. Proposal This proposal aims to be extremely simple. It has 2 parts: 1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites. 2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner. Thoughts? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] RFC: Standard Declaration of tests in eggs
David Fraser wrote: Jim Fulton wrote: Here is a rough draft proposal for declaring tests in eggs: Introduction Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests. Proposal This proposal aims to be extremely simple. It has 2 parts: 1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites. How would this work if for example, you're using an alternative testing framework (like py.test) for your test? I'm not familiar with py.test. I guess it's not based on unittest. Why? Couldn't it at least have a unittest wrapper, like the one I wrote for doctest? Even though I use doctest almost exclusively, I view unittest as a common API that various testing frameworks can and should play with. I certainly think there should be some common API like that and see unittest as the incumbant. It would be nice to be able to bootstrap it :-) What do you mean by that? 2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner. Great, so at least the testing framework could be declared as a dependency My assumption (and my use case) is that you'd run whatever test runner you want and tell it to run tests from some given eggs. So the test runner would already be loaded. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] RFC: Standard Declaration of tests in eggs
At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote: Here is a rough draft proposal for declaring tests in eggs: Introduction Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests. Proposal This proposal aims to be extremely simple. It has 2 parts: 1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites. 2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner. Thoughts? Point #2 is unnecessary, since individual entry points can list extras in square brackets following the module/attribute information. When loading an entry point, these extras automatically get require()'d. Conversely, if the test runner wants to manage the loading process, it can simply inspect the entry point object to determine the names of the extras. Regarding point #1, I'm not sure this is enought to define what's necessary. For example, it should perhaps be stated that the entry point must be in code that will be installed by either the distribution itself, or that is included in the code provided by the entry point's extras. Also, an egg can't provide more than one entry point with the same name, so rather than a 'test_suite' entry point, we probably want an entry point *group*, perhaps something like 'installed.test_suites'. Next, there needs to be reasonable support for dynamic test discovery, such as setuptools' own ScanningLoader. I would suggest that there be an optional entry point for the test loader class to be used to process the other entry points' target objects. The standard unittest protocol for loadTestsFromName() takes two arguments: a name and an object. Passing an empty string for the name, and the object loaded by the test suite entry point, suffices to enable normal behavior for loaders such as ScanningLoader, and I believe 'nose' as well as any other well-behaved unittest extensions. However, I think the spec should try to define what well-behaved means in terms of I/O, result reporting, etc. (Obviously, one requirement is that the test loader must be able to take an empty name string and an object in its loadTestsFromName() method, and return a test suite.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] RFC: Standard Declaration of tests in eggs
At 01:11 PM 1/5/2007 -0500, Jim Fulton wrote: Phillip J. Eby wrote: ... Why can't an entry point invoke a test loader itself? This seems much simpler and more straightforward to me. Because that requires you to write code for something that can adequately be expressed through an existing configuration mechanism. *And* you have to write that code in every project. It's probably a couple of lines. Import a loader and call it. That doesn't seem like a big deal to me. Nice and explicit too. And it's enormously repetitive if you have a lot of projects. Besides, I expect most projects will have a single test suite (function) that could just be named directly. You expect wrongly. :) 'nose' and 'py.test' are popular *precisely because* they do NOT require this. Setuptools changed to emulate them by adding ScanningLoader, even though it originally did the test suite function thing you're proposing. See, I originally wrote all my test suites the way you're proposing, but I changed because this is one of those convention beats configuration situations. It's easier to just conform to a policy and have a tool that applies the policy. Different projects may have different policies, so providing an option for the loader, allows them to follow it. (And it made my life easier, too, as I began doing more small projects instead of a few large ones.) And, if you want your approach to be widely adopted, it would be best to allow projects to follow their own policies without requiring them to add any code, even if it's a one-liner. So, *technically*, there may be no reason to do this, but from a usability, friendliness, compatiblity, marketing, adoption, etc. format I don't think the standard will be successful without allowing this out. People often have heavy psychological investment in existing tools, and it is hundreds of times easier to rationalize adding a line of configuration to setup() than it is to rationalize adding two or three lines of code. One is merely a packaging change, the other is a programming change. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
