Re: [Numpy-discussion] Doctest items
2008/7/2 Ryan May [EMAIL PROTECTED]: To be clear, these aren't tests of the numpy code. The tests would be to make sure the examples still run. Right. I just don't think effort should be put into making examples using matplotlib run as doctests. If the behavior is important, numpy should have a standalone test for it. Still, I'd like to see consistent markup for code snippets. Currently, those are indicated by ''. First and foremost, docstrings (in NumPy, with its test suite) are not tests, but serve as documentation. Running them is simply a way of verifying that we did not make mistakes in the examples. Matplotlib docstrings: Currently, the plotting docstrings have not been written as valid doctests, because I didn't want them riddled with: [matplotlib.lines.Line2D object at 0x33f60d0] How about a slight modification to Fernando's idea: a dummy function that a) Does nothing if matplotlib is not installed b) Otherwise passes through calls to matplotlib, after setting the backend to /dev/null. Any results (lines, figures, etc.) are suppressed. This limits us to very simple graphical examples, but I'm ok with that. If you really need to get hold of the figure, for example, you could always use `figure(); f = gcf()` instead of `f = figure()`. Pasting code: I recommend using the %cpaste magic in IPython. It has been updated to support line-spanning examples. Alan: The latest versions of the docstrings are on the wiki, and there are a number of examples using matplotlib. These are always listed last in the examples section, so that they can be easily ignored by terminal-only users. We'll definitely merge the wiki docstrings to into the source tree before the conference. I've been at a workshop this past week, but on Monday I'll proceed full-steam again. Regards Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Wed, Jul 2, 2008 at 5:47 PM, Stéfan van der Walt [EMAIL PROTECTED] wrote: How about a slight modification to Fernando's idea: a dummy function that a) Does nothing if matplotlib is not installed b) Otherwise passes through calls to matplotlib, after setting the backend to /dev/null. Any results (lines, figures, etc.) are suppressed. That's doable; it turned out to be easy to explicitly specify the execution context of the doctests, so we can add in any number of dummy objects/functions. In my working NumPy, the only globally available items are __builtins__ and np; I was going to update all the doctests that won't run under that context before changing the tester behavior, though. At the moment I don't know anything about configuring matplotlib's backend and all that, but I'm sure it shouldn't be hard to figure out. If somebody was to write a first draft of these dummy objects before I got around to it I wouldn't complain about it, though. ;) (By the way, I'm not assuming that there's a consensus about the matplotlib dummy objects, just saying it turns out to be easy to add if we want to.) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Doctest items
Just a few questions/comments about doctests: 1. Should all doctests be written such that you could start Python, do an import numpy as np, and then type in the examples verbatim? There are a number that currently wouldn't work that way (they depend on the function under test being in the local namespace, for example). 2. In regard to the auto-ignore of plt., etc. in commands: using the existing ellipsis feature of doctest should cover a significant portion of the cases that the auto-ignore was suggested to solve, and it's a very minor change to enable it (whereas the auto-ignore is more involved). If nobody objects, I will enable ellipsis for all doctests (which doesn't cause any obvious problems in existing NumPy tests), and use it to clean up existing doctests where appropriate. If the auto-ignore capability is still needed after that, I'll work on it. 3. When the test suite is run with doctests enabled, some unit tests fail that normally wouldn't, and some doctests fail that shouldn't. There's probably some state that needs to be reset (or otherwise managed) between doctest runs; I'll look into that and provide a fix as soon as possible. I just figured I should mention it in case it causes somebody problems before it gets fixed. Thanks, Alan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 11:56 AM, Alan McIntyre [EMAIL PROTECTED] wrote: Just a few questions/comments about doctests: 1. Should all doctests be written such that you could start Python, do an import numpy as np, and then type in the examples verbatim? There are a number that currently wouldn't work that way (they depend on the function under test being in the local namespace, for example). +1 Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 12:56, Alan McIntyre [EMAIL PROTECTED] wrote: Just a few questions/comments about doctests: 1. Should all doctests be written such that you could start Python, do an import numpy as np, and then type in the examples verbatim? There are a number that currently wouldn't work that way (they depend on the function under test being in the local namespace, for example). 2. In regard to the auto-ignore of plt., etc. in commands: using the existing ellipsis feature of doctest should cover a significant portion of the cases that the auto-ignore was suggested to solve, and it's a very minor change to enable it (whereas the auto-ignore is more involved). If nobody objects, I will enable ellipsis for all doctests (which doesn't cause any obvious problems in existing NumPy tests), and use it to clean up existing doctests where appropriate. +1 If the auto-ignore capability is still needed after that, I'll work on it. It seems to me that the ellipsis mechanism just allows the output to differ. However, matplotlib would still be required because plt.plot() would still be executed. matplotlib should not be a requirement for running the tests. 3. When the test suite is run with doctests enabled, some unit tests fail that normally wouldn't, and some doctests fail that shouldn't. There's probably some state that needs to be reset (or otherwise managed) between doctest runs; I'll look into that and provide a fix as soon as possible. I just figured I should mention it in case it causes somebody problems before it gets fixed. Thank you for the notice. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 2:45 PM, Robert Kern [EMAIL PROTECTED] wrote: If the auto-ignore capability is still needed after that, I'll work on it. It seems to me that the ellipsis mechanism just allows the output to differ. However, matplotlib would still be required because plt.plot() would still be executed. matplotlib should not be a requirement for running the tests. Oops, I misunderstood, then: I thought the intent was to execute the statement but not compare the output (because they returned objects that had their address in the repr). I didn't look before to see how many times this feature would be needed (yeah, should have done that before complaining about using #doctest: +SKIP), but now that I look, I only see one batch of plt. commands, in numpy.lib.function_base.bartlett. In view of that, does it make more sense to use the SKIP directive for the ten plt. lines in that one example? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 14:14, Alan McIntyre [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 2:45 PM, Robert Kern [EMAIL PROTECTED] wrote: If the auto-ignore capability is still needed after that, I'll work on it. It seems to me that the ellipsis mechanism just allows the output to differ. However, matplotlib would still be required because plt.plot() would still be executed. matplotlib should not be a requirement for running the tests. Oops, I misunderstood, then: I thought the intent was to execute the statement but not compare the output (because they returned objects that had their address in the repr). I didn't look before to see how many times this feature would be needed (yeah, should have done that before complaining about using #doctest: +SKIP), but now that I look, I only see one batch of plt. commands, in numpy.lib.function_base.bartlett. In view of that, does it make more sense to use the SKIP directive for the ten plt. lines in that one example? Can it work on an entire section? If not, can we do something that works on a whole section? Everything after Plot the window and its frequency response: is not required for testing. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 3:20 PM, Robert Kern [EMAIL PROTECTED] wrote: Can it work on an entire section? If not, can we do something that works on a whole section? Everything after Plot the window and its frequency response: is not required for testing. It's on a per-line basis at the moment, so each lines needs a #doctest: +SKIP. Changing a directive to apply to multiple lines probably isn't trivial (I haven't really looked into doing that, though). We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 1:30 PM, Alan McIntyre [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 3:20 PM, Robert Kern [EMAIL PROTECTED] wrote: Can it work on an entire section? If not, can we do something that works on a whole section? Everything after Plot the window and its frequency response: is not required for testing. It's on a per-line basis at the moment, so each lines needs a #doctest: +SKIP. Changing a directive to apply to multiple lines probably isn't trivial (I haven't really looked into doing that, though). We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). Would it serve to overload plot with a function that does zippo? Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 3:39 PM, Charles R Harris [EMAIL PROTECTED] wrote: We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). Would it serve to overload plot with a function that does zippo? Probably not in this case; there's an explicit matplotlib import, and then a bunch of method calls on a matplotlib object: from matplotlib import pyplot as plt window = np.bartlett(51) plt.plot(window) plt.title(Bartlett window) and so on. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 3:33 PM, Robert Kern [EMAIL PROTECTED] wrote: That's not a bad idea. Coordinate with Stéfan about the details (if any are left to be decided). Ok, will do. I'll also update all the test documentation I can find so that documenters have a chance of being aware of the doctest assumptions/requirements/capabilities. So, unless anyone else has objections, I'll: 1. Enabled ellipsis for all doctests 2. Update all doctests so that they only assume import numpy as np. I'll also check into restricting the doctest execution environment so that tests that make other assumptions should fail (as part of figuring out the test state pollution problem). ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 14:39, Charles R Harris [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 1:30 PM, Alan McIntyre [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 3:20 PM, Robert Kern [EMAIL PROTECTED] wrote: Can it work on an entire section? If not, can we do something that works on a whole section? Everything after Plot the window and its frequency response: is not required for testing. It's on a per-line basis at the moment, so each lines needs a #doctest: +SKIP. Changing a directive to apply to multiple lines probably isn't trivial (I haven't really looked into doing that, though). We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). Would it serve to overload plot with a function that does zippo? If it's not going to test anything, I would prefer that it not be part of the tests. Admittedly, that's just my sense of aesthetics, not a technical objection. A technical objection would be that some of the matplotlib functions actually do return something, and we would still have to uglify the examples with the ellipsis stuff. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
Tue, 01 Jul 2008 15:30:47 -0400, Alan McIntyre wrote: On Tue, Jul 1, 2008 at 3:20 PM, Robert Kern [EMAIL PROTECTED] wrote: Can it work on an entire section? If not, can we do something that works on a whole section? Everything after Plot the window and its frequency response: is not required for testing. It's on a per-line basis at the moment, so each lines needs a #doctest: +SKIP. Changing a directive to apply to multiple lines probably isn't trivial (I haven't really looked into doing that, though). I think this can be done without too many problems: Looking at the doctest.py source code, the easiest way to change the way how docstrings are parsed into doctests is to subclass `DoctestParser` and override its `parse` method. The `testmod` and `testfile` functions don't take a parser argument, but they appear to be only thin wrappers for instantiating `DocTestFinder` (which does take a parser argument) and then calling a method in `DocTestRunner`. Now, things appear to work a bit differently in nose. There, DocTestParser is instantiated by Doctest.loadTestsFromFile in the doctests.py plugin. I don't see an easy way to override this, except for monkeypatching doctest.DocTestParser or the whole nose plugin with our stuff. All in all, I'd estimate this to be ~100 lines, put in a suitable location. But it's a custom tweak to doctest, so it might break at some point in the future, and I don't love the monkeypatching here... We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 1:41 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: But it's a custom tweak to doctest, so it might break at some point in the future, and I don't love the monkeypatching here... Welcome to the joys of extending doctest/unittest. They hardcoded so much stuff in there that the only way to reuse that code is by copy/paste/monkeypatch. It's absolutely atrocious. We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. How about a much simpler approach? Just pre-populate the globals dict where doctest executes with an object called 'plt' that basically does def noop(*a,**k): pass class dummy(): def __getattr__(self,k): return noop plt = dummy() This would ensure that all calls to plt.anything() silently succeed in the doctests. Granted, we're not testing matplotlib, but it has the benefit of simplicity and of letting us keep consistent formatting, and examples that *users* can still paste into their sessions where plt refers to the real matplotlib. Just an idea... f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
Robert Kern wrote: On Tue, Jul 1, 2008 at 17:50, Fernando Perez [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 1:41 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: But it's a custom tweak to doctest, so it might break at some point in the future, and I don't love the monkeypatching here... Welcome to the joys of extending doctest/unittest. They hardcoded so much stuff in there that the only way to reuse that code is by copy/paste/monkeypatch. It's absolutely atrocious. We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. How about a much simpler approach? Just pre-populate the globals dict where doctest executes with an object called 'plt' that basically does def noop(*a,**k): pass class dummy(): def __getattr__(self,k): return noop plt = dummy() This would ensure that all calls to plt.anything() silently succeed in the doctests. Granted, we're not testing matplotlib, but it has the benefit of simplicity and of letting us keep consistent formatting, and examples that *users* can still paste into their sessions where plt refers to the real matplotlib. It's actually easier for users to paste the non-doctestable examples since they don't have the markers and any stdout the examples produce as a byproduct. I'm with Robert here. It's definitely easier as an example without the . I also don't see the utility of being able to have the matplotlib code as tests of anything. We're not testing matplotlib here and any behavior that matplotlib relies on (and hence tests) should be captured in a test for that behavior separate from matplotlib code. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 4:41 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: All in all, I'd estimate this to be ~100 lines, put in a suitable location. If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. Well, for the moment, it's 100 lines of new code that's needed in order to *avoid* running 10 lines of doctest code in one function's docstring. Maybe down the road there will be more examples that need a bulk don't execute me mechanism, and if we do I'll be glad to work on it, but for right now I need to spend more time increasing coverage. I also agree with Ryan that matplotlib is where the tests for this particular use case of the object returned by bartlett should be. So at the moment I'm inclined to just remove the and come back to it later. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 19:19, Ryan May [EMAIL PROTECTED] wrote: Robert Kern wrote: On Tue, Jul 1, 2008 at 17:50, Fernando Perez [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 1:41 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: But it's a custom tweak to doctest, so it might break at some point in the future, and I don't love the monkeypatching here... Welcome to the joys of extending doctest/unittest. They hardcoded so much stuff in there that the only way to reuse that code is by copy/paste/monkeypatch. It's absolutely atrocious. We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. How about a much simpler approach? Just pre-populate the globals dict where doctest executes with an object called 'plt' that basically does def noop(*a,**k): pass class dummy(): def __getattr__(self,k): return noop plt = dummy() This would ensure that all calls to plt.anything() silently succeed in the doctests. Granted, we're not testing matplotlib, but it has the benefit of simplicity and of letting us keep consistent formatting, and examples that *users* can still paste into their sessions where plt refers to the real matplotlib. It's actually easier for users to paste the non-doctestable examples since they don't have the markers and any stdout the examples produce as a byproduct. I'm with Robert here. It's definitely easier as an example without the . I also don't see the utility of being able to have the matplotlib code as tests of anything. We're not testing matplotlib here and any behavior that matplotlib relies on (and hence tests) should be captured in a test for that behavior separate from matplotlib code. To be clear, these aren't tests of the numpy code. The tests would be to make sure the examples still run. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
Robert Kern wrote: On Tue, Jul 1, 2008 at 19:19, Ryan May [EMAIL PROTECTED] wrote: Robert Kern wrote: On Tue, Jul 1, 2008 at 17:50, Fernando Perez [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 1:41 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: But it's a custom tweak to doctest, so it might break at some point in the future, and I don't love the monkeypatching here... Welcome to the joys of extending doctest/unittest. They hardcoded so much stuff in there that the only way to reuse that code is by copy/paste/monkeypatch. It's absolutely atrocious. We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. How about a much simpler approach? Just pre-populate the globals dict where doctest executes with an object called 'plt' that basically does def noop(*a,**k): pass class dummy(): def __getattr__(self,k): return noop plt = dummy() This would ensure that all calls to plt.anything() silently succeed in the doctests. Granted, we're not testing matplotlib, but it has the benefit of simplicity and of letting us keep consistent formatting, and examples that *users* can still paste into their sessions where plt refers to the real matplotlib. It's actually easier for users to paste the non-doctestable examples since they don't have the markers and any stdout the examples produce as a byproduct. I'm with Robert here. It's definitely easier as an example without the . I also don't see the utility of being able to have the matplotlib code as tests of anything. We're not testing matplotlib here and any behavior that matplotlib relies on (and hence tests) should be captured in a test for that behavior separate from matplotlib code. To be clear, these aren't tests of the numpy code. The tests would be to make sure the examples still run. Right. I just don't think effort should be put into making examples using matplotlib run as doctests. If the behavior is important, numpy should have a standalone test for it. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
