Re: [Numpy-discussion] import numpy is slow
Would it not be possible to import just the necessary module of numpy to meet the necessary functionality of your application. i.e. import numpy.core or whatever you're using you could even do: import numpy.core as numpy I think, to simplify your code, I'm no expert though. Hanni 2008/7/1 Andrew Dalke [EMAIL PROTECTED]: On Jul 1, 2008, at 2:22 AM, Robert Kern wrote: Your use case isn't so typical and so suffers on the import time end of the balance. I'm working on my presentation for EuroSciPy. Isn't so typical seems to be a good summary of my first slide. :) Any chance of cutting down on the number, in order to improve startup costs? Not at this point in time, no. That would break too much code. Understood. Thanks for the response, Andrew [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
Hi, IIRC, il you do import numpy.core as numpy, it starts by importing numpy, so it will be even slower. Matthieu 2008/7/1 Hanni Ali [EMAIL PROTECTED]: Would it not be possible to import just the necessary module of numpy to meet the necessary functionality of your application. i.e. import numpy.core or whatever you're using you could even do: import numpy.core as numpy I think, to simplify your code, I'm no expert though. Hanni 2008/7/1 Andrew Dalke [EMAIL PROTECTED]: On Jul 1, 2008, at 2:22 AM, Robert Kern wrote: Your use case isn't so typical and so suffers on the import time end of the balance. I'm working on my presentation for EuroSciPy. Isn't so typical seems to be a good summary of my first slide. :) Any chance of cutting down on the number, in order to improve startup costs? Not at this point in time, no. That would break too much code. Understood. Thanks for the response, Andrew [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
You are correct, it appears to take slightly longer to import numpy.core and longer again to import numpy.core as numpy I should obviously check first in future. Hanni 2008/7/1 Matthieu Brucher [EMAIL PROTECTED]: Hi, IIRC, il you do import numpy.core as numpy, it starts by importing numpy, so it will be even slower. Matthieu 2008/7/1 Hanni Ali [EMAIL PROTECTED]: Would it not be possible to import just the necessary module of numpy to meet the necessary functionality of your application. i.e. import numpy.core or whatever you're using you could even do: import numpy.core as numpy I think, to simplify your code, I'm no expert though. Hanni 2008/7/1 Andrew Dalke [EMAIL PROTECTED]: On Jul 1, 2008, at 2:22 AM, Robert Kern wrote: Your use case isn't so typical and so suffers on the import time end of the balance. I'm working on my presentation for EuroSciPy. Isn't so typical seems to be a good summary of my first slide. :) Any chance of cutting down on the number, in order to improve startup costs? Not at this point in time, no. That would break too much code. Understood. Thanks for the response, Andrew [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] import numpy is slow
2008/7/1 Hanni Ali [EMAIL PROTECTED]: Would it not be possible to import just the necessary module of numpy to meet the necessary functionality of your application. Matthieu Brucher responded: IIRC, il you do import numpy.core as numpy, it starts by importing numpy, so it will be even slower. which you can see if you start python with the -v option to display imports. import numpy.core import numpy # directory /Library/Frameworks/Python.framework/ Versions/2.5/lib/python2.5/site-packages/numpy # /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ site-packages/numpy/__init__.pyc matches /Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/ __init__.py import numpy # precompiled from /Library/Frameworks/Python.framework/ Versions/2.5/lib/python2.5/site-packages/numpy/__init__.pyc # /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ site-packages/numpy/__config__.pyc matches /Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/ __config__.py import numpy.__config__ # precompiled from /Library/Frameworks/ Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/ __config__.pyc ... and many more Andrew [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: SfePy 00.46.02
I am pleased announce the release of SfePy 00.46.02. SfePy is a finite element analysis software in Python, based primarily on Numpy and SciPy. Mailing lists, issue tracking, mercurial repository: http://sfepy.org Home page: http://sfepy.kme.zcu.cz Major improvements: - alternative short syntax for specifying essential boundary conditions, variables and regions - manufactured solutions tests: - SymPy support - site configuration now via script/config.py + site_cfg.py - new solvers - new terms For more information on this release, see http://sfepy.googlecode.com/svn/web/releases/004602_RELEASE_NOTES.txt If you happen to come to Leipzig for EuroSciPy 2008, see you there! Best regards, Robert Cimrman SfePy developers ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] More pending test framework changes (please give feedback)
On Mon, Jun 30, 2008 at 1:54 PM, Alan McIntyre [EMAIL PROTECTED] wrote: 1. All doctests in NumPy will have the numpy module available in their execution context as np. 2. Turn on the normalized whitespace option for all doctests. Having a doctest fail just because there's a space after your result seems like an unnecessary hassle for documenters. 3. Output will be ignored for each doctest expected output line that contains #random. I figured this can serve both as an ignore flag and indication to the reader that the listed output may differ from what they see if they execute the associated command. So you would be able to do: random.random() 0.1234567890 #random: output may differ on your system And have the example executed but not cause a failure. You could also use this to ignore the SomeObject at 0x1234ABCD output from plot methods as well. Since I didn't see any objections, these changes are now committed. I'll be updating some doctests to take advantage of them later today. Alan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] More pending test framework changes (please give feedback)
On Tue, Jul 1, 2008 at 8:26 AM, Alan McIntyre [EMAIL PROTECTED] wrote: Since I didn't see any objections, these changes are now committed. I'll be updating some doctests to take advantage of them later today. Excellent. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Doctest items
Just a few questions/comments about doctests: 1. Should all doctests be written such that you could start Python, do an import numpy as np, and then type in the examples verbatim? There are a number that currently wouldn't work that way (they depend on the function under test being in the local namespace, for example). 2. In regard to the auto-ignore of plt., etc. in commands: using the existing ellipsis feature of doctest should cover a significant portion of the cases that the auto-ignore was suggested to solve, and it's a very minor change to enable it (whereas the auto-ignore is more involved). If nobody objects, I will enable ellipsis for all doctests (which doesn't cause any obvious problems in existing NumPy tests), and use it to clean up existing doctests where appropriate. If the auto-ignore capability is still needed after that, I'll work on it. 3. When the test suite is run with doctests enabled, some unit tests fail that normally wouldn't, and some doctests fail that shouldn't. There's probably some state that needs to be reset (or otherwise managed) between doctest runs; I'll look into that and provide a fix as soon as possible. I just figured I should mention it in case it causes somebody problems before it gets fixed. Thanks, Alan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Coverage improvement requests
Hi all, This week I'm going to start working on new tests to improve Python code coverage in NumPy (C code coverage will come later in the summer, or maybe even after GSoC). Does anyone have recommendations for particularly important bits of code that need coverage? If not, I'm just going to start on modules with the largest number of uncovered lines, and go from there. Thanks, Alan (Sorry for the flood of posts) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] More pending test framework changes (please give feedback)
On Tue, Jul 1, 2008 at 9:26 AM, Alan McIntyre [EMAIL PROTECTED] wrote: On Mon, Jun 30, 2008 at 1:54 PM, Alan McIntyre [EMAIL PROTECTED] wrote: 1. All doctests in NumPy will have the numpy module available in their execution context as np. 2. Turn on the normalized whitespace option for all doctests. Having a doctest fail just because there's a space after your result seems like an unnecessary hassle for documenters. 3. Output will be ignored for each doctest expected output line that contains #random. I figured this can serve both as an ignore flag and indication to the reader that the listed output may differ from what they see if they execute the associated command. So you would be able to do: random.random() 0.1234567890 #random: output may differ on your system And have the example executed but not cause a failure. You could also use this to ignore the SomeObject at 0x1234ABCD output from plot methods as well. Since I didn't see any objections, these changes are now committed. I'll be updating some doctests to take advantage of them later today. I note that a lot of unit test files import tons of specific functions, numpy.core, etc., etc. Is there any reason not to fix things up to import numpy as np from numpy.testing import * I fixed one file this way, but I wonder if we shouldn't make all of them work like that. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 11:56 AM, Alan McIntyre [EMAIL PROTECTED] wrote: Just a few questions/comments about doctests: 1. Should all doctests be written such that you could start Python, do an import numpy as np, and then type in the examples verbatim? There are a number that currently wouldn't work that way (they depend on the function under test being in the local namespace, for example). +1 Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Coverage improvement requests
On Tue, Jul 1, 2008 at 13:13, Alan McIntyre [EMAIL PROTECTED] wrote: Hi all, This week I'm going to start working on new tests to improve Python code coverage in NumPy (C code coverage will come later in the summer, or maybe even after GSoC). Does anyone have recommendations for particularly important bits of code that need coverage? If not, I'm just going to start on modules with the largest number of uncovered lines, and go from there. numpy.core and numpy.lib are good places to start. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 12:56, Alan McIntyre [EMAIL PROTECTED] wrote: Just a few questions/comments about doctests: 1. Should all doctests be written such that you could start Python, do an import numpy as np, and then type in the examples verbatim? There are a number that currently wouldn't work that way (they depend on the function under test being in the local namespace, for example). 2. In regard to the auto-ignore of plt., etc. in commands: using the existing ellipsis feature of doctest should cover a significant portion of the cases that the auto-ignore was suggested to solve, and it's a very minor change to enable it (whereas the auto-ignore is more involved). If nobody objects, I will enable ellipsis for all doctests (which doesn't cause any obvious problems in existing NumPy tests), and use it to clean up existing doctests where appropriate. +1 If the auto-ignore capability is still needed after that, I'll work on it. It seems to me that the ellipsis mechanism just allows the output to differ. However, matplotlib would still be required because plt.plot() would still be executed. matplotlib should not be a requirement for running the tests. 3. When the test suite is run with doctests enabled, some unit tests fail that normally wouldn't, and some doctests fail that shouldn't. There's probably some state that needs to be reset (or otherwise managed) between doctest runs; I'll look into that and provide a fix as soon as possible. I just figured I should mention it in case it causes somebody problems before it gets fixed. Thank you for the notice. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 2:45 PM, Robert Kern [EMAIL PROTECTED] wrote: If the auto-ignore capability is still needed after that, I'll work on it. It seems to me that the ellipsis mechanism just allows the output to differ. However, matplotlib would still be required because plt.plot() would still be executed. matplotlib should not be a requirement for running the tests. Oops, I misunderstood, then: I thought the intent was to execute the statement but not compare the output (because they returned objects that had their address in the repr). I didn't look before to see how many times this feature would be needed (yeah, should have done that before complaining about using #doctest: +SKIP), but now that I look, I only see one batch of plt. commands, in numpy.lib.function_base.bartlett. In view of that, does it make more sense to use the SKIP directive for the ten plt. lines in that one example? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 14:14, Alan McIntyre [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 2:45 PM, Robert Kern [EMAIL PROTECTED] wrote: If the auto-ignore capability is still needed after that, I'll work on it. It seems to me that the ellipsis mechanism just allows the output to differ. However, matplotlib would still be required because plt.plot() would still be executed. matplotlib should not be a requirement for running the tests. Oops, I misunderstood, then: I thought the intent was to execute the statement but not compare the output (because they returned objects that had their address in the repr). I didn't look before to see how many times this feature would be needed (yeah, should have done that before complaining about using #doctest: +SKIP), but now that I look, I only see one batch of plt. commands, in numpy.lib.function_base.bartlett. In view of that, does it make more sense to use the SKIP directive for the ten plt. lines in that one example? Can it work on an entire section? If not, can we do something that works on a whole section? Everything after Plot the window and its frequency response: is not required for testing. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] More pending test framework changes (please give feedback)
On Tue, Jul 1, 2008 at 2:37 PM, Charles R Harris [EMAIL PROTECTED] wrote: I note that a lot of unit test files import tons of specific functions, numpy.core, etc., etc. Is there any reason not to fix things up to import numpy as np from numpy.testing import * I fixed one file this way, but I wonder if we shouldn't make all of them work like that. Personally, I prefer the imports to be as simple as possible, but I managed to restrain myself from cleaning up test module imports when I was making my changes. ;) If making them somewhat standardized is desirable, I might as well do it while I'm cleaning up and fixing tests. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 3:20 PM, Robert Kern [EMAIL PROTECTED] wrote: Can it work on an entire section? If not, can we do something that works on a whole section? Everything after Plot the window and its frequency response: is not required for testing. It's on a per-line basis at the moment, so each lines needs a #doctest: +SKIP. Changing a directive to apply to multiple lines probably isn't trivial (I haven't really looked into doing that, though). We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] More pending test framework changes (please give feedback)
On Tue, Jul 1, 2008 at 1:21 PM, Alan McIntyre [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 2:37 PM, Charles R Harris [EMAIL PROTECTED] wrote: I note that a lot of unit test files import tons of specific functions, numpy.core, etc., etc. Is there any reason not to fix things up to import numpy as np from numpy.testing import * I fixed one file this way, but I wonder if we shouldn't make all of them work like that. Personally, I prefer the imports to be as simple as possible, but I managed to restrain myself from cleaning up test module imports when I was making my changes. ;) If making them somewhat standardized is desirable, I might as well do it while I'm cleaning up and fixing tests. A lot of the imports seem to have just grown over the years, some even contain duplicates. So I think cleaning up would be a good idea if no one objects. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 1:30 PM, Alan McIntyre [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 3:20 PM, Robert Kern [EMAIL PROTECTED] wrote: Can it work on an entire section? If not, can we do something that works on a whole section? Everything after Plot the window and its frequency response: is not required for testing. It's on a per-line basis at the moment, so each lines needs a #doctest: +SKIP. Changing a directive to apply to multiple lines probably isn't trivial (I haven't really looked into doing that, though). We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). Would it serve to overload plot with a function that does zippo? Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 3:39 PM, Charles R Harris [EMAIL PROTECTED] wrote: We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). Would it serve to overload plot with a function that does zippo? Probably not in this case; there's an explicit matplotlib import, and then a bunch of method calls on a matplotlib object: from matplotlib import pyplot as plt window = np.bartlett(51) plt.plot(window) plt.title(Bartlett window) and so on. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 3:33 PM, Robert Kern [EMAIL PROTECTED] wrote: That's not a bad idea. Coordinate with Stéfan about the details (if any are left to be decided). Ok, will do. I'll also update all the test documentation I can find so that documenters have a chance of being aware of the doctest assumptions/requirements/capabilities. So, unless anyone else has objections, I'll: 1. Enabled ellipsis for all doctests 2. Update all doctests so that they only assume import numpy as np. I'll also check into restricting the doctest execution environment so that tests that make other assumptions should fail (as part of figuring out the test state pollution problem). ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] More pending test framework changes (please give feedback)
On Tue, Jul 1, 2008 at 3:37 PM, Charles R Harris [EMAIL PROTECTED] wrote: A lot of the imports seem to have just grown over the years, some even contain duplicates. So I think cleaning up would be a good idea if no one objects. Ok. As a pre-emptive clarification, I'll only be tweaking imports in unit test files--I don't want to mess with any of the magic that goes on in the package imports. ;) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 14:39, Charles R Harris [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 1:30 PM, Alan McIntyre [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 3:20 PM, Robert Kern [EMAIL PROTECTED] wrote: Can it work on an entire section? If not, can we do something that works on a whole section? Everything after Plot the window and its frequency response: is not required for testing. It's on a per-line basis at the moment, so each lines needs a #doctest: +SKIP. Changing a directive to apply to multiple lines probably isn't trivial (I haven't really looked into doing that, though). We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). Would it serve to overload plot with a function that does zippo? If it's not going to test anything, I would prefer that it not be part of the tests. Admittedly, that's just my sense of aesthetics, not a technical objection. A technical objection would be that some of the matplotlib functions actually do return something, and we would still have to uglify the examples with the ellipsis stuff. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
Tue, 01 Jul 2008 15:30:47 -0400, Alan McIntyre wrote: On Tue, Jul 1, 2008 at 3:20 PM, Robert Kern [EMAIL PROTECTED] wrote: Can it work on an entire section? If not, can we do something that works on a whole section? Everything after Plot the window and its frequency response: is not required for testing. It's on a per-line basis at the moment, so each lines needs a #doctest: +SKIP. Changing a directive to apply to multiple lines probably isn't trivial (I haven't really looked into doing that, though). I think this can be done without too many problems: Looking at the doctest.py source code, the easiest way to change the way how docstrings are parsed into doctests is to subclass `DoctestParser` and override its `parse` method. The `testmod` and `testfile` functions don't take a parser argument, but they appear to be only thin wrappers for instantiating `DocTestFinder` (which does take a parser argument) and then calling a method in `DocTestRunner`. Now, things appear to work a bit differently in nose. There, DocTestParser is instantiated by Doctest.loadTestsFromFile in the doctests.py plugin. I don't see an easy way to override this, except for monkeypatching doctest.DocTestParser or the whole nose plugin with our stuff. All in all, I'd estimate this to be ~100 lines, put in a suitable location. But it's a custom tweak to doctest, so it might break at some point in the future, and I don't love the monkeypatching here... We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Change of behavior in flatten between 1.0.4 and 1.1
Hi -- I have noticed a change in the behavior of numpy.flatten(True) between NumPy 1.0.4 and NumPy 1.1. The change affects 3D arrays. I am wondering if this is a bug or a feature. Here's the change. Note that the output from flatten(True) is different between 1.0.4 and 1.1. === First the preliminary set up: === In [3]: A = numpy.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]], [[9, 10], [11, 12]]]) In [4]: A Out[4]: array([[[ 1, 2], [ 3, 4]], [[ 5, 6], [ 7, 8]], [[ 9, 10], [11, 12]]]) === Now the change: Numpy 1.0.4 === In [5]: A.flatten() Out[5]: array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]) In [6]: A.flatten(True) Out[6]: array([ 1, 5, 9, 2, 6, 10, 3, 7, 11, 4, 8, 12]) === Numpy 1.1 === In [4]: A.flatten() Out[4]: array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]) In [5]: A.flatten(True) Out[5]: array([ 1, 5, 9, 3, 7, 11, 2, 6, 10, 4, 8, 12]) Note that the output of A.flatten(True) is different. Is this a bug or a feature? Cheers, Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change of behavior in flatten between 1.0.4 and 1.1
Tue, 01 Jul 2008 17:18:55 -0400, Stuart Brorson wrote: Hi -- I have noticed a change in the behavior of numpy.flatten(True) between NumPy 1.0.4 and NumPy 1.1. The change affects 3D arrays. I am wondering if this is a bug or a feature. Here's the change. Note that the output from flatten(True) is different between 1.0.4 and 1.1. I think it was this one http://scipy.org/scipy/numpy/ticket/676 The rationale was to make the output from .flatten(1) to be equal to interpreting the data as it would appear in a multidimensional Fortran array (equivalent to reshape(a, (prod(a.shape),), order='F'), IIRC). In 1.0.4 a.flatten(1) only swapped the two first axes and then flattened in C-order. In 1.1, a.flatten(1) == a.transpose().flatten(). To me, it appeared that the behavior in 1.0.4 was incorrect, so I filed the bug (after being bitten by it in real code...) and submitted a patch that got applied. -- Pauli Virtanen === First the preliminary set up: === In [3]: A = numpy.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]], [[9, 10], [11, 12]]]) In [4]: A Out[4]: array([[[ 1, 2], [ 3, 4]], [[ 5, 6], [ 7, 8]], [[ 9, 10], [11, 12]]]) === Now the change: Numpy 1.0.4 === In [5]: A.flatten() Out[5]: array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]) In [6]: A.flatten(True) Out[6]: array([ 1, 5, 9, 2, 6, 10, 3, 7, 11, 4, 8, 12]) Here the two first dimensions are swapped and data is interpreted in C- order. === Numpy 1.1 === In [4]: A.flatten() Out[4]: array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]) In [5]: A.flatten(True) Out[5]: array([ 1, 5, 9, 3, 7, 11, 2, 6, 10, 4, 8, 12]) Here dimensions are transposed, and data is interpreted in C-order. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 1:41 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: But it's a custom tweak to doctest, so it might break at some point in the future, and I don't love the monkeypatching here... Welcome to the joys of extending doctest/unittest. They hardcoded so much stuff in there that the only way to reuse that code is by copy/paste/monkeypatch. It's absolutely atrocious. We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. How about a much simpler approach? Just pre-populate the globals dict where doctest executes with an object called 'plt' that basically does def noop(*a,**k): pass class dummy(): def __getattr__(self,k): return noop plt = dummy() This would ensure that all calls to plt.anything() silently succeed in the doctests. Granted, we're not testing matplotlib, but it has the benefit of simplicity and of letting us keep consistent formatting, and examples that *users* can still paste into their sessions where plt refers to the real matplotlib. Just an idea... f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] problem with NumPy test harness in 1.1.0?
On Tue, Jul 1, 2008 at 17:53, David Brown [EMAIL PROTECTED] wrote: Hi, Our code depends on NumPy and we have been taking advantage of the NumPy test harness to create test scripts for new features as we develop them. As recently as version 1.1.0.dev5064 we had no problem running our tests from the directory in which they live, e.g.: python test_script.py .much output . -- Ran 10 tests in 0.397s OK However, with the released version NumPy 1.1.0 none of our tests work any longer and instead we get this error message: Traceback (most recent call last): File test_mfio.py, line 426, in module NumpyTest().run() File /usr/local/lib/python2.5/site-packages/numpy/testing/ numpytest.py, line 655, in run testcase_pattern=options.testcase_pattern) File /usr/local/lib/python2.5/site-packages/numpy/testing/ numpytest.py, line 575, in test level, verbosity) File /usr/local/lib/python2.5/site-packages/numpy/testing/ numpytest.py, line 453, in _test_suite_from_all_tests importall(this_package) File /usr/local/lib/python2.5/site-packages/numpy/testing/ numpytest.py, line 681, in importall for subpackage_name in os.listdir(package_dir): OSError: [Errno 2] No such file or directory: '' Looking through the archives I have found a reference to the same error at http://www.nabble.com/NumpyTest-problem-td17603890.html. However, this thread is focused on the use of NumpyTest for test scripts within the NumPy source tree and the suggestions do not seem to apply to our situation. My question is whether using NumpyTest for our own tests is considered to be a proper application of the test module. If not why not? You should migrate your tests to some other test runner. NumpyTest and NumpyTestCase will be deprecated in numpy 1.2 and removed as early as numpy 1.3. They were never particularly intended to be used outside of numpy and scipy. We are migrating to using nose as our test runner so we do not have to maintain our own; it's not our core competency. As you can see. If it is, is there some modification we can make to our tests to make them work with NumpyTest as it is now configured? Try NumpyTest().test(level=11, all=False) Or is this possibly a bug in NumPyTest? Possibly a bug. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] problem with NumPy test harness in 1.1.0?
On Tue, Jul 1, 2008 at 4:43 PM, Robert Kern [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 18:39, David Brown [EMAIL PROTECTED] wrote: OK, thanks for the info Robert. Any recommendations for another test harness? numpy and scipy are moving to nose as of numpy 1.2 and scipy 0.7. http://www.somethingaboutorange.com/mrl/projects/nose/ It has been a personal favorite of mine for some time, too. So has ipython recently, for what it's worth. It does make easy a lot of things that should be easy but aren't under unittest, which is a big plus in my view. We also adopted it after advice from Robert, others at Enthought and Titus Brown during a BOF session at last year's SciPy. The fact that nose absorbs/recognizes all regular unittest tests makes the transition fairly easy. Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] problem with NumPy test harness in 1.1.0?
Thanks for the input Fernando. -dave On Jul 1, 2008, at 5:48 PM, Fernando Perez wrote: On Tue, Jul 1, 2008 at 4:43 PM, Robert Kern [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 18:39, David Brown [EMAIL PROTECTED] wrote: OK, thanks for the info Robert. Any recommendations for another test harness? numpy and scipy are moving to nose as of numpy 1.2 and scipy 0.7. http://www.somethingaboutorange.com/mrl/projects/nose/ It has been a personal favorite of mine for some time, too. So has ipython recently, for what it's worth. It does make easy a lot of things that should be easy but aren't under unittest, which is a big plus in my view. We also adopted it after advice from Robert, others at Enthought and Titus Brown during a BOF session at last year's SciPy. The fact that nose absorbs/recognizes all regular unittest tests makes the transition fairly easy. Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
Robert Kern wrote: On Tue, Jul 1, 2008 at 17:50, Fernando Perez [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 1:41 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: But it's a custom tweak to doctest, so it might break at some point in the future, and I don't love the monkeypatching here... Welcome to the joys of extending doctest/unittest. They hardcoded so much stuff in there that the only way to reuse that code is by copy/paste/monkeypatch. It's absolutely atrocious. We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. How about a much simpler approach? Just pre-populate the globals dict where doctest executes with an object called 'plt' that basically does def noop(*a,**k): pass class dummy(): def __getattr__(self,k): return noop plt = dummy() This would ensure that all calls to plt.anything() silently succeed in the doctests. Granted, we're not testing matplotlib, but it has the benefit of simplicity and of letting us keep consistent formatting, and examples that *users* can still paste into their sessions where plt refers to the real matplotlib. It's actually easier for users to paste the non-doctestable examples since they don't have the markers and any stdout the examples produce as a byproduct. I'm with Robert here. It's definitely easier as an example without the . I also don't see the utility of being able to have the matplotlib code as tests of anything. We're not testing matplotlib here and any behavior that matplotlib relies on (and hence tests) should be captured in a test for that behavior separate from matplotlib code. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 4:41 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: All in all, I'd estimate this to be ~100 lines, put in a suitable location. If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. Well, for the moment, it's 100 lines of new code that's needed in order to *avoid* running 10 lines of doctest code in one function's docstring. Maybe down the road there will be more examples that need a bulk don't execute me mechanism, and if we do I'll be glad to work on it, but for right now I need to spend more time increasing coverage. I also agree with Ryan that matplotlib is where the tests for this particular use case of the object returned by bartlett should be. So at the moment I'm inclined to just remove the and come back to it later. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
On Tue, Jul 1, 2008 at 19:19, Ryan May [EMAIL PROTECTED] wrote: Robert Kern wrote: On Tue, Jul 1, 2008 at 17:50, Fernando Perez [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 1:41 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: But it's a custom tweak to doctest, so it might break at some point in the future, and I don't love the monkeypatching here... Welcome to the joys of extending doctest/unittest. They hardcoded so much stuff in there that the only way to reuse that code is by copy/paste/monkeypatch. It's absolutely atrocious. We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. How about a much simpler approach? Just pre-populate the globals dict where doctest executes with an object called 'plt' that basically does def noop(*a,**k): pass class dummy(): def __getattr__(self,k): return noop plt = dummy() This would ensure that all calls to plt.anything() silently succeed in the doctests. Granted, we're not testing matplotlib, but it has the benefit of simplicity and of letting us keep consistent formatting, and examples that *users* can still paste into their sessions where plt refers to the real matplotlib. It's actually easier for users to paste the non-doctestable examples since they don't have the markers and any stdout the examples produce as a byproduct. I'm with Robert here. It's definitely easier as an example without the . I also don't see the utility of being able to have the matplotlib code as tests of anything. We're not testing matplotlib here and any behavior that matplotlib relies on (and hence tests) should be captured in a test for that behavior separate from matplotlib code. To be clear, these aren't tests of the numpy code. The tests would be to make sure the examples still run. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Doctest items
Robert Kern wrote: On Tue, Jul 1, 2008 at 19:19, Ryan May [EMAIL PROTECTED] wrote: Robert Kern wrote: On Tue, Jul 1, 2008 at 17:50, Fernando Perez [EMAIL PROTECTED] wrote: On Tue, Jul 1, 2008 at 1:41 PM, Pauli Virtanen [EMAIL PROTECTED] wrote: But it's a custom tweak to doctest, so it might break at some point in the future, and I don't love the monkeypatching here... Welcome to the joys of extending doctest/unittest. They hardcoded so much stuff in there that the only way to reuse that code is by copy/paste/monkeypatch. It's absolutely atrocious. We could always just make the plotting section one of those it's just an example not a doctest things and remove the (since it doesn't appear to provide any useful test coverage or anything). If possible, I'd like other possibilities be considered first before jumping this route. I think it would be nice to retain the ability to run also the matplotlib examples as (optional) doctests, to make sure also they execute correctly. Also, using two different markups in the documentation to work around a shortcoming of doctest is IMHO not very elegant. How about a much simpler approach? Just pre-populate the globals dict where doctest executes with an object called 'plt' that basically does def noop(*a,**k): pass class dummy(): def __getattr__(self,k): return noop plt = dummy() This would ensure that all calls to plt.anything() silently succeed in the doctests. Granted, we're not testing matplotlib, but it has the benefit of simplicity and of letting us keep consistent formatting, and examples that *users* can still paste into their sessions where plt refers to the real matplotlib. It's actually easier for users to paste the non-doctestable examples since they don't have the markers and any stdout the examples produce as a byproduct. I'm with Robert here. It's definitely easier as an example without the . I also don't see the utility of being able to have the matplotlib code as tests of anything. We're not testing matplotlib here and any behavior that matplotlib relies on (and hence tests) should be captured in a test for that behavior separate from matplotlib code. To be clear, these aren't tests of the numpy code. The tests would be to make sure the examples still run. Right. I just don't think effort should be put into making examples using matplotlib run as doctests. If the behavior is important, numpy should have a standalone test for it. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
