Re: [Numpy-discussion] NumPy 1.8.0rc2 release
Hi,
I found an unexpected difference from numpy 1.7.1 and 1.8.0rc2 with Python
3.3.2 on Ubuntu 12.04 (amd64). Here is the test program:
import numpy as np
print(np.__version__)
K = np.array([[ 0., 0., 0., 0.],
[-0., 0., 0., 0.],
[ 0., -0., 0., 0.],
[ 0., -0., 0., 0.]])
w, V = np.linalg.eigh(K)
print('w')
print(w)
with numpy 1.7.1:
1.7.1
w
[-0. -0. -0. 0.]
with numpy 1.8.0rc2:
1.8.0rc2
w
[ 0. 0. 0. 0.]
Apologies if this is my mistake!
-Andrew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy 1.8.0rc2 release
Yeah, sorry for the confusion. I thought my email didn't go through. I
created a bug report on github instead. Now I find the email was indeed
sent. Sorry about that. Looking at the github activity (see
https://github.com/numpy/numpy/issues/3977 ), I think a fix is almost at
hand thanks to Julian Taylor.
Best,
Andrew
On Fri, Oct 25, 2013 at 7:52 PM, Nathaniel Smith n...@pobox.com wrote:
It's py3 only, see the discussion in #3977.
On 25 Oct 2013 17:45, Charles R Harris charlesr.har...@gmail.com
wrote:
On Fri, Oct 25, 2013 at 7:07 AM, Andrew Straw
dr.andrew.st...@gmail.comwrote:
Hi,
I found an unexpected difference from numpy 1.7.1 and 1.8.0rc2 with
Python 3.3.2 on Ubuntu 12.04 (amd64). Here is the test program:
import numpy as np
print(np.__version__)
K = np.array([[ 0., 0., 0., 0.],
[-0., 0., 0., 0.],
[ 0., -0., 0., 0.],
[ 0., -0., 0., 0.]])
w, V = np.linalg.eigh(K)
print('w')
print(w)
with numpy 1.7.1:
1.7.1
w
[-0. -0. -0. 0.]
with numpy 1.8.0rc2:
1.8.0rc2
w
[ 0. 0. 0. 0.]
This appears to have gotten fixed since rc2:
In [1]: K = np.array([[ 0., 0., 0., 0.
],
[-0., 0., 0., 0.],
[ 0., -0., 0., 0.],
[ 0., -0., 0., 0.]])
In [2]: eigh(K)
Out[2]:
(array([-0., -0., -0., 0.]),
array([[-0.78251031, 0.37104402, 0.00198815, 0.5 ],
[-0.29974269, 0.03728557, 0.81164285, -0.5 ],
[ 0.54246491, 0.46764373, 0.48686874, 0.5 ],
[-0.05969728, -0.80140218, 0.32278596, 0.5 ]]))
In [3]: np.__version__
Out[3]: '1.8.0.dev-ced0a94'
Could you try the current 1.8.x version on github and see if the problem
persists?
Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Compiling for free on Windows32
Fadhley Salim wrote: Thomasm, What want is the current latest Numpy as a win32 .egg file for Python 2.4.4. I'm not bothered how I get there. We've been able to compile * Dont care how I make it as long as it works! Are you aware that the reason numpy is not distributed as an .egg is so that the installer can determine the correct architecture (no SSE, SSE, SSE2) and install the appropriate files? This is not possible with .egg, so you'll either have to ship the no SSE version or ensure that it will be installed on only computers with the appropriate SIMD processor. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
Eric Firing wrote:
Sure enough, that is what I was looking for. (gitweb doesn't seem to
have the annotate [or blame, in git-speak] option, or the graph.)
gitweb does, you have to turn it on, though...
You need to add this to your gitweb.conf, though:
$feature{'blame'}{'default'} = [1];
$feature{'blame'}{'override'} = 1;
I also find pickaxe and snapshot useful:
$feature{'pickaxe'}{'default'} = [1];
$feature{'pickaxe'}{'override'} = 1;
$feature{'snapshot'}{'default'} = ['zip', 'tgz'];
$feature{'snapshot'}{'override'} = 1;
I don't know about the graph. (You mean the gitk --all kind of view? I
saw one JavaScript-y web app that did that, but it was slow and ugly for
any non-trivial repo.)
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] DVCS at PyCon
Matthieu Brucher wrote: One thing about git-svn is that this is not really needed if you just use git and I installed git from source on many linuxes and clusters and it just works, as it is just pure C. I usually just use git-svn on my laptop/workstation, where I install the Debian/Ubuntu packages, and I create the git repository, upload to github.com or somewhere else and just work with the git repository. But I agree that if it installs git-svn and it doesn't just work, it's a big problem. I was inquiring the use of git with the use of one of our internal svn repositories, just to have a feeling about it :( My opinion is that attempting to use git-svn to get a feeling of git is not a good idea. There's too much slowness of svn involved, too much pain of trying to learn git while also trying to learn git-svn (which itself has corner cases and such that pure git doesn't) and there's no bidirectional 1:1 mapping between branches (that I've found), eliminating a huge part of the joy of git -- cheap branches. Better to start developing on a pure git project to get a feel. You can't go wrong with sympy, for example. :) -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] N-D array interface page is out of date
Gael Varoquaux wrote: On Sun, Mar 08, 2009 at 10:18:11PM -0700, Andrew Straw wrote: OK, I now have a password (thanks Gaël), but I don't have edit permissions on that page. So I'm attaching a patch against that page source that incorporates the stuff that was on the old page that's not in the new page. I'm happy to apply this myself if someone gives me edit permissions. I wasn't able to check out all the ReST formatting with the online editor because I don't have edit permissions. You don't have permissions to the corresonding page in the doc wiki: http://docs.scipy.org/numpy/docs/numpy-docs/reference/arrays.interface.rst/ ? I am a bit lost, if this is the case. OK, thanks for the pointer. Somehow I navigated to a view of that page that I could not edit. I have now uploaded my changes, including bits that didn't hadn't yet made it into the Sphinx-based documentation from the original page on numpy.scipy.org. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy documentation editor - retrieve password?
Hi, I created a login for the numpy documentation editor but cannot remember my password. Would it be possible to have it sent to me or a new one generated? It would be great to have a button on the website so that I could do this myself, but if that's too much pain, my username is AndrewStraw. Thanks, Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] N-D array interface page is out of date
Pauli Virtanen wrote: Hi, Fri, 06 Mar 2009 09:16:33 -0800, Andrew Straw wrote: I have updated http://numpy.scipy.org/array_interface.shtml to have a giant warning first paragraph describing how that information is outdated. Additionally, I have updated http://numpy.scipy.org/ to point people to the buffer interface described in PEP 3118 and implemented in Python 2.6/3.0. Furthermore, I have suggested Cython has a way to write code for older Pythons that will automatically support the buffer interface in newer Pythons. If you have knowledge about these matters (Travis O. and Dag, especially), I'd appreciate it if you could read over the pages to ensure everything is actually correct. I wonder if it would make sense to redirect the page here: http://docs.scipy.org/doc/numpy/reference/arrays.interface.html so that it would be easier to edit etc. in the future? Yes, great idea. I just updated the page to point to the page you linked (which I didn't know existed -- thanks for pointing it out). Also, I have made several changes to arrays.interface.rst which I will upload once my password situation gets resolved. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy.scipy.org
Hi all, I have been doing some editing of http://numpy.scipy.org . In general, however, lots of this page is redundant and outdated compared to lots of other documentation that has now sprung up. Shall we kill this page off, redirect it to another page, or continue updating it? (For this latter option, patches are welcome.) -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] N-D array interface page is out of date
Andrew Straw wrote:
Pauli Virtanen wrote:
Hi,
Fri, 06 Mar 2009 09:16:33 -0800, Andrew Straw wrote:
I have updated http://numpy.scipy.org/array_interface.shtml to have a
giant warning first paragraph describing how that information is
outdated. Additionally, I have updated http://numpy.scipy.org/ to point
people to the buffer interface described in PEP 3118 and implemented in
Python 2.6/3.0. Furthermore, I have suggested Cython has a way to write
code for older Pythons that will automatically support the buffer
interface in newer Pythons.
If you have knowledge about these matters (Travis O. and Dag,
especially), I'd appreciate it if you could read over the pages to
ensure everything is actually correct.
I wonder if it would make sense to redirect the page here:
http://docs.scipy.org/doc/numpy/reference/arrays.interface.html
so that it would be easier to edit etc. in the future?
Yes, great idea. I just updated the page to point to the page you linked
(which I didn't know existed -- thanks for pointing it out).
Also, I have made several changes to arrays.interface.rst which I will
upload once my password situation gets resolved.
OK, I now have a password (thanks Gaël), but I don't have edit
permissions on that page. So I'm attaching a patch against that page
source that incorporates the stuff that was on the old page that's not
in the new page.
I'm happy to apply this myself if someone gives me edit permissions. I
wasn't able to check out all the ReST formatting with the online editor
because I don't have edit permissions.
-Andrew
diff --git a/scipy-website/numpy-doc/arrays.interface.rst b/scipy-website/numpy-doc/arrays.interface.rst
index e3b12ce..4894dd6 100644
--- a/scipy-website/numpy-doc/arrays.interface.rst
+++ b/scipy-website/numpy-doc/arrays.interface.rst
@@ -206,6 +206,33 @@ array using only one attribute lookup and a well-defined C-structure.
must also not reallocate their memory if other objects are
referencing them.
+The PyArrayInterface structure is defined in ``numpy/ndarrayobject.h``
+as::
+
+ typedef struct {
+int two; /* contains the integer 2 -- simple sanity check */
+int nd; /* number of dimensions */
+char typekind;/* kind in array --- character code of typestr */
+int itemsize; /* size of each element */
+int flags;/* flags indicating how the data should be interpreted */
+ /* must set ARR_HAS_DESCR bit to validate descr */
+Py_intptr_t *shape; /* A length-nd array of shape information */
+Py_intptr_t *strides; /* A length-nd array of stride information */
+void *data; /* A pointer to the first element of the array */
+PyObject *descr; /* NULL or data-description (same as descr key
+ of __array_interface__) -- must set ARR_HAS_DESCR
+ flag or this will be ignored. */
+ } PyArrayInterface;
+
+The flags member may consist of 5 bits showing how the data should be
+interpreted and one bit showing how the Interface should be
+interpreted. The data-bits are :const:`CONTIGUOUS` (0x1),
+:const:`FORTRAN` (0x2), :const:`ALIGNED` (0x100), :const:`NOTSWAPPED`
+(0x200), and :const:`WRITEABLE` (0x400). A final flag
+:const:`ARR_HAS_DESCR` (0x800) indicates whether or not this structure
+has the arrdescr field. The field should not be accessed unless this
+flag is present.
+
.. admonition:: New since June 16, 2006:
In the past most implementations used the desc member of the
@@ -216,3 +243,92 @@ array using only one attribute lookup and a well-defined C-structure.
reference to the object when the :ctype:`PyCObject` is created using
:ctype:`PyCObject_FromVoidPtrAndDesc`.
+
+Type description examples
+=
+
+For clarity it is useful to provide some examples of the type
+description and corresponding :data:`__array_interface__` 'descr'
+entries. Thanks to Scott Gilbert for these examples:
+
+In every case, the 'descr' key is optional, but of course provides
+more information which may be important for various applications::
+
+ * Float data
+ typestr == 'f4'
+ descr == [('','f4')]
+
+ * Complex double
+ typestr == 'c8'
+ descr == [('real','f4'), ('imag','f4')]
+
+ * RGB Pixel data
+ typestr == '|V3'
+ descr == [('r','|u1'), ('g','|u1'), ('b','|u1')]
+
+ * Mixed endian (weird but could happen).
+ typestr == '|V8' (or 'u8')
+ descr == [('big','i4'), ('little','i4')]
+
+ * Nested structure
+ struct {
+ int ival;
+ struct {
+ unsigned short sval;
+ unsigned char bval;
+ unsigned char cval;
+ } sub;
+ }
+ typestr == '|V8' (or 'u8' if you want)
+ descr == [('ival','i4'), ('sub', [('sval','u2'), ('bval','|u1'), ('cval
Re: [Numpy-discussion] N-D array interface page is out of date
Hi, I have updated http://numpy.scipy.org/array_interface.shtml to have a giant warning first paragraph describing how that information is outdated. Additionally, I have updated http://numpy.scipy.org/ to point people to the buffer interface described in PEP 3118 and implemented in Python 2.6/3.0. Furthermore, I have suggested Cython has a way to write code for older Pythons that will automatically support the buffer interface in newer Pythons. If you have knowledge about these matters (Travis O. and Dag, especially), I'd appreciate it if you could read over the pages to ensure everything is actually correct. Thanks, Andrew Stéfan van der Walt wrote: 2009/2/3 Andrew Straw straw...@astraw.com: Can someone with appropriate permissions fix the page or give me the appropriate permissions so I can do it? I think even deleting the page is better than keeping it as-is. Who all has editing access to this page? Is it hosted on scipy.org? Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is there a faster way to get a buffer interface than ndarray.tostring()?
Given what you're doing, may I also suggest having a look at
http://code.astraw.com/projects/motmot/wxglvideo.html
-Andrew
Chris Colbert wrote:
As an update for any future googlers:
the problem was with revpixels = pixeldata[::-1,:,;:-1] which apparently
returns an array that is discontinuous in memory. What Lisandro
suggested worked. pixeldata[::-1,:,;:-1].copy() returns a continuous
array object which natively implements a single-segment buffer
interface. i.e. no buffer(revpixels) is needed; just simply revpixels.
array.copy() is also 50% faster than array.tostring() on my machine.
Chris
On Tue, Feb 24, 2009 at 9:27 PM, Chris Colbert sccolb...@gmail.com
mailto:sccolb...@gmail.com wrote:
thanks for both answers!
Lisandro, you're right, I should have declared the array outside the
loop. Thanks for catching that!
Robert, as always, thanks for the answer. Quick and to the point!
You've helped me more than once on the enthought list :)
On Tue, Feb 24, 2009 at 8:06 PM, Lisandro Dalcin dalc...@gmail.com
mailto:dalc...@gmail.com wrote:
When you do pixeldata[::-1,:,::-1], you just got a new array with
different strides, but now non-contiguous... So I believe you really
need a fresh copy of the data... tostring() copies, but could be
slow... try to use
revpixels = pixeldata[::-1,:,::-1].copy()
...
rgbBMP = wx.BitmapFromBuffer(640, 480, buffer(revpixels))
Perhaps you could optimize this by declaring revpixels outside de
loop, and then inside de loop doing
revpixels[...] = pixeldata[::-1,:,::-1]
This way you will save the mem allocation of revpixels at each
step of the loop.
On Tue, Feb 24, 2009 at 9:15 PM, Chris Colbert
sccolb...@gmail.com mailto:sccolb...@gmail.com wrote:
Hi all,
I'm new to mailing list and relatively new (~1 year) to
python/numpy. I
would appreciate any insight any of you may have here. The
last 8 hours of
digging through the docs has left me, finally, stuck.
I am making a wxPython program that includes webcam
functionality. The
script below just brings up a simple display window that shows
the webcam
feed. The problem I am running into is in the
WebCamWorker.run() method.
This method reads the raw pixel data (in string form) from the
webcam
buffer. This data (thank you microsoft!) comes in BGR and
bottom-to-top. I
feed this buffer to the array constructor and then swap the
pixel order to
RGB and top-to-bottom. This all happens very fast, ~1ms on a
Quad Core, and
the majority of that time is spent constructing the array (the
numpy pixel
swapping is on the order of E-5s !). The tostring() method,
however, takes a
whopping 10ms to execute. Unfortunately, the
wx.BitmapFromBuffer needs a
single segment buffer interface as an argument. Is there a way
to expose my
numpy array as a buffer to cut down on this conversion time?
I have tried sending the array to the PIL Image.fromarray(),
but that
eventually requires me to do a Image.tostring() anyway, which
negates any
benefit.
Thanks in advance for the help!
S. Chris Colbert
Rehabilitation Robotics Laboratory
University of South Florida
Code ###
import VideoCapture
import wx
import time
import threading
import numpy as np
import Image
class WebCamWorker(threading.Thread):
_abort = 0
def __init__(self, parent):
super(WebCamWorker, self).__init__()
self._parent = parent
self.panel = wx.Panel(parent)
def run(self):
while not self._abort:
#numpy arrays reverse pixel data that comes in
from CCD as BGR
and bottom to top
pixeldata = np.ndarray((480,640,3),
buffer=webcam.getBuffer()[0], dtype='u1')
revpixels = pixeldata[::-1,:,::-1].tostring()
#tostring is
an order of magnitude slower than the entire array
manipulation. need a
faster method.
rgbBMP = wx.BitmapFromBuffer(640, 480, revpixels)
dc = wx.ClientDC(self.panel)
dc.DrawBitmap(rgbBMP, 0, 0)
#b = time.clock()
#print
Re: [Numpy-discussion] small suggestion for numpy.testing utils
Darren, What's the difference between asanyarray(y) and array(y, copy=False, subok=True)? I thought asanyarray would also do what you want. -Andrew Darren Dale wrote: On Sun, Feb 22, 2009 at 3:22 PM, Darren Dale dsdal...@gmail.com mailto:dsdal...@gmail.com wrote: On Sun, Feb 22, 2009 at 3:17 PM, Darren Dale dsdal...@gmail.com mailto:dsdal...@gmail.com wrote: Hello, I am using numpy's assert_array_equal and assert_array_almost_equal to unit test my physical quantities package. I made a single minor change to assert_array_compare that I think might make these functions more useful to ndarray subclasses, and thought maybe they could be useful to numpy itself. I tried applying this diff to numpy and running the test suite, and instead of 9 known failures I got 1 known failure, 11 skips, 2 errors and 2 failures. Perhaps it is possible that by not forcing the input arrays to be ndarray instances, some additional numpy features are exposed. Thanks, Darren $ svn diff Index: numpy/testing/utils.py === --- numpy/testing/utils.py (revision 6370) +++ numpy/testing/utils.py (working copy) @@ -240,9 +240,9 @@ def assert_array_compare(comparison, x, y, err_msg='', verbose=True, header=''): -from numpy.core import asarray, isnan, any -x = asarray(x) -y = asarray(y) +from numpy.core import array, isnan, any +x = array(x, copy=False, subok=True) +y = array(y, copy=False, subok=True) def isnumber(x): return x.dtype.char in '?bhilqpBHILQPfdgFDG' Actually, my svn checkout was not up to date. With this patch applied, I get 1 known failure and 11 skips. I just double checked and I think I get the same results running the svn 6456 test suite with and without this patch applied. I tried posting an enhancement request at the trac website, but I cant file the ticket because I get 500 Internal Server Error, so I'm posting it here. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] parallel compilation of numpy
David Cournapeau wrote: * Integration with setuptools and eggs, which enables things like namespace packages. This is not. eggs are not specified, and totally implementation defined. I tried some time ago to add an egg builder to scons, but I gave up. And I don't think you can reuse the setuptools code, as everything is coupled. It's an interesting idea to build Python package distributions without distutils. For pure Python installables, if all you seek better is distutils, the bar seems fairly low. For compiled stuff, it still doesn't seem too bad. Of course, it is easy to say this without having tried... So, what do you mean by an egg, in the context of it being hard to produce? An .egg zip file, an .egg directory, and/or a normal distutils package with an .egg-info/ sibling? Since .egg-info/ is now part of distutils, this should now be specified... or? [This, though, does point out a conceptual problem with setuptools for me -- it does a zillion things some of which I like a lot (e.g. installing console scripts and gui scripts, a simple plugin architecture) and others I don't care about as long as they don't break things for me (e.g. installing multiple version of packages side-by-side, a problem which is much more sanely solved by setting PYTHONPATH, or its sophisticated cousin, virtualenv). And all of these things go by one name setuptools and sometimes even eggs, even though people often use those words to discuss totally different features. Hence my question above.] -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] parallel compilation of numpy
David Cournapeau wrote: On Thu, Feb 19, 2009 at 3:24 PM, Andrew Straw straw...@astraw.com wrote: It's an interesting idea to build Python package distributions without distutils. For pure Python installables, if all you seek better is distutils, the bar seems fairly low. :) Being better than distutils is not difficult, indeed - that is if you don't care about backward compatibility with distutils (which I don't personally - since distutils is implementation defined, I don't see any way to be backward compatible and be a significant improvement at the same time). Maybe if you need a level of backward compatibility, (and really, to gain a decent audience for this idea, I think you do need some level of backward compatibility) the new tool could emit setup.py files for consumption by distutils as a fallback plan. I could imagine auto-generating a setup.py is easier than emulating distutils, particularly if your concept starts simple. Furthermore, if you're not opposed to dropping in your own distutils monkeypatches, like lots of other packages, you probably could do anything you wanted. For example, bypassing the build_ext command and injecting the built products into the distutils install command. This monkeypatching idea is not even so unpalatable if one remembers that monkeypatching, which is quite common, makes it literally impossible to emulate distutils without being distutils. I will refrain from speaking about setuptools :) But the above problem is the same for distutils and setuptools, and exactly the fundamental issue. If distutils was conceived as a set of lousely coupled modules for tools, build, distribution, it would have been fixable. In the case of setuptools, it was a conscious decision to force setuptools on python world, This reminds of Linus' criticism of svn: its goal is to be a better cvs. Said dripping with incredulity due to his perception of the fatal flaws of CVS. Well, I think (parts of) setuptools are better than distutils, but by being distutils+, it will always share the same flawed genetic material... ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0; Ubuntu packages
Fernando Perez wrote: On Wed, Feb 11, 2009 at 6:17 PM, David Cournapeau courn...@gmail.com wrote: Unfortunately, it does require some work, because hardy uses g77 instead of gfortran, so the source package has to be different (once hardy is done, all the one below would be easy, though). I am not sure how to do that with PPA (the doc is not great). OK, thanks for the info. This is already very useful. What exactly is the expected problem and how would I verify that I'm not getting hit by it? The context is that I have built your packages on Hardy and got no FTBFS errors in a clean sbuild machine and was about to upload this to my lab's repo but saw your email. (Also, I have been using scipy 0.7.0rc2-ish apparently descended from the same Debian source package as yours for a couple weeks and have noted no problems of late.) nosetest scipy on an amd64 with lots of installed packages shows: Ran 3211 tests in 356.572s FAILED (SKIP=13, errors=9) Where the errors are all known failures. The same test on the i386 sbuild shows: Ran 3211 tests in 312.464s FAILED (SKIP=16, errors=204) Most of the failures appear to be from weave/scxx and I suspect have to do with some package not being installed -- if that's true, it's probably a packaging bug and not a scipy bug. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0; Ubuntu packages
David Cournapeau wrote: Andrew Straw wrote: Fernando Perez wrote: On Wed, Feb 11, 2009 at 6:17 PM, David Cournapeau courn...@gmail.com wrote: Unfortunately, it does require some work, because hardy uses g77 instead of gfortran, so the source package has to be different (once hardy is done, all the one below would be easy, though). I am not sure how to do that with PPA (the doc is not great). OK, thanks for the info. This is already very useful. What exactly is the expected problem and how would I verify that I'm not getting hit by it? I want to follow as closely as possible the official debian/ubuntu packages. Ideally, any package produced on the PPA superseded by an official package (from 1.2.1-1~ppaN to 1.2.1-1) should be identical to the superseding package. Hardy default fortran ABI is g77, not gfortran, so I have to use g77 for hardy - and the ppa package, limited to intrepid, use gfortran (since intrepid ABI is gfortran's). (Warning: this email is a little over-detailed on the packaging details front. Believe it or not, I'm not discussing the details of Debian packaging for fun, but rather my questions have practical importance to me -- I don't want to break all my lab's scipy installations. :) This doesn't make sense to me. I built the .deb on an a clean, minimal sbuild (a chroot with only a very few basic packages installed, somewhat mimicing Ubuntu's PPA builder). It built from your unmodified .dsc, which auto-downloads the declared dependencies (and nothing else). It passes the tests. To be very explicit -- I didn't specify to use g77 at any point. (As implied by my previous statement of using your unmodified .dsc, I used only the debian/rules and debian/control in your package.) To understand your statement about identical, I will operationally define identical for .debs to mean that they were built from the same .dsc. Of course, in the case you describe above, it can't be _exactly_ the same .dsc because the version numbers in debian/changelog must change and consequently so must the checksums and GPG signature in the .dsc file, and presumably a different person will sign it. Also, there will be timestamp differences and such for two .debs built from the exact same .dsc, but we can ignore those. In this case, I don't see why an official package, which meets this operational definition of identical, wouldn't work on Hardy, as it would be built from nearly an identical .dsc on nearly an identical clean build environment. (Of course, there will never be an official package of this for Hardy, but that's not the point.) In your case, you built a package which uses gfortran ABI on Hardy - it works, but is not really acceptable for an official package - and thus, when an upgrade from ppa to official happens, you won't get the same package. Why is it not really acceptable? As long as it builds and works and doesn't break anything, why would Ubuntu maintainers care if it uses the gfortran ABI? Also, in practical terms, what upgrade? A) Hardy will not upgrade python-scipy. It's against policy for a released distribution to upgrade software without a security reason. B) Imaging for a moment that there would be an upgrade, why do you think it would break? In the rpm world, you can use conditional on distribution version/type int the spec file (which is the control + changelog + rules in one file), but AFAIK, you can't do that with debian, or at least have not found the relevant doc. I don't understand what you're saying. My understanding is that at the beginning of each distribution (Hardy, Intrepid, Lenny), the maintainers decide on a C++ (and I guess Fortran, but I'm not sure) ABI and fix the toolchain to build this ABI. From then on, everything is built with this ABI. And the point of a signed .dsc file and a clean sbuild/pbuilder is that any .deb that gets built will be contingent on the files in debian/* because that's cryptographically signed in the .dsc file. So, if you trust the archive master and his computer (by trusting his keys in your apt keyring), you trust that the .deb was built from the .dsc. An the .dsc is signed by the maintainer. So there's a cryptographic chain of trust to those control + changelog + rules files. I'm still not sure if I'm not getting your worry, though... Thanks, Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] Numpy 1.2.1 and Scipy 0.7.0; Ubuntu packages
OK, I think you're concerned about compatibility of Python extensions using fortran. We don't use any (that I know of), so I'm going to stop worrying about this and upload .debs from your .dsc (or very close) to my repository... ...except for one last question: If Hardy uses the g77 ABI but I'm building scipy with gfortran, shouldn't there be an ABI issue with my ATLAS? Shouldn't I get lots of test failures with scipy? I don't. David Cournapeau wrote: My main rationale to provide PPA is to avoid the never-ending queue of emails about missing symbols, etc... because I am tired of it, because it gives a bad mouth taste to users, because I would like to deal with more interesting issues. Well, I appreciate you doing that. Packaging is a thankless job... and when everything works on your own computer, it's hard to work up the motivation to make it work on others'. To understand your statement about identical, I will operationally define identical for .debs to mean that they were built from the same .dsc. I has an even larger definition of identical: same control/rules/patches is identical for me. This is the only other point I want to make: if you're building from the same .dsc, it means you're using the same control/rules/patches. (That's why I brought up the checksums and the signatures.) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] N-D array interface page is out of date
Regarding http://numpy.scipy.org/array_interface.shtml : I just noticed that this out of date page is now featuring in recent discussions about the future of Numpy in Ubuntu: https://bugs.launchpad.net/ubuntu/+source/python-numpy/+bug/309215 Can someone with appropriate permissions fix the page or give me the appropriate permissions so I can do it? I think even deleting the page is better than keeping it as-is. -Andrew Andrew Straw wrote: Hi, I just noticed that the N-D array interface page is outdated and doesn't mention the buffer interface that is standard with Python 2.6 and Python 3.0: http://numpy.scipy.org/array_interface.shtml This page is linked to from http://numpy.scipy.org/ I suggest, at the minimum, modifying the page with really annoying blinking red letters at the top (or other suitable warning) that this is deprecated at that people should use http://www.python.org/dev/peps/pep-3118/ instead. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] N-D array interface page is out of date
Hi, I just noticed that the N-D array interface page is outdated and doesn't mention the buffer interface that is standard with Python 2.6 and Python 3.0: http://numpy.scipy.org/array_interface.shtml This page is linked to from http://numpy.scipy.org/ I suggest, at the minimum, modifying the page with really annoying blinking red letters at the top (or other suitable warning) that this is deprecated at that people should use http://www.python.org/dev/peps/pep-3118/ instead. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Singular Matrix problem with Matplitlib in Numpy (Windows - AMD64)
John Hunter wrote: Andrew, since you are the original author of the isnan port, could you patch the branch and the trunk to take care of this? Done in r6791 and r6792. Sorry for the trouble. Now I just hope we don't get a problem with long long, although now if _ISOC99_SOURCE is defined, we'll preferentially use int64_t out of stdint.h, so I should think this is more portable on sane platforms. This one of many reasons why I stick to Python... -Andrew JDH On Fri, Jan 16, 2009 at 8:07 AM, George g...@emss.co.za wrote: Hello. I am terribly sorry. I was mistaken last night. I had the latest Matplotlib version 0.98.5.2 and I thought the bug was fixed but it hasn't. Let me explain. In the file MPL_isnan.h line 26 there is a declaration: typedef long int MPL_Int64 This is fine for Linux 64-bit, but NOT for Windows XP 64-bit! For Windows the declaration should be: typedef long longMPL_Int64 This bug has caused me a LOT of late nights and last night was one of them. The declaration is correct for Linux 64-bit and I guess Matplotlib was developed on Linux because of this declaration. That is also why I thought the bug was fixed but this morning I realised that I was looking at the wrong console. So, in summary. For Matplotlib 0.98.5.2 and Numpy 1.2.1 to work without any problems. This means compiling and using Numpy and Matplotlib on Windows XP 64-bit using AMD 64-bit compile environment, change line 26 in the file MPL_isnan.h from long int to long long.\ I also previously suggested switching MKL and ACML etc. but with this change everything is fine. One can choose any math library and it works. Writing a small test application using sizeof on different platforms highlights the problem. Thanks. George. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Efficient removal of duplicates
Hanno Klemm wrote: Hi, I the following problem: I have a relatively long array of points [(x0,y0), (x1,y1), ...]. Apparently, I have some duplicate entries, which prevents the Delaunay triangulation algorithm from completing its task. Question, is there an efficent way, of getting rid of the duplicate entries? All I can think of involves loops. Thanks and regards, Hanno One idea is to create a view of the original array with a shape of (N,) and elements with a dtype that encompases both xn, yn. Then use numpy.unique() to find the unique entries, and create a view of that array with your original dtype. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] test results of 1.2.0rc1
Hi, with numpy 1.2.0rc1 running 'python -c import numpy; numpy.test()' on my Ubuntu Hardy amd64 machine results in 1721 tests being run and 1 skipped. So far, so good. However, if I run numpy.test(10,10,all=True), I get 1846 tests with: the message FAILED (SKIP=1, errors=8, failures=68) Furthermore, there are several matplotlib windows that pop up, many of which are non-reassuringly blank: Bartlett window frequency response (twice -- I guess the 2nd is actually for the Blackman window), Hamming window frequency response, Kaiser window, Kaiser window frequency response, sinc function. Additionally, the linspace and logspace tests each generate a plots with green and blue dots at Y values of 0.0 and 0.5, but it would be nice to have an axes title. Should I be concerned that there are so many errors and failures with the numpy test suite? Or am I just running it with unintended settings? If these tests should pass, I will attempt to find time to generate bug reports for them, although I don't think there's anything particularly weird about my setup. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] rc1 update
Jarrod Millman wrote: On Thu, Aug 28, 2008 at 7:48 AM, Travis E. Oliphant [EMAIL PROTECTED] wrote: +1 on what Andrew said. I don't really care that much, but I do think API is better than FEATURE. I would think that there may be times when we change the API but not the features (e.g., renaming something). I won't bring this up again, so if no one else finds this compelling consider it dropped. As I said, I'm +0 on that aspect. So I guess that makes Travis +0, too. ;) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advice on converting iterator into array efficiently
Alan Jackson wrote:
Looking for advice on a good way to handle this problem.
I'm dealing with large tables (Gigabyte large). I would like to
efficiently subset values from one column based on the values in
another column, and get arrays out of the operation. For example,
say I have 2 columns, energy and collection. Collection is
basically an index that flags values that go together, so all the
energy values with a collection value of 18 belong together. I'd
like to be able to set up an iterator on collection that would
hand me an array of energy on each iteration :
if table is all my data, then something like
for c in table['collection'] :
e = c['energy']
... do array operations on e
I've been playing with pytables, and they help, but I can't quite
seem to get there. I can get an iterator for energy within a collection,
but I can't figure out an efficient way to get an array out.
What I have so far is
for h in np.unique(table.col('collection')) :
rows = table.where('collection == c')
for row in rows :
print c,' : ', row['energy']
but I really want to convert rows['energy'] to an array.
I've thought about building a nasty set of pointers and whatnot -
I did it once in perl - but I'm hoping to avoid that.
I do stuff like this all the time:
t = table[:] # convert to structured array
collections = np.unique(t['collection'])
for collection in collections:
cond = t['collection'] == collection
energy_this_collection = t['energy'][cond]
HTH,
Andrew
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [REVIEW] Update NumPy API format to support updates that don't break binary compatibility
Looking at the code, but not testing it -- this looks fine to me. (I wrote the original NPY_VERSION stuff and sent it to Travis, who modified and included it.) I have added a couple of extremely minor points to the code review tool -- as much as a chance to play with the tool as to comment on the code. -Andrew Stéfan van der Walt wrote: The current NumPy API number, stored as NPY_VERSION in the header files, needs to be incremented every time the NumPy C-API changes. The counter tells developers with exactly which revision of the API they are dealing. NumPy does some checking to make sure that it does not run against an old version of the API. Currently, we have no way of distinguishing between changes that break binary compatibility and those that don't. The proposed fix breaks the version number up into two counters -- one that gets increased when binary compatibility is broken, and another when the API is changed without breaking compatibility. Backward compatibility with packages such as Matplotlib is maintained by renaming NPY_VERSION to NPY_BINARY_VERSION. Please review the proposed change at http://codereview.appspot.com/2946 Regards Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C-API change for 1.2
Robert Kern wrote: On Sat, Aug 16, 2008 at 04:34, Jon Wright [EMAIL PROTECTED] wrote: Travis, Stéfan, I missed Travis mail previously. Are you *really* sure you want force all C code which uses numpy arrays to be recompiled? If you mean that all your matplotlib/PIL/pyopengl/etc users are going to have to make a co-ordinated upgrade, then this seems to be a grave mistake. FWIW, neither PIL nor PyOpenGL have C code which uses numpy arrays, so they are entirely unaffected. And this does not require an *upgrade* of the actually affected packages, just a rebuild of the binary. I'll also point out that PEP 3118 will make this unnecessary in the future for many applications. http://www.python.org/dev/peps/pep-3118/ From what I can tell ( http://svn.python.org/view?rev=61491view=rev ), this is already in Python 2.6. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Instructions on building from source
Eric Firing wrote: Andrew Straw wrote: Just for reference, you can find the build dependencies of any Debian source package by looking at its .dsc file. For numpy, that can be found at http://packages.debian.org/sid/python-numpy Currently (version 1.1.0, debian version 1:1.1.0-3), that list is: Build-Depends: cdbs (= 0.4.43), python-all-dev, python-all-dbg, python-central (= 0.6), gfortran (= 4:4.2), libblas-dev [!arm !m68k], liblapack-dev [!arm !m68k], debhelper (= 5.0.38), patchutils, python-docutils, libfftw3-dev Build-Conflicts: atlas3-base-dev, libatlas-3dnow-dev, libatlas-base-dev, libatlas-headers, libatlas-sse-dev, libatlas-sse2-dev Do you know why atlas is not used, and is even listed as a conflict? I have libatlas-sse2 etc. installed on ubuntu hardy, and I routinely build numpy from source. Maybe the debian specification is for lowest-common-denominator hardware? The way it's supposed to work, as far as I understand it, is that atlas is not required at build time but when installed later automatically speeds up the blas routines. (Upon installation of libatlas3gf-sse2, libblas.so.3gf is pointed to /usr/lib/sse2/atlas/libblas.so.3gf from libblas.so.3gf = /usr/lib/libblas.so.3gf). I have not verified that any of this actually happens. So, please take this with a grain of salt. Especially since my answer differs from Robert's. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Instructions on building from source
Robert Kern wrote: On Wed, Jul 23, 2008 at 16:56, Fernando Perez [EMAIL PROTECTED] wrote: Howdy, I was just trying to explain to a new user how to build numpy from source on ubuntu and I realized that there's not much info on this front in the source tree. Scipy has a nice INSTALL.txt that even lists the names of the debian/ubuntu packages needed for the build (which can be a useful guide on other distros). Should we have a stripped-down copy of this doc somewhere in the top-level directory of numpy? Just for reference, you can find the build dependencies of any Debian source package by looking at its .dsc file. For numpy, that can be found at http://packages.debian.org/sid/python-numpy Currently (version 1.1.0, debian version 1:1.1.0-3), that list is: Build-Depends: cdbs (= 0.4.43), python-all-dev, python-all-dbg, python-central (= 0.6), gfortran (= 4:4.2), libblas-dev [!arm !m68k], liblapack-dev [!arm !m68k], debhelper (= 5.0.38), patchutils, python-docutils, libfftw3-dev Build-Conflicts: atlas3-base-dev, libatlas-3dnow-dev, libatlas-base-dev, libatlas-headers, libatlas-sse-dev, libatlas-sse2-dev Some of that stuff (cdbs, debhelper, patchutils) is specific to the Debian build process and wouldn't be necessary for simply compiling numpy itself. And on a Debian (derivative) system, you can stall those with apt-get build-dep python-numpy. This will only install the build dependencies for the version of python-numpy which is listed in your apt sources.list, but 99% of the time, that should be sufficient. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FFT's IFFT's on images
Mike Sarahan wrote: I agree that the components are very small, and in a numeric sense, I wouldn't worry at all about them, but the image result is simply noise, albeit periodic-looking noise. Fernando Perez and John Hunter have written a nice FFT image denoising example: http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/py4science/examples/fft_imdenoise.py?view=markup with documentation, even: http://matplotlib.svn.sourceforge.net/viewvc/matplotlib/trunk/py4science/workbook/fft_imdenoise.tex?view=markup ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] nose changes checked in
Stéfan van der Walt wrote: 2008/6/18 Alan McIntyre [EMAIL PROTECTED]: Is next release referring to 1.2 or the release after that? If it's the release after 1.2, then I assume that 1.2 must still be able to run all its tests without nose. Alternatively, we could distribute Nose inside of NumPy for one release? I suppose the Debian guys would shoot us :) No, I don't think they'll shoot anyone. :) It can just be disabled with a Debian specific patch to numpy. And since nose is LGPL (and therefore DFSG kosher), I don't think there'd be any reason to even bother re-making the source tarball. I'm quite sure the Debian packagers have dealt with worse. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Switching to nose test framework (was: NumpyTest problem)
Alan McIntyre wrote: On Mon, Jun 9, 2008 at 3:41 AM, Stéfan van der Walt [EMAIL PROTECTED] wrote: I suggest we also remove ParametricTestCase now. I don't mind converting the existing uses (looks like it's only used 5 times) to something else, it was causing trouble for me with nose anyway--whenever the test module imported it, nose wanted to run it since it inherited from TestCase. So if nobody really, really needs it I'll make ParametricTestCase go away. I'm using it in some of my code, but I'll happily switch to nose. It will make my life easier, however, if I can see how you've converted it. If you do this, can you indicate what svn revision made the switch? Thanks, Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Buildbot dying horrible, slow death.
Charles R Harris wrote: Dear Friends, Our wonderful buildbot is in declining health. The Mac can't update from svn, Andrew's machines are offline, the Sparcs have lost their spark, and bsd_64 is suffering from tolist() syndrome: AFAIK, I don't have any machines on the buildbot... Is there another Andrew with offline machines? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: NumPy 1.1.0
Thanks, Jarrod. Should I replace the old numpy 1.0.4 information at http://www.scipy.org/Download with the 1.1.0? It's still listing 1.0.4, but I wonder if there's some compatibility with scipy 0.6 issue that should cause it to stay at 1.0.4. In either case, I think the page should be updated -- particularly as searching Google for numpy download results in that page as the first hit. -Andrew Jarrod Millman wrote: I'm pleased to announce the release of NumPy 1.1.0. NumPy is the fundamental package needed for scientific computing with Python. It contains: * a powerful N-dimensional array object * sophisticated (broadcasting) functions * basic linear algebra functions * basic Fourier transforms * sophisticated random number capabilities * tools for integrating Fortran code. Besides it's obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide-variety of databases. This is the first minor release since the 1.0 release in October 2006. There are a few major changes, which introduce some minor API breakage. In addition this release includes tremendous improvements in terms of bug-fixing, testing, and documentation. For information, please see the release notes: http://sourceforge.net/project/shownotes.php?release_id=602575group_id=1369 Thank you to everybody who contributed to this release. Enjoy, ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.1.0rc1 tagged
Jarrod Millman wrote: Please test the release candidate: svn co http://svn.scipy.org/svn/numpy/tags/1.1.0rc1 1.1.0rc1 Thanks, Jarrod. I have packaged SVN trunk from r5189 and made a Debian source package (based on a slightly old version the Debian Python Modules Team's numpy package with all patches removed) and Ubuntu Hardy (8.04) binary packages. These are available at: http://debs.astraw.com/hardy/ In particular, you can just grab the .deb for your architecture for Ubuntu Hardy: * i386: http://debs.astraw.com/hardy/python-numpy_1.1.0~dev5189-0ads1_i386.deb * amd64: http://debs.astraw.com/hardy/python-numpy_1.1.0~dev5189-0ads1_amd64.deb All numpy, tests pass on both architectures in my hands, and I shall begin testing my various codes with this release. Other Ubunteers who don't want to bother compiling from source are also welcome to try these packages. (I chose the trunk rather than the RC tag because my understanding is that fixes to the final 1.1.0 are going to the trunk, and David Cournapeau has made a couple commits. Also, I released after I packaged this up that I forgot to touch the date in debian/changelog -- apologies!) -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] searchsorted() and memory cache
I will post any new insights as I continue to work on this...
OK, I save isolated a sample of my data that illustrates the terrible
performance with the binarysearch. I have uploaded it as a pytables file
to http://astraw.com/framenumbers.h5 in case anyone wants to have a look
themselves. Here's an example of the type of benchmark I've been running:
import fastsearch.downsamp
import fastsearch.binarysearch
import tables
h5=tables.openFile('framenumbers.h5',mode='r')
framenumbers=h5.root.framenumbers.read()
keys=h5.root.keys.read()
h5.close()
def bench( implementation ):
for key in keys:
implementation.index( key )
downsamp = fastsearch.downsamp.DownSampledPreSearcher( framenumbers )
binary = fastsearch.binarysearch.BinarySearcher( framenumbers )
# The next two lines are IPython-specific, and the 2nd takes a looong time:
%timeit bench(downsamp)
%timeit bench(binary)
Running the above gives:
In [14]: %timeit bench(downsamp)
10 loops, best of 3: 64 ms per loop
In [15]: %timeit bench(binary)
10 loops, best of 3: 184 s per loop
Quite a difference (a factor of about 3000)! At this point, I haven't
delved into the dataset to see what makes it so pathological --
performance is nowhere near this bad for the binary search algorithm
with other sets of keys.
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] searchsorted() and memory cache
Charles R Harris wrote: On Wed, May 14, 2008 at 8:09 AM, Andrew Straw [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Quite a difference (a factor of about 3000)! At this point, I haven't delved into the dataset to see what makes it so pathological -- performance is nowhere near this bad for the binary search algorithm with other sets of keys. It can't be that bad Andrew, something else is going on. And 191 MB isn's *that* big, I expect it should bit in memory with no problem. I agree the performance difference seems beyond what one would expect due to cache misses alone. I'm at a loss to propose other explanations, though. Ideas? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] searchsorted() and memory cache
Aha, I've found the problem -- my values were int64 and my keys were uint64. Switching to the same data type immediately fixes the issue! It's not a memory cache issue at all. Perhaps searchsorted() should emit a warning if the keys require casting... I can't believe how bad the hit was. -Andrew Charles R Harris wrote: On Wed, May 14, 2008 at 2:00 PM, Andrew Straw [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Charles R Harris wrote: On Wed, May 14, 2008 at 8:09 AM, Andrew Straw [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Quite a difference (a factor of about 3000)! At this point, I haven't delved into the dataset to see what makes it so pathological -- performance is nowhere near this bad for the binary search algorithm with other sets of keys. It can't be that bad Andrew, something else is going on. And 191 MB isn's *that* big, I expect it should bit in memory with no problem. I agree the performance difference seems beyond what one would expect due to cache misses alone. I'm at a loss to propose other explanations, though. Ideas? I just searched for 2**25/10 keys in a 2**25 array of reals. It took less than a second when vectorized. In a python loop it took about 7.7 seconds. The only thing I can think of is that the search isn't getting any cpu cycles for some reason. How much memory is it using? Do you have any nans and such in the data? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] searchsorted() and memory cache
Thanks for all the comments on my original question. I was more offline than intended after I sent it until now, so I'm sorry I wasn't immediately able to participate in the discussion. Anyhow, after working on this a bit more, I came up with a few implementations of search algorithms doing just what I needed with the same interface available using bazaar and launchpad at http://launchpad.net/~astraw/+junk/fastsearch (MIT license). I have attached the output of the plot_comparisons.py benchmarking script to this email (note that this benchmarking is pretty crude). For the problem I originally wrote about, I get what a nearly unbelievable speedup of ~250x using the fastsearch.downsamp.DownSampledPreSearcher class, which is very similar in spirit to Charles' suggestion. It takes 1000 values from the original array to create a new first-level array that is itself localized in memory and points to a more localized region of the full original array. Also, I get a similar (though slightly slower) result using AVL trees using the fastsearch.avlsearch.AvlSearcher class, which uses pyavl ( http://sourceforge.net/projects/pyavl ). Using the benchmarking code included in the bzr branch, I don't get anything like this speedup (e.g. the attached figure), so I'm not sure exactly what's going on at this point, but I'm not going to argue with a 250x speedup, so the fastsearch.downsamp code is now being put to use in one of my projects. Stefan -- I think your code simply implements the classic binary search -- I don't see how it will reduce cache misses. Anyhow, perhaps someone will find the above useful. I guess it would still be a substantial amount of work to make a numpy-types-aware implementation of AVL trees or similar algorithms. These sorts of binary search trees seem like the right way to solve this problem and thus there might be an interesting project in this. I imagine that a numpy-types-aware Cython might make such implementation significantly easier and still blazingly fast compared to the binary search implemented in searchsorted() given today's cached memory architectures. -Andrew Andrew Straw wrote: I've got a big element array (25 million int64s) that searchsorted() takes a long time to grind through. After a bit of digging in the literature and the numpy source code, I believe that searchsorted() is implementing a classic binary search, which is pretty bad in terms of cache misses. There are several modern implementations of binary search which arrange items in memory such that cache misses are much more rare. Clearly making such an indexing arrangement would take time, but in my particular case, I can spare the time to create an index if searching was faster, since I'd make the index once but do the searching many times. Is there an implementation of such an algorithm that works easilty with numpy? Also, can you offer any advice, suggestions, and comments to me if I attempted to implement such an algorithm? Thanks, Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion inline: fastsearch.png___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] searchsorted() and memory cache
I've got a big element array (25 million int64s) that searchsorted() takes a long time to grind through. After a bit of digging in the literature and the numpy source code, I believe that searchsorted() is implementing a classic binary search, which is pretty bad in terms of cache misses. There are several modern implementations of binary search which arrange items in memory such that cache misses are much more rare. Clearly making such an indexing arrangement would take time, but in my particular case, I can spare the time to create an index if searching was faster, since I'd make the index once but do the searching many times. Is there an implementation of such an algorithm that works easilty with numpy? Also, can you offer any advice, suggestions, and comments to me if I attempted to implement such an algorithm? Thanks, Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] python memory use
Robin wrote: Hi, I am starting to push the limits of the available memory and I'd like to understand a bit better how Python handles memory... This is why I switched to 64 bit linux and never looked back. If I try to allocate something too big for the available memory I often get a MemoryError exception. However, in other situations, Python memory use continues to grow until the machine falls over. I was hoping to understand the difference between those cases. I don't know what falls over mean. It could be that you're getting swap death -- the kernel starts attempting to use virtual memory (hard disk) for some of the RAM. This would be characterized by your CPU use dropping to near-zero, your hard disk grinding away, and your swap space use increasing. The MemoryError simply means that Python made a request for memory that the kernel didn't grant. There's something else you might run into -- the maximum memory size of a process before the kernel kills that process. On linux i686, IIRC this limit is 3 GB. I'm not sure why you get different behavior on different runs. FWIW, with 64 bit linux the worst that happens to me now is swap death, which can be forestalled by adding lots of RAM. From what I've read Python never returns memory to the OS (is this right?) so the second case, python is holding on to memory that it isn't really using (for objects that have been destroyed). I guess my question is why doesn't it reuse the memory freed from object deletions instead of requesting more - and even then when requesting more, why does it continue until the machine falls over and not cause a MemoryError? It's hard to say without knowing what your code does. A first guess is that you're allocating lots of memory without allowing it to be freed. Specifically, you may have references to objects which you no longer need, and you should eliminate those references and allow them to be garbage collected. In some cases, circular references can be hard for python to detect, so you might want to play around with the gc module and judicious use of the del statement. Note also that IPython keeps references to past results by default (the history). While investigating this I found this script: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/511474 which does wonders for my code. I was wondering if this function should be included in Numpy as it seems to provide an important feature, or perhaps an entry on the wiki (in Cookbook section?) I don't think it belongs in numpy per se, and I'm not sure of the necessity of a spot on the scipy cookbook given that it's in the python cookbook. Perhaps more useful would be starting a page called MemoryIssues on the scipy wiki -- I imagine this subject, as a whole, is of particular interest for many in the numpy/scipy crowd. Certainly adding a link and description to that recipe would be useful in that context. But please, feel free to add to or edit the wiki as you see fit -- if you think something will be useful, by all means, go ahead and do it. I think there are enough eyes on the wiki that it's fairly self-regulating. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fast histogram
Hi Zach,
I have a similar loop which I wrote using scipy.weave. This was my first
foray into weave, and I had to dig through the intermediate C sources to
find the macros that did the indexing in the way I make use of here, but
this snipped may get you started. There are 2 functions, which each do
the same thing, but one is in python, the other is in C. Note that this
is for a 3D histogram -- presumably you could remove B and C from this
example.
I'm sure there are better (more documented) ways to do this using weave
-- but I had this code written, it works, and it appears it may be
useful to you... (Sorry it's not documented, however.)
-Andrew
Zachary Pincus wrote:
Hi,
How about a combination of sort, followed by searchsorted right/left
using the bin boundaries as keys? The difference of the two
resulting vectors is the bin value. Something like:
In [1]: data = arange(100)
In [2]: bins = [0,10,50,70,100]
In [3]: lind = data.searchsorted(bins)
In [4]: print lind[1:] - lind[:-1]
[10 40 20 30]
This won't be as fast as a c implementation, but at least avoids the
loop.
This is, more or less, what the current numpy.histogram does, no? I
was hoping to avoid the O(n log n) sorting, because the image arrays
are pretty big, and numpy.histogram doesn't get close to video rate
for me...
Perhaps, though, some of the slow-down from numpy.histogram is from
other overhead, and not the sorting. I'll try this, but I think I'll
probably just have to write the c loop...
Zach
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
from scipy import weave
import numpy
def increment_slow(final_array_shape, A,B,C):
counts = numpy.zeros(final_array_shape, dtype=numpy.uint64)
for i in range(len(A)):
a=A[i]; b=B[i]; c=C[i]
counts[a,b,c] += 1
return counts
def increment_fast(final_array_shape, idxa,idxb,idxc):
counts = numpy.zeros(final_array_shape, dtype=numpy.uint64)
assert len(idxa.shape)==1
assert len(idxa)==len(idxb)
assert len(idxa)==len(idxc)
code = r
for (int i=0; iNidxa[0]; i++) {
COUNTS3( IDXA1(i), IDXB1(i), IDXC1(i) )++;
}
weave.inline( code, ['counts', 'idxa','idxb','idxc'])
return counts
#increment = increment_slow
increment = increment_fast
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpyx.pyx (recent svn) works?
This is off-topic and should be directed to the pyrex/cython list, but since we're on the subject: I suppose the following is true, but let me ask, since I have not used Cython. Please correct me if I'm wrong. I have a bunch of pyrex compiled .pyx code. If I start adding some Cython compiled code, all the extensions will live happily without colliding, even if I don't re-compile my old .pyx files with Cython. (I can't imagine how they'd conflict, since they both translate to .c extensions without anything to collide. Or are there linker-level symbols that collide or something difficult?) -Andrew Fernando Perez wrote: On Tue, Apr 8, 2008 at 4:38 PM, Travis E. Oliphant [EMAIL PROTECTED] wrote: I say just add it. We should move forward with Cython. More important is to see if random actually builds with Cython right now. There was an issue that I recall from a few weeks ago that Cython could not build the pyrex extension in NumPy. OK, I'll play with random for a bit. BTW, the original 'bug' that started this thread is due to a change in Cython's casting behavior explained here: http://wiki.cython.org/DifferencesFromPyrex it's fixed with a simple extra (void *) cast as shown here: print 'Printing array info for ndarray at 0x%0lx'% \ (Py_intptr_tvoid *arr,) I just committed that code to the bzr branch. In summary: 1. Do you want me to obliterate the old numpy/doc/pyrex and replace it with numpy/doc/cython? That would make it clear what the tool to use is... 2. I'll work on random and report shortly. Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Compile Numpy in VC++8
Matthieu Brucher wrote: 2008/4/3, Chris Barker [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Robert Kern wrote: Just since that has been discussed a LOT, for years, I want to be clear: Different versions of Microsoft's compiler use different libraries for the standard C library. Some simple Python extension modules compiled with a different compiler than the Python interpreter will usually work. Do SOME modules USUALLY work just because of what routines in the standard library they happen to call? So it's just a matter of luck that a given module may not trigger a conflict between the two runtime libs? Some parts of the C interface are fixed by the standard. Rely on them and you should be OK (safe perhaps for some memory functions, but I never saw a problem there). The other parts should never be trusted (as the I/O stuff). These are the rules I follow. I have also had success compiling external code that required VS 2008 into a shared library (.dll) and then calling that using ctypes. I'm not sure if I was just lucky that this worked in my case, or if the windows linker can deal with the issues resulting from mixing C runtimes when using .dlls. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] image to array doubt
[EMAIL PROTECTED] wrote: Robin wrote I'm not sure why they would be doing this - to me it looks they might be using Image as a convenient way to store some other kind of data... thanks Robin, I am wondering if there is a more straightforward way to do these.. especially the vector to image function D Check out scipy.misc.pilutil.imread() and imsave() ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Median again
Considering that many of the statistical functions (mean, std, median) must iterate over all the data and that people (or at least myself) typically call them sequentially on the same data, it may make sense to make a super-function with less repetition. Instead of: x_mean = np.mean(x) x_median = np.median(x) x_std = np.std(x) x_min = np.min(x) x_max = np.max(x) We do: x_stats = np.get_descriptive_stats(x, stats=['mean','median','std','min','max'],axis=-1) And x_stats is a dictionary with 'mean','meadian','std','min', 'max' keys. The implementation could reduce the number of iterations over the data in this case. The implementation wouldn't have to be optimized initially, but could be gradually sped up once the interface is in place. I bring this up now to suggest such an idea as a more-general alternative to the medianwithaxis function proposed. What do you think? (Perhaps something like this already exists?) And, finally, this all surely belongs in scipy, but we already have stuff in numpy that can't be removed without seriously breaking backwards compatibility... -Andrew Matthew Brett wrote: Hi, median moved mediandim0 implementation of medianwithaxis or similar, with same call signature as mean. Deprecation warning for use of median, and return of mediandim0 for now. Eventual move of median to return medianwithaxis. This would confuse people even more, I'm afraid. First they're said that median() is deprecated, and then later on it becomes the standard function to use. I would actually prefer a short pain rather than a long one. I was thinking the warning could be something like: The current and previous version of numpy use a version of median that is not consistent with other summary functions such as mean. The calling convention of median will change in a future version of numpy to match that of the other summary functions. This compatible future version is implemented as medianwithaxis, and will become the default implementation of median. Please change any code using median to call medianwithaxis specifically, to maintain compatibility with future numpy APIs. I would certainly like median to take the axis keyword. The axis keyword (and its friends) could be added to 1.0.5 with the default being 1 instead of None, so that it keeps compatibility with the 1.0 API. Then, with 1.1 (an API-breaking release) the default can be changed to None to restore consistency with mean, etc. But that would be very surprising to a new user, and might lead to some hard to track down silent bugs at a later date. Matthew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] load movie frames in python?
I'm pretty sure there's code floating around the pyglet mailing list. I'd be happy to add it to http://code.astraw.com/projects/motmot/wiki/pygarrayimage if it seems reasonable. (pygarrayimage goes from numpy array to pyglet texture). Brian Blais wrote: On Jan 29, 2008, at Jan 29:8:24 PM, Andrew Straw wrote: I'd suggest pyglet's avbin library. great suggestion! I never would have thought to do that. Do you happen to know how to convert a player.texture into a numpy.array? there is ImageData, but I can't seem to figure out how to do the conversion. bb -- Brian Blais [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] http://web.bryant.edu/~bblais http://web.bryant.edu/%7Ebblais ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Is __array_interface__ supposed to work on numpy scalars?
Hi, I'm forwarding a bug from PyOpenGL. The developer, Mike Fletcher, is having troubles accessing a numpy scalar with the __array_interface__. Is this supposed to work? Or should __array_interface__ trigger an AttributeError on a numpy scalar? Note that I haven't done any digging on this myself... -Andrew ---BeginMessage--- Bugs item #1827190, was opened at 2007-11-06 17:35 Message generated for change (Comment added) made by mcfletch You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105988aid=1827190group_id=5988 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: GL Group: v3.0.0 Status: Pending Resolution: None Priority: 5 Private: No Submitted By: Chris Waters (crwaters) Assigned to: Mike C. Fletcher (mcfletch) Summary: Default (Numpy) array return values not accepted Initial Comment: The values returned by the default array type, numpy, are not accepted by OpenGL; more importantly, they are not accepted by the respective set/bind functions. This output is from the attached test file, run with the latest CVS revision: glGenTextures(1) - 1 (type 'long') glGenTextures(2) - [2 3] (list: type 'numpy.ndarray', elements: type 'numpy.uint32') Calling: glBindTexture(GL_TEXTURE_2D, 1) (created from glGenTextures(1)) No Exceptions Calling: glBindTexture(GL_TEXTURE_2D, 2) (created from glGenTextures(2), element 0) Exception Caught: argument 2: type 'exceptions.TypeError': wrong type The returned type of the array is numpy.ndarray, with each element having the type numpy.uint32. This element type is also not immediately convertable to a function argument type such as GLuint. The return type of glGenTextures(1), however, is of the type long due to the special-case functionality. This is not the case for functions that do not handle special cases similar to this, such as OpenGL.GL.EXT.framebuffer_object.glGenFramebuffersEXT A quick global work-around is to change the array type to ctypes after importing OpenGL: from OpenGL.arrays import formathandler formathandler.FormatHandler.chooseOutput( 'ctypesarrays' ) -- Comment By: Mike C. Fletcher (mcfletch) Date: 2008-01-07 13:32 Message: Logged In: YES user_id=34901 Originator: NO I've added an optional flag to the top-level module which allows for using numpy scalars. ALLOW_NUMPY_SCALARS which when true allows your test case to succeed. Coded in Python, however, it's rather slow (and rather poorly implemented). I looked into implementing this using the __array_interface__ on scalars, but the data-pointer there appears to be randomly generated. Without that, a conversion at the Python level and then passing onto the original function seems the only solution. I doubt we'll get a *good* solution to this in the near term. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105988aid=1827190group_id=5988 - Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace ___ PyOpenGL Homepage http://pyopengl.sourceforge.net ___ PyOpenGL-Devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/pyopengl-devel ---End Message--- ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is __array_interface__ supposed to work on numpy scalars?
Travis E. Oliphant wrote: Andrew Straw wrote: Hi, I'm forwarding a bug from PyOpenGL. The developer, Mike Fletcher, is having troubles accessing a numpy scalar with the __array_interface__. Is this supposed to work? Or should __array_interface__ trigger an AttributeError on a numpy scalar? Note that I haven't done any digging on this myself... Yes, the __array_interface__ approach works (but read-only). In fact, what happens is that a 0-d array is created and you are given the information for it. Thanks. How is the array reference kept alive? Maybe that's Mike's issue? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] parallel numpy (by Brian Granger) - any info?
dmitrey wrote: The only one thing I'm very interested in for now - why the most simplest matrix operations are not implemented to be parallel in numpy yet (for several-CPU computers, like my AMD Athlon X2). For what it's worth, sometimes I *want* my numpy operations to happen only on one core. (So that I can do more important stuff with the others.) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array)
I have added a page to the wiki describing this issue: http://scipy.org/numpy_warts_and_gotchas I'll link it into the main documentation pages over the next few days, but I ask for a review the following text for correctness and clarity: (You can simply edit the wiki page or post your reply here and I'll do it.) Like most of numpy, allclose() uses the broadcasting rules when performing its operation. This leads to the following behavior: a=32 b=numpy.array([]) numpy.allclose(a,b) True Upon closer inspection, we can see that the broadcasting rules cause a to become a zero-dimensional array like b. The default truth value of a zero-dimensional array is True, so the following holds and illustrates how the above result is consistent with numpy's rules. Andrew Straw wrote: Apologies if I've missed the discussion of this, but I was recently surprised by the following behavior (in svn trunk 4673). The following code runs without triggering the assertion. import numpy as np print np.__version__ a=np.int32(42) b=np.array([],dtype=np.int32) assert np.allclose(a,b) Is this expected behavior of numpy or is this a bug I should report? Thanks, Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array)
Thanks, I updated the page. Charles R Harris wrote: On Jan 4, 2008 12:27 PM, Andrew Straw [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I have added a page to the wiki describing this issue: http://scipy.org/numpy_warts_and_gotchas I'll link it into the main documentation pages over the next few days, but I ask for a review the following text for correctness and clarity: (You can simply edit the wiki page or post your reply here and I'll do it.) Like most of numpy, allclose() uses the broadcasting rules when performing its operation. This leads to the following behavior: a=32 b=numpy.array([]) numpy.allclose(a,b) True Upon closer inspection, we can see that the broadcasting rules cause a to become a zero-dimensional array like b. The default truth value of a It is not the dimensions, it's the fact that the array is empty, so that anything said about it's non-existent elements is true, i.e., x in empty - anything is always true because x in empty is always false. So we also have the following: In [1]: allclose(32,[[]]) Out[1]: True The other reason is that because of broadcasting, the shape of the arrays may be immaterial. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] unexpected behavior with allclose( scalar, empty array)
Apologies if I've missed the discussion of this, but I was recently surprised by the following behavior (in svn trunk 4673). The following code runs without triggering the assertion. import numpy as np print np.__version__ a=np.int32(42) b=np.array([],dtype=np.int32) assert np.allclose(a,b) Is this expected behavior of numpy or is this a bug I should report? Thanks, Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] unexpected behavior with allclose( scalar, empty array)
Matthew Brett wrote: So, currently we have all and allclose giving the same answer: In [19]: a = array([]) In [20]: b = array([1]) In [21]: all(a == b) Out[21]: True In [22]: allclose(a, b) Out[22]: True Would we want the answers to be different? No. I wasn't thinking correctly, previously. It's unfortunate that all of us immediately think the given answer is wrong. Maybe we need allclose_sameshape() (my ability to name stuff is terrible, but you get the idea). Regardless of the name issue, I have no idea how this is viewed against the no-namespace-bloat principle. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] where construct
or is logical or. You want | which is bitwise/elementwise or. Also, watch the order of operations -- | has higher precedence than . Thus, you want where( (a1) | (b3), b,c) Ross Harder wrote: What's the correct way to do something like this? a=array( (0,1,1,0) ) b=array( (4,3,2,1) ) c=array( (1,2,3,4) ) where( (a1 or b3), b,c) Python throws a ValueError I would expect to get an array that looks like [4,2,2,1] I think Thanks, Ross Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
An idea that occurred to me after reading Fernando's email. A function could be called at numpy import time that specifically checks for the instruction set on the CPU running and makes sure that is completely covers the instruction set available through all the various calls, including to BLAS. If this kind of thing were added, numpy could fail with a loud warning rather than dying with mysterious errors later on. The trouble would seem that you can switch your BLAS shared library without re-compiling numpy, so numpy would have to do a run-time query of ATLAS, etc. for compilation issues. Which is likely library-dependent, and furthermore, not having looked into BLAS implementations, I'm not sure that (m)any of them provide such information. Do they? Is this idea technically possible? -Andrew Fernando Perez wrote: On Dec 10, 2007 4:41 PM, Robert Kern [EMAIL PROTECTED] wrote: The current situation is untenable. I will gladly accept a slow BLAS for an official binary that won't segfault anywhere. We can look for a faster BLAS later. Just to add a note to this: John Hunter and I just finished teaching a python workshop here in Boulder, and one attendee had a recurring all-out crash on WinXP. Eventually John was able to track it to a bad BLAS call, but the death was an 'illegal instruction'. We then noticed that this was on an older Pentium III laptop, and I'd be willing to bet that the problem is an ATLAS compiled with SSE2 support. The PIII chip only has plain SSE, not SSE2, and that's the kind of crash I've seen when accidentally running code compiled in my office machine (a P4) on my laptop (a similarly old PIII). It may very well be that it's OK to ship binaries with ATLAS, but just to build them without any fancy instruction support (no SSE, SSE2 or anything else of that kind, just plain x87 code). Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
According to the QEMU website, QEMU does not (yet) emulate SSE on x86 target, so a Windows installation on a QEMU virtual machine may be a good way to build binaries free of these issues. http://fabrice.bellard.free.fr/qemu/qemu-tech.html -Andrew Travis E. Oliphant wrote: Fernando Perez wrote: On Dec 10, 2007 4:41 PM, Robert Kern [EMAIL PROTECTED] wrote: The current situation is untenable. I will gladly accept a slow BLAS for an official binary that won't segfault anywhere. We can look for a faster BLAS later. Just to add a note to this: John Hunter and I just finished teaching a python workshop here in Boulder, and one attendee had a recurring all-out crash on WinXP. Eventually John was able to track it to a bad BLAS call, but the death was an 'illegal instruction'. We then noticed that this was on an older Pentium III laptop, and I'd be willing to bet that the problem is an ATLAS compiled with SSE2 support. The PIII chip only has plain SSE, not SSE2, and that's the kind of crash I've seen when accidentally running code compiled in my office machine (a P4) on my laptop (a similarly old PIII). It may very well be that it's OK to ship binaries with ATLAS, but just to build them without any fancy instruction support (no SSE, SSE2 or anything else of that kind, just plain x87 code). I think this is what the default should be (but plain SSE allowed). However, since I have moved, the machine I was using to build official binaries has switched and that is probably at the core of the problem. Also, I've tried to build ATLAS 3.8.0 without SSE without success (when I'm on a machine that has it). It would be useful to track which binaries are giving people problems as I built the most recent ones on a VM against an old version of ATLAS (3.6.0) that has been compiled on windows for a long time. I'm happy to upload a better binary of NumPy (if I can figure out which one is giving people grief and how to create a decent one). -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Loading a GB file into array
Hi all, I haven't done any serious testing in the past couple years, but for this particular task -- drawing frames using OpenGL without ever skipping a video update -- it is my impression that as of a few Ubuntu releases ago (Edgy?) Windows still beat linux. Just now, I have investigated on 2.6.22-14-generic x86_64 as pacakged by Ubuntu 7.10, and I didn't skip a frame out of 1500 at 60 Hz. That's not much testing, but it is certainly better performance than I've seen in the recent past, so I'll certainly be doing some more testing soon. Oh, how I'd love to never be forced to use Windows again. Leaving my computer displaying moving images overnight, (and tomorrow at lab on a 200 Hz display), Andrew Gael Varoquaux wrote: On Tue, Dec 04, 2007 at 02:13:53PM +0900, David Cournapeau wrote: With recent kernels, you can get really good latency if you do it right (around 1-2 ms worst case under high load, including high IO pressure). As you can see on my page, I indeed measured less than 1ms latency on Linux under load with kernel more than a year old. These things have gotten much better recently and with a premptible kernel you should be able to get 1ms easily. Going below 0.5ms without using a realtime OS (ie a realtime kernel, under linux) is really pushing it. Cheers, Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] OT: A Way to Approximate and Compress a 3DSurface
Christopher Barker wrote: For data interpolation: 2D-Delaunay triangulation based method (I think you can find one in the scipy cookbook). yup -- but then you need the decimation to remove the unneeded points. I don't think Scipy has that. The sandbox does, thanks to Robert Kern. (And I should really submit a patch to move this into the main scipy.) http://www.scipy.org/Cookbook/Matplotlib/Gridding_irregularly_spaced_data ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C-API Documentation?
Thomas Schreiner wrote: Am I doing anything wrong in this program? It's crashing immediately after the before line, using Borland C++ Builder 6 and numpy-1.0.3.1.win32-py2.4. You have to call import_array() before using the C API. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Pickle, pytables, and sqlite - loading and saving recarray's
Gael Varoquaux wrote: On Thu, Jul 19, 2007 at 09:42:42PM -0500, Vincent Nijs wrote: I'd luv to hear from people using sqlite, pytables, and cPickle about their experiences. I was about to point you to this discussion: http://projects.scipy.org/pipermail/scipy-user/2007-April/011724.html but I see that you participated in it. I store data from each of my experimental run with pytables. What I like about it is the hierarchical organization of the data which allows me to save a complete description of the experiment, with strings, and extensible data structures. Another thing I like is that I can load this in Matlab (I can provide enhanced script for hdf5, if somebody wants them), and I think it is possible to read hdf5 in Origin. I don't use these software, but some colleagues do. I want that Matlab script! I have colleagues with whom the least common denominator is currently .mat files. I'd be much happier if it was hdf5 files. Can you post it on the scipy wiki cookbook? (Or the pytables wiki?) Cheers! Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Unhandled floating point exception running test in numpy-1.0.3 and svn 3875
john, there was a bug that made it into debian sarge whereby a SIGFPE wasn't trapped in the appropriate place and ended up causing problems similar to what you describe. the difficulty in debugging is that you're after whatever triggers the FPE in the first place (or the bug that lets it go untrapped), but you only get notified when the kernel kills your program for not trapping it. I wrote a little page about the Debian sarge issue: http://code.astraw.com/debian_sarge_libc.html John Ollinger wrote: rex rex at nosyntax.com writes: There doesn't appear to be a problem with recent versions of the software. In particular, ATLAS 3.7.33 does not cause an error. Is there some reason for you to use such old software? (gcc 3.3.1 kernel 2.4.21)? What platform are you building for? -rex I think have resolved this problem. First off, I building on such an old system because the applications I write are used at a number of labs across the campus and most of these labs are in nontechnical departments with minimal linux support. As a result, the usually run the version that was current when the machine was purchased. If I want people to use my code, I have to supply them with a tar file that contains everything they need, including the dependencies for numpy, scipy, wxPython, vtk and itk. I try to make a general build on the oldest version I have access to in case I miss something. The motherboard on my desktop died last week, so I was forced to use the older system for a couple of weeks, which is what prompted me to update numpy from numeric. The floating exception is definitely not caused by the numpy or scipy builds. The same builds run correctly on one of our newer systems (2.6.9). I rebuilt everything on my desktop (including gcc. The new box on my desk is now running 2.6.28 with gcc 4.1, so I had to build gcc 3.6 anyway). The new build has the Floating point exception, but in a different, later test (three tests after the matvec test). Then I rebuilt a new version of gcc (3.6 rather than 3.3 and built numpy again. The floating point exception still occurred but this time at the cdouble test, the third from last. The fact that the build runs find with nonoptimized lapack libraries made me wonder about the threading support. I found an article at http://www.ibm.com/developerworks/eserver/library/es-033104.html which said that SUSE 2.4.21 used a backported version of the new threading package in version 2.6. The exceptions always occur on complex operations, so it isn't a stretch to assume that threads are in play when they occur. John p.s. My desktop is now running selinux, which is denying access to the numpy shareable libries. The error message is cannot restore segment prot after reloc. The numpy setup script should probably set the context for the libraries. I am going to post this on a separate thread since other people will probably be encountering it. For those googling this, the command is chcon -t texrel_shlib_t filename ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] average of array containing NaN
Giorgio F. Gilestro wrote: I find myself in a situation where an array may contain not-Numbers that I set as NaN. Yet, whatever operation I do on that array( average, sum...) will threat the NaN as infinite values rather then ignoring them as I'd like it'd do. Am I missing something? Is this a bug or a feature? :-) You may be interested in masked arrays. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] annoying numpy string to float conversion behaviour
Torgil Svensson wrote: OS-specific routines (probably the c-library, haven't looked). I think python should be consistent regarding this across platforms but I don't know if different c-libraries generates different strings for special numbers. Anyone? Windows and Linux certainly generate different strings for special numbers from current Python, and I guess the origin is the libc on those platforms. But, as Python is moving away from the libc for file IO in Python 3K, perhaps string representation of floats would be considered, too. (In fact for all I know, perhaps it has already been considered.) Maybe you should email the python-3k-dev list? -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] VMWare Virtual Appliance of Ubuntu with numpy, scipy, matplotlib, and ipython available
This is a note to announce the availability of a VMWare Virtual Appliance with Ubuntu linux with numpy, scipy, matplotlib, and ipython installed. This should make it relatively easy to try out the software. The VMWare Player and VMWare Server are available for no cost from http://www.vmware.com/products/player/ and http://www.vmware.com/products/server/ The download URL is: http://mosca.caltech.edu/outgoing/Ubuntu%207.04%20for%20scientific%20computing%20in%20Python.zip The username is ubuntu and the password is abc123. The network will share the host's interface using NAT. The md5sum is 4191e13abda1154c94e685ffdc0f829b. I have updated http://scipy.org/Download with this information. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Vista installer?
OK, I have placed an Ubuntu 7.04 image with stock numpy, scipy, matplotlib, and ipython at http://mosca.caltech.edu/outgoing/Ubuntu%207.04%20for%20scientific%20computing%20in%20Python.zip The md5sum is 4191e13abda1154c94e685ffdc0f829b. Note: I haven't tested this at all on any computer other than the one which I created the virtual appliance on. I'm posting now because it will take me a while to download the file (it's 1 GB) in order to test. -Andrew Andrew Straw wrote: Hi Ryan, I use VMware server on my linux box to host several more linux images. I will see if I can whip you up a Ubuntu Feisty i386 image with the big 4 - numpy/scipy/matplotlib/ipython. If I understand their docs correctly, I have virtual appliances for previously existing images already... If it's that easy, this should take just a few minutes. I'll see what I can do and post the results. -Andrew Ryan Krauss wrote: I need to plot things using matplotlib, so I don't think it works for me without X. Thanks though. Ryan On 5/28/07, David Cournapeau [EMAIL PROTECTED] wrote: Ryan Krauss wrote: I have this more or less working, the only problem is that my guest Ubuntu OS doesn't have Scipy/Numpy/IPython and I can't get basic networking. So, I can't install anything. Does anyone have a VMWare virtual appliance with Scipy/Numpy/IPython/Matplotlib already installed? I posted a question to the VMWare forum about networking, but will welcome any help from here. I use vmware to test various packages: it is a barebone version (only command line), dunno if this is of any interest for you ? David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Vista installer?
Andrew Straw wrote: OK, I have placed an Ubuntu 7.04 image with stock numpy, scipy, matplotlib, and ipython at http://mosca.caltech.edu/outgoing/Ubuntu%207.04%20for%20scientific%20computing%20in%20Python.zip The md5sum is 4191e13abda1154c94e685ffdc0f829b. Note: I haven't tested this at all on any computer other than the one which I created the virtual appliance on. I'm posting now because it will take me a while to download the file (it's 1 GB) in order to test. After downloading the 1GB, it does seems to work on my laptop. (On a Mac running VMware Fusion, even! Sweet! I thought this job would require booting my desktop.) The username is ubuntu and the password is abc123. I set the network up to share the host's interface with NAT. Hmm, as David suggests, this might be a pretty good way to make an easy-to-try scipy environment, particularly for Windows users. I'm happy to continue to (allow Caltech to) host this, but also happy if we move it, or an improved version, somewhere else. For example, I could imagine wanting auto login turned on and some kind of icon on the desktop that says click here for interactive python prompt. -Andrew -Andrew Andrew Straw wrote: Hi Ryan, I use VMware server on my linux box to host several more linux images. I will see if I can whip you up a Ubuntu Feisty i386 image with the big 4 - numpy/scipy/matplotlib/ipython. If I understand their docs correctly, I have virtual appliances for previously existing images already... If it's that easy, this should take just a few minutes. I'll see what I can do and post the results. -Andrew Ryan Krauss wrote: I need to plot things using matplotlib, so I don't think it works for me without X. Thanks though. Ryan On 5/28/07, David Cournapeau [EMAIL PROTECTED] wrote: Ryan Krauss wrote: I have this more or less working, the only problem is that my guest Ubuntu OS doesn't have Scipy/Numpy/IPython and I can't get basic networking. So, I can't install anything. Does anyone have a VMWare virtual appliance with Scipy/Numpy/IPython/Matplotlib already installed? I posted a question to the VMWare forum about networking, but will welcome any help from here. I use vmware to test various packages: it is a barebone version (only command line), dunno if this is of any interest for you ? David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Vista installer?
Hi Ryan, I use VMware server on my linux box to host several more linux images. I will see if I can whip you up a Ubuntu Feisty i386 image with the big 4 - numpy/scipy/matplotlib/ipython. If I understand their docs correctly, I have virtual appliances for previously existing images already... If it's that easy, this should take just a few minutes. I'll see what I can do and post the results. -Andrew Ryan Krauss wrote: I need to plot things using matplotlib, so I don't think it works for me without X. Thanks though. Ryan On 5/28/07, David Cournapeau [EMAIL PROTECTED] wrote: Ryan Krauss wrote: I have this more or less working, the only problem is that my guest Ubuntu OS doesn't have Scipy/Numpy/IPython and I can't get basic networking. So, I can't install anything. Does anyone have a VMWare virtual appliance with Scipy/Numpy/IPython/Matplotlib already installed? I posted a question to the VMWare forum about networking, but will welcome any help from here. I use vmware to test various packages: it is a barebone version (only command line), dunno if this is of any interest for you ? David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Vista installer?
Charles R Harris wrote: On 5/24/07, *Ryan Krauss* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I am trying to use Numpy/Scipy for a class I am teaching this summer. I have one student running Vista. Is there an installer that works for Vista? Running the exe file from webpage gives errors about not being able to create various folders and files. I think this is from Vista being very restrictive about which files and folders are writable. Is anyone out there running Numpy/Scipy in Vista? If so, how did you get it to work? Install Ubuntu? ;) I've heard nothing but complaints and nasty words from co-workers stuck with new computers and trying to use Vista as a development platform for scientific work. Just a follow-up, vmware is now given away and Ubuntu has always been free, and I guess most computers with Vista have native support for virtualization. So, this way you could run Ubuntu and Vista simultaneously without a speed loss on either or partitioning the drive. Of course, if it were me I'd put Ubuntu as the host OS and probably never boot the guest OS. ;) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy array sharing between processes?
Charles R Harris wrote: I'll pitch in a few donuts (and my eternal gratitude) for an example of shared memory use using numpy arrays that is cross platform, or at least works in linux, mac, and windows. I wonder if you could mmap a file and use it as common memory? Yes, that's the basic idea. Now for the example that works on those platforms... Forking in python under linux leads to copies because anything that accesses an object changes its reference count. I'm not sure what you're trying to say here. If it's shared memory, it's not copied -- that's the whole point. I don't really care how I spawn the multiple processes, and indeed forking is one way. Pipes are easy and could be used for synchronization. True. But they're not going to be very fast. (I'd like to send streams of realtime images between different processes.) Would python threading work for you? That's what I use now and what I'd like to get away from because 1) the GIL sucks and 2) (bug-free) threading is hard. -Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] bug in http://www.scipy.org/NumPy_for_Matlab_Users
No, the nth index of a Python sequence is a[n], where n starts from zero. Thus, if I want the nth dimension of array a, I want a.shape[n]. I reverted the page to its original form and added a couple explanatory comments about zero vs one based indexing. dmitrey wrote: now there is MATLABNDArray Matrix size(a,n) a.shape[n] a.shape[n] but it should be size(a,n) a.shape[n-1] a.shape[n-1] WBR, D. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Python issue of Computing in Science and Engineering available
The May/June issue of Computing in Science and Engineering http://computer.org/cise: is out and has a Python theme. Many folks we know and love from the community and mailing lists contribute to the issue. Read articles by Paul Dubois and Travis Oliphant for free online. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] building numpy with atlas on ubuntu edgy
rex wrote: Keith Goodman [EMAIL PROTECTED] [2007-04-18 10:49]: I'd like to compile atlas so that I can take full advantage of my core 2 duo. If your use is entirely non-commercial you can use Intel's MKL with built-in optimized BLAS and LAPACK and avoid the need for ATLAS. Just to clarify, my understanding is that if you buy a developer's license, you can also use it for commercial use, including distributing binaries. (Otherwise it would seem kind of silly for Intel to invest so much in their performance libraries and compilers.) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] building numpy with atlas on ubuntu edgy
Christian K wrote: David Cournapeau wrote: On Ubuntu and debian, you do NOT need any site.cfg to compile numpy with atlas support. Just install the package atlas3-base-dev, and you are done. The reason is that when *compiling* a software which needs atlas, the linker will try to find libblas.so in /usr/lib, not in /usr/lib/sse2. If you install atlas3-base-dev, the package will install those at the correct locations. I have updated the instructions for Ubuntu (also works for debian) on the wiki a few days ago: Indeed, installing atlas3-base-dev helps. I only had atlas3-base, atlas3-sse and atlas3-sse2-dev installed. Sorry for the noise. Hmm, atlas3-sse2-dev doesn't depend on atlas3-base-dev? That sounds like a bug... Off to investigate and possibly file a bug report with Debian or Ubuntu, Andrew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Nonblocking Plots with Matplotlib
Bill Baxter wrote: On 3/15/07, Bill Baxter [EMAIL PROTECTED] wrote: Thanks, Sebastian. I'll take a look at Pyro. Hadn't heard of it. I'm using just xmlrpclib with pickle right now. I took a look at Pyro -- it looks nice. The only thing I couldn't find, though, is how decouple the wx GUI on the server side from the Pyro remote call handler. Both wx and Pyro want to run a main loop. With the XML-RPC, I could use twisted and its wxreactor class. That does all the necessary magic under the hood to run both loops. Basically all you have to do to make it work is: class MainApp(wx.App, twisted.web.xmlrpc.XMLRPC): ... twisted.internet.wxreactor.install() app = MainApp() twisted.internet.reactor.registerWxApp(app) twisted.internet.reactor.run() And then you're good to go. reactor.run() takes care of both main loops somehow. Do you know of any good examples showing how to do that sort of thing with Pyro? It must be possible. I mean it's the exact same sort of thing you'd need if you're writing a simple GUI internet chat program. My googling has turned up nothing, though. It is possible to do this with Pyro. I think Pyro can auto-background (my terminology not theirs) if you allow it to do threading. Otherwise, you could handle requests by 1) putting Pyro in a thread you manage or 2) a GUI timer calling Pyro's daemon.handleRequests(...) fairly often. But I hear nothing but good things about Twisted, and your code works, so I say find some bigger fish to fry (especially since I can see potential issues with each of the above options...). ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Matplotlib-users] Nonblocking Plots with Matplotlib
Bill, very cool. Also, thanks for showing me how Twisted can be used like Pyro, more-or-less, I think. (If I understand your code from my 1 minute perusal.) On Mac OS X, there's one issue I don't have time to follow any further: sys.executable points to /Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/Contents/MacOS/Python whereas /Library/Frameworks/Python.framework/Versions/Current/bin/python is the file actually on my path. For some reason, when I run the latter ezplot is found, when the former, it is not. Thus, your auto-spawning of a plotserver instance fails on my installation. Other than that, the example you gave works as advertised and looks great. (Ohh, those anti-aliased lines look better and better the more I suffer through my colleagues' aliased plots...) Bill Baxter wrote: Howdy Folks, I was missing the good ole days of using Matlab back at the Uni when I could debug my code, stop at breakpoints and plot various data without fear of blocking the interpreter process. Using ipython -pylab is what has been suggested to me in the past, but the problem is I don't do my debugging from ipython. I have a very nice IDE that works very well, and it has a lovely interactive debugging prompt that I can use to probe my code when stopped at a breakpoint. It's great except I can't really use matplotlib for debugging there because it causes things to freeze up. So I've come up with a decent (though not perfect) solution for quickie interactive plots which is to run matplotlib in a separate process. I call the result it 'ezplot'. The first alpha version of this is now available at the Cheeseshop. (I made an egg too, so if you have setuptools you can do easy_install ezplot.) The basic usage is like so: In [1]: import ezplot In [2]: p = ezplot.Plotter() In [3]: p.plot([1,2,3],[1,4,9],marker='o') Connecting to server... waiting... connected to plotserver 0.1.0a1 on http://localhost:8397 Out[3]: True In [4]: from numpy import * In [5]: x = linspace(-5,5,20) In [13]: p.clf() Out[13]: True In [14]: p.plot(x, x*x*log(x*x+0.01)) (Imagine lovely plots popping up on your screen as these commands are typed.) The only return values you get back are True (success...probably) or False (failure...for sure). So no fancy plot object manipulation is possible. But you can do basic plots no problem. The nice part is that this (unlike ipython's built-in -pylab threading mojo) should work just as well from wherever you're using python. Whether it's ipython (no -pylab) or Idle, or a plain MS-DOS console, or WingIDE's debug probe, or SPE, or a PyCrust shell or whatever. It doesn't matter because all the client is doing is packing up data and shipping over a socket. All the GUI plotting mojo happens in a completely separate process. There are plenty of ways this could be made better, but for me, for now, this probably does pretty much all I need, so it's back to Real Work. But if anyone is interested in making improvements to this, let me know. Here's a short list of things that could be improved: * Right now I assume use of the wxAGG backend for matplotlib. Don't know how much work it would be to support other back ends (or how to go about it, really). wxAGG is what I always use. * Returning more error/exception info from the server would be nice * Returning full fledged proxy plot objects would be nice too, but I suspect that's a huge effort * SOAP may be better for this than xmlrpclib but I just couldn't get it to work (SOAPpy + Twisted). * A little more safety would be nice. Anyone know how to make a Twisted xmlrpc server not accept connections from anywhere except localhost? * There's a little glitch in that the spawned plot server dies with the parent that created it. Maybe there's a flag to subprocess.Popen to fix that? * Sometimes when you click on Exit Server, if there are plot windows open it hangs while shutting down. Only tested on Win32 but there's nothing much platform specific in there. Give it a try and let me know what you think! --bb - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fortran order arrays to and from numpy arrays
Alexander Schmolck wrote: 2. Despite this overhead, copying around large arrays (e.g. =1e5 elements) in above way causes notable additional overhead. Whilst I don't think there's a sane way to avoid copying by sharing data between numpy and matlab the copying could likely be done better. Alex, what do you think about hybrid arrays? http://www.mail-archive.com/numpy-discussion@lists.sourceforge.net/msg03748.html ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Greek Letters
Robert Kern wrote: On Windows, you may be out of luck. I don't know of any fully-Unicode-capable terminal. The lack of a decent console application is one of the most problematic issues I face whenever attempting to do serious programming in Windows. I wish I knew of a better terminal program. Here's one that seems like it might work, but I haven't tried it yet: http://software.jessies.org/terminator ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Profiling numpy ? (parts written in C)
I added a ticket for Francesc's enhancement: http://projects.scipy.org/scipy/numpy/ticket/403 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
