Re: [Numpy-discussion] python import question
On Fri, May 18, 2012 at 5:49 PM, Chao YUE wrote: > Previously I have installed numpy 1.5.1. and then I used pip install > --upgrade numpy > to install numpy 1.6.1 > Why was the old 1.5.1 installation in /usr/lib/pymodules/python2.7? I have in the past used 'pip uninstall package' a couple of times in a row in order to remove old versions of packages, then run 'pip install package'. Old packages and pip seem to be a problem if you have used 'easy_install' to install the older packages. Kindest regards, Tim ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] python import question
On Friday, May 18, 2012, Chao YUE wrote: > Dear all, > > This is only a small python import question. I think I'm right but just > want some confirmation. > > Previously I have installed numpy 1.5.1. and then I used pip install > --upgrade numpy > to install numpy 1.6.1 > > But when I try to import numpy as np within ipython shell, I still get the > version 1.5.1 > > then I checked my sys.path: > > In [21]: sys.path > Out[21]: > ['', > '/usr/local/bin', > '/usr/local/lib/python2.7/dist-packages/pupynere-1.0.15-py2.7.egg', > '/usr/lib/pymodules/python2.7', > > '/usr/local/lib/python2.7/dist-packages/scikits.statsmodels-0.3.1-py2.7.egg', > > '/usr/local/lib/python2.7/dist-packages/Shapely-1.2.13-py2.7-linux-i686.egg', > > '/usr/local/lib/python2.7/dist-packages/pandas-0.7.3-py2.7-linux-i686.egg', > '/home/chaoyue/python/python_lib', > '/usr/lib/python2.7', > '/usr/lib/python2.7/plat-linux2', > '/usr/lib/python2.7/lib-tk', > '/usr/lib/python2.7/lib-old', > '/usr/lib/python2.7/lib-dynload', > '/usr/local/lib/python2.7/dist-packages', > '/usr/lib/python2.7/dist-packages', > '/usr/lib/python2.7/dist-packages/PIL', > '/usr/lib/pymodules/python2.7/gtk-2.0', > '/usr/lib/python2.7/dist-packages/gst-0.10', > '/usr/lib/python2.7/dist-packages/gtk-2.0', > '/usr/lib/pymodules/python2.7/ubuntuone-client', > '/usr/lib/pymodules/python2.7/ubuntuone-control-panel', > '/usr/lib/pymodules/python2.7/ubuntuone-storage-protocol', > '/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode', > '/usr/local/lib/python2.7/dist-packages/IPython/extensions'] > > Actually I found I have numpy 1.5.1 in /usr/lib/pymodules/python2.7 > > and numpy 1.6.1 in /usr/local/lib/python2.7/dist-packages/numpy/ > > but because the first path is before the second one in sys.path, so > ipython imports only the first one and ignore the second one. > Then I delete the directory of /usr/lib/pymodules/python2.7/numpy and redo > the import, I get the version 1.6.1 > > This means that import will try to find the first occurrence of the module > and will ignore the ones with same name in later occurrences? > > cheers, > > Chao > > Yes. This is actually very common. The $PATH environment variable works the same way for finding executables. Ben Root > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
The best way to keep in the loop is to comment on this list and pay attention to threads that discuss it.Thank you for speaking up, as I was aware of your significant use of the current masked array in NumPy, but it is good when you can articulate your use-cases and APIs that are helpful or annoying to you. -Travis On May 18, 2012, at 4:54 PM, Doutriaux, Charles wrote: > Travis, > > We have a significant user base for masked arrays, with a lot of > "real-life" experience, use-cases and data. > > We would really like to get involved on this, please keep us in the loop. > > C. > > > On 5/18/12 2:47 PM, "Travis Oliphant" wrote: > >> Hey all, >> >> After reading all the discussion around masked arrays and getting input >> from as many people as possible, it is clear that there is still >> disagreement about what to do, but there have been some fruitful >> discussions that ensued. >> >> This isn't really new as there was significant disagreement about what to >> do when the masked array code was initially checked in to master. So, >> in order to move forward, Mark and I are going to work together with >> whomever else is willing to help with an effort that is in the spirit of >> my third proposal but has a few adjustments. >> >> The idea will be fleshed out in more detail as it progresses, but the >> basic concept is to create an (experimental) ndmasked object in NumPy 1.7 >> and leave the actual ndarray object unchanged. While the details need >> to be worked out here, a goal is to have the C-API work with both >> ndmasked arrays and arrayobjects (possibly by defining a base-class >> C-level structure that both ndarrays inherit from). This might also >> be a good way for Dag to experiment with his ideas as well but that is >> not an explicit goal. >> >> One way this could work, for example is to have PyArrayObject * be the >> base-class array (essentially the same C-structure we have now with a >> HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject >> * as well but add more members to the C-structure. I think this is >> the easiest thing to do and requires the least amount of code-change. >> It is also possible to define an abstract base-class PyArrayObject * >> that both ndarray and ndmasked inherit from. That way ndarray and >> ndmasked are siblings even though the ndarray would essentially *be* the >> PyArrayObject * --- just with a different type-hierarchy on the python >> side. >> >> This work will take some time and, therefore, I don't expect 1.7 to be >> released prior to SciPy Austin with an end of June target date. The >> timing will largely depend on what time is available from people >> interested in resolving the situation. Mark and I will have some >> availability for this work in June but not a great deal (about 2 >> man-weeks total between us).If there are others who can step in and >> help, it will help accelerate the process. >> >> Best regards, >> >> -Travis >> >> >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
Travis, We have a significant user base for masked arrays, with a lot of "real-life" experience, use-cases and data. We would really like to get involved on this, please keep us in the loop. C. On 5/18/12 2:47 PM, "Travis Oliphant" wrote: >Hey all, > >After reading all the discussion around masked arrays and getting input >from as many people as possible, it is clear that there is still >disagreement about what to do, but there have been some fruitful >discussions that ensued. > >This isn't really new as there was significant disagreement about what to >do when the masked array code was initially checked in to master. So, >in order to move forward, Mark and I are going to work together with >whomever else is willing to help with an effort that is in the spirit of >my third proposal but has a few adjustments. > >The idea will be fleshed out in more detail as it progresses, but the >basic concept is to create an (experimental) ndmasked object in NumPy 1.7 >and leave the actual ndarray object unchanged. While the details need >to be worked out here, a goal is to have the C-API work with both >ndmasked arrays and arrayobjects (possibly by defining a base-class >C-level structure that both ndarrays inherit from). This might also >be a good way for Dag to experiment with his ideas as well but that is >not an explicit goal. > >One way this could work, for example is to have PyArrayObject * be the >base-class array (essentially the same C-structure we have now with a >HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject >* as well but add more members to the C-structure. I think this is >the easiest thing to do and requires the least amount of code-change. > It is also possible to define an abstract base-class PyArrayObject * >that both ndarray and ndmasked inherit from. That way ndarray and >ndmasked are siblings even though the ndarray would essentially *be* the >PyArrayObject * --- just with a different type-hierarchy on the python >side. > >This work will take some time and, therefore, I don't expect 1.7 to be >released prior to SciPy Austin with an end of June target date. The >timing will largely depend on what time is available from people >interested in resolving the situation. Mark and I will have some >availability for this work in June but not a great deal (about 2 >man-weeks total between us).If there are others who can step in and >help, it will help accelerate the process. > >Best regards, > >-Travis > > > >___ >NumPy-Discussion mailing list >NumPy-Discussion@scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] python import question
Dear all, This is only a small python import question. I think I'm right but just want some confirmation. Previously I have installed numpy 1.5.1. and then I used pip install --upgrade numpy to install numpy 1.6.1 But when I try to import numpy as np within ipython shell, I still get the version 1.5.1 then I checked my sys.path: In [21]: sys.path Out[21]: ['', '/usr/local/bin', '/usr/local/lib/python2.7/dist-packages/pupynere-1.0.15-py2.7.egg', '/usr/lib/pymodules/python2.7', '/usr/local/lib/python2.7/dist-packages/scikits.statsmodels-0.3.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/Shapely-1.2.13-py2.7-linux-i686.egg', '/usr/local/lib/python2.7/dist-packages/pandas-0.7.3-py2.7-linux-i686.egg', '/home/chaoyue/python/python_lib', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/pymodules/python2.7/gtk-2.0', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7/ubuntuone-client', '/usr/lib/pymodules/python2.7/ubuntuone-control-panel', '/usr/lib/pymodules/python2.7/ubuntuone-storage-protocol', '/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode', '/usr/local/lib/python2.7/dist-packages/IPython/extensions'] Actually I found I have numpy 1.5.1 in /usr/lib/pymodules/python2.7 and numpy 1.6.1 in /usr/local/lib/python2.7/dist-packages/numpy/ but because the first path is before the second one in sys.path, so ipython imports only the first one and ignore the second one. Then I delete the directory of /usr/lib/pymodules/python2.7/numpy and redo the import, I get the version 1.6.1 This means that import will try to find the first occurrence of the module and will ignore the ones with same name in later occurrences? cheers, Chao -- *** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Masked Array for NumPy 1.7
Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).If there are others who can step in and help, it will help accelerate the process. Best regards, -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] pre-PEP for making creative forking of NumPy less destructive
On Fri, May 18, 2012 at 6:59 PM, Dag Sverre Seljebotn wrote: > On 05/18/2012 05:00 PM, Henry Gomersall wrote: >> On Fri, 2012-05-18 at 14:45 +0200, Dag Sverre Seljebotn wrote: >>> I would focus on the 'polymorphic C API' spin. PyObject_GetItem is >>> polymorphic, but there is no standard way for 3rd party libraries to >>> make such functions. >>> >>> So let's find a C API that's NOT about arrays at all and show how some >>> polymorphism may help. >>> >> so memory mapped IO or something? > > A C API towards a Python extension that's in wide use; an analog to the > NumPy C API in a different domain. Something like a web server or > database Python extension module which also exports a C API for writing > other Python extension modules against it. > > I'm not even sure if such a thing exists, in which case NumPy would > indeed be a special case. numpy *is* pretty unique in this regard. The need for this style of polymorphism at this level is even rarer. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] pre-PEP for making creative forking of NumPy less destructive
On 05/18/2012 05:00 PM, Henry Gomersall wrote: > On Fri, 2012-05-18 at 14:45 +0200, Dag Sverre Seljebotn wrote: >> I would focus on the 'polymorphic C API' spin. PyObject_GetItem is >> polymorphic, but there is no standard way for 3rd party libraries to >> make such functions. >> >> So let's find a C API that's NOT about arrays at all and show how some >> polymorphism may help. >> > so memory mapped IO or something? A C API towards a Python extension that's in wide use; an analog to the NumPy C API in a different domain. Something like a web server or database Python extension module which also exports a C API for writing other Python extension modules against it. I'm not even sure if such a thing exists, in which case NumPy would indeed be a special case. Dag ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] pre-PEP for making creative forking of NumPy less destructive
On Fri, 2012-05-18 at 14:45 +0200, Dag Sverre Seljebotn wrote: > I would focus on the 'polymorphic C API' spin. PyObject_GetItem is > polymorphic, but there is no standard way for 3rd party libraries to > make such functions. > > So let's find a C API that's NOT about arrays at all and show how some > polymorphism may help. > so memory mapped IO or something? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] pre-PEP for making creative forking of NumPy less destructive
Henry Gomersall wrote: >On Fri, 2012-05-18 at 12:48 +0100, mark florisson wrote: >> If we can find even more examples, preferably outside of the >> scientific community, where related projects face a similar >situation, >> it may help people understand that this is not "a Numpy problem". > >Buffer Objects in OpenGL? There is already PEP 3118 though. I would focus on the 'polymorphic C API' spin. PyObject_GetItem is polymorphic, but there is no standard way for 3rd party libraries to make such functions. So let's find a C API that's NOT about arrays at all and show how some polymorphism may help. Of course there's a couple of non-array Cython usecases too. Dag > >___ >NumPy-Discussion mailing list >NumPy-Discussion@scipy.org >http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] pre-PEP for making creative forking of NumPy less destructive
On Fri, 2012-05-18 at 12:48 +0100, mark florisson wrote: > If we can find even more examples, preferably outside of the > scientific community, where related projects face a similar situation, > it may help people understand that this is not "a Numpy problem". Buffer Objects in OpenGL? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] pre-PEP for making creative forking of NumPy less destructive
On 17 May 2012 23:53, Dag Sverre Seljebotn wrote:
> I'm repeating myself a bit, but my previous thread of this ended up
> being about something else, and also since then I've been on an
> expedition to the hostile waters of python-dev.
>
> I'm crazy enough to believe that I'm proposing a technical solution to
> alleviate the problems we've faced as a community the past year. No,
> this will NOT be about NA, and certainly not governance, but do please
> allow me one paragraph of musings before the meaty stuff.
>
> I believe the Achilles heel of NumPy is the C API and the PyArrayObject.
> The reliance we all have on the NumPy C API means there can in practice
> only be one "array" type per Python process. This makes people *very*
> afraid of creative forking or new competing array libraries (since they
> just can't live in parallel -- like Cython and Pyrex can!), and every
> new feature has to go into ndarray to fully realise itself. This in turn
> means that experimentation with new features has to happen within one or
> a few release cycles, it cannot happen in the wild and by competition
> and by seeing what works over the course of years before finally making
> it into upstream. Finally, if any new great idea can really only be
> implemented decently if it also impacts thousands of users...that's bad
> both for morale and developer recruitment.
>
> The meat:
>
> There's already of course been work on making the NumPy C API work
> through an indirection layer to make a more stable ABI. This is about
> changing the ideas of how that indirection should happen, so that you
> could in theory implement the C API independently of NumPy.
>
> You could for instance make a "mini-NumPy" that only contains the bare
> essentials, and load that in the same process as the real NumPy, and use
> the C API against objects from both libraries.
>
> I'll assume that we can get a PEP through by waving a magic wand, since
> that makes it easier to focus on essentials. There's many ugly or less
> ugly hacks to make it work on any existing CPython [1], and they
> wouldn't be so ugly if there's PEP blessing for the general idea.
>
> Imagine if PyTypeObject grew an extra pointer "tp_customslots", which
> pointed to an array of these:
>
> typedef struct {
> unsigned long tpe_id;
> void *tpe_data;
> } PyTypeObjectCustomSlot;
>
> The ID space is partitioned to anyone who asks, and NumPy is given a
> large chunk. To insert a "custom slot", you stick it in this list. And
> you search it linearly for, say, PYTYPE_CUSTOM_NUMPY_SLOT (each type
> will typically have 0-3 entries so the search is very fast).
>
> I've benchmarked something very similar recently, and the overhead in a
> "hot" situation is on the order of 4-6 cycles. (As for cache, you can at
> least stick the slot array right next to the type object in memory.)
>
> Now, a NumPy array would populate this list with 1-2 entries pointing to
> tables of function pointers for the NumPy C API. This lookup through the
> PyTypeObject would in part replace the current import_array() mechanism.
>
> I'd actually propose two such custom slots for ndarray for starters:
>
> a) One PEP 3118-like binary description that exposes raw data pointers
> (without the PEP 3118 red tape)
>
> b) A function pointer table for a suitable subset of the NumPy C API
> (obviously not array construction and so on)
>
> The all-important PyArray_DATA/DIMS/... would be macros that try for a)
> first, but fall back to b). Things like PyArray_Check would actually
> check for support of these slots, "duck typing", rather than the Python
> type (of course, this could only be done at a major revision like NumPy
> 2.0 or 3.0).
>
> The overhead should be on the order of 5 cycles per C API call. That
> should be fine for anything but the use of PyArray_DATA inside a tight
> loop (which is a bad idea anyway).
>
> For now I just want to establish if there's support for this general
> idea, and see if I can get some weight behind a PEP (and ideally a
> co-author), which would make this a general approach and something more
> than an ugly NumPy specific hack. We'd also have good use for such a PEP
> in Cython (and, I believe, Numba/SciPy in CEP 1000).
Well, you have my vote, but you already knew that. I'd also be willing
to co-author any PEP etc, but I'm sensing it may be more useful to
have support from people from different projects. Personally, I think
if this is to succeed, we first need to fix the design to work for
subclasses (I think one may just want to memcpy the interface
information over to the subclass, e.g. through a convenience function
that allows one to add more as well). If we have a solid idea of the
technical implementation, we should actually implement it and present
the benchmarks, comparing the results to capsules as attributes (and
to the _PyType_Lookup approach).
If we have come that far, then even at that point people may argue
that one could do the same thing through a metaclass, and tha
