Re: [Numpy-discussion] pickable ndarray subclass

[Stefan van der Walt]

Hi Christiaan

On Sun, Apr 15, 2007 at 02:03:49PM +0900, Christian K wrote:
> could someone please provide example code for how to make a subclassed ndarray
> pickable? I don't quite understand the docs of ndarray.__reduce__.
> My subclassed ndarray has just one additional attribute.
> 

__reduce__ is part of the pickle protocol, which is described at

http://docs.python.org/lib/node321.html

You need to modify __reduce__ to store the complete state of your
custom object, as well as __setstate__ which restores the state on
unpickling.

See the attached code as an example.  There, I have an InfoArray,
which is a subclassed numpy.ndarray.  The InfoArray contains an
additional member, info, which is a dictionary.

The __reduce__ method calls ndarray's __reduce__, then adds to the
result the state of the InfoArray.  Conversely, the __setstate__
method calls ndarray's __setstate__ as well as restoring the info
member.

Regards
Stéfan
import numpy as N

class InfoArray(N.ndarray):
def __new__(subtype, data, dtype=None, info=None, copy=False):
subtype.__defaultinfo = info

if copy:
data = N.array(data,dtype=dtype)
else:
data = N.asarray(data,dtype=dtype)

data = data.view(subtype)
return data

def __array_finalize__(self,obj):
if not hasattr(self, "info"):
# The object does not already have an `.info` attribute: we use the default
self.info = getattr(obj,'info',self.__defaultinfo)

def __reduce__(self):
object_state = list(N.ndarray.__reduce__(self))
subclass_state = (self.info,)
object_state[2] = (object_state[2],subclass_state)
return tuple(object_state)

def __setstate__(self,state):
nd_state, own_state = state
N.ndarray.__setstate__(self,nd_state)

info, = own_state
self.info = info

def __repr__(self):
desc="""\
array(data=
  %(data)s,
  tag=%(tag)s)"""
return desc % {'data': str(self), 'tag':self.info }

import numpy as N
x = InfoArray(N.arange(10), info={'name':'x'})
print repr(x)
y = InfoArray(N.arange(10), info={'name':'y'})
assert (x.info['name']=='x')
assert (y.info['name']=='y')
z = N.sqrt(x)
print repr(z)

# -- demonstrate that instance of subclass can be pickled --

import pickle
print "\n\nBefore pickling:"
print repr(z)

jar = pickle.dumps(z)
z = pickle.loads(jar)

print "\n\nAfter pickling:"
print repr(z)

___
Numpy-discussion mailing list
[EMAIL PROTECTED]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: PyQwt3D-0.1.4 released

[Gerard Vermeulen]

What is PyQwt3D?

- it is a set of Python bindings for the QwtPlot3D C++ class library
  which extends the Qt framework with widgets for 3D data visualization.
  PyQwt3D inherits the snappy feel from QwtPlot3D.
  The examples at http://pyqwt.sourceforge.net/pyqwt3d-examples.html
  show how easy it is to make a 3D plot and how to save a 3D plot to
  an image or an (E)PS/PDF file.

- it requires and extends PyQt, a set of Python bindings for Qt.

- it supports the use of PyQt, Qt, QwtPlot3D, and NumPy or SciPy in a
  GUI Python application or in an interactive Python session.

- it runs on POSIX, Mac OS X and Windows platforms (practically any
  platform supported by Qt and Python).

The home page of PyQwt3D is http://pyqwt.sourceforge.net.

PyQwt3D-0.1.4 is a minor release providing:
- support for SIP-4.6
- a binary installer for Windows.

PyQwt3D-0.1.4 supports:
1. Python-2.5.x, -2.4.x, or -2.3.x
2. PyQt-4.2 and -4.1.x, or -3.17.x
3. SIP-4.6, or -4.5.x
4. Qt-4.2.x, Qt-4.1.x, Qt-3.3.x, or -3.2.x
5. QwtPlot3D-0.2.6


Enjoy -- Gerard Vermeulen
___
Numpy-discussion mailing list
[EMAIL PROTECTED]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] newbie question - large dataset

[Steve Staneff]

Many thanks to all those who responded to my question.  Not being a real 
programmer, sometimes it's good to be reminded 
of the basics - such as timing various sections of code.  It turns out that 
most of the time in the loop was spent on 
two of the functions.  I've rewritten the first to be pre-calculated (a real 
oversight on my part), and looked at 
improving the second as much as possible.

Some of the other time in the loop is being spent going back to the db server 
for data, and I'll look into using 
pytables as a way of speeding that up (possibly by initializing temporary 
tables on each of my client machines).

Thanks again,

Steve

> On 4/7/07, Steve Staneff <[EMAIL PROTECTED]> wrote:
> 
>>Hi,
>>
>>I'm looking for a better solution to managing a very large calculation.
>>Set A is composed of tuples a, each of the form a = [float, string]; set B
>>is composed of tuples of similar structure (b = [float, string]).  For
>>each possible combination of a and b I'm calculating c, of the form c =
>>f(a,b) = [g(a[0], b[0]), h(a[1], b[1])] where g() and h() are non-trivial
>>functions.
> 
___
Numpy-discussion mailing list
[EMAIL PROTECTED]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] building numpy with Intel MS compiler and FFT

[Ray Schumacher]

Thanks Rex,
I'll give it a try next week.

 >I've compiled both Numpy and Python 2.5 with the Intel compiler. On a
 >Core 2 Duo, at least, the speed increase on Pybench was ~49%, even
 >before compiling Python with icc. My post about it was on 25 Jan, and
 >has subject: Compiling Python with icc

___
Numpy-discussion mailing list
[EMAIL PROTECTED]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] building numpy with atlas on ubuntu edgy

[David Cournapeau]

Christian K wrote:
>
> Thanks, but that didn't help:
>
> atlas_info:
>   libraries lapack not found in /usr/lib/sse2
>   libraries f77blas,cblas,atlas not found in /usr/lib/atlas
>   libraries lapack_atlas not found in /usr/lib/atlas
>   libraries lapack not found in /usr/lib/sse2
>   libraries f77blas,cblas,atlas not found in /usr/lib
>   libraries lapack_atlas not found in /usr/lib
> numpy.distutils.system_info.atlas_info
> /media/hda6/home/ck/prog/scipy/numpy/numpy/distutils/system_info.py:903:
> UserWarning:
> *
> Could not find lapack library within the ATLAS installation.
> *
>
>   warnings.warn(message)
>   FOUND:
> libraries = ['f77blas', 'cblas', 'atlas']
> library_dirs = ['/usr/lib/sse2']
> language = c
> define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
>
> lapack_info:
>   libraries lapack not found in /usr/lib/sse2
>   libraries lapack not found in /usr/lib
>   NOT AVAILABLE
>
>
> Christian
>
> ___
> Numpy-discussion mailing list
> Numpy-discussion@scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
On Ubuntu and debian, you do NOT need any site.cfg to compile numpy with 
atlas support. Just install the package atlas3-base-dev, and you are 
done. The reason is that when *compiling* a software which needs atlas, 
the linker will try to find libblas.so in /usr/lib, not in 
/usr/lib/sse2. If you install atlas3-base-dev, the package will install 
those at the correct locations. I have updated the instructions for 
Ubuntu (also works for debian) on the wiki a few days ago:

http://www.scipy.org/Installing_SciPy/Linux#head-c5a062b2ecf76f746d78cfcde1dae00ae26109fe

Note that if you have also optimized version installed (sse, sse2), it 
will use those automatically when you launch numpy (you can check by 
doing ldd on the file numpy/core/_dotblas.so inside your numpy installation.

cheers,

David


___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] building numpy with atlas on ubuntu edgy

[Charles R Harris]

On 4/15/07, David Cournapeau <[EMAIL PROTECTED]> wrote:


Christian K wrote:
>
> Thanks, but that didn't help:
>
> atlas_info:
>   libraries lapack not found in /usr/lib/sse2
>   libraries f77blas,cblas,atlas not found in /usr/lib/atlas
>   libraries lapack_atlas not found in /usr/lib/atlas
>   libraries lapack not found in /usr/lib/sse2
>   libraries f77blas,cblas,atlas not found in /usr/lib
>   libraries lapack_atlas not found in /usr/lib
> numpy.distutils.system_info.atlas_info
> /media/hda6/home/ck/prog/scipy/numpy/numpy/distutils/system_info.py:903:
> UserWarning:
> *
> Could not find lapack library within the ATLAS installation.
> *
>
>   warnings.warn(message)
>   FOUND:
> libraries = ['f77blas', 'cblas', 'atlas']
> library_dirs = ['/usr/lib/sse2']
> language = c
> define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
>
> lapack_info:
>   libraries lapack not found in /usr/lib/sse2
>   libraries lapack not found in /usr/lib
>   NOT AVAILABLE
>
>
> Christian
>
> ___
> Numpy-discussion mailing list
> Numpy-discussion@scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
On Ubuntu and debian, you do NOT need any site.cfg to compile numpy with
atlas support. Just install the package atlas3-base-dev, and you are
done. The reason is that when *compiling* a software which needs atlas,
the linker will try to find libblas.so in /usr/lib, not in
/usr/lib/sse2. If you install atlas3-base-dev, the package will install
those at the correct locations. I have updated the instructions for
Ubuntu (also works for debian) on the wiki a few days ago:


http://www.scipy.org/Installing_SciPy/Linux#head-c5a062b2ecf76f746d78cfcde1dae00ae26109fe

Note that if you have also optimized version installed (sse, sse2), it
will use those automatically when you launch numpy (you can check by
doing ldd on the file numpy/core/_dotblas.so inside your numpy
installation.



Be aware that on recent 64 bit Intel processors running a 64 bit OS the base
Atlas package appears to be buggy. You might be better off building your own
version, on a fast machine it won't take long.

Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] building numpy with atlas on ubuntu edgy

[David Cournapeau]

Charles R Harris wrote:
>
>
> On 4/15/07, *David Cournapeau* <[EMAIL PROTECTED] 
> > wrote:
>
> Christian K wrote:
> >
> > Thanks, but that didn't help:
> >
> > atlas_info:
> >   libraries lapack not found in /usr/lib/sse2
> >   libraries f77blas,cblas,atlas not found in /usr/lib/atlas
> >   libraries lapack_atlas not found in /usr/lib/atlas
> >   libraries lapack not found in /usr/lib/sse2
> >   libraries f77blas,cblas,atlas not found in /usr/lib
> >   libraries lapack_atlas not found in /usr/lib
> > numpy.distutils.system_info.atlas_info
> >
> /media/hda6/home/ck/prog/scipy/numpy/numpy/distutils/system_info.py:903:
> > UserWarning:
> >
> *
> > Could not find lapack library within the ATLAS installation.
> >
> *
> >
> >   warnings.warn(message)
> >   FOUND:
> > libraries = ['f77blas', 'cblas', 'atlas']
> > library_dirs = ['/usr/lib/sse2']
> > language = c
> > define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
> >
> > lapack_info:
> >   libraries lapack not found in /usr/lib/sse2
> >   libraries lapack not found in /usr/lib
> >   NOT AVAILABLE
> >
> >
> > Christian
> >
> > ___
> > Numpy-discussion mailing list
> > Numpy-discussion@scipy.org 
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> On Ubuntu and debian, you do NOT need any site.cfg to compile
> numpy with
> atlas support. Just install the package atlas3-base-dev, and you are
> done. The reason is that when *compiling* a software which needs
> atlas,
> the linker will try to find libblas.so in /usr/lib, not in
> /usr/lib/sse2. If you install atlas3-base-dev, the package will
> install
> those at the correct locations. I have updated the instructions for
> Ubuntu (also works for debian) on the wiki a few days ago:
>
> 
> http://www.scipy.org/Installing_SciPy/Linux#head-c5a062b2ecf76f746d78cfcde1dae00ae26109fe
>
> Note that if you have also optimized version installed (sse, sse2), it
> will use those automatically when you launch numpy (you can check by
> doing ldd on the file numpy/core/_dotblas.so inside your numpy
> installation.
>
>
> Be aware that on recent 64 bit Intel processors running a 64 bit OS 
> the base Atlas package appears to be buggy. You might be better off 
> building your own version, on a fast machine it won't take long.
I do not use any 64 bits machines myself, so I didn't know about this. 
Compiling atlas may take a long time on fast machine if you don't have 
arch defaults, though (it takes several hours for my workstation which 
has two xeon @ 3.2 ghz).

My impression is that using 64 bits OS is still a bit rough, anyway.

David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion