[Numpy-discussion] numpy FFT memory accumulation
I am using fftRes = abs(fft.rfft(data_array[end-2**15:end])) to do running analysis on streaming data. The N never changes. It sucks memory up at ~1MB/sec with 70kHz data rate and 290 ffts/sec. (Interestingly, Numeric FFT accumulates much slower..) (Commenting out that one line stops memory growth.) What can one do to alleviate this? Can I del() some numpy object or such? It's a bit of an issue for a program that needs to run for weeks. It's purpose is simply to argmax() the largest bin, which always falls within a small range - do I have another, better option than fft? Cheers, Ray Schumacher Blue Cove Interactive ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy version and numpy.asarray behavior issue
I have 2 PCs with 2 different installs: ActivePython 2.4.3 Build 12 with numpy version '1.0b1' and Enthought 2.4.3 (1.0.0 #69) with numpy version '0.9.7.2476' The attached runs Ok on numpy v1.0, but on Enthought's, print a1[0] gives: IndexError: 0-d arrays can't be indexed. It seems that the 0.9.7 numpy.asarray is not creating a true array from Dummy class in the code below. Enthought only comes with 0.9.9.2706 (now). When was the asarray behavior supported, or, is there some other issue I'm missing? I'll use Activestate's distro if needed, but I'd like to keep Enthought for that one... def fromaddress(address, dtype, shape, strides=None): Create a numpy array from an integer address, a dtype or dtype string, a shape tuple, and possibly strides. import numpy # Make sure our dtype is a dtype, not just f or whatever. dtype = numpy.dtype(dtype) class Dummy(object): pass d = Dummy() d.__array_interface__ = dict( data = (address, False), typestr = dtype.str, descr = dtype.descr, shape = shape, strides = strides, version = 3, ) return numpy.asarray(d) Thanks, Ray nFromAddress.py def fromaddress(address, dtype, shape, strides=None): Create a numpy array from an integer address, a dtype or dtype string, a shape tuple, and possibly strides. import numpy # Make sure our dtype is a dtype, not just f or whatever. dtype = numpy.dtype(dtype) class Dummy(object): pass d = Dummy() d.__array_interface__ = dict( data = (address, False), typestr = dtype.str, descr = dtype.descr, shape = shape, strides = strides, version = 3, ) return numpy.asarray(d) ##Numeric example, with address kludge import Numeric, numpy, ctypes, string a0 = Numeric.zeros((1), Numeric.Int16) nAddress = int(string.split(repr(a0.__copy__))[-1][:-1], 16) tmp=(ctypes.c_long*1)(0) ctypes.memmove(tmp, nAddress+8, 4) nAddress = tmp[0] a1 = fromaddress(nAddress, numpy.int16, (1,)) a0[0] = 5 print a1[0] ## numpy example a2 = numpy.zeros(1, numpy.int16) nAddress = a2.__array_interface__['data'][0] nType = a2.__array_interface__['typestr'] nShape = a2.__array_interface__['shape'] a3 = fromaddress(nAddress, nType, nShape) a2[0] = 5 print a3[0]___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy version and numpy.asarray behavior issue
I just installed 1.0.3.1 on top of Enthought's and asarray() works. But... Although the creation of an array from an address via a Dummy class is kosher in one process (as in the previous attachment), it fails across processes - the array is created, but gives a Python has generated errors window if the second process even attempts to read. It can seem to work across processes with mmap.mmap() and tags (used in Windows) def arrSharedMemory(shape, dtype, tag=PySharedMemory, access=None): ## Windows only ! share memory between different ## processes if same `tag` is used. itemsize = N.dtype(dtype).itemsize count = N.product(shape) size = count * itemsize import mmap sharedmem = mmap.mmap(0, size, tag, access) a=N.frombuffer(sharedmem, dtype, count) a.shape = shape return a I guess I'll use mmap unless someone can point out otherwise Thanks, Ray nFromAddress.py ##Numeric example, with address kludge import Numeric, numpy, ctypes, subprocess from time import clock, sleep a = Numeric.zeros((4), Numeric.Int16) nAddress = int(repr(.__copy__).split()[-1][:-1], 16) tmp=(ctypes.c_long*1)(0) ctypes.memmove(tmp, nAddress+8, 4) nAddress = tmp[0] a = numpy.zeros(4, numpy.int16) nAddress = a.__array_interface__['data'][0] print nAddress pid = subprocess.Popen( [r'C:\python24\python.exe', ['nFromAddress2.py '+str(nAddress)] ]).pid while clock()5: sleep(.1) if a[0]!=0: ## wait for a change... print a0[0] nFromAddress.py import numpy import time, sys def fromaddress(address, dtype, shape, strides=None): Create a numpy array from an integer address, a dtype or dtype string, a shape tuple, and possibly strides. # Make sure our dtype is a dtype, not just f or whatever. dtype = numpy.dtype(dtype) class Dummy(object): pass d = Dummy() d.__array_interface__ = dict( data = (address, False), typestr = dtype.str, descr = dtype.descr, shape = shape, strides = strides, version = 3, ) return numpy.asarray(d) nAddress = sys.argv[1] print 'recvd addr', nAddress a3 = fromaddress(nAddress, numpy.int16, (4,)) ## any of the following cause a Python/Windows error on access print a3 #a3[0] = 5___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
Thanks all: At 10:00 AM 10/10/2007, Robert Kern wrote: Something like the following should suffice (untested, though I've done similar things with ctypes before): I tested, successfully: nFromAddress.py def fromaddress(address, dtype, shape, strides=None): Create a numpy array from an integer address, a dtype or dtype string, a shape tuple, and possibly strides. import numpy # Make sure our dtype is a dtype, not just f or whatever. dtype = numpy.dtype(dtype) class Dummy(object): pass d = Dummy() d.__array_interface__ = dict( data = (address, False), typestr = dtype.str, descr = dtype.descr, shape = shape, strides = strides, version = 3, ) return numpy.asarray(d) ## Numeric example, with address kludge import Numeric, numpy, ctypes, string a0 = Numeric.zeros((1), Numeric.Int16) nAddress = int(string.split(repr(a0.__copy__))[-1][:-1], 16) tmp=(ctypes.c_long*1)(0) ctypes.memmove(tmp, nAddress+8, 4) nAddress = tmp[0] a1 = fromaddress(nAddress, numpy.int16, (1,)) ## explicit type a0[0] = 5 print a1[0] ## numpy example a2 = numpy.zeros(1, numpy.int16) nAddress = a2.__array_interface__['data'][0] nType = a2.__array_interface__['typestr'] nShape = a2.__array_interface__['shape'] a3 = fromaddress(nAddress, nType, nShape) a2[0] = 5 print a3[0] So, now with little effort the relevant info can be passed over pipes, shared memory, etc. and shares/views created in other processes, since they are not objects but ints and strings. David Cournapeau Wrote: Basically, you cannot expect file descriptors (or even file handles: the standard ones from C library fopen) to cross dll boundaries if the dll do not have exactly the same runtime. It sounds like there is a general dilemma: no one with Python 2.4 or 2.5 can reliably expect to compile extensions/modules if they did not install the 7.1 compiler in time. The 2.6 seems to use VC 2005 Express, I don't know about py3000(?), with associated upgrade issues. It would be nice if the build bots could also compile suggested modules/extentions. Thanks again, Ray ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
On 10/9/07, Sebastian Haase replied: Did you find that locks or semaphores were needed? Maybe that's why it crashed ;-) !? But for simple use it seems fine. I just did some code (below) that does read/write to the array AFAP, and there is no crash, or any other issue (Win2000, py2.4, numpy 1.0b1). Without the print statements, it does max both processors; with printing I/O only 58%. Both processes can modify the array without issue either. I'll experiment with I had seen the Win mmap in this thread: http://objectmix.com/python/127666-shared-memory-pointer.html and here: http://www.codeproject.com/cpp/embedpython_2.asp Note also that the Python mmap docs read In either case you must provide a file descriptor for a file opened for update. and no mention of the integer zero descriptor option. Access options behave as presented. Because *NIX has MAP_SHARED as an option you'd think that there might be cross-platform share behavior with some platform checking if statements. Without a tag though, how does another process reference the same memory on NIX, a filename? (It seems) But I had the same issue as Mark Heslep http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422 of creating a numpy array from a raw address (not a c_array). I assume this is a different issue, but haven't looked into it yet. Yes, a different methodology attempt. It would be interesting to know anyway how to create a numpy array from an address; it's probably buried in the undocumented C-API that I don't grok, and likely frowned upon. Thanks, Ray ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy/Windows shared arrays between processes?
Is anyone sharing arrays between processes on Windows? I tried compiling the posh sources (once, so far) with the new MS toolkit and failed... What other solutions are in use? Have a second process create an array view from an address would suffice for this particular purpose. I could pass the address as a parameter of the second process's argv. I've also tried things like pb=pythonapi.PyBuffer_FromReadWriteMemory(9508824, 9*sizeof(c_int)) N.frombuffer(pb, N.int32) which fails since pb is and int. What are my options? Ray Schumacher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy array sharing between processes? (and ctypes)
While investigating ctypes and numpy for sharing, I saw that the example on http://www.scipy.org/Cookbook/Ctypes#head-7def99d882618b52956c6334e08e085e297cb0c6 does not quite work. However, with numpy.version.version=='1.0b1', ActivePython 2.4.3 Build 12: import numpy as N from ctypes import * x = N.zeros((3, 3), dtype=N.float64) xdataptr = N.intp(x.__array_interface__['data'])[0] y = (c_double * x.size).from_address(xdataptr) y[0] = 123. y[4] = 456. y[8] = 789 print N.diag(x) Works for me... I can then do: import numpy.core.multiarray as MA xBuf = MA.getbuffer(x) z = MA.frombuffer(xBuf).reshape((3,3)) z array([[ 123.,0.,0.], [ 0., 456.,0.], [ 0.,0., 789.]]) z[0,1] = 99 z array([[ 123., 99.,0.], [ 0., 456.,0.], [ 0.,0., 789.]]) x array([[ 123., 99.,0.], [ 0., 456.,0.], [ 0.,0., 789.]]) y[1] 99.0 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] zoom FFT with numpy? (Nadav Horesh)
Hi Nadev, A long time ago I translated a free code of chirp z transform (zoom fft) into python. Thanks, I'll try it out. I did, however read before on the differences: From Numerix http://www.numerix-dsp.com/zoomfft.html: One common question is : Is the zoom FFT the same as the chirp z-transform. The answer is : Absolutely not. The FFT calculates the FFT at N equally spaced points around the unit circle in the z-plane, the chirp z-transform modifies the locations of these points along a contour that can lie anywhere on the z-plane. In contrast, the zoom-FFT uses digital down conversion techniques to localise the standard FFT to a narrow band of frequencies that are centered on a higher frequency. The chirp z-transform is often used to analyze signals such as speech, that have certain frequency domain charactgeristics. The zoom-FFT is used to reduce the sample rate required when analysing narrowband signals - E.G. in HF communications. http://www-gatago.com/comp/dsp/34830442.html I just saw was good reading too. It will be interesting, and the code is appreciated! Also, czt.c might be particularly fast if compiled with the Intel FFT lib and weave.blitz(). Again, the goal is increased f resolution within a known small band for the ~same CPU cycles... Ray ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] zoom FFT with numpy?
We'd like to do what most call a zoom FFT; we only are interested in the frequencies of say, 6kHZ to 9kHz with a given N, and so the computations from DC to 6kHz are wasted CPU time. Can this be done without additional numpy pre-filtering computations? If explicit filtering is needed to baseband the data, is it worth it? It sounds like we need to generate cosine data once for each band and N, then multiple with the data for each FFT. Has anyone coded this up before? I couldn't find any references other than http://www.dsprelated.com/showmessage/38917/1.php and http://www.numerix-dsp.com/siglib/examples/test_zft.c (which uses Numerix). Ray ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
