Re: [Numpy-discussion] PEP: named axis
Hello list, I am not sure, if I understood everything of the discussion on the named-axis-idea of numpy-arrays, since I am only a *user* of numpy. I never subclassed the numpy-array-class ;-) However, I have the need to store meta-information for my arrays. I do this with a stand-alone class with the name 'Wave' that stores its data in a n-dimensional numpy-array as a member. The meta-information I store (using dicts and lists) is * coordinateLabel per axis * x0 per axis * dx per axis This concept is taken from the data structures in the commercial software IGOR, that are also called 'Waves'. An example would be an image I took with a microscope. The data would be 2d, say shape = (640, 480) holding the intesity information per pixel. x0 could then be [-1e-6, -2e-6] and dx [100e-9, 100e-9] meaning that the image's pixel index [0,0] corresponds to a position of -1micrometer/-2micrometer and the pixels have a spacing of 100nanometers. coordinateLabels would be ['x(m)', 'y(m)']. If I have a movie, the data would be 3d with x0 = [-1e-6, -2e-6, 0], dx = [100e-9, 100e-9, 100e-3] and coordinateLabels = ['x(m)', 'y(m)', 't(s)'] for a frame rate of 10 fps. What I would like to say with this is the following (as a user...) : * Meta-information is often necessary * A string-label per axis is often not enough. Scaling is also important * I like the idea of a most-basic-as-possible numpy-array. In my opinion, the meta-data-management should be done by another (sub-?) class. This way, numpy-arrays are simple enough for new users (as I was roughly two years ago...). I would be very interested in a class that *uses* numpy-arrays to provide a datastructure for physical data with coordinate labels and scaling. Regards, Lars Friedrich -- Dipl.-Ing. Lars Friedrich Bio- and Nano-Photonics Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-Köhler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: lfrie...@imtek.de ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] histogramdd memory needs
Hi, 2) Is there a way to use another algorithm (at the cost of performance) that uses less memory during calculation so that I can generate bigger histograms? You could work through your array block by block. Simply fix the range and generate an histogram for each slice of 100k data and sum them up at the end. Thank you for your answer. I sliced the (original) data into blocks. However, when I do this, I need at least twice the memory for the whole histogram (one for the temporary result and one for accumulating the total result). Assuming my histogram has a size of (280**3)*8 = 176 (megabytes) this does not help, I think. What I will try next is to compute smaller parts of the big histogram and combine them at the end. (Slice the histogram into blocks) Is it this, that you were recommending? Lars ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fourier with single precision
Thank you for your comments! I will try this fftw3-scipy approach and see how much faster I can get. Maybe this is enough for me...? Lars ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fourier with single precision
Hello, thanks for your comments. If I got you right, I should look for a FFT-code that uses SSE (what does this actually stand for?), which means that it vectorizes 32bit-single-operations into larger chunks that make efficient use of recent CPUs. You mentioned FFTW and MKL. Is this www.fftw.org and the 'intel math kernel library'? If I would like to use one of them, is numpy the right place to put it in? Does anyone know, if it is possible to switch on SSE support (at compile time) in the fftpack.c that numpy uses? Thanks Lars ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fourier with single precision
Hello, David Cournapeau wrote: As far as I can read from the fft code in numpy, only double is supported at the moment, unfortunately. Note that you can get some speed by using scipy.fftpack methods instead, if scipy is an option for you. What I understood is that numpy uses FFTPACK's algorithms. From www.netlib.org/fftpack (is this the right address?) I took that there is a single-precision and double-precision-version of the algorithms. How hard would it be (for example for me...) to add the single-precision versions to numpy? I am not a decent C-hacker, but if someone tells me, that this task is not *too* hard, I would start looking more closely at the code... Would it make sense, that if one passes an array of dtype = numpy.float32 to the fft function, a complex64 is returned, and if one passes an array of dtype = numpy.float64, a complex128 is returned? Lars ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] fourier with single precision
Hello, is there a way to tell numpy.fft.fft2 to use complex64 instead of complex128 as output dtype to speed the up transformation? Thanks Lars ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] tofile speed
Hello,
I tried the following:
### start code
a = N.random.rand(100)
myFile = file('test.bin', 'wb')
for i in range(100):
a.tofile(myFile)
myFile.close()
### end code
And this gives roughly 50 MB/s on my office-machine but only 6.5 MB/s on
the machine that I was reporting about.
Both computers use Python 2.4.3 with enthought 1.0.0 and numpy 1.0.1
So I think I will go and check the harddisk-drivers. array.tofile does
not seem to be the problem and actually seems to be very fast. Any other
recommendations?
Thanks
Lars
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] tofile speed
Hello everyone, thank you for the replies. Sebastian, the chunk size is roughly 4*10^6 samples, with two byte per sample, this is about 8MB. I can vary this size, but increasing it only helps for much smaller values. For example, when I use a size of 100 Samples, I am much too slow. It gets better for 1000 Samples, 1 Samples and so on. But since I already reached a chunksize in the region of megabytes, I have difficulties to increase my buffer size further. I also have the feeling that increasing does not help in this size region. (correct me if I am wrong...) Chuck, I am using a Windows XP system with a new (few months old) Maxtor SATA-drive. Lars ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] tofile speed
Hello everyone, I am using array.tofile successfully for a data-acqusition-streaming application. I mean that I do the following: for a long time: temp = dataAcquisisionDevice.getData() temp.tofile(myDataFile) temp is a numpy array that is used for storing the data temporarily. The data acquisition device is acquiring continuously and writing the data to a buffer from which I can read with .getData(). This works fine, but of course, when I turn the sample rate higher, there is a point when temp.toFile is too slow. The dataAcquisitionDevice's buffer will run full before I can fetch the data again. (temp has a size of ~Mbyte, and the for loop has a period of ~0.5 seconds so that increasing the chunk size won't help) I have no idea how efficient array.tofile() is. Maybe it is terribly efficient and what I see is just the limitation of my hardware (harddisk). Currently I can stream with roughly 4 Mbyte/s, which is quite fast, I guess. However, if anyone can point me to a way to write my data to harddisk faster, I would be very happy! Thanks Lars -- Dipl.-Ing. Lars Friedrich Photonic Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-Köhler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
