Re: [Numpy-discussion] Shared memory ndarrays (update)

2011-04-11 Thread Sebastian Haase
On Mon, Apr 11, 2011 at 7:05 AM, Gael Varoquaux gael.varoqu...@normalesup.org wrote: Hey Sturla, It's really great that you are still working on that. I'll test the code under Linux. The scipy community has moved to github. If I create a repository under github and put the code on it, would

Re: [Numpy-discussion] Shared memory ndarrays (update)

2011-04-11 Thread Zbigniew Jędrzejewski-Szmek
On 04/11/2011 09:21 AM, Sebastian Haase wrote: I have a non python question: for Java there seems to exist a module/package/class called nio http://download.oracle.com/javase/1.4.2/docs/api/java/nio/MappedByteBuffer.html public abstract class MappedByteBufferextends ByteBuffer A

Re: [Numpy-discussion] Classes as records within an numpy array

2011-04-11 Thread Zbigniew Jędrzejewski-Szmek
On 04/11/2011 05:40 AM, David Crisp wrote: On Mon, Apr 11, 2011 at 1:17 PM, David Crisp david.cr...@gmail.com wrote: On Mon, Apr 11, 2011 at 11:00 AM, Sturla Molden stu...@molden.no wrote: Den 11.04.2011 02:01, skrev David Crisp: Can anybody guide me through this problem? I dont know how

Re: [Numpy-discussion] Shared memory ndarrays (update)

2011-04-11 Thread srean
Hi everyone, I was looking up the options that are available for shared memory arrays and this thread came up at the right time. The doc says that multiprocessing .Array(...) gives a shared memory array. But from the code it seems to me that it is actually using a mmap. Is that correct a

Re: [Numpy-discussion] Shared memory ndarrays (update)

2011-04-11 Thread srean
Apologies for adding to my own post. multiprocessing.Array(...) uses an anonymous mmapped file. I am not sure if that means it is resident on RAM or the swap device. But my original question remains, what are the pros and cons of using it versus numpy mmapped arrays. If multiprocessing.Array is

Re: [Numpy-discussion] Shared memory ndarrays (update)

2011-04-11 Thread Sturla Molden
Shared memory is memory mapping from the paging file (i.e. RAM), not a file on disk. They can have a name or be anonymous. I have explained why we need named shared memory before. If you didn't understand it, try to pass an instance of |multiprocessing.Array over | |multiprocessing.Queue.

Re: [Numpy-discussion] Shared memory ndarrays (update)

2011-04-11 Thread Sturla Molden
Den 11.04.2011 14:58, skrev Zbigniew Jędrzejewski-Szmek: Hi, it could, but you'd have to do the parsing of data yourself. So nothing fancy unless you want to reimplement numpy in Java :) Not really. Only the data buffer is stored in shared memory. If you can pass the required fields to Java

[Numpy-discussion] Odd numerical difference between Numpy 1.5.1 and Numpy 1.5.1

2011-04-11 Thread Skipper Seabold
All, We noticed some failing tests for statsmodels between numpy 1.5.1 and numpy = 1.6.0. These are the versions where I noticed the change. It seems that when you divide a float array and multiply by a boolean array the answers are different (unless the others are also off by some floating

Re: [Numpy-discussion] Shared memory ndarrays (update)

2011-04-11 Thread srean
Got you and thanks a lot for the explanation. I am not using Queues so I think I am safe for the time being. Given that you have worked a lot on these issues, would you recommend plain mmapped numpy arrays over multiprocessing.Array Thanks again -- srean On Mon, Apr 11, 2011 at 1:36 PM, Sturla

Re: [Numpy-discussion] Shared memory ndarrays (update)

2011-04-11 Thread Sturla Molden
Den 11.04.2011 01:20, skrev Sturla Molden: Changes: - 64-bit support. - Memory leak on Linux/Unix should be gone (monkey patch for os._exit). - Added a global lock as there are callbacks to Python (the GIL is not sufficient serialization). I will also add a barrier synchronization

Re: [Numpy-discussion] Shared memory ndarrays (update)

2011-04-11 Thread Sturla Molden
Den 11.04.2011 21:15, skrev srean: Got you and thanks a lot for the explanation. I am not using Queues so I think I am safe for the time being. Given that you have worked a lot on these issues, would you recommend plain mmapped numpy arrays over |multiprocessing.Array| | With multiprocessing

Re: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1 and Numpy 1.5.1

2011-04-11 Thread Robert Kern
On Mon, Apr 11, 2011 at 13:54, Skipper Seabold jsseab...@gmail.com wrote: All, We noticed some failing tests for statsmodels between numpy 1.5.1 and numpy = 1.6.0. These are the versions where I noticed the change. It seems that when you divide a float array and multiply by a boolean array

Re: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1 and Numpy 1.5.1

2011-04-11 Thread Charles R Harris
On Mon, Apr 11, 2011 at 12:54 PM, Skipper Seabold jsseab...@gmail.comwrote: All, We noticed some failing tests for statsmodels between numpy 1.5.1 and numpy = 1.6.0. These are the versions where I noticed the change. It seems that when you divide a float array and multiply by a boolean

Re: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1 and Numpy 1.5.1

2011-04-11 Thread Mark Wiebe
On Mon, Apr 11, 2011 at 12:37 PM, Robert Kern robert.k...@gmail.com wrote: On Mon, Apr 11, 2011 at 13:54, Skipper Seabold jsseab...@gmail.com wrote: All, We noticed some failing tests for statsmodels between numpy 1.5.1 and numpy = 1.6.0. These are the versions where I noticed the

Re: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1 and Numpy 1.5.1

2011-04-11 Thread Charles R Harris
On Mon, Apr 11, 2011 at 2:31 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Apr 11, 2011 at 12:37 PM, Robert Kern robert.k...@gmail.comwrote: On Mon, Apr 11, 2011 at 13:54, Skipper Seabold jsseab...@gmail.com wrote: All, We noticed some failing tests for statsmodels between numpy 1.5.1

[Numpy-discussion] Extending numpy statistics functions (like mean)

2011-04-11 Thread Sergio Pascual
Hi list. For mi application, I would like to implement some new statistics functions over numpy arrays, such as truncated mean. Ideally this new function should have the same arguments than numpy.mean: axis, dtype and out. Is there a way of writing this function that doesn't imply writing it in C

Re: [Numpy-discussion] Extending numpy statistics functions (like mean)

2011-04-11 Thread Sergio Pascual
What I have is some C++ functions that implement statistic functions. What I need is some kind of ufunc where I can plug my functions. But I doesn't seem to exist an ufunc that operates on a N-d array and turns it into a number. 2011/4/12 Keith Goodman kwgood...@gmail.com: On Mon, Apr 11, 2011

Re: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1 and Numpy 1.5.1

2011-04-11 Thread Travis Oliphant
On Apr 11, 2011, at 3:55 PM, Charles R Harris wrote: On Mon, Apr 11, 2011 at 2:31 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Apr 11, 2011 at 12:37 PM, Robert Kern robert.k...@gmail.com wrote: On Mon, Apr 11, 2011 at 13:54, Skipper Seabold jsseab...@gmail.com wrote: All, We

Re: [Numpy-discussion] Odd numerical difference between Numpy 1.5.1 and Numpy 1.5.1

2011-04-11 Thread Mark Wiebe
On Mon, Apr 11, 2011 at 8:48 PM, Travis Oliphant oliph...@enthought.comwrote: On Apr 11, 2011, at 3:55 PM, Charles R Harris wrote: snip I agree with Charles. Let's take the needed time and work this through. This is the sort of thing I was a bit nervous about with the changes made to the