Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
Ray Schumacher wrote: > At 05:22 AM 10/9/2007, David Cournapeau wrote: > >> Could not this be because you compiled the posh sources with a >> compiler/runtime which is different than the other extensions and python >> interpreter ? >> > > It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and > anyone else who didn't download it in time) are now seemingly SOL > since it is no longer available. I saw much discussion of this as > well, but even 2.5 is now "fixed" on 7.1 and reports of compiling > distutil modules with the new MS SDK and having them work at all with > 2.4 were very mixed. I also tried GCC and had a litany of other > errors with the posh. > Basically, you cannot expect file descriptors (or even file handles: the standard ones from C library fopen) to cross dll boundaries if the dll do not have exactly the same runtime. The only solution is to convert your file descriptor to the non standard HANDLE things from the Win32 API, and pass this at the API level. Now, I have no idea if this works also for mmap files; maybe this is broken too. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes? CODE...
# numpyShared.py import numpy as N def arrSharedMemory(shape, dtype, tag="PySharedMemory", access=None): """ Windows only ! share memory between different processes if same `tag` is used. """ itemsize = N.dtype(dtype).itemsize count = N.product(shape) size = count * itemsize import mmap sharedmem = mmap.mmap(0, size, tag, access) a=N.frombuffer(sharedmem, dtype, count) a.shape = shape return a if __name__ == '__main__': ## test it ## produce a mish-mash of output on one shell screen import random import time import subprocess, sys import mmap if len(sys.argv)<2: pid = subprocess.Popen([r'C:\python24\python.exe', ['numpyShared.py rjs100']]).pid ## the parent process a1 = arrSharedMemory((1,), N.int32, tag='rjs100', access=mmap.ACCESS_WRITE) time.sleep(.7) now=time.time(); while time.time()http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
Ray S wrote: > Yes, a different methodology attempt. It would be interesting to know > anyway how to create a numpy array from an address; it's probably > buried in the undocumented C-API that I don't grok, and likely > frowned upon. Make a dummy object that exposes the __array_interface__ attribute filled with the appropriate information: http://numpy.scipy.org/array_interface.shtml Something like the following should suffice (untested, though I've done similar things with ctypes before): import numpy def fromaddress(address, dtype, shape, strides=None): """ Create a numpy array from an integer address, a dtype, a shape tuple, and possibly strides. """ # Make sure our dtype is a dtype, not just "f" or whatever. dtype = numpy.dtype(dtype) class Dummy(object): pass d = Dummy() d.__array_interface__ = dict( data = (address, False), typestr = dtype.str, descr = dtype.descr, shape = shape, strides = strides, version = 3, ) return numpy.asarray(d) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
On 10/9/07, Sebastian Haase replied: > > Did you find that locks > > or semaphores were needed? > Maybe that's why it crashed ;-) !? But for simple use it seems fine. I just did some code (below) that does read/write to the array AFAP, and there is no crash, or any other issue (Win2000, py2.4, numpy 1.0b1). Without the print statements, it does max both processors; with printing I/O only 58%. Both processes can modify the array without issue either. I'll experiment with I had seen the Win mmap in this thread: http://objectmix.com/python/127666-shared-memory-pointer.html and here: http://www.codeproject.com/cpp/embedpython_2.asp Note also that the Python mmap docs read "In either case you must provide a file descriptor for a file opened for update." and no mention of the integer zero descriptor option. Access options behave as presented. Because *NIX has MAP_SHARED as an option you'd think that there might be cross-platform share behavior with some platform checking if statements. Without a tag though, how does another process reference the same memory on NIX, a filename? (It seems) > > But I had the same > > issue as Mark Heslep > > http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422 > > of creating a numpy array from a raw address (not a c_array). >I assume this is a different issue, but haven't looked into it yet. Yes, a different methodology attempt. It would be interesting to know anyway how to create a numpy array from an address; it's probably buried in the undocumented C-API that I don't grok, and likely frowned upon. Thanks, Ray ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Warnings
How can I get the line number of where a numpy warning message is envoked in my code? Cheers Tommy ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] diff of find of diff trick
Hello, As a converting MATLAB user I am bringing over some of my code. A core function I need is to sample the probability distribution of a given set of data. Sometimes these sets are large so I would like to make it as efficient as possible. (the data are integers representing members of a discrete space) In MATLAB the best way I found was the "diff of find of diff" trick which resulted in the completely vectorised solution (below). Does it make sense to translate this into numpy? I don't have a feel yet for what is fast/slow - are the functions below built in and so quick (I thought diff might not be). Is there a better/more pythonic way to do it? function Pr=prob(data, nR) Pr= zeros(nR,1); % diff of find of diff trick for counting number of elements temp = sort(data(data>0));% want vector excluding P(0) dtemp= diff([temp;max(temp)+1]); count = diff(find([1;dtemp])); indx = temp(dtemp>0); Pr(indx)= count ./ numel(data);% probability Thanks Robin ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
On 10/9/07, Ray Schumacher <[EMAIL PROTECTED]> wrote: > At 05:22 AM 10/9/2007, David Cournapeau wrote: > >Could not this be because you compiled the posh sources with a > >compiler/runtime which is different than the other extensions and python > >interpreter ? > > It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and > anyone else who didn't download it in time) are now seemingly SOL > since it is no longer available. I saw much discussion of this as > well, but even 2.5 is now "fixed" on 7.1 and reports of compiling > distutil modules with the new MS SDK and having them work at all with > 2.4 were very mixed. I also tried GCC and had a litany of other > errors with the posh. > > >Sebastian Haase added: > >I was in fact experimenting with this. The solution seemed to lie in > >"simple" memmap as it is implemented in Windows: > > > I had just found and started to write some tests with that MS > function. If I can truly write to the array in one process and > instantly read it in the other I'll be happy. Did you find that locks > or semaphores were needed? Maybe that's why it crashed ;-) !? But for simple use it seems fine. > > >(( I have to mention, that I could crash a process while testing this ... )) > > That was one of my first results! I also found that using ctypes to > create arrays from the other process's address and laying a numpy > array on top was prone to that in experimentation. But I had the same > issue as Mark Heslep > http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422 > of creating a numpy array from a raw address (not a c_array). > I assume this is a different issue, but haven't looked into it yet. -Sebastian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
At 05:22 AM 10/9/2007, David Cournapeau wrote: >Could not this be because you compiled the posh sources with a >compiler/runtime which is different than the other extensions and python >interpreter ? It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and anyone else who didn't download it in time) are now seemingly SOL since it is no longer available. I saw much discussion of this as well, but even 2.5 is now "fixed" on 7.1 and reports of compiling distutil modules with the new MS SDK and having them work at all with 2.4 were very mixed. I also tried GCC and had a litany of other errors with the posh. >Sebastian Haase added: >I was in fact experimenting with this. The solution seemed to lie in >"simple" memmap as it is implemented in Windows: I had just found and started to write some tests with that MS function. If I can truly write to the array in one process and instantly read it in the other I'll be happy. Did you find that locks or semaphores were needed? >(( I have to mention, that I could crash a process while testing this ... )) That was one of my first results! I also found that using ctypes to create arrays from the other process's address and laying a numpy array on top was prone to that in experimentation. But I had the same issue as Mark Heslep http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422 of creating a numpy array from a raw address (not a c_array). Thanks, Ray Schumacher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
On 10/9/07, Sebastian Haase <[EMAIL PROTECTED]> wrote: > > > As I recollect, the tag thing was the key for turning the mmap into a > "not really memmaped file", that is, a memmap without a corresponding > file on the disk. All this is windows specific, I have never seen this tag thing on other OS. > In other words, isn't a mmap ( without(!) tag ) always bound to a > "real" file in the file system ? No: mmap makes the link between a memory area and a file descriptor. Nothing force the file descriptor to be bound to a "real" file. POSIX defines shm_open to create shared memory file descriptor. AFAIK, this is the standard way to share memory between processes on modern unices: first create a memory area using shm_open, and then mmap it. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
On 10/9/07, David Cournapeau <[EMAIL PROTECTED]> wrote: > Sebastian Haase wrote: > > Hi! > > I was in fact experimenting with this. The solution seemed to lie in > > "simple" memmap as it is implemented in Windows: > > > > import numpy as N > > def arrSharedMemory(shape, dtype, tag="PriithonSharedMemory"): > > """ > > Windows only ! > > share memory between different processes if same `tag` is used. > > """ > > itemsize = N.dtype(dtype).itemsize > > count = N.product(shape) > > size = count * itemsize > > > > import mmap > > sharedmem = mmap.mmap(0, size, tag) > > a=N.frombuffer(sharedmem, dtype, count) > > a.shape = shape > > return a > > > > For explaintion look up the microsoft site for the mmap documentation. > > And/or the Python-doc for mmap. > > (( I have to mention, that I could crash a process while testing this ... )) > > > > If anyone here would know an equivalent way of doing this on > > Linux/OS-X we were back to a cross-platfrom function. > > > AFAIK, the tag thing is pretty much windows specific, so why not just > ignoring it on non windows platforms ? (or interpreting the tag argument > as the flag argument for mmap, which would be consistent with python > mmap API ?) > As I recollect, the tag thing was the key for turning the mmap into a "not really memmaped file", that is, a memmap without a corresponding file on the disk. In other words, isn't a mmap ( without(!) tag ) always bound to a "real" file in the file system ? -Sebastian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
Sebastian Haase wrote: > Hi! > I was in fact experimenting with this. The solution seemed to lie in > "simple" memmap as it is implemented in Windows: > > import numpy as N > def arrSharedMemory(shape, dtype, tag="PriithonSharedMemory"): > """ > Windows only ! > share memory between different processes if same `tag` is used. > """ > itemsize = N.dtype(dtype).itemsize > count = N.product(shape) > size = count * itemsize > > import mmap > sharedmem = mmap.mmap(0, size, tag) > a=N.frombuffer(sharedmem, dtype, count) > a.shape = shape > return a > > For explaintion look up the microsoft site for the mmap documentation. > And/or the Python-doc for mmap. > (( I have to mention, that I could crash a process while testing this ... )) > > If anyone here would know an equivalent way of doing this on > Linux/OS-X we were back to a cross-platfrom function. > AFAIK, the tag thing is pretty much windows specific, so why not just ignoring it on non windows platforms ? (or interpreting the tag argument as the flag argument for mmap, which would be consistent with python mmap API ?) cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] beginner question: rank-1 arrays
Alan G Isaac schrieb: > On Mon, 8 Oct 2007, Robin apparently wrote: >> However in my code (I am converting from MATLAB) it is >> important to maintain 2d arrays, and keep the difference >> between row and column vectors. > > How about using matrices? > help(numpy.mat) > > hth, > Alan Isaac > Robin, Alan is right, you want numpy matrices which are always 2d. Check out numpy.matlib; if you replace from numpy import [whatever] by from numpy.matlib import [whatever] you get everything there is in numpy, and things like ones() zeros() empty() etc. will always be 2d matrices. -sven ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy/Windows shared arrays between processes?
Hi! I was in fact experimenting with this. The solution seemed to lie in "simple" memmap as it is implemented in Windows: import numpy as N def arrSharedMemory(shape, dtype, tag="PriithonSharedMemory"): """ Windows only ! share memory between different processes if same `tag` is used. """ itemsize = N.dtype(dtype).itemsize count = N.product(shape) size = count * itemsize import mmap sharedmem = mmap.mmap(0, size, tag) a=N.frombuffer(sharedmem, dtype, count) a.shape = shape return a For explaintion look up the microsoft site for the mmap documentation. And/or the Python-doc for mmap. (( I have to mention, that I could crash a process while testing this ... )) If anyone here would know an equivalent way of doing this on Linux/OS-X we were back to a cross-platfrom function. Hope this helps, Sebastian Haase On 10/9/07, David Cournapeau <[EMAIL PROTECTED]> wrote: > Ray S wrote: > > Is anyone sharing arrays between processes on Windows? > > I tried compiling the posh sources (once, so far) with the new MS > > toolkit and failed... > > What other solutions are in use? > > > > Have a second process create an array view from an address would > > suffice for this particular purpose. I could pass the address as a > > parameter of the second process's argv. > > > > I've also tried things like > > pb=pythonapi.PyBuffer_FromReadWriteMemory(9508824, 9*sizeof(c_int)) > > N.frombuffer(pb, N.int32) > > which fails since pb is and int. What are my options? > > > (disclaimer: I know nothing about windows idiosyncraties) > > Could not this be because you compiled the posh sources with a > compiler/runtime which is different than the other extensions and python > interpreter ? I don't know the details, but since most of the posix > functions related to files and processes are broken beyond despair in > windows, and in particular, many posix handles cannot cross dll > boundaries compiled by different compilers, I would not be surprised if > this cause some trouble. > > The fact that POSH is said to be posix-only on python.org > (http://wiki.python.org/moin/ParallelProcessing) would imply that people > do not care much about windows, too (but again, this is just from > reading what posh is about; I have never used it personnally). > > cheers, > > David > > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
