Re: [Numpy-discussion] Memmap with multiprocessing
On Tue, Apr 26, 2011 at 6:21 PM, Ralf Gommers ralf.gomm...@googlemail.com wrote: On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes totonixs...@gmail.com wrote: Hi, Has anyone confirmed if this is a bug? Should I post this in the bug tracker? I see the same thing with recent master. Something very strange is going on in the memmap.__array_finalize__ method under Windows. Can you file a bug? Ralf Hi Ralf, Done http://projects.scipy.org/numpy/ticket/1809 Thanks! Thanks! On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes totonixs...@gmail.com wrote: Hi all, I'm having a error using memmap objects shared among processes created by the multprocessing module. This error only happen in Windows with numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac OS X it doesn't happen. This error is demonstrated by this little example script here https://gist.github.com/929168 , and the traceback is bellow (between traceback tags): traceback Process Process-1: Traceback (most recent call last): File C:\Python26\Lib\multiprocessing\process.py, line 232, in _bootstrap self.run() File C:\Python26\Lib\multiprocessing\process.py, line 88, in run self._target(*self._args, **self._kwargs) File C:\Documents and Settings\phamorim\Desktop\test.py, line 7, in print_ma trix print matrix File C:\Python26\Lib\site-packages\numpy\core\numeric.py, line 1379, in arra y_str return array2string(a, max_line_width, precision, suppress_small, ' ', , s tr) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 309, in ar ray2string separator, prefix) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 189, in _a rray2string data = _leading_trailing(a) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 162, in _l eading_trailing min(len(a), _summaryEdgeItems))] File C:\Python26\Lib\site-packages\numpy\core\memmap.py, line 257, in __arra y_finalize__ self.filename = obj.filename AttributeError: 'memmap' object has no attribute 'filename' Exception AttributeError: AttributeError('NoneType' object has no attribute 'te ll',) in bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16) ignored /traceback I don't know if it's a bug, but I thought it's is import to report because the version 1.4.1 was working and 1.5.0 and above was not. Thanks! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memmap with multiprocessing
I don't think this was working correctly in numpy 1.4 either. The underlying problem seems to be that instance attributes of ndarray subtypes get lost during pickling: import pickle import numpy as np class aarray(np.ndarray): def __new__(subtype): self = np.ndarray.__new__(subtype, (1,)) self.attr = 'attr' return self def __array_finalize__(self, obj): self.attr = getattr(obj, 'attr', None) a = aarray() b = pickle.loads(a.dumps()) assert a.attr == b.attr, (a.attr, b.attr) AssertionError: ('attr', None) Christoph On 4/26/2011 2:21 PM, Ralf Gommers wrote: On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes totonixs...@gmail.com wrote: Hi, Has anyone confirmed if this is a bug? Should I post this in the bug tracker? I see the same thing with recent master. Something very strange is going on in the memmap.__array_finalize__ method under Windows. Can you file a bug? Ralf Thanks! On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes totonixs...@gmail.com wrote: Hi all, I'm having a error using memmap objects shared among processes created by the multprocessing module. This error only happen in Windows with numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac OS X it doesn't happen. This error is demonstrated by this little example script here https://gist.github.com/929168 , and the traceback is bellow (betweentraceback tags): traceback Process Process-1: Traceback (most recent call last): File C:\Python26\Lib\multiprocessing\process.py, line 232, in _bootstrap self.run() File C:\Python26\Lib\multiprocessing\process.py, line 88, in run self._target(*self._args, **self._kwargs) File C:\Documents and Settings\phamorim\Desktop\test.py, line 7, in print_ma trix print matrix File C:\Python26\Lib\site-packages\numpy\core\numeric.py, line 1379, in arra y_str return array2string(a, max_line_width, precision, suppress_small, ' ', , s tr) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 309, in ar ray2string separator, prefix) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 189, in _a rray2string data = _leading_trailing(a) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 162, in _l eading_trailing min(len(a), _summaryEdgeItems))] File C:\Python26\Lib\site-packages\numpy\core\memmap.py, line 257, in __arra y_finalize__ self.filename = obj.filename AttributeError: 'memmap' object has no attribute 'filename' Exception AttributeError: AttributeError('NoneType' object has no attribute 'te ll',) inbound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16) ignored /traceback I don't know if it's a bug, but I thought it's is import to report because the version 1.4.1 was working and 1.5.0 and above was not. Thanks! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memmap with multiprocessing
On Wed, Apr 27, 2011 at 4:07 PM, Christoph Gohlke cgoh...@uci.edu wrote: I don't think this was working correctly in numpy 1.4 either. The underlying problem seems to be that instance attributes of ndarray subtypes get lost during pickling: import pickle import numpy as np class aarray(np.ndarray): def __new__(subtype): self = np.ndarray.__new__(subtype, (1,)) self.attr = 'attr' return self def __array_finalize__(self, obj): self.attr = getattr(obj, 'attr', None) a = aarray() b = pickle.loads(a.dumps()) assert a.attr == b.attr, (a.attr, b.attr) AssertionError: ('attr', None) Christoph possibly related to this ticket: http://projects.scipy.org/numpy/ticket/1452 On 4/26/2011 2:21 PM, Ralf Gommers wrote: On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes totonixs...@gmail.com wrote: Hi, Has anyone confirmed if this is a bug? Should I post this in the bug tracker? I see the same thing with recent master. Something very strange is going on in the memmap.__array_finalize__ method under Windows. Can you file a bug? Ralf Thanks! On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes totonixs...@gmail.com wrote: Hi all, I'm having a error using memmap objects shared among processes created by the multprocessing module. This error only happen in Windows with numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac OS X it doesn't happen. This error is demonstrated by this little example script here https://gist.github.com/929168 , and the traceback is bellow (betweentraceback tags): traceback Process Process-1: Traceback (most recent call last): File C:\Python26\Lib\multiprocessing\process.py, line 232, in _bootstrap self.run() File C:\Python26\Lib\multiprocessing\process.py, line 88, in run self._target(*self._args, **self._kwargs) File C:\Documents and Settings\phamorim\Desktop\test.py, line 7, in print_ma trix print matrix File C:\Python26\Lib\site-packages\numpy\core\numeric.py, line 1379, in arra y_str return array2string(a, max_line_width, precision, suppress_small, ' ', , s tr) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 309, in ar ray2string separator, prefix) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 189, in _a rray2string data = _leading_trailing(a) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 162, in _l eading_trailing min(len(a), _summaryEdgeItems))] File C:\Python26\Lib\site-packages\numpy\core\memmap.py, line 257, in __arra y_finalize__ self.filename = obj.filename AttributeError: 'memmap' object has no attribute 'filename' Exception AttributeError: AttributeError('NoneType' object has no attribute 'te ll',) inbound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16) ignored /traceback I don't know if it's a bug, but I thought it's is import to report because the version 1.4.1 was working and 1.5.0 and above was not. Thanks! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memmap with multiprocessing
On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes totonixs...@gmail.com wrote: Hi, Has anyone confirmed if this is a bug? Should I post this in the bug tracker? I see the same thing with recent master. Something very strange is going on in the memmap.__array_finalize__ method under Windows. Can you file a bug? Ralf Thanks! On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes totonixs...@gmail.com wrote: Hi all, I'm having a error using memmap objects shared among processes created by the multprocessing module. This error only happen in Windows with numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac OS X it doesn't happen. This error is demonstrated by this little example script here https://gist.github.com/929168 , and the traceback is bellow (between traceback tags): traceback Process Process-1: Traceback (most recent call last): File C:\Python26\Lib\multiprocessing\process.py, line 232, in _bootstrap self.run() File C:\Python26\Lib\multiprocessing\process.py, line 88, in run self._target(*self._args, **self._kwargs) File C:\Documents and Settings\phamorim\Desktop\test.py, line 7, in print_ma trix print matrix File C:\Python26\Lib\site-packages\numpy\core\numeric.py, line 1379, in arra y_str return array2string(a, max_line_width, precision, suppress_small, ' ', , s tr) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 309, in ar ray2string separator, prefix) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 189, in _a rray2string data = _leading_trailing(a) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 162, in _l eading_trailing min(len(a), _summaryEdgeItems))] File C:\Python26\Lib\site-packages\numpy\core\memmap.py, line 257, in __arra y_finalize__ self.filename = obj.filename AttributeError: 'memmap' object has no attribute 'filename' Exception AttributeError: AttributeError('NoneType' object has no attribute 'te ll',) in bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16) ignored /traceback I don't know if it's a bug, but I thought it's is import to report because the version 1.4.1 was working and 1.5.0 and above was not. Thanks! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memmap with multiprocessing
On Tue, Apr 26, 2011 at 4:21 PM, Ralf Gommers ralf.gomm...@googlemail.comwrote: On Mon, Apr 25, 2011 at 1:16 PM, Thiago Franco Moraes totonixs...@gmail.com wrote: Hi, Has anyone confirmed if this is a bug? Should I post this in the bug tracker? I see the same thing with recent master. Something very strange is going on in the memmap.__array_finalize__ method under Windows. Can you file a bug? Just a confirmation: I get the error with numpy 1.5.1, Python 2.7.1 in Windows XP, but not in Mac OSX or Linux (Ubuntu). Warren Ralf Thanks! On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes totonixs...@gmail.com wrote: Hi all, I'm having a error using memmap objects shared among processes created by the multprocessing module. This error only happen in Windows with numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac OS X it doesn't happen. This error is demonstrated by this little example script here https://gist.github.com/929168 , and the traceback is bellow (between traceback tags): traceback Process Process-1: Traceback (most recent call last): File C:\Python26\Lib\multiprocessing\process.py, line 232, in _bootstrap self.run() File C:\Python26\Lib\multiprocessing\process.py, line 88, in run self._target(*self._args, **self._kwargs) File C:\Documents and Settings\phamorim\Desktop\test.py, line 7, in print_ma trix print matrix File C:\Python26\Lib\site-packages\numpy\core\numeric.py, line 1379, in arra y_str return array2string(a, max_line_width, precision, suppress_small, ' ', , s tr) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 309, in ar ray2string separator, prefix) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 189, in _a rray2string data = _leading_trailing(a) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 162, in _l eading_trailing min(len(a), _summaryEdgeItems))] File C:\Python26\Lib\site-packages\numpy\core\memmap.py, line 257, in __arra y_finalize__ self.filename = obj.filename AttributeError: 'memmap' object has no attribute 'filename' Exception AttributeError: AttributeError('NoneType' object has no attribute 'te ll',) in bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16) ignored /traceback I don't know if it's a bug, but I thought it's is import to report because the version 1.4.1 was working and 1.5.0 and above was not. Thanks! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memmap with multiprocessing
Hi, Has anyone confirmed if this is a bug? Should I post this in the bug tracker? Thanks! On Tue, Apr 19, 2011 at 9:01 PM, Thiago Franco de Moraes totonixs...@gmail.com wrote: Hi all, I'm having a error using memmap objects shared among processes created by the multprocessing module. This error only happen in Windows with numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac OS X it doesn't happen. This error is demonstrated by this little example script here https://gist.github.com/929168 , and the traceback is bellow (between traceback tags): traceback Process Process-1: Traceback (most recent call last): File C:\Python26\Lib\multiprocessing\process.py, line 232, in _bootstrap self.run() File C:\Python26\Lib\multiprocessing\process.py, line 88, in run self._target(*self._args, **self._kwargs) File C:\Documents and Settings\phamorim\Desktop\test.py, line 7, in print_ma trix print matrix File C:\Python26\Lib\site-packages\numpy\core\numeric.py, line 1379, in arra y_str return array2string(a, max_line_width, precision, suppress_small, ' ', , s tr) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 309, in ar ray2string separator, prefix) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 189, in _a rray2string data = _leading_trailing(a) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 162, in _l eading_trailing min(len(a), _summaryEdgeItems))] File C:\Python26\Lib\site-packages\numpy\core\memmap.py, line 257, in __arra y_finalize__ self.filename = obj.filename AttributeError: 'memmap' object has no attribute 'filename' Exception AttributeError: AttributeError('NoneType' object has no attribute 'te ll',) in bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16) ignored /traceback I don't know if it's a bug, but I thought it's is import to report because the version 1.4.1 was working and 1.5.0 and above was not. Thanks! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Memmap with multiprocessing
Hi all, I'm having a error using memmap objects shared among processes created by the multprocessing module. This error only happen in Windows with numpy 1.5 or above, in numpy 1.4.1 it doesn't happen, In Linux and Mac OS X it doesn't happen. This error is demonstrated by this little example script here https://gist.github.com/929168 , and the traceback is bellow (between traceback tags): traceback Process Process-1: Traceback (most recent call last): File C:\Python26\Lib\multiprocessing\process.py, line 232, in _bootstrap self.run() File C:\Python26\Lib\multiprocessing\process.py, line 88, in run self._target(*self._args, **self._kwargs) File C:\Documents and Settings\phamorim\Desktop\test.py, line 7, in print_ma trix print matrix File C:\Python26\Lib\site-packages\numpy\core\numeric.py, line 1379, in arra y_str return array2string(a, max_line_width, precision, suppress_small, ' ', , s tr) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 309, in ar ray2string separator, prefix) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 189, in _a rray2string data = _leading_trailing(a) File C:\Python26\Lib\site-packages\numpy\core\arrayprint.py, line 162, in _l eading_trailing min(len(a), _summaryEdgeItems))] File C:\Python26\Lib\site-packages\numpy\core\memmap.py, line 257, in __arra y_finalize__ self.filename = obj.filename AttributeError: 'memmap' object has no attribute 'filename' Exception AttributeError: AttributeError('NoneType' object has no attribute 'te ll',) in bound method memmap.__del__ of memmap([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int16) ignored /traceback I don't know if it's a bug, but I thought it's is import to report because the version 1.4.1 was working and 1.5.0 and above was not. Thanks! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion