[issue14130] memoryview: add multi-dimensional indexing and slicing
Ian Beaver added the comment: Its not multi-dimensional slicing to get a subset of objects as in Numpy, but more the ability to slice a buffer containing a multi-dimensional array as raw bytes. Buffer objects in Python2.7 are dimensionality naive so it works fine. You were correct that I was testing against Python3.2, in Python3.3 the slicing of ndim 1 works, however only for reading from the buffer. I still can't write back into a memoryview object with ndim 1 in Python 3.3. Python 2.7.3: import numpy as np arr = np.zeros(shape=(100,100)) type(arr.data) type 'buffer' arr.data[0:10] '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' Python 3.2.3: import numpy as np arr = np.zeros(shape=(100,100)) type(arr.data) class 'memoryview' arr.data[0:10] Traceback (most recent call last): File stdin, line 1, in module NotImplementedError Python 3.3.3: import numpy as np arr = np.zeros(shape=(100,100)) type(arr.data) class 'memoryview' arr.data[0:10] memory at 0x7faaf1d03a48 However to write data back into a buffer: Python 2.7.3: import numpy as np arr = np.zeros(shape=(100,100)) arr.data[0:10] = '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' Python 3.2.3: import numpy as np arr = np.zeros(shape=(100,100)) arr.data[0:10] = b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' Traceback (most recent call last): File stdin, line 1, in module NotImplementedError Python 3.3.3: import numpy as np arr = np.zeros(shape=(100,100)) arr.data[0:10] = b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' Traceback (most recent call last): File stdin, line 1, in module NotImplementedError: memoryview assignments are currently restricted to ndim = 1 Also the slice in Python3.3 is not the same as just returning a chunk of raw bytes from the memory buffer, instead of a bytes object the indexing behaves similar to numpy array indexes and you get the (sub) array items back as Python objects. Python2.7.3: import numpy as np arr = np.zeros(shape=(100,100)) arr.data[0:10] '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' len(bytes(arr.data[0:10])) 10 Python3.3.3: import numpy as np arr = np.zeros(shape=(100,100)) arr.data[0:10] memory at 0x7f109a71ea48 len(bytes(arr.data[0:10])) 8000 This is not a big deal in my case since I already have numpy arrays I can just use bytes(arr.flat[start:end]) to scan through the array contents as byte chunks, but that would not be possible with just a memoryview object like it was with the Python2 buffer object without converting it to something else or dropping to ctypes and iterating over the memory addresses and dereferencing the contents. So in Python3.3 its halfway to the functionality in Python2.7, I can send chunks of the data through a compressed or encrypted stream, but I can't rebuild the data on the other side without first creating a bytearray and eating the cost of a copy into a memoryview. All I really need is a way to reconstruct the original memoryview buffer in memory from a stream of bytes without having to make a temporary object first and then copy its contents into the final memoryview object when it is complete. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14130 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14130] memoryview: add multi-dimensional indexing and slicing
Ian Beaver added the comment: If there is any way to get this implemented, it is needed. For one, the docs on memoryview make no mention that indexing and slicing doesn't work with multi-dimensional data which led me to believe it was supported until I tried using it. A second reason is currently this represents a loss of functionality from the buffer type in python2. In porting code using the buffer type in python2 to python3, you get a very unhelpful NotImplementedError with no description when trying to slice a memoryview. There is no workaround but to call tobytes() and copy the data in memory to an object that supports slicing, but for very large objects this defeats the primary purpose of using buffers in the first place, which is to avoid memory copies. -- nosy: +undercoveridiot ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14130 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1230540] sys.excepthook doesn't work in threads
Ian Beaver undercoverid...@gmail.com added the comment: I found that the workaround suggested doesn't work when you have a subclass of threading.Thread and you want to catch everything in the module that contains the class to a common log. Say you have a module with a socket server that spawns a thread on accept and you want to log anything that tracebacks in the module. This seems to work: import sys import logging from functools import wraps def myExceptHook(type, value, tb): Redirect tracebacks to error log import traceback rawreport = traceback.format_exception(type, value, tb) report = '\n'.join(rawreport) log.error(report) sys.excepthook = myExceptHook def use_my_excepthook(view): Redirect any unexpected tracebacks @wraps(view) def run(*args, **kwargs): try: return view(*args, **kwargs) except: sys.excepthook(*sys.exc_info()) return run Then in your thread subclass: class MyThread(threading.Thread): def __init__(self, socket_conn): threading.Thread.__init__(self) self.my_conn = socket_conn @use_my_excepthook def run(self): Do stuff -- nosy: +undercoveridiot ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1230540 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1230540] sys.excepthook doesn't work in threads
Ian Beaver undercoverid...@gmail.com added the comment: Instead of using decorators, this is a slightly simpler modification to the proposed workaround that allows for any subclassed run method to also be caught. def installThreadExcepthook(): Workaround for sys.excepthook thread bug From http://spyced.blogspot.com/2007/06/workaround-for-sysexcepthook-bug.html (https://sourceforge.net/tracker/?func=detailatid=105470aid=1230540group_id=5470). Call once from __main__ before creating any threads. If using psyco, call psyco.cannotcompile(threading.Thread.run) since this replaces a new-style class method. init_old = threading.Thread.__init__ def init(self, *args, **kwargs): init_old(self, *args, **kwargs) run_old = self.run def run_with_except_hook(*args, **kw): try: run_old(*args, **kw) except (KeyboardInterrupt, SystemExit): raise except: sys.excepthook(*sys.exc_info()) self.run = run_with_except_hook threading.Thread.__init__ = init -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1230540 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
