[issue41732] Custom iterator to memoryview - performance improvement
Diogo Flores added the comment: Thank you Dong-hee Na and Victor. @Stefan Krah: I am sorry that you didn't had the chace to try out the patch before it got merged, however iff you are curious about the any aspect of the implementation (or the motivation behind it) I am happy to discuss it in depth. -- ___ Python tracker <https://bugs.python.org/issue41732> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41732] Custom iterator to memoryview - performance improvement
Change by Diogo Flores : -- keywords: +patch pull_requests: +21203 stage: -> patch review pull_request: https://github.com/python/cpython/pull/22119 ___ Python tracker <https://bugs.python.org/issue41732> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41732] Custom iterator to memoryview - performance improvement
New submission from Diogo Flores : Hi, Here's a patch that adds an iterator to memoryview, which makes iterating over memoryview objs considerably faster. The following values result from a compilation with debug ON. Without patch: cpython: ./python.exe -m timeit -s "a=memoryview(b'x'*100)" "for x in a: pass" 5 loops, best of 5: 98 msec per loop cpython: ./python.exe -m timeit -s "a=b'x'*100" "for x in a: pass" 5 loops, best of 5: 68.6 msec per loop With patch: cpython: ./python.exe -m timeit -s "a=memoryview(b'x'*100)" "for x in a: pass" 5 loops, best of 5: 68.1 msec per loop cpython: ./python.exe -m timeit -s "a=b'x'*100" "for x in a: pass" 5 loops, best of 5: 70 msec per loop Please let me know your thoughts regarding it. Diogo -- messages: 376460 nosy: dxflores priority: normal severity: normal status: open title: Custom iterator to memoryview - performance improvement versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue41732> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39959] Bug on multiprocessing.shared_memory
Diogo Flores added the comment: I have tried a different approach using https://gitlab.com/tenzing/shared-array and I got it to perform well on Linux. Basically, the code above places all numpy arrays in /dev/shm which allows you to access and modify them from any number of processes without creating any copies; for deleting is equally simple - The code provides a SharedArray.list() to list all objects that itself placed in /dev/shm and so one can just iterate over the list and delete each element. (An easier approach is to use PathLib and just unlik all shared memory objects in /dev/shm) I guess a solution based on Mat's code could be adapted to try and solve the shared-memory problems. I look forward for further discussion on the subject. Diogo -- ___ Python tracker <https://bugs.python.org/issue39959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39959] Bug on multiprocessing.shared_memory
Diogo Flores added the comment: Any update on this issue? -- title: (Possible) bug on multiprocessing.shared_memory -> Bug on multiprocessing.shared_memory ___ Python tracker <https://bugs.python.org/issue39959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39959] (Possible) bug on multiprocessing.shared_memory
Diogo Flores added the comment: Follow up - tested on Linux (The first solution). The solution presented below will fix the problem with the caveat that the base process (the one that creates the shared-memory obj) must outlive any process that use the shared-memory. The rationale is that unless *this* process is creating a new shared-memory object (as opposed to attaching itself to an already existing one), then there is no point to register itself to be tracked. By making this small change, the problem I mentioned when I opened this issue disappears. # # https://github.com/python/cpython/blob/master/Lib/multiprocessing/shared_memory.py#L116 by changing: from .resource_tracker import register register(self._name, "shared_memory") # To: if create: from .resource_tracker import register register(self._name, "shared_memory") # To retain the ability for the base process to be able to exit before those that use the shared-memory obj that the base process itself created (the current/problematic implementation), as well as fix the issue, I suggest the following approach: When (and only when) a new shared-memory obj is created, such is registered on a new class variable of the resource-tracker, hence it can always be accessed and closed/unlinked by any process later on - this differs from the current approach, where each process that wants to access the shared-memory obj is being registered on the resource-tracker. I look forward for any discussion on the subject. Thank you, Diogo -- ___ Python tracker <https://bugs.python.org/issue39959> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39971] Error on documentation - Quick fix.
Diogo Flores added the comment: Agreed, Eric. Nevertheless, I think that this example might raise unnecessary confusion to someone new to Python, and hence the reason why I decided to open the issue. Anyhow, I trust your judgement and please feel free to leave things as is and close this issue. PS: I really the work you did with dataclasses, and I actually explored the idea further with a small (300 LoC) library that adds runtime type checking for instances of dataclasses - if you are ever curious you can find the code on my GitHub at /dxflores/invis. Thank you, Diogo -- ___ Python tracker <https://bugs.python.org/issue39971> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39971] Error on documentation - Quick fix.
New submission from Diogo Flores : Hello, A code example from the 'functional programming how-to' raises an error. The simplest fix would be to remove the Ellipsis object from the 'line_list'. Thank you, Diogo Please check below for the commands issued: # https://docs.python.org/3/howto/functional.html#generator-expressions-and-list-comprehensions >>> line_list = [' line 1\n', 'line 2 \n', ...] >>> >>> # Generator expression -- returns iterator >>> stripped_iter = (line.strip() for line in line_list) >>> >>> # List comprehension -- returns list >>> stripped_list = [line.strip() for line in line_list] Traceback (most recent call last): File "", line 1, in File "", line 1, in AttributeError: 'ellipsis' object has no attribute 'strip' -- assignee: docs@python components: Documentation messages: 364247 nosy: docs@python, dxflores priority: normal severity: normal status: open title: Error on documentation - Quick fix. versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue39971> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39959] (Possible) bug on multiprocessing.shared_memory
New submission from Diogo Flores :
Hello,
I came across with what seems like a bug (or at least a disagreement with the
current documentation).
Discussion:
I expected that after creating a numpy-array (on terminal 1), which is backed
by shared memory, I would be able to use it in other terminals until I would
call `shm.unlink()` (on terminal 1), at which point, the memory block would be
released and no longer accessible.
What happened is that after accessing the numpy-array from terminal 2, I called
'close()' on the local 'existing_shm' instance and exited the interpreter,
which displayed the `warning` seen below.
After, I tried to access the same shared memory block from terminal 3, and a
FileNotFoundError was raised. (The same error was also raised when I tried to
call 'shm.unlink()' on terminal 1, after calling 'close()' on terminal 2.)
It seems that calling `close()` on an instance, destroys further access to the
shared memory block from any point, while what I expected was to be able to
access the array (i.e. on terminal 2), modify it, "close" my access to it, and
after be able to access the modified array on i.e. terminal 3.
If the error is on my side I apologize for raising this issue and I would
appreciate for clarification on what I am doing wrong.
Thank you.
Diogo
Please check below for the commands issued:
## Terminal 1
>>> from multiprocessing import shared_memory
>>> import numpy as np
>>>
>>> a = np.array([x for x in range(10)])
>>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
>>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
>>> b[:] = a[:]
>>>
>>> shm.name
'psm_592ec635'
## Terminal 2
>>> from multiprocessing import shared_memory
>>> import numpy as np
>>>
>>> existing_shm = shared_memory.SharedMemory('psm_592ec635')
>>> c = np.ndarray((10,), dtype=np.int64, buffer=existing_shm.buf)
>>>
>>> c
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>>
>>> del c
>>> existing_shm.close()
>>>
>>> exit()
~: /usr/lib/python3.8/multiprocessing/resource_tracker.py:203: UserWarning:
resource_tracker: There appear to be 1 leaked shared_memory objects to clean up
at shutdown
warnings.warn('resource_tracker: There appear to be %d '
## Finally, on terminal 3
>>> from multiprocessing import shared_memory
>>> import numpy as np
>>>
>>> existing_shm = shared_memory.SharedMemory('psm_592ec635')
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.8/multiprocessing/shared_memory.py", line 100, in
__init__
self._fd = _posixshmem.shm_open(
FileNotFoundError: [Errno 2] No such file or directory: '/psm_592ec635'
--
components: Library (Lib)
messages: 364121
nosy: dxflores
priority: normal
severity: normal
status: open
title: (Possible) bug on multiprocessing.shared_memory
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue39959>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
