[issue40860] Exception in multiprocessing/context.py under load
Arkady added the comment: "documentation of thread safety" I find it surprising that a module called "multiprocessing" has not a thread safe API. If this is inevitable, I guess that's the life. I expect nothing less that a bold bright red font at the top of the document page. Protecting a call to join() with a mutex would impact latency of the API: the slowest subprocess can win the race. -- ___ Python tracker <https://bugs.python.org/issue40860> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40860] Exception in multiprocessing/context.py under load
Arkady added the comment: I have switched to os.fork() I am doing something like this https://gist.github.com/larytet/3ca9f9a32b1dc089a24cb7011455141f -- ___ Python tracker <https://bugs.python.org/issue40860> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37788] fix for bpo-36402 (threading._shutdown() race condition) causes reference leak
Arkady added the comment: I have reproduced a similar memory leak in the multiprocessing This is the trace: 913 memory blocks: 33232 Bytes: File "/usr/lib/python3.7/threading.py", line 890;self._bootstrap_inner(); File "/usr/lib/python3.7/threading.py", line 914;self._set_tstate_lock(); File "/usr/lib/python3.7/threading.py", line 904;self._tstate_lock = _set_sentinel(); -- nosy: +Arkady M ___ Python tracker <https://bugs.python.org/issue37788> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40860] Exception in multiprocessing/context.py under load
Arkady added the comment: There is a memory leak every time call to join() fails (is https://bugs.python.org/issue37788 relevant?) -- ___ Python tracker <https://bugs.python.org/issue40860> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40860] Exception in multiprocessing/context.py under load
Arkady added the comment: A workaround is to synchronize the call to Process.start() diff --git a/main.py b/main.py index d09dc53..49d68f0 100644 --- a/main.py +++ b/main.py @@ -26,17 +26,24 @@ def load_cpu(deadline): while time.time() - start < 0.2*deadline: math.pow(random.randint(0, 1), random.randint(0, 1)) +def join_process(job, timeout): +time_start = time.time() +while time.time()-time_start < timeout and job.is_alive(): +time.sleep(0.1 * timeout) +continue + job_counter = 0 +lock = threading.Lock() def spawn_job(deadline): ''' Creat a new Process, call join(), process errors ''' global job_counter time_start = time.time() -job = multiprocessing.Process(target=load_cpu, args=(deadline, )) -job.start() -# timeout=None in the call to join() solves the problem -job.join(deadline) +with lock: +job = multiprocessing.Process(target=load_cpu, args=(deadline, )) +job.start() +join_process(job, deadline) -- ___ Python tracker <https://bugs.python.org/issue40860> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40860] Exception in multiprocessing/context.py under load
Arkady added the comment: Update. I have reproduced the problem in the code not calling Process.join() at all. It require more time and more load, but eventually Process.start() crashes. Posted a question on https://stackoverflow.com/questions/62276345/call-to-pythons-mutliprocessing-process-join-fails The problem reproduces in MacOS as well. -- ___ Python tracker <https://bugs.python.org/issue40860> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40860] Exception in multiprocessing/context.py under load
Arkady added the comment: Is there any news about? This 50 lines sample reproduces the problem https://github.com/larytet-py/multiprocess Please let me know if more information is needed. -- ___ Python tracker <https://bugs.python.org/issue40860> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40860] Exception in multiprocessing/context.py under load
Arkady added the comment: The problem is likely in the call to multiprocessing.Process.join() with timeout. If I use timeout=None the code works. -- ___ Python tracker <https://bugs.python.org/issue40860> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40860] Exception in multiprocessing/context.py under load
New submission from Arkady M :
I am running an HTTP server (socketserver.ThreadingMixIn,
http.server.HTTPServer) in a Docker container (FROM ubuntu:19.10)
Occasionally I get an exception:
Exception happened during processing of request from ('172.17.0.1', 35756)
Traceback (most recent call last):
File "/usr/lib/python3.7/socketserver.py", line 650, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.7/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "service.py", line 221, in __init__
super(UrlExtractorServer, self).__init__(*args, **kwargs)
File "/usr/lib/python3.7/socketserver.py", line 720, in __init__
self.handle()
File "/usr/lib/python3.7/http/server.py", line 426, in handle
self.handle_one_request()
File "/usr/lib/python3.7/http/server.py", line 414, in handle_one_request
method()
File "service.py", line 488, in do_POST
self._post_extract(url)
File "service.py", line 459, in _post_extract
extracted_links, err_msg = self._extract_links(transaction_id,
attachment_id, zip_password, data)
File "service.py", line 403, in _extract_links
error, results = call_timeout(process_deadline,
extractor.extract_links_binary_multiprocess, args=data)
File "service.py", line 175, in call_timeout
manager = multiprocessing.Manager()
File "/usr/lib/python3.7/multiprocessing/context.py", line 56, in Manager
m.start()
File "/usr/lib/python3.7/multiprocessing/managers.py", line 563, in start
self._process.start()
File "/usr/lib/python3.7/multiprocessing/process.py", line 111, in start
_cleanup()
File "/usr/lib/python3.7/multiprocessing/process.py", line 56, in _cleanup
if p._popen.poll() is not None:
AttributeError: 'NoneType' object has no attribute 'poll'
I am in the process of preparingof of a reasonably simple piece of code
demonstrating the problem.
Meanwhile the following can be important. In the code below I ma getting the
elapse < timeout (20 times from 70K). In all case psutil.Process() returned
psutil.NoSuchProcess
time_start = time.time()
job = multiprocessing.Process(target=func, args=(args, results),
kwargs=kwargs)
job.start()
job.join(timeout)
elapsed = time.time()-time_start
if job.is_alive():
try:
process = psutil.Process(job.pid)
process_error = f"pid {job.pid} status {process.status} {process}"
except Exception as e:
process_error = f"psutil.Process() failed {e}"
if elapsed < timeout:
print("elapsed < timeout")
--
messages: 370695
nosy: Arkady M
priority: normal
severity: normal
status: open
title: Exception in multiprocessing/context.py under load
type: crash
versions: Python 3.7
___
Python tracker
<https://bugs.python.org/issue40860>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28899] Symbols doesn't match in VS for python.exe and python35.dll
Changes by Arkady “KindDragon” Shapkin <kinddrago...@gmail.com>: -- resolution: -> not a bug ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28899> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28899] Symbols doesn't match in VS for python.exe and python35.dll
Arkady “KindDragon” Shapkin added the comment: That was my fault -- status: open -> closed ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28899> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28899] Symbols doesn't match in VS for python.exe and python35.dll
New submission from Arkady “KindDragon” Shapkin: I installed Python 3.5.2 with PDB, but they can't be loaded. If I try to load them manually Visual Studio displays message "A matching symbol file was not found in this folder." -- components: Windows messages: 282658 nosy: Arkady “KindDragon” Shapkin, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Symbols doesn't match in VS for python.exe and python35.dll versions: Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28899> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13207] os.path.expanduser breaks when using unicode character in the username
Arkady “KindDragon” Shapkin added the comment: At least Python 2.7 should return in locale.getpreferredencoding() encoding -- nosy: +Arkady “KindDragon” Shapkin ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue13207> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11577] testcase for exception binhex.Error
New submission from Arkady Koplyarov akoplya...@rim.com: Testcase for exception binhex.Error to increase test coverage. -- components: Library (Lib), Tests files: testcase_for_binhex_module.patch keywords: patch messages: 131159 nosy: arkady.koplyarov priority: normal severity: normal status: open title: testcase for exception binhex.Error versions: Python 2.7 Added file: http://bugs.python.org/file21248/testcase_for_binhex_module.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11577 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11577] testcase for exception binhex.Error
Arkady Koplyarov akoplya...@rim.com added the comment: The testcase updated accordingly to Nick Coghlan's suggestion to use assertRaises() since the testcase deals with an exception. -- Added file: http://bugs.python.org/file21252/testcase_for_binhex_module_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11577 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11577] testcase for exception binhex.Error
Arkady Koplyarov akoplya...@rim.com added the comment:
The testcase provided shows up a resource leakage:
-
C:\_cpython\cpythonPCbuild\python_d.exe -m test.regrtest test_binhex
[1/1] test_binhex
C:\_cpython\cpython\lib\unittest\case.py:574: ResourceWarning: unclosed file
_io.BufferedWriter name='@test_5592_tmp2'
callableObj(*args, **kwargs)
1 test OK.
-
The resource leakage occurs in module binhex.py in binhex(inp,out)
BinHex.__init__() _writeinfo() when the exception binhex.Error is raised in
_writeinfo() at the code line:
raise Error('Filename too long')
The issue is that when the exception is thrown the file is left unclosed.
One of possible fixes is to catch the thrown exception and close the unclosed
file in the BinHex.__init__().
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11577
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11577] testcase for exception binhex.Error
Arkady Koplyarov akoplya...@rim.com added the comment: Well, I believe that in BinHex.__init__() I cannot just wrap the _writeinfo call into a try-finally block, and close ofp in the finally. I see that in the case of normal operation when the exception is not thrown, the output file descriptor ofp is used in binhex(inp,out) as a result of BinHex.__init__() and so need remain open until binhex(inp,out) close it explicitly. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11577 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
