[issue30698] asyncio sslproto do not shutdown ssl layer cleanly
Grzegorz Grzywacz <grzgrzg...@gmail.com> added the comment: No, this PR originally fix similar but different issue. However it also fix issue29406 as a side effect. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue30698> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27144] concurrent.futures.as_completed() memory inefficiency
Grzegorz Grzywacz added the comment: Tests are reusing finished futures. `_yield_and_decref` function do not clear waiters in finished futures. In the initial merge i propose to clear waiters but after review we decide it should be removed. I am confused now, should we change tests or restore initial `_yield_and_decref` function. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30549] ProcessPoolExecutor hangs forever if the object raises on __getstate__
Grzegorz Grzywacz added the comment: This is already reported and patch was proposed. Here: #30006 -- nosy: +grzgrzgrz3 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30549> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30541] Add restricted mocks to the python unittest mocking framework
Grzegorz Grzywacz added the comment:
Existing mock implementation already has that feature. Mock attributes can be
limited with `spec` attribute.
>>> inner_m = Mock(spec=["method2"], **{"method2.return_value": 1})
>>> m = Mock(spec=["method1"], **{"method1.return_value": inner_m})
>>>
>>> m.method1().method2()
1
>>>
>>> m.method1().attr
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.5/unittest/mock.py", line 580, in __getattr__
raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'attr'
--
nosy: +grzgrzgrz3
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue30541>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30698] asyncio sslproto do not shutdown ssl layer cleanly
Grzegorz Grzywacz added the comment: No one yet responded, maybe this is unclear. I will clarify what is going on, why i made this change, what we gain from this and why this is not ideal solution. I will focus on ssl layer shutdown as this issue regards. We have connection asyncio <-> nginx Lets see first situation asyncio initiates shutdown: 1. ideal: shutdown asyncio --> nginx <-- shutdown Ideal situation asyncio sending shutdown and nginx replies back. This is how it works before attached PR. 2. we can't relay on nginx shutdown asyncio --> nginx ||| shutdown At this point everything looks great, but what will happen when nginx do not sent shutdown - we will wait forever. We have this situation here #29406. Attached PR "fix" this problem (note is not ideal fix, more like workaround): 3. with fix: shutdown asyncio --> nginx ||? shutdown asyncio is sending shutdown ssl data to nginx but not waiting for nginx response, transport is closed anyway. I think ideal will be to wait for certain amount of time for response like Nikolay in #29406 propose. This will allow to implement SSL downgrade to plain text. Second situation, nginx sent ssl eof. 1. before fix: shutdown nginx --> asyncio ||| shutdown In this case we are receiving nginx shutdown and correctly process it but after that, shutdown callback will close the transport before shutdown is sent back. Asyncio will try to send this data but fail due to closed transport. There is another issue should be not possible to write to closed transport. We are getting false-positive result to write. I do not analyze this deeper, maybe there is a reason to it. 2. after fix: shutdown nginx --> asyncio <-- shutdown This is clean, shutdown callback in _SSLPipe is removed. We close transport in ssl protocol. I think connections between _SSLPipe and ssl protocol has design problems, _SSLPipe should be, as name suggest, only a pipe. Callback for handshake and shutdown feels wrong. Ssl protocol based on _SSLPipie mode and state can figure out when to call connection_made or close transport. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30698> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29406] asyncio SSL contexts leak sockets after calling close with certain Apache servers
Grzegorz Grzywacz added the comment: This is not problem with madis-data.ncep.noaa.gov not doing ssl shutdown, this is problem with asyncio not doing it. Patch from this #30698 issue fix this too. -- nosy: +grzgrzgrz3 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29406> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30698] asyncio sslproto do not shutdown ssl layer cleanly
Changes by Grzegorz Grzywacz <grzgrzg...@gmail.com>: -- pull_requests: +2320 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30698> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30698] asyncio sslproto do not shutdown ssl layer cleanly
New submission from Grzegorz Grzywacz: Asyncio on shutdown do not send shutdown confirmation to the other side. _SSLPipe after doing unwrap is calling shutdown callback where transportis closed and quit ssldata wont be sent. -- components: asyncio messages: 296295 nosy: grzgrzgrz3, yselivanov priority: normal severity: normal status: open title: asyncio sslproto do not shutdown ssl layer cleanly type: behavior versions: Python 3.5, Python 3.6, Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30698> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30595] test_queue_feeder_donot_stop_onexc() of test_multiprocessing_spawn fails randomly on x86 Windows7 3.x
Grzegorz Grzywacz added the comment: of course it should be `if not`: diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py index dda03dd..514f991 100644 --- a/Lib/multiprocessing/queues.py +++ b/Lib/multiprocessing/queues.py @@ -101,7 +101,7 @@ class Queue(object): try: if block: timeout = deadline - time.time() -if timeout < 0 or not self._poll(timeout): +if not self._poll(timeout): raise Empty elif not self._poll(): raise Empty -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30595> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30595] test_queue_feeder_donot_stop_onexc() of test_multiprocessing_spawn fails randomly on x86 Windows7 3.x
Grzegorz Grzywacz added the comment: Looks like build bot is too slow for timeout=0.1. I am guessing `0.1` is too low because we have wrong condition in Queue.get. It should be. diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py index dda03dd..42e9884 100644 --- a/Lib/multiprocessing/queues.py +++ b/Lib/multiprocessing/queues.py @@ -101,7 +101,7 @@ class Queue(object): try: if block: timeout = deadline - time.time() -if timeout < 0 or not self._poll(timeout): +if self._poll(timeout): raise Empty elif not self._poll(): raise Empty If we successfully acquired self._rlock, we should poll no matter how long acquire took. -- nosy: +grzgrzgrz3 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30595> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30514] test_poplib replace asyncore
Changes by Grzegorz Grzywacz <grzgrzg...@gmail.com>: -- pull_requests: +1947 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30514> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28533] Replace asyncore
Grzegorz Grzywacz added the comment: ./Lib/test/test_poplib.py sub-issue issue30514 Fixed issue number from previous comment -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28533] Replace asyncore
Grzegorz Grzywacz added the comment: ./Lib/test/test_poplib.py sub-issue issue28533 -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30514] test_poplib replace asyncore
New submission from Grzegorz Grzywacz: sub-issue of issue28533 -- components: Tests messages: 294770 nosy: grzgrzgrz3 priority: normal severity: normal status: open title: test_poplib replace asyncore versions: Python 3.6, Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30514> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28533] Replace asyncore
Grzegorz Grzywacz added the comment: I would like to work on this issue. I think it's a good idea to split this task into few parts/PR. Let me start from ./Lib/test/test_poplib.py. What about rewriting pop3 server stub using asyncio, i think requests could be handled synchronously, there will be no benefits from asynchronous. Please let me know what you think. -- nosy: +grzgrzgrz3 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder
Changes by Grzegorz Grzywacz <grzgrzg...@gmail.com>: -- pull_requests: +1778 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30414> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder
New submission from Grzegorz Grzywacz: multiprocessing.Queue is running background thread feeder. Feeder serialize and sends buffered data to pipe. The issue is with exception handling, feeder is catching all exceptions but out of main loop, so after exception is handled feeder is not going back to loop - thread finish. If feeder thread is not running any Queue.put will execute without exceptions but message not gonna be delivered. Solution is to move exception handling inside main loop. I will provide PR. I have run performance tests (found: #17025) and submitted patch do not affect performance. -- components: Library (Lib) messages: 294044 nosy: grzgrzgrz3 priority: normal severity: normal status: open title: multiprocesing.Queue silently ignore messages after exc in _feeder versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30414> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30357] test_thread.test_save_exception_state_on_error(): Unhandled exception in thread: AMD64 Debian root 2.7
Grzegorz Grzywacz added the comment: I think this do not solve this issue yet. There is still posibillity that different tests/testrunners spawn threads and 'fool' testcase. I think we should not relay on `thread._count` value where it's possible. For master branch `thread._set_sentinel()` can be used. For 2.7 i don't know simple solution. I would like to know your opinion it is worth changing? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30357> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30357] test_thread.test_save_exception_state_on_error(): Unhandled exception in thread: AMD64 Debian root 2.7
Changes by Grzegorz Grzywacz <grzgrzg...@gmail.com>: -- pull_requests: +1676 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30357> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30357] test_thread.test_save_exception_state_on_error(): Unhandled exception in thread: AMD64 Debian root 2.7
Grzegorz Grzywacz added the comment: Problem is with test test_thread.ThreadRunningTests.test_save_exception_state_on_error when other tests leave threads runnig. test_save_exception_state_on_error relay on thread._get_count(), if this value decrease test assume thread is finished with is not always correct (other threads finish - started by different test). Fix is to make sure each test wait for all threads to finsh. -- nosy: +grzgrzgrz3 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30357> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27144] concurrent.futures.as_completed() memory inefficiency
Grzegorz Grzywacz added the comment: > We just ran into the exact same issue here in Google using a > ThreadPoolExecutor.map call Looks like map got simillar issue. I created GitHub PR with map fixed. > Concurrent package was added in 3.2. How backport it 2.7? There is official, unofficial 2.7 backport. Git: https://github.com/agronholm/pythonfutures Pip: futures Ubuntu package: python-concurrent.futures -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27144] concurrent.futures.as_completed() memory inefficiency
Changes by Grzegorz Grzywacz <grzgrzg...@gmail.com>: -- pull_requests: +1658 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27144] concurrent.futures.as_completed() memory inefficiency
Changes by Grzegorz Grzywacz <grzgrzg...@gmail.com>: -- keywords: +patch Added file: http://bugs.python.org/file43038/issue27144.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27144] concurrent.futures.as_completed() memory inefficiency
New submission from Grzegorz Grzywacz: as_complite generator keeps reference of all passed futures until StopIteration. It may lead to serious memory inefficiency. Solution is to remove reference from lists and yield future ad-hoc. I have submitted patch and reproduce sample. I can create backport for older versions if needed. -- components: Library (Lib) files: reproduce.py messages: 266552 nosy: bquinlan, grzgrzgrz3 priority: normal severity: normal status: open title: concurrent.futures.as_completed() memory inefficiency type: resource usage versions: Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6 Added file: http://bugs.python.org/file43037/reproduce.py ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27144> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
