[issue46812] Thread starvation with threading.Condition
New submission from Mark Gordon : When using Condition variables to manage access to shared resources you can run into starvation issues due to the thread that just gave up a resource (making a call to notify/notify_all) having priority on immediately reacquiring that resource before any of the waiting threads get a chance. The issue appears to arise because unlike the Lock implementation Condition variables are implemented partly in Python and a thread must hold the GIL when it reacquires its underlying condition variable lock. Coupled with Python's predictable switch interval this means that if a thread notifies others of a resource being available and then shortly after attempts to reacquire that resource it will be able to do so since it will have held the GIL the entire time. This can lead to some threads being entirely starved (forever) for access to a shared resource. This came up in a real world situation for me when I had multiple threads trying to access a shared database connection repeatedly without blocking between accesses. Some threads were never getting a connection leading to unexpected timeouts. See https://github.com/sqlalchemy/sqlalchemy/issues/7679 Here's a simple example of this issue using the queue.Queue implementation: https://gist.github.com/msg555/36a10bb5a0c0fe8c89c89d8c05d00e21 Similar example just using Condition variables directly: https://gist.github.com/msg555/dd491078cf10dbabbe7b1cd142644910 Analagous C++ implementation. On Linux 5.13 this is still not _that_ fair but does not completely starve threads: https://gist.github.com/msg555/14d8029b910704a42d372004d3afa465 Thoughts: - Is this something that's worth fixing? The behavior at the very least is surprising and I was unable to find discussion or documentation of it. - Can Condition variables be implemented using standard C libraries? (e.g. pthreads) Maybe at least this can happen when using the standard threading.Lock as the Condition variables lock? - I mocked up a fair Condition variable implementation at https://github.com/msg555/fairsync/blob/main/fairsync/condition.py. However fairness comes at its own overhead of additional context switching. Tested on Python 3.7-3.10 -- components: Library (Lib) messages: 413629 nosy: msg555 priority: normal severity: normal status: open title: Thread starvation with threading.Condition type: behavior versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue46812> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40320] Add ability to specify instance of contextvars context to Task() & asyncio.create_task()
Change by Mark Gordon : -- keywords: +patch nosy: +msg555 nosy_count: 3.0 -> 4.0 pull_requests: +25251 stage: -> patch review pull_request: https://github.com/python/cpython/pull/26664 ___ Python tracker <https://bugs.python.org/issue40320> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43910] cgi.parse_header does not handle escaping correctly
Change by Mark Gordon : -- keywords: +patch pull_requests: +24236 stage: -> patch review pull_request: https://github.com/python/cpython/pull/25519 ___ Python tracker <https://bugs.python.org/issue43910> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43910] cgi.parse_header does not handle escaping correctly
New submission from Mark Gordon :
cgi.parse_header incorrectly handles unescaping of quoted-strings
Note that you can find the related RFCs to how HTTP encodes the Content-Type
header at https://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html and further
discussion on how quoted-string is defined at
https://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-16.html#rfc.section.3.2.1.p.3.
The way parse_header is written it has no context to be able to tell if a
backslash is escaping a double quote or if the backslash is actually the
escaped character and the double quote is free-standing, unescaped. For this
reason it fails to parse values that have a backslash literal at the end. e.g.
the following Content-Type will fail to be parsed
a/b; foo="moo\\"; bar=baz
Example run on current cpython master demonstrating the bug:
Python 3.10.0a7+ (heads/master:660592f67c, Apr 21 2021, 22:51:04) [GCC 9.3.0]
on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cgi
>>> query = 'a; foo="moo"; bar=cow'
>>> print(query)
a; foo="moo\\"; bar=cow
>>> cgi.parse_header(query)
('a', {'foo': '"moo"; bar=cow'})
--
components: Library (Lib)
messages: 391580
nosy: msg555
priority: normal
severity: normal
status: open
title: cgi.parse_header does not handle escaping correctly
type: behavior
versions: Python 3.10
___
Python tracker
<https://bugs.python.org/issue43910>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
