[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Andrew Svetlov]

Changes by Andrew Svetlov :


--
stage: test needed -> committed/rejected

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Andrew Svetlov]

Andrew Svetlov added the comment:

Committed. Thanks.

--
resolution:  -> fixed
stage:  -> test needed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Roundup Robot]

Roundup Robot added the comment:

New changeset 70cef0a160cf by Andrew Svetlov in branch 'default':
Issue #16284: Prevent keeping unnecessary references to worker functions in 
concurrent.futures ThreadPoolExecutor.
http://hg.python.org/cpython/rev/70cef0a160cf

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Taras Lyapun]

Taras Lyapun added the comment:

Added comments to patch

--
nosy: +lyapun
Added file: http://bugs.python.org/file27850/issue16284_with_comments.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Andrew Svetlov]

Andrew Svetlov added the comment:

Updated patch to execute tests only for CPython.

--
nosy: +asvetlov
Added file: http://bugs.python.org/file27820/kill_reference_3.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Richard Oudkerk]

Richard Oudkerk added the comment:

LGTM

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Brian Quinlan]

Brian Quinlan added the comment:

The concurrent.futures stuff looks good to me.

Could you add a comment explaining why the delete is necessary? And, as Antoine 
said, the test should be CPython only.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Antoine Pitrou]

Antoine Pitrou added the comment:

Sounds fine to me. You might want to make the test CPython-specific.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Mark Dickinson]

Changes by Mark Dickinson :


--
nosy: +jnoller, sbt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Mark Dickinson]

Mark Dickinson added the comment:

A new patch (with tests), and a fuller explanation:

At work, we've got Python talking to a customer's existing COM library; we're 
using Thomas Heller's 'comtypes' library to do that.  Unfortunately, comtypes 
depends quite a lot on __del__-time cleanup, so reference counting matters.  
(I'm well aware that this isn't the recommended way to deal with resource 
cleanup in Python, but rewriting the existing infrastructure isn't a realistic 
option here.)

Anyway, it turned out that the concurrent.futures executors were indirectly 
holding onto references to COM objects, causing issues with our application.

The attached patch adds a few 'del' statements to remove references that are no 
longer needed.  For the ProcessExecutor, some of those 'del' statements had to 
go into the multiprocessing.Queue implementation.

The troublesome pattern (in both multiprocessing and futures) takes the form 
(simplified):


def my_worker_function(...):
...
while :
obj = blocking_wait_for_next_item()
do_processing(obj)
...

The issue is that the reference to obj is kept until the completion of the next 
blocking wait call.  I'm suggesting just adding an extra 'del obj' after 
'do_processing(obj)'.

--
components: +Library (Lib)
Added file: http://bugs.python.org/file27633/kill_reference_2.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Ralf Schmitt]

Changes by Ralf Schmitt :


--
nosy: +schmir

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

[Mark Dickinson]

New submission from Mark Dickinson:

The ThreadPoolExecutor unnecessarily keeps references to _WorkItem objects.  
With the attached patch (which lacks a test), all tests still pass, and the 
references are removed as soon as they're no longer needed.

--
files: kill_reference.diff
keywords: patch
messages: 173318
nosy: bquinlan, mark.dickinson
priority: normal
severity: normal
status: open
title: concurrent.futures ThreadPoolExecutor keeps unnecessary references to 
worker functions.
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file27619/kill_reference.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com