[issue14119] Ability to adjust queue size in Executors

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Issue 29595 has a more complete PR, so closing this issue as duplicate.

--
nosy: +pitrou
resolution:  -> duplicate
stage: patch review -> resolved
status: open -> closed
superseder:  -> Expose max_queue_size in ThreadPoolExecutor

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Michael Hrivnak]

Michael Hrivnak  added the comment:

My project also has a use case for this, very similar to the others described. 
Here's what we want:

with ThreadPoolExecutor(queue_size=500) as executor:
  for item in parse_a_long_list_of_work(somefile.xml):
executor.submit(Job(item))

I do not want to parse the entire list of work items and load them into memory 
at once. It is preferable for the main thread running the above code to block 
on submit() when the queue size is above some threshold.

It's a classic case of the producer and consumer operating at different speeds. 
In the past, a Queue object has been the way to connect such a producer and 
consumer. The various Executor classes do not provide an easy way to consume 
from a provided Queue object, so giving them that capability would be a 
reasonable alternative to having the submit() method block.

--
nosy: +mhrivnak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Camilla Montonen]

Changes by Camilla Montonen :


--
nosy: +Winterflower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Vinay Anantharaman]

Vinay Anantharaman added the comment:

I did a code reading myself and I noticed that task_done is not called as well. 
Is there a reason?

--
nosy: +Vinay Anantharaman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Patrik Dufresne]

Patrik Dufresne added the comment:

Any update on this subject ?

Also had to monkey patch the implementation to avoid consuming all the system 
memory.

--
nosy: +Patrik Dufresne

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Alexander Mohr]

Alexander Mohr added the comment:

adding support for internal queue size is critical to avoid chewing through all 
your memory when you have a LOT of tasks.  I just hit this issue myself.  If we 
could have a simple parameter to set the max queue size this would help 
tremendously!

--
nosy: +thehesiod

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Brian Quinlan]

Brian Quinlan added the comment:

Can't you accomplish what you want using add_done_callback?

e.g.

# Pseudocode
class MyExecutor(ThreadPoolExecutor):
  def __init__(self):
self._count = 0

  def _decrement(self):
with self._some_lock:
  self._count -= 1

  def submit(self, fn, *args, **kwargs):
f = super(self).submit(fn, *args, **kwargs)
with self._some_lock:
  self._count += 1
f.add_done_callback(self._decrement)

  @property
  def num_pending_futures(self):
return self._count

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Victor Varvariuc]

Victor Varvariuc added the comment:

Hi!

Looks like your pseudocode will work as a workaround instead of monkey-patching!

Still the my suggestion to add the line to code stays.
self._count should be always equal to the length of self._work_queue? If yes, 
why duplication. If no - which one to use, why duplication? Also there is an 
additional lock.

http://docs.python.org/3.3/library/queue.html#queue.Queue.task_done - there is 
a special method, why not using it?

Looks like you think that `work_queue.task_done()` should not be added. I don't 
understand why, but you decide what's better for Python.

Thank you for your time!
Victor

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Victor Varvariuc]

Victor Varvariuc added the comment:

Maybe I should have created another issue for this, but without this issue 
being solved, the new issue will not help much.
Here it is:
http://hg.python.org/cpython/file/37caaf21f827/Lib/concurrent/futures/thread.py#l63
After running an work item `work_queue.task_done()` is not called.
So it's difficult to know if worker threads have any more work to do.
http://stackoverflow.com/questions/20965754/determine-if-worker-threads-are-doing-any-work?noredirect=1#comment31495804_20965754

--
nosy: +Victor.Varvariuc

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Brian Quinlan]

Brian Quinlan added the comment:

Hi Victor,

I don't understand your problem. Could you be very specific in your description?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Victor Varvariuc]

Victor Varvariuc added the comment:

Hi Brian,

In one my projects I had to monkey-patch module `concurrent.futures.thread:60`( 
http://hg.python.org/cpython/file/37caaf21f827/Lib/concurrent/futures/thread.py#l60)
 with:

def _worker(executor_reference, work_queue):
try:
while True:
work_item = work_queue.get(block=True)
if work_item is not None:
work_item.run()
work_queue.task_done()  # -- added this line
continue
executor = executor_reference()
# Exit if:
#   - The interpreter is shutting down OR
#   - The executor that owns the worker has been collected OR
#   - The executor that owns the worker has been shutdown.
if futures_thread._shutdown or executor is None or 
executor._shutdown:
# Notice other workers
work_queue.put(None)
return
del executor
except BaseException:
futures_thread._base.LOGGER.critical('Exception in worker', 
exc_info=True)

This helps me to control the state of the work queue -- I can see if there are 
any work items still being processed:

if executor._work_queue.unfinished_tasks:
# executor is still producing something
...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Brian Quinlan]

Brian Quinlan br...@sweetapp.com added the comment:

I've had people request that they be able control the order of processed work 
submissions. So a more general way to solve your problem might be to make the 
two executors take an optional Queue argument in their constructors.

You'd have to explain in detail in the document how the queues are used.

What do you think?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Nam Nguyen]

Nam Nguyen bits...@gmail.com added the comment:

+1

That was actually what I did. I replaced the internal queue with another one 
whose limit was properly set.

If you are busy to write one, let me find some time to create another patch.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Brian: ping.  Since this is an enhancement, if you are going to accept it it 
would be nice to get it into 3.3, which means committing it before June 23rd.

--
nosy: +r.david.murray
stage:  - patch review
versions: +Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Brian Quinlan]

Brian Quinlan br...@sweetapp.com added the comment:

Hey Nam,

I'm not sure that I understand. You want ThreadPoolExecutor.submit to block if 
there are too many work items in the queue? Are you sure that this happens 
currently with ProcessPoolExecutor? I can't see why it would.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Nam Nguyen]

Nam Nguyen bits...@gmail.com added the comment:

Currently, ProcessionPoolExecutor has this line in its constructor:

self._call_queue = multiprocessing.Queue(self._max_workers +
 EXTRA_QUEUED_CALLS)

where EXTRA_QUEUED_CALLS is 1.

And yes, it would be best to export a method so that the app developer can set 
the queue size for themselves. In my case, I would want to limit the queue so 
that I dont run out of memory. Others might not want the queue to block, and 
hence would prefer an unlimited queue.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Brian Quinlan]

Brian Quinlan br...@sweetapp.com added the comment:

The queue that you identified i.e.

self._call_queue = multiprocessing.Queue(self._max_workers +
 EXTRA_QUEUED_CALLS)

does not get considered during submit() - are you sure that it somehow causes 
submit() to block.

Could you explain what your use case is such that you can run out of memory?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Nam Nguyen]

Nam Nguyen bits...@gmail.com added the comment:

I used the ThreadPoolExecutor to process a large number (bounded) of 
input/output jobs. Because there were too many of them and the worker threads 
could not process them fast enough to drain them from the queue, the queue kept 
increasing in size.

It was okay for the script, though, to block, waiting for the queue to drain, 
before submitting new jobs. So I needed to limit the queue size.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Brian Quinlan]

Changes by Brian Quinlan br...@sweetapp.com:


--
assignee:  - bquinlan
nosy: +bquinlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Nam Nguyen]

Changes by Nam Nguyen bits...@gmail.com:


--
keywords: +patch
Added file: http://bugs.python.org/file24666/executor-queue-size.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Nam Nguyen]

New submission from Nam Nguyen bits...@gmail.com:

I am running into a memory consumption issue with concurrent.futures module. 
Its Executors do not have a public API to adjust their queue size.  and the 
queues are created unbounded initially.

It would be helpful to have some public method or a parameter at construction 
time to limit this queue size.

--
components: Library (Lib)
messages: 154175
nosy: Nam.Nguyen
priority: normal
severity: normal
status: open
title: Ability to adjust queue size in Executors
type: enhancement

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14119] Ability to adjust queue size in Executors

[Nam Nguyen]

Nam Nguyen bits...@gmail.com added the comment:

By the way, ProcessPoolExecutor actually sets its queue size to a reasonable 
number but ThreadPoolExecutor does not.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com