[issue14119] Ability to adjust queue size in Executors

Antoine Pitrou added the comment: Issue 29595 has a more complete PR, so closing this issue as duplicate. -- nosy: +pitrou resolution: -> duplicate stage: patch review -> resolved status: open -> closed superseder: -> Expose max_queue_size in ThreadPoolExecutor

[issue14119] Ability to adjust queue size in Executors

Michael Hrivnak added the comment: My project also has a use case for this, very similar to the others described. Here's what we want: with ThreadPoolExecutor(queue_size=500) as executor: for item in parse_a_long_list_of_work(somefile.xml):

[issue14119] Ability to adjust queue size in Executors

Changes by Camilla Montonen : -- nosy: +Winterflower ___ Python tracker ___ ___

[issue14119] Ability to adjust queue size in Executors

Vinay Anantharaman added the comment: I did a code reading myself and I noticed that task_done is not called as well. Is there a reason? -- nosy: +Vinay Anantharaman ___ Python tracker

[issue14119] Ability to adjust queue size in Executors

Patrik Dufresne added the comment: Any update on this subject ? Also had to monkey patch the implementation to avoid consuming all the system memory. -- nosy: +Patrik Dufresne ___ Python tracker

[issue14119] Ability to adjust queue size in Executors

Alexander Mohr added the comment: adding support for internal queue size is critical to avoid chewing through all your memory when you have a LOT of tasks. I just hit this issue myself. If we could have a simple parameter to set the max queue size this would help tremendously! --

[issue14119] Ability to adjust queue size in Executors

Brian Quinlan added the comment: Can't you accomplish what you want using add_done_callback? e.g. # Pseudocode class MyExecutor(ThreadPoolExecutor): def __init__(self): self._count = 0 def _decrement(self): with self._some_lock: self._count -= 1 def submit(self, fn,

[issue14119] Ability to adjust queue size in Executors

Victor Varvariuc added the comment: Hi! Looks like your pseudocode will work as a workaround instead of monkey-patching! Still the my suggestion to add the line to code stays. self._count should be always equal to the length of self._work_queue? If yes, why duplication. If no - which one to

[issue14119] Ability to adjust queue size in Executors

Victor Varvariuc added the comment: Maybe I should have created another issue for this, but without this issue being solved, the new issue will not help much. Here it is: http://hg.python.org/cpython/file/37caaf21f827/Lib/concurrent/futures/thread.py#l63 After running an work item

[issue14119] Ability to adjust queue size in Executors

Brian Quinlan added the comment: Hi Victor, I don't understand your problem. Could you be very specific in your description? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14119 ___

[issue14119] Ability to adjust queue size in Executors

Victor Varvariuc added the comment: Hi Brian, In one my projects I had to monkey-patch module `concurrent.futures.thread:60`( http://hg.python.org/cpython/file/37caaf21f827/Lib/concurrent/futures/thread.py#l60) with: def _worker(executor_reference, work_queue): try: while True:

[issue14119] Ability to adjust queue size in Executors

Brian Quinlan br...@sweetapp.com added the comment: I've had people request that they be able control the order of processed work submissions. So a more general way to solve your problem might be to make the two executors take an optional Queue argument in their constructors. You'd have to

[issue14119] Ability to adjust queue size in Executors

Nam Nguyen bits...@gmail.com added the comment: +1 That was actually what I did. I replaced the internal queue with another one whose limit was properly set. If you are busy to write one, let me find some time to create another patch. -- ___

[issue14119] Ability to adjust queue size in Executors

R. David Murray rdmur...@bitdance.com added the comment: Brian: ping. Since this is an enhancement, if you are going to accept it it would be nice to get it into 3.3, which means committing it before June 23rd. -- nosy: +r.david.murray stage: - patch review versions: +Python 3.3

[issue14119] Ability to adjust queue size in Executors

Brian Quinlan br...@sweetapp.com added the comment: Hey Nam, I'm not sure that I understand. You want ThreadPoolExecutor.submit to block if there are too many work items in the queue? Are you sure that this happens currently with ProcessPoolExecutor? I can't see why it would. --

[issue14119] Ability to adjust queue size in Executors

Nam Nguyen bits...@gmail.com added the comment: Currently, ProcessionPoolExecutor has this line in its constructor: self._call_queue = multiprocessing.Queue(self._max_workers + EXTRA_QUEUED_CALLS) where EXTRA_QUEUED_CALLS is 1. And yes, it

[issue14119] Ability to adjust queue size in Executors

Brian Quinlan br...@sweetapp.com added the comment: The queue that you identified i.e. self._call_queue = multiprocessing.Queue(self._max_workers + EXTRA_QUEUED_CALLS) does not get considered during submit() - are you sure that it somehow causes

[issue14119] Ability to adjust queue size in Executors

Nam Nguyen bits...@gmail.com added the comment: I used the ThreadPoolExecutor to process a large number (bounded) of input/output jobs. Because there were too many of them and the worker threads could not process them fast enough to drain them from the queue, the queue kept increasing in

[issue14119] Ability to adjust queue size in Executors

Changes by Brian Quinlan br...@sweetapp.com: -- assignee: - bquinlan nosy: +bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14119 ___ ___

[issue14119] Ability to adjust queue size in Executors

Changes by Nam Nguyen bits...@gmail.com: -- keywords: +patch Added file: http://bugs.python.org/file24666/executor-queue-size.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14119 ___

[issue14119] Ability to adjust queue size in Executors

New submission from Nam Nguyen bits...@gmail.com: I am running into a memory consumption issue with concurrent.futures module. Its Executors do not have a public API to adjust their queue size. and the queues are created unbounded initially. It would be helpful to have some public method or

[issue14119] Ability to adjust queue size in Executors

Nam Nguyen bits...@gmail.com added the comment: By the way, ProcessPoolExecutor actually sets its queue size to a reasonable number but ThreadPoolExecutor does not. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14119