[issue39645] Expand concurrent.futures.Future's public API
Brian Quinlan added the comment: I'll try to take a look at this before the end of the week, but I'm currently swamped with other life stuff :-( -- ___ Python tracker <https://bugs.python.org/issue39645> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39205] Hang when interpreter exits after ProcessPoolExecutor.shutdown(wait=False)
Brian Quinlan added the comment: New changeset 884eb89d4a5cc8e023deaa65001dfa74a436694c by Brian Quinlan in branch 'master': bpo-39205: Tests that highlight a hang on ProcessPoolExecutor shutdown (#18221) https://github.com/python/cpython/commit/884eb89d4a5cc8e023deaa65001dfa74a436694c -- ___ Python tracker <https://bugs.python.org/issue39205> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39205] Hang when interpreter exits after ProcessPoolExecutor.shutdown(wait=False)
Change by Brian Quinlan : -- keywords: +patch pull_requests: +17601 stage: -> patch review pull_request: https://github.com/python/cpython/pull/18221 ___ Python tracker <https://bugs.python.org/issue39205> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39205] Hang when interpreter exits after ProcessPoolExecutor.shutdown(wait=False)
New submission from Brian Quinlan :
```
from concurrent.futures import ProcessPoolExecutor
import time
t = ProcessPoolExecutor(max_workers=3)
t.map(time.sleep, [1,2,3])
t.shutdown(wait=False)
```
Results in this exception and then a hang (i.e. Python doesn't terminate):
```
Exception in thread QueueManagerThread:
Traceback (most recent call last):
File "/usr/local/google/home/bquinlan/cpython/Lib/threading.py", line 944, in
_bootstrap_inner
self.run()
File "/usr/local/google/home/bquinlan/cpython/Lib/threading.py", line 882, in
run
self._target(*self._args, **self._kwargs)
File
"/usr/local/google/home/bquinlan/cpython/Lib/concurrent/futures/process.py",
line 352, in _queue_management_worker
_add_call_item_to_queue(pending_work_items,
File
"/usr/local/google/home/bquinlan/cpython/Lib/concurrent/futures/process.py",
line 280, in _add_call_item_to_queue
call_queue.put(_CallItem(work_id,
File "/usr/local/google/home/bquinlan/cpython/Lib/multiprocessing/queues.py",
line 82, in put
raise ValueError(f"Queue {self!r} is closed")
ValueError: Queue is closed
```
--
assignee: bquinlan
messages: 359257
nosy: bquinlan
priority: normal
severity: normal
status: open
title: Hang when interpreter exits after
ProcessPoolExecutor.shutdown(wait=False)
type: behavior
versions: Python 3.9
___
Python tracker
<https://bugs.python.org/issue39205>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31783] Race condition in ThreadPoolExecutor when scheduling new jobs while the interpreter shuts down
Brian Quinlan added the comment: Can I add "needs backport to 3.8" and "needs backport to 3.7" labels now or do I have to use cherry_picker at this point? On Mon, Jul 1, 2019 at 3:55 PM Ned Deily wrote: > > Ned Deily added the comment: > > > I don't know what the backport policy is. > > It does seem that the devguide does not give much guidance on this; I've > opened an issue about it (https://github.com/python/devguide/issues/503). > But, in general, if the fix is potentially beneficial and does not add > undue risk or an incompatibility, we would generally consider backporting > it to the currently active maintenance branches; at the moment, that would > be 3.8 (in beta phase) and 3.7 (maintenance nmode). We have a lot of > buildbot tests that show non-deterministic failures, some possibly due to > concurrent.futures. If there is a chance that this fix might mitigate > those, I'd like to consider it for backporting. > > -- > > ___ > Python tracker > <https://bugs.python.org/issue31783> > ___ > > -- ___ Python tracker <https://bugs.python.org/issue31783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31783] Race condition in ThreadPoolExecutor when scheduling new jobs while the interpreter shuts down
Brian Quinlan added the comment: I don't know what the backport policy is. The bug is only theoretical AFAIK i.e. someone noticed it through code observation but it has not appeared in the wild. On Mon, Jul 1, 2019 at 3:25 PM Ned Deily wrote: > > Ned Deily added the comment: > > Brian, should this fix be backported to 3.8 and 3.7? As it stands now, it > will only be showing up in Python 3.9 unless you add the backport labels to > the original PR. Unless it cna be shown to be a security issue, it would > not be appropriate to backport to 3.6 at this stage. > > -- > nosy: +ned.deily > > ___ > Python tracker > <https://bugs.python.org/issue31783> > ___ > > -- ___ Python tracker <https://bugs.python.org/issue31783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31783] Race condition in ThreadPoolExecutor when scheduling new jobs while the interpreter shuts down
Change by Brian Quinlan : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue31783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31783] Race condition in ThreadPoolExecutor when scheduling new jobs while the interpreter shuts down
Brian Quinlan added the comment: New changeset 242c26f53edb965e9808dd918089e664c0223407 by Brian Quinlan in branch 'master': bpo-31783: Fix a race condition creating workers during shutdown (#13171) https://github.com/python/cpython/commit/242c26f53edb965e9808dd918089e664c0223407 -- ___ Python tracker <https://bugs.python.org/issue31783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37294] concurrent.futures.ProcessPoolExecutor and multiprocessing.pool.Pool fail with super
Change by Brian Quinlan : -- resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> function changed when pickle bound method object ___ Python tracker <https://bugs.python.org/issue37294> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37208] Weird exception behaviour in ProcessPoolExecutor
Change by Brian Quinlan : -- stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue37208> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37208] Weird exception behaviour in ProcessPoolExecutor
Change by Brian Quinlan : -- resolution: -> duplicate superseder: -> picke cannot dump Exception subclasses with different super() args ___ Python tracker <https://bugs.python.org/issue37208> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37287] picke cannot dump Exception subclasses with different super() args
Change by Brian Quinlan : -- title: picke cannot dump exceptions subclasses with different super() args -> picke cannot dump Exception subclasses with different super() args ___ Python tracker <https://bugs.python.org/issue37287> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37287] picke cannot dump exceptions subclasses with different super() args
New submission from Brian Quinlan :
$ ./python.exe nopickle.py
TypeError: __init__() missing 1 required positional argument: 'num'
The issue is that the arguments passed to Exception.__init__ (via `super()`)
are collected into `args` and then serialized by pickle e.g.
>>> PickleBreaker(5).args
()
>>> PickleBreaker(5).__reduce_ex__(3)
(, (), {'num': 5})
>>> # The 1st index is the `args` tuple
Then, during load, the `args` tuple is used to initialize the Exception i.e.
PickleBreaker(), which results in the `TypeError`
See https://github.com/python/cpython/blob/master/Modules/_pickle.c#L6769
--
components: Library (Lib)
files: nopickle.py
messages: 345647
nosy: bquinlan
priority: normal
severity: normal
status: open
title: picke cannot dump exceptions subclasses with different super() args
versions: Python 3.7, Python 3.8, Python 3.9
Added file: https://bugs.python.org/file48420/nopickle.py
___
Python tracker
<https://bugs.python.org/issue37287>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37208] Weird exception behaviour in ProcessPoolExecutor
Brian Quinlan added the comment: That's a super interesting bug! It looks like this issue is that your exception can't be round-tripped using pickle i.e. >>> class PoolBreaker(Exception): ... def __init__(self, num): ... super().__init__() ... self.num = num ... >>> import pickle >>> pickle.loads(pickle.dumps(PoolBreaker(5))) Traceback (most recent call last): File "", line 1, in TypeError: __init__() missing 1 required positional argument: 'num' -- assignee: -> bquinlan ___ Python tracker <https://bugs.python.org/issue37208> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24980] Allow for providing 'on_new_thread' callback to 'concurrent.futures.ThreadPoolExecutor'
Brian Quinlan added the comment: Joshua, I'm closing this since I haven't heard from you in a month. Please re-open if you use case isn't handled by `initializer` and `initargs`. -- assignee: -> bquinlan resolution: -> out of date stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue24980> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors
Brian Quinlan added the comment: We can bike shed over the name in the PR ;-) -- ___ Python tracker <https://bugs.python.org/issue36780> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26903] ProcessPoolExecutor(max_workers=64) crashes on Windows
Change by Brian Quinlan : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue26903> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32679] concurrent.futures should store full sys.exc_info()
Brian Quinlan added the comment: My understanding is that tracebacks have a pretty large memory profile so I'd rather not keep them alive. Correct me if I'm wrong about that. -- ___ Python tracker <https://bugs.python.org/issue32679> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24882] ThreadPoolExecutor doesn't reuse threads until #threads == max_workers
Brian Quinlan added the comment: After playing with it for a while, https://github.com/python/cpython/pull/6375 seems reasonable to me. It needs tests and some documentation. Antoine, are you still -1 because of the complexity increase? -- ___ Python tracker <https://bugs.python.org/issue24882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24882] ThreadPoolExecutor doesn't reuse threads until #threads == max_workers
Brian Quinlan added the comment: When I first wrote and started using ThreadPoolExecutor, I had a lot of code like this: with ThreadPoolExecutor(max_workers=500) as e: e.map(download, images) I didn't expect that `images` would be a large list but, if it was, I wanted all of the downloads to happen in parallel. I didn't want to have to explicitly take into account the list size when starting the executor (e.g. max_works=min(500, len(images))) but I also didn't want to create 500 threads up front when I only needed a few. My use case involved transient ThreadPoolExecutors so I didn't have to worry about idle threads. In principle, I'd be OK with trying to avoid unnecessary thread creation if the implementation can be simple and efficient enough. https://github.com/python/cpython/pull/6375 seems simple enough but I haven't convinced myself that it works yet ;-) -- ___ Python tracker <https://bugs.python.org/issue24882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30006] Deadlocks in `concurrent.futures.ProcessPoolExecutor`
Brian Quinlan added the comment: Was this fixed by https://github.com/python/cpython/pull/3895 ? -- ___ Python tracker <https://bugs.python.org/issue30006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36395] Add deferred single-threaded/fake executor to concurrent.futures
Brian Quinlan added the comment: Brian, I was looking for an example where the current executor isn't sufficient for testing i.e. a useful test that would be difficult to write with a real executor but would be easier with a fake. Maybe you have such an example from your tests? -- ___ Python tracker <https://bugs.python.org/issue36395> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31783] Race condition in ThreadPoolExecutor when scheduling new jobs while the interpreter shuts down
Change by Brian Quinlan : -- pull_requests: +13087 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue31783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31783] Race condition in ThreadPoolExecutor when scheduling new jobs while the interpreter shuts down
Brian Quinlan added the comment: I think that ProcessPoolExecutor might have a similar race condition - but not in exactly this code path since it would only be with the queue management thread (which is only started once). -- ___ Python tracker <https://bugs.python.org/issue31783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31783] Race condition in ThreadPoolExecutor when scheduling new jobs while the interpreter shuts down
Brian Quinlan added the comment:
Great report Steven!
I was able to reproduce this with the attached patch (just adds some sleeps and
prints) and this script:
from threading import current_thread
from concurrent.futures import ThreadPoolExecutor
from time import sleep
pool = ThreadPoolExecutor(100)
def f():
print("I'm running in: ", current_thread().name)
def g():
print("I'm running in: ", current_thread().name)
for _ in range(100):
pool.submit(f)
sleep(0.1)
pool.submit(g)
sleep(1.5)
The output for me was:
Creating new thread: ThreadPoolExecutor-0_0
I'm running in: ThreadPoolExecutor-0_0
Setting _shutdown
Killing 1 workers
Creating new thread: ThreadPoolExecutor-0_1
I'm running in: ThreadPoolExecutor-0_1
So another thread was created *after* shutdown.
It seems like the most obvious way to fix this is by adding a lock for the
global _shutdown variable.
--
keywords: +patch
Added file: https://bugs.python.org/file48311/find-race.diff
___
Python tracker
<https://bugs.python.org/issue31783>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31783] Race condition in ThreadPoolExecutor when scheduling new jobs while the interpreter shuts down
Change by Brian Quinlan : -- assignee: -> bquinlan ___ Python tracker <https://bugs.python.org/issue31783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22729] concurrent.futures `wait` and `as_completed` depend on private api
Brian Quinlan added the comment: How did the experiment go? Are people still interested in this? -- ___ Python tracker <https://bugs.python.org/issue22729> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22630] `concurrent.futures.Future.set_running_or_notify_cancel` does not notify cancel
Brian Quinlan added the comment: Ben, do you still think that your patch is relevant or shall we close this bug? -- ___ Python tracker <https://bugs.python.org/issue22630> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24980] Allow for providing 'on_new_thread' callback to 'concurrent.futures.ThreadPoolExecutor'
Brian Quinlan added the comment: Do the `initializer` and `initargs` parameters deal with this use case for you? https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor -- ___ Python tracker <https://bugs.python.org/issue24980> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24767] can concurrent.futures len(Executor) return the number of tasks?
Brian Quinlan added the comment: If we supported this, aren't we promising that we will always materialize the iterator passed to map? I think that we'd need a really strong use-case for this to be worth-while. -- ___ Python tracker <https://bugs.python.org/issue24767> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36395] Add deferred single-threaded/fake executor to concurrent.futures
Brian Quinlan added the comment: Hey Brian, I understand the non-determinism. I was wondering if you had a non-theoretical example i.e. some case where the non-determinism had impacted a real test that you wrote? -- ___ Python tracker <https://bugs.python.org/issue36395> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors
Brian Quinlan added the comment: Hey Hrvoje, I agree that #1 is the correct approach. `disown` might not be the best name - maybe `allow_shutdown` or something. But we can bike shed about that later. Are you interested in writing a patch? -- ___ Python tracker <https://bugs.python.org/issue36780> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24195] Add `Executor.filter` to concurrent.futures
Brian Quinlan added the comment: Hey Ethan, I'm really sorry about dropping the ball on this. I've been burnt out on Python stuff for the last couple of years. When we left this, it looked like the -1s were in the majority and no one new has jumped on to support `filter`. If you wanted to add this, I wouldn't object. But I've been inactive so long that I don't think that I should make the decision. -- ___ Python tracker <https://bugs.python.org/issue24195> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23697] Module level map & submit for concurrent.futures
Brian Quinlan added the comment: Using a default executor could be dangerous because it could lead to deadlocks. For example: mylib.py def my_func(): tsubmit(...) tsubmit(...) tsubmit(somelib.some_func, ...) somelib.py -- def some_func(): tsubmit(...) # Potential deadlock if no more free threads. -- ___ Python tracker <https://bugs.python.org/issue23697> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22361] Ability to join() threads in concurrent.futures.ThreadPoolExecutor
Brian Quinlan added the comment: So you actually use the result of ex.submit i.e. use the resulting future? If you don't then it might be easier to just create your own thread. -- ___ Python tracker <https://bugs.python.org/issue22361> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36395] Add deferred single-threaded/fake executor to concurrent.futures
Brian Quinlan added the comment: Do you have a example that you could share? I can't think of any other fakes in the standard library and I'm hesitant to be the person who adds the first one ;-) -- ___ Python tracker <https://bugs.python.org/issue36395> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26374] concurrent_futures Executor.map semantics better specified in docs
Change by Brian Quinlan : -- stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue26374> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26374] concurrent_futures Executor.map semantics better specified in docs
Brian Quinlan added the comment: Can we close this bug then? -- ___ Python tracker <https://bugs.python.org/issue26374> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36395] Add deferred single-threaded/fake executor to concurrent.futures
Brian Quinlan added the comment: Hey Brian, why can't you use threads in your unit tests? Are you worried about non-determinism or resource usage? Could you make a ThreadPoolExecutor with a single worker? -- ___ Python tracker <https://bugs.python.org/issue36395> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26903] ProcessPoolExecutor(max_workers=64) crashes on Windows
Change by Brian Quinlan : -- keywords: +patch pull_requests: +13045 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue26903> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors
Brian Quinlan added the comment: OK, I completely disagree with my statement: """If you added this as an argument to shutdown() then you'd probably also have to add it as an option to the constructors (for people using Executors as context managers). But, if you have to add it to the constructor anyway, you may as well *only* add it to the constructor.""" This functionality wouldn't apply to context manager use (since the point of the context manager is to ensure resource clean-up). So maybe another argument to shutdown() would make sense -- ___ Python tracker <https://bugs.python.org/issue36780> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26903] ProcessPoolExecutor(max_workers=64) crashes on Windows
Brian Quinlan added the comment: BTW, the 61 process limit comes from: 63 - - -- ___ Python tracker <https://bugs.python.org/issue26903> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26903] ProcessPoolExecutor(max_workers=64) crashes on Windows
Change by Brian Quinlan : -- assignee: -> bquinlan ___ Python tracker <https://bugs.python.org/issue26903> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26903] ProcessPoolExecutor(max_workers=64) crashes on Windows
Brian Quinlan added the comment: If no one has short-term plans to improve multiprocessing.connection.wait, then I'll update the docs to list this limitation, ensure that ProcessPoolExecutor never defaults to >60 processes on windows and raises a ValueError if the user explicitly passes a larger number. -- ___ Python tracker <https://bugs.python.org/issue26903> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors
Brian Quinlan added the comment: >> The current behavior is explicitly documented, so presumably >> it can't be (easily) changed And it isn't clear that it would be desirable to change this even if it were possible - doing structured resource clean-up seems consistent with the rest of Python. That being said, I can see the user case for this. If you added this as an argument to shutdown() then you'd probably also have to add it as an option to the constructors (for people using Executors as context managers). But, if you have to add it to the constructor anyway, you may as well *only* add it to the constructor. I'll let an active maintainer weigh in on whether, on balance, they think that this functionality is worth complicating the API. -- ___ Python tracker <https://bugs.python.org/issue36780> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24195] Add `Executor.filter` to concurrent.futures
Brian Quinlan added the comment: Actually it immediately converts the iterable into a list. Recall: def filter(self, fn, iterable, timeout=None): l = list(iterable) # iterable = list return (item for (item, keep) in zip(l, self.map(fn, l, timeout)) if keep) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24195 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24195] Add `Executor.filter` to concurrent.futures
Brian Quinlan added the comment: Ethan: I'm -0 so I'd happily go with the consensus. Does anyone have a strong opinion? Ram: What did you mean by Possible issues? Did you mean that to be an example using the executor.map() technique? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24195 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24195] Add `Executor.filter` to concurrent.futures
Brian Quinlan added the comment: This feature seems unnecessary to me but couldn't the implementation be simplified to work in terms of map? i.e. (pseudocode): def filter(self, fn, iterable, timeout=None): l = list(iterable) return (item for (item, keep) in zip(l, self.map(fn, l, timeout)) if keep) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24195 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21362] concurrent.futures does not validate that max_workers is proper
Brian Quinlan added the comment: Committed, thanks for the fix! -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21362 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21362] concurrent.futures does not validate that max_workers is proper
Brian Quinlan added the comment: The patch looks good to me too. I'll commit it soon. -- assignee: - bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21362 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Brian Quinlan added the comment: Thanks very much for the patch Glenn! -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Brian Quinlan added the comment: Oops, no. That was junk due to my sloppiness. I’ll fix it in a minute. On Jan 31, 2014, at 5:03 PM, STINNER Victor rep...@bugs.python.org wrote: STINNER Victor added the comment: New changeset 0bcf23a52d55 by Brian Quinlan in branch 'default': Issue #20319: concurrent.futures.wait() can block forever even if Futures have completed http://hg.python.org/cpython/rev/0bcf23a52d55 Hum, the change also contains: +Fix warning message when `os.chdir()` fails inside + `test.support.temp_cwd()`. Patch by Chris Jerdonek. Is it related to this issue? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Changes by Brian Quinlan br...@sweetapp.com: -- assignee: - bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Brian Quinlan added the comment: I'm looking at futures.patch. I don't understand why these blocks are helpful -_create_and_install_waiters has two call sites and both (as_completed and wait) call _create_and_install_waiters from within an _AcquireFutures context manager: -pending_count = sum( -f._state not in [CANCELLED_AND_NOTIFIED, FINISHED] for f in fs) +with _AcquireFutures(fs): +pending_count = sum( +f._state not in [CANCELLED_AND_NOTIFIED, FINISHED] +for f in fs) -for f in fs: -f._waiters.append(waiter) +with _AcquireFutures(fs): +for f in fs: +f._waiters.append(waiter) It seems more plausible that the locks around the removals are fixing the bug but I don't see how. I'll look into it some more. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20319] concurrent.futures.wait() can block forever even if Futures have completed
Brian Quinlan added the comment: I'm not currently working on concurrent.futures but I do look at patches and bug reports. I'll take a look at this and Issue20297 sometime this week. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20319 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14119] Ability to adjust queue size in Executors
Brian Quinlan added the comment: Can't you accomplish what you want using add_done_callback? e.g. # Pseudocode class MyExecutor(ThreadPoolExecutor): def __init__(self): self._count = 0 def _decrement(self): with self._some_lock: self._count -= 1 def submit(self, fn, *args, **kwargs): f = super(self).submit(fn, *args, **kwargs) with self._some_lock: self._count += 1 f.add_done_callback(self._decrement) @property def num_pending_futures(self): return self._count -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14119 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14119] Ability to adjust queue size in Executors
Brian Quinlan added the comment: Hi Victor, I don't understand your problem. Could you be very specific in your description? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14119 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11161] futures.ProcessPoolExecutor hangs
Changes by Brian Quinlan br...@sweetapp.com: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11161 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13785] Make concurrent.futures.Future state public
Brian Quinlan added the comment: Any progress on this or can I close the bug? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13785 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18699] What is Future.running() for in PEP 3148 / concurrent.futures.Future?
Brian Quinlan added the comment: The only thing that I've used .running() for is for UI reasons. But it is easier to just iterative over your futures and call a method than to install a callback and handle the state yourself. -- nosy: +bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18699 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17874] ProcessPoolExecutor in interactive shell doesn't work in Windows
Brian Quinlan added the comment: Take a look at http://bugs.python.org/issue11161 and its fix. Do you think that just saying that ProcessPoolExecutor will not work from the interactive shell is sufficient? -- nosy: +bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17874 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7200] multiprocessing deadlock on Mac OS X when queue collected before process terminates
Brian Quinlan added the comment: OK, working as intended. -- resolution: - invalid ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7200 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10918] **kwargs unnecessarily restricted in concurrent.futures 'submit' API
Brian Quinlan added the comment: I think that we are happy to not fix this. -- resolution: accepted - wont fix status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10918 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.
Brian Quinlan added the comment: The concurrent.futures stuff looks good to me. Could you add a comment explaining why the delete is necessary? And, as Antoine said, the test should be CPython only. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16284 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15966] concurrent.futures: Executor.submit keyword arguments may not be called 'fn' (or 'self')
Brian Quinlan added the comment: This has come up before. Did you actually bang into this? Because the fix seems pretty ugly to me and the work-around (using functools.partial) is pretty easy. But, if people are actually hitting this, then your is probably the best that we can do. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15966 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13785] Make concurrent.futures.Future state public
Changes by Brian Quinlan br...@sweetapp.com: -- stage: - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13785 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8792] Support Apache extensions to XML-RPC in xmlrpclib
Brian Quinlan added the comment: I'm closing this since the filer hasn't specified exactly what they want. -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8792 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11271] concurrent.futures.ProcessPoolExecutor.map() doesn't batch function arguments by chunks
Changes by Brian Quinlan br...@sweetapp.com: -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11271 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13785] Make concurrent.futures.Future state public
Brian Quinlan br...@sweetapp.com added the comment: My current thinking is that adding a state property probably isn't worth it given the fact that you can construct this yourself and the requirement for a history property is too specialized. The callback idea seams reasonable and sufficient for your use case. I think that you'd probably want to *not* specify that state change that you want i.e. any state change triggers the function. What do you think? (BTW, I'm on leave right now so it might take a while to respond) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13785 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14119] Ability to adjust queue size in Executors
Brian Quinlan br...@sweetapp.com added the comment: I've had people request that they be able control the order of processed work submissions. So a more general way to solve your problem might be to make the two executors take an optional Queue argument in their constructors. You'd have to explain in detail in the document how the queues are used. What do you think? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14119 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14119] Ability to adjust queue size in Executors
Brian Quinlan br...@sweetapp.com added the comment: Hey Nam, I'm not sure that I understand. You want ThreadPoolExecutor.submit to block if there are too many work items in the queue? Are you sure that this happens currently with ProcessPoolExecutor? I can't see why it would. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14119 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15015] Access to non-existing future attribute in error path of futures
Brian Quinlan br...@sweetapp.com added the comment: Thanks for the patch! -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15015 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14119] Ability to adjust queue size in Executors
Brian Quinlan br...@sweetapp.com added the comment: The queue that you identified i.e. self._call_queue = multiprocessing.Queue(self._max_workers + EXTRA_QUEUED_CALLS) does not get considered during submit() - are you sure that it somehow causes submit() to block. Could you explain what your use case is such that you can run out of memory? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14119 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15015] Access to non-existing future attribute in error path of futures
Changes by Brian Quinlan br...@sweetapp.com: -- assignee: - bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15015 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14214] test_concurrent_futures hangs
Brian Quinlan br...@sweetapp.com added the comment: Could you run just the test_concurrent_futures test, hit ctrl-C at the point where it hangs, and send the traceback here? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14214 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14119] Ability to adjust queue size in Executors
Changes by Brian Quinlan br...@sweetapp.com: -- assignee: - bquinlan nosy: +bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14119 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11271] concurrent.futures.ProcessPoolExecutor.map() doesn't batch function arguments by chunks
Brian Quinlan br...@sweetapp.com added the comment: I'm closing this since tbrink didn't respond to pitrou's comments. -- resolution: - out of date ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11271 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14148] Option to kill stuck workers in a multiprocessing pool
Changes by Brian Quinlan br...@sweetapp.com: -- nosy: +bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14148 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13785] Make concurrent.futures.Future state public
Brian Quinlan br...@sweetapp.com added the comment: I guess the question is: why do you need to know the state in that form? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13785 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13785] Make concurrent.futures.Future state public
Brian Quinlan br...@sweetapp.com added the comment: Thanks for the patch! 1. The fetching the state feature seems reasonable but I think that explaining the difference between CANCELLED and CANCELLED_AND_NOTIFIED is going to be hard. Maybe you could look at how Doc/library/concurrent.futures.rst would need to be updated to see if we can provide a reasonable description of the different states? 2. Having the future record the history of its state transitions seems potentially useful but there is no precedent for this in other Python objects where it would also be useful. Maybe you could make the executors take a Futures factory and then provide a subclass that does that? Not sure that I like that either but... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13785 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5788] datetime.timedelta is inconvenient to use...
Brian Quinlan br...@sweetapp.com added the comment: You'll probably get more traction if you file a new bug. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5788 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11271] concurrent.futures.ProcessPoolExecutor.map() slower than multiprocessing.Pool.map() for fast function argument
Brian Quinlan br...@sweetapp.com added the comment: On my crappy computer, ProcessPoolExecutor.map adds 3ms of added execution time per call using your test framework. What is your use case where that is too much? That being said, I don't have any objections to making improvements. If you want to pursue this, could you attach a working map_comprison.py? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11271 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11864] sporadic failure in test_concurrent_futures
Changes by Brian Quinlan br...@sweetapp.com: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11864 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Brian Quinlan br...@sweetapp.com added the comment: Under what circumstances do we expect a ProcessPoolExecutor child process to be killed outside of the control of the ProcessPoolExecutor? If the user kills a child then maybe all we want to do is raise an exception rather than deadlock as a convenience. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Brian Quinlan br...@sweetapp.com added the comment: Killed by the user, or by an automatic device (such as the Linux OOM killer), or crashed. Crashed would be bad - it would indicate a bug in the ProcessPoolExecutor code. If the user kills a child then maybe all we want to do is raise an exception rather than deadlock as a convenience. That's what the patch does, roughly. Right. But instead of trying to recover, it might be better to fail very loudly i.e. - fail every non-finished future - kill every child process in the ProcessPoolExecutor - set the ProcessPoolExecutor as shutdown so no new work can be scheduled - raise in wait(), as_completed(), etc. It really dependents on whether we view the situation as expected (and try our best to let the user recover) or as an aberrant situation that the user can't reasonably recover from. Cheers, Brian -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9205 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11864] sporadic failure in test_concurrent_futures
Brian Quinlan br...@sweetapp.com added the comment: It looks like the timing is too sensitive. -- assignee: - bquinlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11864 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11777] Executor.map does not submit futures until iter.next() is called
Brian Quinlan br...@sweetapp.com added the comment: Nice catch. I hadn't noticed that the docs are lying :-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11777 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11777] Executor.map does not submit futures until iter.next() is called
Changes by Brian Quinlan br...@sweetapp.com: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11777 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11777] Executor.map does not submit futures until iter.next() is called
Brian Quinlan br...@sweetapp.com added the comment: I think that it surprising behavior, especially considering that asking for the *first* element in the iterator causes *all* of the futures to be created. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11777 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11777] Executor.map does not submit futures until iter.next() is called
New submission from Brian Quinlan br...@sweetapp.com: from concurrent import futures with futures.ThreadPoolExecutor(max_workers=5) as e: e.map(print, range(10)) # No output -- assignee: bquinlan components: Library (Lib) messages: 133104 nosy: bquinlan priority: normal severity: normal status: open title: Executor.map does not submit futures until iter.next() is called type: behavior versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11777 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10632] multiprocessing generates a fatal error
Brian Quinlan br...@sweetapp.com added the comment: Filing a new bug might have been a mistake. Once the investigation in issue 10517 isolated the failure as being in a different module, I thought it best to file a new bug with a minimal repro case. Fill free to cleanup. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10632 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11724] concurrent.futures: executor.submit() runs until completion even when future times out or is canceled
Brian Quinlan br...@sweetapp.com added the comment: This is the expected behavior. Future.cancel() returns a bool indicating if the future was cancelled or not. If it returns False then there is no way to stop it. The behavior that you are seeing for shutdown is documented as: shutdown(wait=True) ... If wait is True then this method will not return until all the pending futures are done executing and the resources associated with the executor have been freed. If wait is False then this method will return immediately and the resources associated with the executor will be freed when all pending futures are done executing. Regardless of the value of wait, the entire Python program will not exit until all pending futures are done executing. -- resolution: - invalid ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11724 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10632] multiprocessing generates a fatal error
Brian Quinlan br...@sweetapp.com added the comment: No, I wasn't able to replicate. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10632 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Brian Quinlan br...@sweetapp.com added the comment: Your approach seems workable but your patch allows the interpreter to exit while work items are still being processed. See the comment at the top of concurrent/futures/thread.py. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Brian Quinlan br...@sweetapp.com added the comment: Sorry, I didn't read an error message very carefully. When I apply your patch I see: from concurrent.futures import * from time import * t = ThreadPoolExecutor(5) t.submit(sleep, 100) Future at 0x8acc94c state=running ctrl-D Error in atexit._run_exitfuncs: NameError: global name 'thread' is not defined Does that not happen in your environment? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11635] concurrent.futures uses polling
Brian Quinlan br...@sweetapp.com added the comment: I would suggest that you base your patch on 3.3/default. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11635 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11161] futures.ProcessPoolExecutor hangs
Brian Quinlan br...@sweetapp.com added the comment: ProcessPoolExecutor is not expected to work with any interactive shell on Windows because it uses the multiprocessing module behind the scenes, which has that limitation. I'm reclassifying this as a documentation bug since either a reference to http://docs.python.org/dev/library/multiprocessing.html#programming-guidelines should appear in the futures docs or (better) the content should be rewritten specifically for ProcessPoolExecutor. -- components: +Documentation -IDLE ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11161 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10918] **kwargs unnecessarily restricted in concurrent.futures 'submit' API
Brian Quinlan br...@sweetapp.com added the comment: Sorry for taking so long to reply - I was on holidays until today. This is an incompatible API change (since people may be providing fn by keyword) so we should probably hold off until 3.3. I also don't really like that the signature for submit will become less readable. And the fix is pretty trivial i.e. functools.partial. So really I'm -0 on this. But if anyone cares enough, I'll happily review and apply patches that change the submit signature to the one that Adrian proposed. def sugar(*args, **kw): return args[0].submit(args[1], args[2:], kw) -- resolution: wont fix - accepted status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10918 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10918] **kwargs unnecessarily restricted in API
Brian Quinlan br...@sweetapp.com added the comment:
Arian,
This seems like such an unimportant edge case that I don't want to mess with
the API just to accommodate it.
If you really need to pass an fn keyword argument, use:
.submit(foo, 1, 2, **{'fn': bar})
--
resolution: - wont fix
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10918
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10918] **kwargs unnecessarily restricted in API
Brian Quinlan br...@sweetapp.com added the comment: Good point! I'd suggest functools.partial. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10918 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
