--- El vie, 10/12/10, Brian Quinlan
escribió:
On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote:
I have a process running for a long time, and
which
may use futures of different max_workers count. I
think it
is not too far-fetched to create a new futures object
each
time. Yet, the execution becomes slower after each
call, for
example with http://freehackers.org/~tnagy/futures_test.py:
"""
import concurrent.futures
from queue import Queue
import datetime
class counter(object):
def __init__(self, fut):
self.fut = fut
def run(self):
def
look_busy(num, obj):
tot = 0
for x in range(num):
tot += x
obj.out_q.put(tot)
start =
datetime.datetime.utcnow()
self.count = 0
self.out_q =
Queue(0)
for x in
range(1000):
self.count += 1
self.fut.submit(look_busy, self.count,
self)
while
self.count:
self.count -= 1
self.out_q.get()
delta =
datetime.datetime.utcnow() - start
print(delta.total_seconds())
fut =
concurrent.futures.ThreadPoolExecutor(max_workers=20)
for x in range(100):
# comment the following line
fut =
concurrent.futures.ThreadPoolExecutor(max_workers=20)
c = counter(fut)
c.run()
"""
The runtime grows after each step:
0.216451
0.225186
0.223725
0.222274
0.230964
0.240531
0.24137
0.252393
0.249948
0.257153
...
Is there a mistake in this piece of code?
There is no mistake that I can see but I suspect that
the
circular references that you are building are causing
the
ThreadPoolExecutor to take a long time to be
collected. Try
adding:
c = counter(fut)
c.run()
+ fut.shutdown()
Even if that fixes your problem, I still don't fully
understand this because I would expect the runtime to
fall
after a while as ThreadPoolExecutors are collected.
The shutdown call is indeed a good fix :-) Here is the time
response of the calls to counter() when shutdown is not
called:
http://www.freehackers.org/~tnagy/runtime_futures.png
After trying to stop the program by using CTRL+C, the
following error may appear, after which the process cannot
be interrupted:
"""
19:18:12 /tmp/build> python3.2 futures_test.py
0.389657
0.417173
0.416513
0.421424
0.449666
0.482273
^CTraceback (most recent call last):
File "futures_test.py", line 36, in <module>
c.run()
File "futures_test.py", line 22, in run
self.fut.submit(look_busy, self.count, self)
File
"/usr/local/lib/python3.2/concurrent/futures/thread.py",
line 114, in submit
self._work_queue.put(w)
File "/usr/local/lib/python3.2/queue.py", line 135, in
put
self.not_full.acquire()
KeyboardInterrupt
"""
It is not expected, is it?
