--- El vie, 10/12/10, Brian Quinlan escribió: > On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: > > I have a process running for a long time, and which > may use futures of different max_workers count. I think it > is not too far-fetched to create a new futures object each > time. Yet, the execution becomes slower after each call, for > example with http://freehackers.org/~tnagy/futures_test.py: > > > > """ > > import concurrent.futures > > from queue import Queue > > import datetime > > > > class counter(object): > > def __init__(self, fut): > > self.fut = fut > > > > def run(self): > > def > look_busy(num, obj): > > > tot = 0 > > > for x in range(num): > > > tot += x > > > obj.out_q.put(tot) > > > > start = > datetime.datetime.utcnow() > > self.count = 0 > > self.out_q = > Queue(0) > > for x in > range(1000): > > > self.count += 1 > > > self.fut.submit(look_busy, self.count, > self) > > > > while > self.count: > > > self.count -= 1 > > > self.out_q.get() > > > > delta = > datetime.datetime.utcnow() - start > > > print(delta.total_seconds()) > > > > fut = > concurrent.futures.ThreadPoolExecutor(max_workers=20) > > for x in range(100): > > # comment the following line > > fut = > concurrent.futures.ThreadPoolExecutor(max_workers=20) > > c = counter(fut) > > c.run() > > """ > > > > The runtime grows after each step: > > 0.216451 > > 0.225186 > > 0.223725 > > 0.222274 > > 0.230964 > > 0.240531 > > 0.24137 > > 0.252393 > > 0.249948 > > 0.257153 > > ... > > > > Is there a mistake in this piece of code? > > There is no mistake that I can see but I suspect that the > circular references that you are building are causing the > ThreadPoolExecutor to take a long time to be collected. Try > adding: > > c = counter(fut) > c.run() > + fut.shutdown() > > Even if that fixes your problem, I still don't fully > understand this because I would expect the runtime to fall > after a while as ThreadPoolExecutors are collected.
The shutdown call is indeed a good fix :-) Here is the time response of the calls to counter() when shutdown is not called: http://www.freehackers.org/~tnagy/runtime_futures.png After trying to stop the program by using CTRL+C, the following error may appear, after which the process cannot be interrupted: """ 19:18:12 /tmp/build> python3.2 futures_test.py 0.389657 0.417173 0.416513 0.421424 0.449666 0.482273 ^CTraceback (most recent call last): File "futures_test.py", line 36, in <module> c.run() File "futures_test.py", line 22, in run self.fut.submit(look_busy, self.count, self) File "/usr/local/lib/python3.2/concurrent/futures/thread.py", line 114, in submit self._work_queue.put(w) File "/usr/local/lib/python3.2/queue.py", line 135, in put self.not_full.acquire() KeyboardInterrupt """ It is not expected, is it? Thomas _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
