Nevermind. While building the test case I figured out where I went wrong. I had a subprocess pool that I forgot to close and it just grew out of hand.
I checked and twisted is running at 4 processes. Sounds better. Thanks for all of the help. On Mon, Mar 2, 2015 at 1:50 PM, Glyph Lefkowitz <gl...@twistedmatrix.com> wrote: > On Mar 2, 2015, at 10:36 AM, Al Niessner <al.niess...@gmail.com> wrote: > > > Platform: > CPU: Intel (2x with 8 core each then hyper-threading gets us to 32 virtual) > Ubuntu 14.04 > Python 3.4 (yes, I know) > Twisted 13.2.0 > > > I use the vanilla twisted.internet.reactor.run() with the > factory twisted.web.server.Site(root). It all runs just fine (seemingly) > but,when I was snooping around on my machine, I had a ton of threads lying > about. After cleaning up, I restarted my twisted application and found that > I have 151 python3 threads lying about. I doubled checked and I have just > twisted application running. > > I browsed the documentation and the twisted thread pool claims to default > between 5 and 10. However, I do abuse the deferToThread() cannot find any > simple documentation on how that might feed back to the 151 threads. I > thought maybe I need to exit the deferred returned from deferToThread() in > some special way to clean it all up. However, I can find no documentation > for it. > > I have been googling about with many random search terms trying to puzzle > the 151 threads from the documentation that tells me I should expect 5 to > 10. > > Can anyone please help understand why I have 151 threads and where they > are coming from? Luckily they are all idle so the machine runs just fine. > > The best story I can come up with is that there is one master thread on > core-0 and then I get 10 threads in the pool for each of the other physical > cores. However, the story is just numerology. > > > There are some bugs in the thread pool implementation (hopefully I'll get > around to finishing my fix for soon, see < > https://twistedmatrix.com/trac/ticket/2673> for details) which may cause > this issue, but given how far out of whack those numbers are, my > inclination is to think it's something else in your environment. > > Can you put your full code online somewhere so that we can test it? > Ideally after fixing it to be a minimal reproducer? :) > > -g > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > >
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
