On Dec 14, 2011, at 7:38 PM, Augusto Mecking Caringi wrote: > On Wed, Dec 14, 2011 at 9:49 PM, Andrew Bennetts <[email protected]> > wrote: >> Augusto Mecking Caringi wrote: >>>>>> om[3082188716].c >> […] >>> str(2917687304 >>> 53120905B 1par 'ype: Test\nDate: 2011-12-14 >>> 16:45:30.995217\nEvent-Subtype: FooBar\nContent-Type: >>> application/json\n\n{\n ') >> >> So it appears you have 53 megabyte string that's referenced from your TLS >> connection, presumably in a buffer of data waiting to be sent? At the moment >> this memory analysis was captured, do you know how much of this data the >> client >> has received? > > Yes... I have a 53MB string referenced in my TLS connection, but *all* > the data has already been sent. > > At the moment that this memory was captured all the data has been sent. > >> Also, how was this data passed to the transport — in one big 53MB write >> call, or >> via many smaller calls, or via a pull producer, or…? > > Via many smaller calls. One call for each event, in a code like that > (running in a thread): > > def stressTestTask(): > time.sleep(1) > c = provider_reference > for i in range(1000000): > event = c.createTestEvent() > reactor.callFromThread(c.transport.write, event) > > I have two components in my system: the server and the provider, both > of them written in Python/Twisted. This capture was from a "stress > test provider" that send hundreds of thousands of events to the server > without any "sleep". And I captured this memory analysis after the > server has received all the data. > >> This looks like an HTTP-like protocol, so possibly there's a library between >> your code and the transport that means you don't know the direct answer to >> that >> question. But whatever details you can give about how you are passing the >> data >> you generate to the connection will help locate the problem, I think. > > Yes, this a HTTP-like protocol, but there is no library, the protocol > was designed by me. > > Thanks.
If everything you've said here is accurate, this is definitely a bug in Twisted. It would be really valuable if you could boil down your example to be as minimal as possible - ideally a unit test, but failing that, a script which could be trivially run to leak an arbitrary amount of memory would be almost as good. Does your callFromThread(c.transport.write, ...) code reliably trigger this error when talking to a very simple protocol, like an Echo or Discard protocol? Thanks a lot for helping us track this down, -glyph _______________________________________________ Twisted-Python mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
