On Aug 26, 11:16 pm, Chad Etzel <c...@twitter.com> wrote:
> Hi Steve,
>
> What was the solution?
Well, Chad, I won't lie to you, it was totally embarrassing: I made a
pass through my code to make sure that I was properly closing all of
the HTTP connections so that my crawler wasn't leaking connections.
I have one method that crawls a (possibly signed) URI so that I can
properly rate-limit my calls. This method was responsible for
returning an org.apache.http.HttpEntity and it was doing the
following:
HttpResponse resp = client.execute(get);
int status = resp.getStatusLine().getStatusCode();
if(status == 200) {
HttpEntity ret = resp.getEntity();
ret.consumeContent();
return ret;
}
It gets the entity from the response, consumes its content completely
to make sure the connection is cleared and then returns the entity to
the caller so that it could read the content and get JSON out of it.
That poor bugger could get in a couple of reads of the stuff that was
pulled down before the socket was closed and then boom!
I refactored things so that the translation from an entity to JSON is
done in this method, since that's what everyone who uses this method
was going to do first anyways, so now the cleanup is in the right
place.
Definitely not my best moment as a programmer (not my worst,
either :-). Let this be a message to you kids: never clean up after
yourselves, it only leads to you looking dumb in public!
Steve
