What would be the difference between 1. using the global agent with `maxSockets = Inifinity` 2. using a new agent for each request (so called `agent: false` to request package)
Why is 2 a fix and not 1? On Thursday, November 6, 2014 at 12:07:04 PM UTC-5, James Nylen wrote: > > The trick that hyperquest uses to disable the node core pooling behavior > is to pass `agent: false` to the options. The same thing works with > request. > > More discussion here: https://github.com/request/request/issues/1231 > > On Saturday, November 2, 2013 11:16:10 PM UTC-5, Jarrett Cruger wrote: >> >> Hey Dan, >> >> I'd try using hyperquest <https://github.com/substack/hyperquest> and >> see if it solves your issue. It will prevent you from being limited by >> socket pooling and you will just be at the mercy of the operating system. >> It seems like you already took care of your ulimits so you should be fine. >> Let me know how it goes. >> >> Cheers, >> Jarrett >> >> On Friday, November 1, 2013 5:53:18 PM UTC-4, Dan wrote: >>> >>> Hey, >>> >>> I'm coding a proxy server checker. >>> >>> In order to check a proxy, I need to connect to it using "request" >>> module, and perform several requests. >>> >>> The goal is to be able to check thousands of proxies in short amount of >>> time. >>> >>> I was hoping to test 1,000 proxies concurrently. I did the following >>> configs: >>> http.globalAgent.maxSockets = 100000; >>> events.EventEmitter.prototype._maxListeners = 100000; >>> >>> and I execute 1,000 requests using async.queue. >>> >>> I also set the open file descriptors to a really high limit... Another >>> thing I do is I set a timeout for each proxy for 30 secs... >>> >>> Anyway, what happens is that the first requests are performed well and >>> then I start getting a lot of ETIMEDOUT & ESOCKETTIMEDOUT. >>> >>> I checked the amout of open files for my node process: >>> pidof node >>> lsof -p 3730 | grep TCP | wc -l >>> >>> And I check it periodically during the execution. It looks like I always >>> get stuck on average of 30 sockets. It seems like something is limiting my >>> requests to work. And then, when a turn of a request comes, it simply >>> refuses to work because it's already timedout. >>> >>> Any suggestions? >>> >>> Thanks. >>> >> -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/569d69c8-6566-41ee-8e20-23bf996e9027%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
