The trick that hyperquest uses to disable the node core pooling behavior is to pass `agent: false` to the options. The same thing works with request.
More discussion here: https://github.com/request/request/issues/1231 On Saturday, November 2, 2013 11:16:10 PM UTC-5, Jarrett Cruger wrote: > > Hey Dan, > > I'd try using hyperquest <https://github.com/substack/hyperquest> and see > if it solves your issue. It will prevent you from being limited by socket > pooling and you will just be at the mercy of the operating system. It seems > like you already took care of your ulimits so you should be fine. Let me > know how it goes. > > Cheers, > Jarrett > > On Friday, November 1, 2013 5:53:18 PM UTC-4, Dan wrote: >> >> Hey, >> >> I'm coding a proxy server checker. >> >> In order to check a proxy, I need to connect to it using "request" >> module, and perform several requests. >> >> The goal is to be able to check thousands of proxies in short amount of >> time. >> >> I was hoping to test 1,000 proxies concurrently. I did the following >> configs: >> http.globalAgent.maxSockets = 100000; >> events.EventEmitter.prototype._maxListeners = 100000; >> >> and I execute 1,000 requests using async.queue. >> >> I also set the open file descriptors to a really high limit... Another >> thing I do is I set a timeout for each proxy for 30 secs... >> >> Anyway, what happens is that the first requests are performed well and >> then I start getting a lot of ETIMEDOUT & ESOCKETTIMEDOUT. >> >> I checked the amout of open files for my node process: >> pidof node >> lsof -p 3730 | grep TCP | wc -l >> >> And I check it periodically during the execution. It looks like I always >> get stuck on average of 30 sockets. It seems like something is limiting my >> requests to work. And then, when a turn of a request comes, it simply >> refuses to work because it's already timedout. >> >> Any suggestions? >> >> Thanks. >> > -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/c42d1cb0-2532-4cb7-bcd9-d0e58509a5a7%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
