Request can be limiting it, see pool.maxSockets. I'd suggest to use 'net' module for that and implement http protocol manually. Request module is a bit too heavy for that.
On Saturday, November 2, 2013 1:53:18 AM UTC+4, Dan wrote: > > Hey, > > I'm coding a proxy server checker. > > In order to check a proxy, I need to connect to it using "request" module, > and perform several requests. > > The goal is to be able to check thousands of proxies in short amount of > time. > > I was hoping to test 1,000 proxies concurrently. I did the following > configs: > http.globalAgent.maxSockets = 100000; > events.EventEmitter.prototype._maxListeners = 100000; > > and I execute 1,000 requests using async.queue. > > I also set the open file descriptors to a really high limit... Another > thing I do is I set a timeout for each proxy for 30 secs... > > Anyway, what happens is that the first requests are performed well and > then I start getting a lot of ETIMEDOUT & ESOCKETTIMEDOUT. > > I checked the amout of open files for my node process: > pidof node > lsof -p 3730 | grep TCP | wc -l > > And I check it periodically during the execution. It looks like I always > get stuck on average of 30 sockets. It seems like something is limiting my > requests to work. And then, when a turn of a request comes, it simply > refuses to work because it's already timedout. > > Any suggestions? > > Thanks. > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
