Hey Dan, I'd try using hyperquest <https://github.com/substack/hyperquest> and see if it solves your issue. It will prevent you from being limited by socket pooling and you will just be at the mercy of the operating system. It seems like you already took care of your ulimits so you should be fine. Let me know how it goes.
Cheers, Jarrett On Friday, November 1, 2013 5:53:18 PM UTC-4, Dan wrote: > > Hey, > > I'm coding a proxy server checker. > > In order to check a proxy, I need to connect to it using "request" module, > and perform several requests. > > The goal is to be able to check thousands of proxies in short amount of > time. > > I was hoping to test 1,000 proxies concurrently. I did the following > configs: > http.globalAgent.maxSockets = 100000; > events.EventEmitter.prototype._maxListeners = 100000; > > and I execute 1,000 requests using async.queue. > > I also set the open file descriptors to a really high limit... Another > thing I do is I set a timeout for each proxy for 30 secs... > > Anyway, what happens is that the first requests are performed well and > then I start getting a lot of ETIMEDOUT & ESOCKETTIMEDOUT. > > I checked the amout of open files for my node process: > pidof node > lsof -p 3730 | grep TCP | wc -l > > And I check it periodically during the execution. It looks like I always > get stuck on average of 30 sockets. It seems like something is limiting my > requests to work. And then, when a turn of a request comes, it simply > refuses to work because it's already timedout. > > Any suggestions? > > Thanks. > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
