I just found the solution: http://stackoverflow.com/questions/19995855/increase-max-open-files-for-ubuntu-upstart-initctl
The problem is as scrapyd starts as a service, it will ignore any configuration in '/etc/security/limits.conf'. I solved the problem by adding this line to '/etc/init/scrapyd.conf': limit nofile 524288 524288 On Tue, Jan 7, 2014 at 1:33 PM, Pablo Hoffman <[email protected]> wrote: > How did you increased the ulimit?. Are you sure it was increased properly? > > > On Tue, Jan 7, 2014 at 3:48 PM, Ali Bozorgkhan <[email protected]>wrote: > >> Hi, >> >> I am running 25 jobs on a single machine, each with 50 concurrent >> requests. After a 10 minutes or so, I get this error in my proxy middleware: >> >> Couldn't bind: 24: Too many open files. >> >> I tried increasing the ulimit on the system but no luck. Is there anyway >> I could fix it? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "scrapy-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "scrapy-users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/scrapy-users/rArexq4tI7I/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
