I have 2 scrapyd instances with max_proc=8 each. I am aware for CONCURRENT_REQUESTS_PER_IP and CONCURRENT_REQUESTS_PER_DOMAIN settings but i read in a previous post that they are for each slot/proc. I tested it and even if i set 1 on both settings its keep downloads 8x2 concurrent items of same domain. Is there some way to limit concurrent requests when i have max_proc > 1 in scrapyd?
-- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
