Hello, in most cases Splash will be the bottleneck, not Scrapy.
Probably you will need one instance of Scrapy and a cluster of Splash. Or maybe you will need a cluster of "Scrapy+Splash" instances. A good start point for a cluster of Splash is to use aquarium https://github.com/TeamHG-Memex/aquarium On Thursday, April 21, 2016 at 6:01:17 PM UTC+3, Alan Kavanagh wrote: > > Hi guys, > > I'm trying to determine whether using Scrapy-cluster and Splash is a > viable option. Has anyone done this before? > > Thanks > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
