Hi I'm currently starting to work on the same kind of use case. I found this article which does not recommend mongodb : https://blog.scrapinghub.com/2013/05/13/mongo-bad-for-scraped-data/
They say that you'll have the same lock contention with mongodb : the article was written in 2013 so maybe it's not the case anymore. And they migrated to HBase which seems to be the right backend, It is used also in the Apache Nutch project. -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
