Hi All, I have been working on a multi-user and distributed system based on Scrapy framework. I have uploaded the code at https://github.com/aaldaber/Distributed-Multi-User-Scrapy-System-with-a-Web-UI. The system uses RabbitMQ for link queues, MongoDB for storage, and scrapyd daemons for workers. The Web interface is done using Django Framework. The system was built to extract mainly text data from news sites and social networks for further analysis.
Posting here in order to share with people who might be in need of such a system. I would be pleased to hear any comments or suggestions regarding the application. Kind regards, Aibek -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
