What about either scrapy's pipelines, so you transform the data before putting it into your final db, or something like Luigi, a python library for creating data pipelines?
If the transforms are simple and local, I'd go with scrapy's methods. You can separate the logic outside of the spider itself using the middleware. If you need to augment the data, especially with third party or external services [like joining it to data in another database], I'd use luigi, as it has some good stuff for that. On Mon, Aug 1, 2016 at 3:26 AM, Rakesh Kumar <rakesh.cybot...@gmail.com> wrote: > I've had a search around SO and the web in general, and haven't been able > to find a solution to my issue; so I thought I would raise a question here. > > I'm currently playing around with Scrapy, and am successfully retrieving > the data I need from my target website. > > The creation of the spider is part of a larger project, and so I need to > work out a way of getting the results from the spider into my website. > > I don't want to access the database with the spider, because there is a > variety of logic that would need to be applied to the data before it was > stored, and I wouldn't want to have to repeat that logic elsewhere. > > What I'm thinking is perhaps a webservice, that receives a JSON object as > a request; and returns a JSON object with the result. > > I'm fairly new to Scrapy and Python in a websetting, but I have read the > Scrapy docs. Whilst there is a webservice for Scrapy, that seems to be more > to do with the monitoring of spiders; and not returning results. > > Any help and thoughts would be greatly appreciated. > > Thanks, > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to scrapy-users+unsubscr...@googlegroups.com. > To post to this group, send email to scrapy-users@googlegroups.com. > Visit this group at https://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
