Return the response.url in the item as well and move the subprocess call to a pipeline. Is there something else which you want to achieve?
On Fri, Jan 2, 2015 at 6:14 PM, Gaurang shah <[email protected]> wrote: > Hi Guys, > > I am writing a scrapper which will download youtube video based on the > criteria you have provided. I am using youtube-dl module to download the > video. However facing following issue my code is not executing further > until whole video is downloaded. > > Is there any way to create a pipeline to download videos. As we created to > download images ? > > > following is the code. > > > class FetchDetails(CrawlSpider): > name = "youtube" > allowed_domains = ["www.youtube.com"] > start_urls = [" > https://www.youtube.com/results?search_query=scrapy&page=1";] > > def parse(self, response): > hxs = HtmlXPathSelector(response) > base_url = "https://www.youtube.com"; > video_links = > hxs.select("//h3[@class='yt-lockup-title']/a/@href").extract() # > href="/watch?v=1EFnX1UkXVU" > for video in video_links: > if "watch" in video: > request = Request(base_url+video, > callback=self.download_video) > yield request > > def download_video(self, response): > item = YoutubeItem() > > selector = Selector(response) > view_count = > selector.xpath("//div[@class='watch-view-count']/text()")[0].extract().strip() > # video_url = > selector.xpath("//video[contains(@class,'html5-main-video')]/@src").extract() > > # item['video_url'] = video_url > item['view_count'] = view_count > > subprocess.call(["youtube-dl", response.url]) > print view_count > > yield item > > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
