Hi Yasoon. now I am putting video url in item as well. however not sure where exactly I should put my subprocess.call function in pipeline ?
Gaurang Shah Blog: qtp-help.blogspot.com Mobile: +91 738756556 On Sat, Jan 3, 2015 at 1:56 AM, Yasoob Khalid <[email protected]> wrote: > Return the response.url in the item as well and move the subprocess call > to a pipeline. Is there something else which you want to achieve? > > On Fri, Jan 2, 2015 at 6:14 PM, Gaurang shah <[email protected]> > wrote: > >> Hi Guys, >> >> I am writing a scrapper which will download youtube video based on the >> criteria you have provided. I am using youtube-dl module to download the >> video. However facing following issue my code is not executing further >> until whole video is downloaded. >> >> Is there any way to create a pipeline to download videos. As we created >> to download images ? >> >> >> following is the code. >> >> >> class FetchDetails(CrawlSpider): >> name = "youtube" >> allowed_domains = ["www.youtube.com"] >> start_urls = [" >> https://www.youtube.com/results?search_query=scrapy&page=1";] >> >> def parse(self, response): >> hxs = HtmlXPathSelector(response) >> base_url = "https://www.youtube.com"; >> video_links = >> hxs.select("//h3[@class='yt-lockup-title']/a/@href").extract() # >> href="/watch?v=1EFnX1UkXVU" >> for video in video_links: >> if "watch" in video: >> request = Request(base_url+video, >> callback=self.download_video) >> yield request >> >> def download_video(self, response): >> item = YoutubeItem() >> >> selector = Selector(response) >> view_count = >> selector.xpath("//div[@class='watch-view-count']/text()")[0].extract().strip() >> # video_url = >> selector.xpath("//video[contains(@class,'html5-main-video')]/@src").extract() >> >> # item['video_url'] = video_url >> item['view_count'] = view_count >> >> subprocess.call(["youtube-dl", response.url]) >> print view_count >> >> yield item >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "scrapy-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "scrapy-users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/scrapy-users/3fqqBEmnNYE/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
