Hello, Have you tried using CrawlSpider instead of the base Spider class? https://doc.scrapy.org/en/latest/topics/spiders.html#crawlspider
Jim On Saturday, March 11, 2017 at 2:24:15 AM UTC-5, rana fge wrote: > > hello i am new in programming and scrapy. Trying to learn scrapy i try > scrap some items. but unable to do the scrap next page item, please help > how parse next link url in this situation. > Here is my code: > import scrapy > > from scrapy.linkextractors import LinkExtractor > class BdJobs(scrapy.Spider): > name = 'bdjobs' > allowed_domains = ['BdJobs.com'] > start_urls = [ > 'http://jobs.bdjobs.com/', > 'http://jobs.bdjobs.com/jobsearch.asp?fcatId=8&icatId=' > ] > #rules=( Rule(LinkExtractor(allow()), callback='parse', follow=True)) > > def parse(self, response): > for title in response.xpath('//div[@class="job-title-text"]/a'): > yield { > 'titles': title.xpath('./text()').extract()[0].strip() > } > > here is output: > [ > {"titles": "Senior Software Engineer (.Net)"}, > {"titles": "Java programmer"}, > {"titles": "VLSI Design Engineer (Japan)"}, > {"titles": "Assistant Executive (Computer Lab-Evening programs)"}, > {"titles": "IT Officer, Business System Management"}, > {"titles": "Executive, IT"}, > {"titles": "Officer, IT"}, > {"titles": "Laravel PHP Developer"}, > {"titles": "Executive - IT (EDISON Footwear)"}, > {"titles": "Software Engineer (PHP/ MySQL)"}, > {"titles": "Software Engineer [Back End]"}, > {"titles": "Full Stack Developer"}, > {"titles": "Mobile Application Developer (iOS/ Android)"}, > {"titles": "Head of IT Security Operations"}, > {"titles": "Database Administrator, Senior Analyst"}, > {"titles": "Infrastructure Delivery Senior Analyst, Network Security"}, > {"titles": "Head of IT Support Operations"}, > {"titles": "Hardware Engineer"}, > {"titles": "JavaScript/ Coffee Script Programmer"}, > {"titles": "Trainer - Auto CAD"}, > {"titles": "ASSISTENT PRODUCTION OFFICER"}, > {"titles": "Customer Relationship Executive"}, > {"titles": "Head of Sales"}, > {"titles": "Sample Master"}, > {"titles": "Manager/ AGM (Finance & Accounts)"}, > {"titles": "Night Aiditor"}, > {"titles": "Officer- Poultry"}, > {"titles": "Business Analyst"}, > {"titles": "Sr. Executive - Sales & Marketing (Sewing Thread)"}, > {"titles": "Civil Engineer"}, > {"titles": "Executive Director-HR"}, > {"titles": "Sr. Executive (MIS & Internal Audit)"}, > {"titles": "Manager, Health & Safety"}, > {"titles": "Computer Engineer (Diploma)"}, > {"titles": "Sr. Manager/ Manager, Procurement"}, > {"titles": "Specialist, Content"}, > {"titles": "Manager, Warranty and Maintenance"}, > {"titles": "Asst. Manager - Compliance"}, > {"titles": "Officer/Sr. Officer/Asst. Manager (Store)"}, > {"titles": "Manager, Maintenance (Sewing)"} > ] > > note: I have attached a image file for conveniance > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
