give us an example of the page you want to crawl to tell you what is the
best way to do it instead of putting the element index manually.
you can post the HTML div if you don't want to share the website.
On Wednesday, February 25, 2015 at 10:17:30 AM UTC+2, Anitha Raji wrote:
>
> I am using scrapy to crawl a website. Here my specification is to get
> address and phone no from a xyz website. But i did it statically. But i
> want to get data dynamically.
>
> *Here is my code of spider (mySpider.py)*
>
> import scrapy
> from test_proj.items import myItem
> class mySpider(scrapy.Spider):
> name = "xyz"
> allowed_domains = ['your domain name']
> start_urls = ['your web addr']
>
> def parse(self, response):
>
> scrapy_item = myItem()
> addResp = response.xpath("//div[@class = 'entry-content
> post-content bottom-3']/p[7]")
> scrapy_item['addr'] = addResp.xpath('text()').extract()
> phoneResp = response.xpath("//div[@class = 'entry-content
> post-content bottom-3']/p[8]")
> scrapy_item['phone'] = phoneResp.xpath('text()').extract()
> yield scrapy_item
>
> *this is my items.py*
>
> import scrapy
> class myItem(scrapy.Item):
>
> addr = scrapy.Field()
> phone = scrapy.Field(
> )
>
> my question is I provide the <p> tag number [p7] in response.xpath()
> statically to get address from that xyz website. but if we are scraping
> data from a complex site, it is not possible to calculate the element
> number and we cannot give it accordingly. so we need to give the element
> number [p7] dynamically. is there any way to achieve this task? if you
> have any ideas please share with me. Thanks
>
>
>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
