I am using scrapy to crawl a website. Here my specification is to get
address and phone no from a xyz website. But i did it statically. But i
want to get data dynamically.
*Here is my code of spider (mySpider.py)*
import scrapy
from test_proj.items import myItem
class mySpider(scrapy.Spider):
name = "xyz"
allowed_domains = ['your domain name']
start_urls = ['your web addr']
def parse(self, response):
scrapy_item = myItem()
addResp = response.xpath("//div[@class = 'entry-content post-content
bottom-3']/p[7]")
scrapy_item['addr'] = addResp.xpath('text()').extract()
phoneResp = response.xpath("//div[@class = 'entry-content
post-content bottom-3']/p[8]")
scrapy_item['phone'] = phoneResp.xpath('text()').extract()
yield scrapy_item
*this is my items.py*
import scrapy
class myItem(scrapy.Item):
addr = scrapy.Field()
phone = scrapy.Field(
)
my question is I provide the <p> tag number [p7] in response.xpath()
statically to get address from that xyz website. but if we are scraping
data from a complex site, it is not possible to calculate the element
number and we cannot give it accordingly. so we need to give the element
number [p7] dynamically. is there any way to achieve this task? if you have
any ideas please share with me. Thanks
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
