Hi,

for testing purpose, i wrote a small scrapy script that should find next 
page based on <a> text.
However if i was able to do it in utf-8, i encountrer some issue with web 
page encoded in "windows-1250" while my scrapy script and by default text 
is written in utf-8

let's have a look at: https://www.vsetkyfirmy.sk/autoskoly

the bottom pagination display "Next page" in local language "Ďalšie >> 
<https://www.vsetkyfirmy.sk/autoskoly/strana_2.html>" and i would like to 
retrieve the complete url of this <a> so if we are on the first page: 
https://www.vsetkyfirmy.sk/autoskoly/strana_2.html, if we are on the page 
2, https://www.vsetkyfirmy.sk/autoskoly/strana_3.html, etc...

however this webpage is encoded in "windows-1250" and in my scrapy script 
i'm confused as i use utf-8 and the following code to retrieve the <a> url:
t = Selector(response).xpath('//*[text()[contains(., 
"Ďalšie")]]/@href').extract()

but once done...scrapy says:
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL 
bytes or control characters

So what should i do to achieve what i want ?

thx

-- 
You received this message because you are subscribed to the Google Groups 
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to scrapy-users+unsubscr...@googlegroups.com.
To post to this group, send email to scrapy-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

Reply via email to