Hi Guys
I known it is one difficult question for crawler and I just want to
know is it possible to nutch's crawler.
The page structure of website I want to crawl is like this
-----> Page 1
| http://a.com/next?pg=1
search box --> url redirector -|----> Page 2
http://a.com/search?x=1&y=2&z=3 http://a.com/b | http://a.com/next?pg=2
http://a.com/search?x=2&y=2&z=4 -----> Page ...
http://a.com/next?pg=.
I added the URLs from search box into nutch crawler's feed. Say now
crawler is processing http://a.com/search?x=1&y=2&z=3, and it is
redirected to http://a.com/b, the response is OK now, so page 1
(http://a.com/next) is fetched and parsed. Page 1 contains the
pagination information like
1
<a href="http://a.com/next?pg=2">2</a>
<a href="http://a.com/next?pg=3">3</a>
<a href="http://a.com/next?pg=4">4</a>
Unfortunately, page 2,3,4 is un-fetchable to crawler. Because url
redirector stores query options into cookie/session, and pagination
get the information from there. I mean the real url of page 2 is
http://a.com/search?x=1&y=2&z=3&pg=2.
Any solution for this scenario?
Regards
/Jack
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers