Hi Guys

I known it is one difficult question for crawler and I just want to
know is it possible to nutch's crawler.

The page structure of website I want to crawl is like this

                                                                      
       -----> Page 1
                                                                      
       |     http://a.com/next?pg=1
         search box                          --> url redirector   -|----> Page 2
http://a.com/search?x=1&y=2&z=3    http://a.com/b   |     http://a.com/next?pg=2
http://a.com/search?x=2&y=2&z=4                            -----> Page ...
                                                                      
             http://a.com/next?pg=.


I added the URLs from search box into nutch crawler's feed. Say now
crawler is processing http://a.com/search?x=1&y=2&z=3, and it is
redirected to http://a.com/b, the response is OK now, so page 1
(http://a.com/next) is fetched and parsed. Page 1 contains the
pagination information like

1
<a href="http://a.com/next?pg=2";>2</a>
<a href="http://a.com/next?pg=3";>3</a>
<a href="http://a.com/next?pg=4";>4</a>

Unfortunately, page 2,3,4 is un-fetchable to crawler. Because url
redirector stores query options into cookie/session, and pagination
get the information from there. I mean the real url of page 2 is
http://a.com/search?x=1&y=2&z=3&pg=2.

Any solution for this scenario?


Regards
/Jack
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers

Reply via email to