I have the same conditions like you meet.

I think to crawle a dynamic page is black hole for crawler.

we could not get all necessary parameters which need to post to a form .

and to fetch dynamic page , we need to identify the duplicate page.

2005/9/26, Jack Tang <[EMAIL PROTECTED]>:
>
> Hi Guys
>
> I known it is one difficult question for crawler and I just want to
> know is it possible to nutch's crawler.
>
> The page structure of website I want to crawl is like this
>
>
> -----> Page 1
>
> | http://a.com/next?pg=1
> search box --> url redirector -|----> Page 2
> http://a.com/search?x=1&y=2&z=3 http://a.com/b | http://a.com/next?pg=2
> http://a.com/search?x=2&y=2&z=4 -----> Page ...
>
> http://a.com/next?pg=.
>
>
> I added the URLs from search box into nutch crawler's feed. Say now
> crawler is processing http://a.com/search?x=1&y=2&z=3, and it is
> redirected to http://a.com/b, the response is OK now, so page 1
> (http://a.com/next) is fetched and parsed. Page 1 contains the
> pagination information like
>
> 1
> <a href="http://a.com/next?pg=2";>2</a>
> <a href="http://a.com/next?pg=3";>3</a>
> <a href="http://a.com/next?pg=4";>4</a>
>
> Unfortunately, page 2,3,4 is un-fetchable to crawler. Because url
> redirector stores query options into cookie/session, and pagination
> get the information from there. I mean the real url of page 2 is
> http://a.com/search?x=1&y=2&z=3&pg=2.
>
> Any solution for this scenario?
>
>
> Regards
> /Jack
> --
> Keep Discovering ... ...
> http://www.jroller.com/page/jmars
>

Reply via email to