I have the same conditions like you meet. I think to crawle a dynamic page is black hole for crawler.
we could not get all necessary parameters which need to post to a form . and to fetch dynamic page , we need to identify the duplicate page. 2005/9/26, Jack Tang <[EMAIL PROTECTED]>: > > Hi Guys > > I known it is one difficult question for crawler and I just want to > know is it possible to nutch's crawler. > > The page structure of website I want to crawl is like this > > > -----> Page 1 > > | http://a.com/next?pg=1 > search box --> url redirector -|----> Page 2 > http://a.com/search?x=1&y=2&z=3 http://a.com/b | http://a.com/next?pg=2 > http://a.com/search?x=2&y=2&z=4 -----> Page ... > > http://a.com/next?pg=. > > > I added the URLs from search box into nutch crawler's feed. Say now > crawler is processing http://a.com/search?x=1&y=2&z=3, and it is > redirected to http://a.com/b, the response is OK now, so page 1 > (http://a.com/next) is fetched and parsed. Page 1 contains the > pagination information like > > 1 > <a href="http://a.com/next?pg=2">2</a> > <a href="http://a.com/next?pg=3">3</a> > <a href="http://a.com/next?pg=4">4</a> > > Unfortunately, page 2,3,4 is un-fetchable to crawler. Because url > redirector stores query options into cookie/session, and pagination > get the information from there. I mean the real url of page 2 is > http://a.com/search?x=1&y=2&z=3&pg=2. > > Any solution for this scenario? > > > Regards > /Jack > -- > Keep Discovering ... ... > http://www.jroller.com/page/jmars >
