To crawl dynamic pages you need to know the dynamics structure of each website separately.

Or, as I used to do it, just crawl anything in small enough chunks, and when something goes wrong, look at the said website, determine why it happened, modify the urlfilter and repeat the process. This is feasible process if you can dedicate yourself full time for up to a million pages or so. In my experience, anything over 1 million pages, with dynamic pages in it, is just plain asking the fetchlists to go into infinity. I've encountered static pages that are creative with the apache extensions and throw nutch into infinity. I've encountered huge websites with loads of static files that also mix up the desired fetching range.

I know that if you are big user (several dedicated machines in a data center with fast connection...) you probably don't care about this, your crawler will run over any website, with 50-500 threads the default three retry times, and the problem will solve itself out. But, can something be done for the rest or us, please?

A simple <max-host-pages>500</max-host-pages> would really be appreciated..

Regards,
EM



Transbuerg Tian wrote:

I have the same conditions like you meet.

I think to crawle a dynamic page is black hole for crawler.

we could not get all necessary parameters which need to post to a form .

and to fetch dynamic page , we need to identify the duplicate page.

2005/9/26, Jack Tang <[EMAIL PROTECTED]>:
Hi Guys

I known it is one difficult question for crawler and I just want to
know is it possible to nutch's crawler.

The page structure of website I want to crawl is like this


-----> Page 1

| http://a.com/next?pg=1
search box --> url redirector -|----> Page 2
http://a.com/search?x=1&y=2&z=3 http://a.com/b | http://a.com/next?pg=2
http://a.com/search?x=2&y=2&z=4 -----> Page ...

http://a.com/next?pg=.


I added the URLs from search box into nutch crawler's feed. Say now
crawler is processing http://a.com/search?x=1&y=2&z=3, and it is
redirected to http://a.com/b, the response is OK now, so page 1
(http://a.com/next) is fetched and parsed. Page 1 contains the
pagination information like

1
<a href="http://a.com/next?pg=2";>2</a>
<a href="http://a.com/next?pg=3";>3</a>
<a href="http://a.com/next?pg=4";>4</a>

Unfortunately, page 2,3,4 is un-fetchable to crawler. Because url
redirector stores query options into cookie/session, and pagination
get the information from there. I mean the real url of page 2 is
http://a.com/search?x=1&y=2&z=3&pg=2.

Any solution for this scenario?


Regards
/Jack
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars





-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers

Reply via email to