To crawl dynamic pages you need to know the dynamics structure of each
website separately.
Or, as I used to do it, just crawl anything in small enough chunks, and
when something goes wrong, look at the said website, determine why it
happened, modify the urlfilter and repeat the process. This is feasible
process if you can dedicate yourself full time for up to a million pages
or so. In my experience, anything over 1 million pages, with dynamic
pages in it, is just plain asking the fetchlists to go into infinity.
I've encountered static pages that are creative with the apache
extensions and throw nutch into infinity. I've encountered huge websites
with loads of static files that also mix up the desired fetching range.
I know that if you are big user (several dedicated machines in a data
center with fast connection...) you probably don't care about this, your
crawler will run over any website, with 50-500 threads the default three
retry times, and the problem will solve itself out. But, can something
be done for the rest or us, please?
A simple <max-host-pages>500</max-host-pages> would really be appreciated..
Regards,
EM
Transbuerg Tian wrote:
I have the same conditions like you meet.
I think to crawle a dynamic page is black hole for crawler.
we could not get all necessary parameters which need to post to a form .
and to fetch dynamic page , we need to identify the duplicate page.
2005/9/26, Jack Tang <[EMAIL PROTECTED]>:
Hi Guys
I known it is one difficult question for crawler and I just want to
know is it possible to nutch's crawler.
The page structure of website I want to crawl is like this
-----> Page 1
| http://a.com/next?pg=1
search box --> url redirector -|----> Page 2
http://a.com/search?x=1&y=2&z=3 http://a.com/b | http://a.com/next?pg=2
http://a.com/search?x=2&y=2&z=4 -----> Page ...
http://a.com/next?pg=.
I added the URLs from search box into nutch crawler's feed. Say now
crawler is processing http://a.com/search?x=1&y=2&z=3, and it is
redirected to http://a.com/b, the response is OK now, so page 1
(http://a.com/next) is fetched and parsed. Page 1 contains the
pagination information like
1
<a href="http://a.com/next?pg=2">2</a>
<a href="http://a.com/next?pg=3">3</a>
<a href="http://a.com/next?pg=4">4</a>
Unfortunately, page 2,3,4 is un-fetchable to crawler. Because url
redirector stores query options into cookie/session, and pagination
get the information from there. I mean the real url of page 2 is
http://a.com/search?x=1&y=2&z=3&pg=2.
Any solution for this scenario?
Regards
/Jack
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers