Hi EM

On 9/26/05, EM <[EMAIL PROTECTED]> wrote:
> To crawl dynamic pages you need to know the dynamics structure of each
> website separately.
>
> Or, as I used to do it, just crawl anything in small enough chunks, and
> when something goes wrong, look at the said website, determine why it
> happened, modify the urlfilter and repeat the process.

Thank you for you advice. I figured out the reason, and I thought
"urlfilter and repeat the process" cannot solve the problem. because
nutch's crawler cannot stored "page context" if "url redirector"
exists.

> This is feasible
> process if you can dedicate yourself full time for up to a million pages
> or so. In my experience, anything over 1 million pages, with dynamic
> pages in it, is just plain asking the fetchlists to go into infinity.
> I've encountered static pages that are creative with the apache
> extensions and throw nutch into infinity. I've encountered huge websites
> with loads of static files that also mix up the desired fetching range.
>
> I know that if you are big user (several dedicated machines in a data
> center with fast connection...) you probably don't care about this, your
> crawler will run over any website, with 50-500 threads the default three
> retry times, and the problem will solve itself out. But, can something
> be done for the rest or us, please?
No, I don't think so. Some web designer put the "url director" as 
obstacle of search engine. It is common in China. And you cannot get
conten of these websites at all.

> A simple <max-host-pages>500</max-host-pages> would really be appreciated..
>
> Regards,
> EM
>
>
>
> Transbuerg Tian wrote:
>
> >I have the same conditions like you meet.
> >
> >I think to crawle a dynamic page is black hole for crawler.
> >
> >we could not get all necessary parameters which need to post to a form .
> >
> >and to fetch dynamic page , we need to identify the duplicate page.
> >
> >2005/9/26, Jack Tang <[EMAIL PROTECTED]>:
> >
> >
> >>Hi Guys
> >>
> >>I known it is one difficult question for crawler and I just want to
> >>know is it possible to nutch's crawler.
> >>
> >>The page structure of website I want to crawl is like this
> >>
> >>
> >>-----> Page 1
> >>
> >>| http://a.com/next?pg=1
> >>search box --> url redirector -|----> Page 2
> >>http://a.com/search?x=1&y=2&z=3 http://a.com/b | http://a.com/next?pg=2
> >>http://a.com/search?x=2&y=2&z=4 -----> Page ...
> >>
> >>http://a.com/next?pg=.
> >>
> >>
> >>I added the URLs from search box into nutch crawler's feed. Say now
> >>crawler is processing http://a.com/search?x=1&y=2&z=3, and it is
> >>redirected to http://a.com/b, the response is OK now, so page 1
> >>(http://a.com/next) is fetched and parsed. Page 1 contains the
> >>pagination information like
> >>
> >>1
> >><a href="http://a.com/next?pg=2";>2</a>
> >><a href="http://a.com/next?pg=3";>3</a>
> >><a href="http://a.com/next?pg=4";>4</a>
> >>
> >>Unfortunately, page 2,3,4 is un-fetchable to crawler. Because url
> >>redirector stores query options into cookie/session, and pagination
> >>get the information from there. I mean the real url of page 2 is
> >>http://a.com/search?x=1&y=2&z=3&pg=2.
> >>
> >>Any solution for this scenario?
> >>
> >>
> >>Regards
> >>/Jack
> >>--
> >>Keep Discovering ... ...
> >>http://www.jroller.com/page/jmars
> >>
> >>
> >>
> >
> >
> >
>
>


--
Keep Discovering ... ...
http://www.jroller.com/page/jmars


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers

Reply via email to