Hi EM On 9/26/05, EM <[EMAIL PROTECTED]> wrote: > To crawl dynamic pages you need to know the dynamics structure of each > website separately. > > Or, as I used to do it, just crawl anything in small enough chunks, and > when something goes wrong, look at the said website, determine why it > happened, modify the urlfilter and repeat the process.
Thank you for you advice. I figured out the reason, and I thought "urlfilter and repeat the process" cannot solve the problem. because nutch's crawler cannot stored "page context" if "url redirector" exists. > This is feasible > process if you can dedicate yourself full time for up to a million pages > or so. In my experience, anything over 1 million pages, with dynamic > pages in it, is just plain asking the fetchlists to go into infinity. > I've encountered static pages that are creative with the apache > extensions and throw nutch into infinity. I've encountered huge websites > with loads of static files that also mix up the desired fetching range. > > I know that if you are big user (several dedicated machines in a data > center with fast connection...) you probably don't care about this, your > crawler will run over any website, with 50-500 threads the default three > retry times, and the problem will solve itself out. But, can something > be done for the rest or us, please? No, I don't think so. Some web designer put the "url director" as obstacle of search engine. It is common in China. And you cannot get conten of these websites at all. > A simple <max-host-pages>500</max-host-pages> would really be appreciated.. > > Regards, > EM > > > > Transbuerg Tian wrote: > > >I have the same conditions like you meet. > > > >I think to crawle a dynamic page is black hole for crawler. > > > >we could not get all necessary parameters which need to post to a form . > > > >and to fetch dynamic page , we need to identify the duplicate page. > > > >2005/9/26, Jack Tang <[EMAIL PROTECTED]>: > > > > > >>Hi Guys > >> > >>I known it is one difficult question for crawler and I just want to > >>know is it possible to nutch's crawler. > >> > >>The page structure of website I want to crawl is like this > >> > >> > >>-----> Page 1 > >> > >>| http://a.com/next?pg=1 > >>search box --> url redirector -|----> Page 2 > >>http://a.com/search?x=1&y=2&z=3 http://a.com/b | http://a.com/next?pg=2 > >>http://a.com/search?x=2&y=2&z=4 -----> Page ... > >> > >>http://a.com/next?pg=. > >> > >> > >>I added the URLs from search box into nutch crawler's feed. Say now > >>crawler is processing http://a.com/search?x=1&y=2&z=3, and it is > >>redirected to http://a.com/b, the response is OK now, so page 1 > >>(http://a.com/next) is fetched and parsed. Page 1 contains the > >>pagination information like > >> > >>1 > >><a href="http://a.com/next?pg=2">2</a> > >><a href="http://a.com/next?pg=3">3</a> > >><a href="http://a.com/next?pg=4">4</a> > >> > >>Unfortunately, page 2,3,4 is un-fetchable to crawler. Because url > >>redirector stores query options into cookie/session, and pagination > >>get the information from there. I mean the real url of page 2 is > >>http://a.com/search?x=1&y=2&z=3&pg=2. > >> > >>Any solution for this scenario? > >> > >> > >>Regards > >>/Jack > >>-- > >>Keep Discovering ... ... > >>http://www.jroller.com/page/jmars > >> > >> > >> > > > > > > > > -- Keep Discovering ... ... http://www.jroller.com/page/jmars ------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
