Hi EM On 9/26/05, EM <[EMAIL PROTECTED]> wrote: > > >> > >>I know that if you are big user (several dedicated machines in a data > >>center with fast connection...) you probably don't care about this, your > >>crawler will run over any website, with 50-500 threads the default three > >>retry times, and the problem will solve itself out. But, can something > >>be done for the rest or us, please? > >> > >> > >No, I don't think so. Some web designer put the "url director" as > >obstacle of search engine. It is common in China. And you cannot get > >conten of these websites at all. > > > > > Maybe I wasn't totally clear, with 10 seconds of timeout , the fetcher > will jump over bunch of pages on the same host. Any obstacles will be > pretty much ignored because these pages won't be fetched and pages > leading from them won't be fetched also. On large scale, search engine > traps or not, the fetcher will play rough an get over them in 3 runs > (actually a bit more since some pages will be fetched). This is of > course the case if you don't need 100% of the pages, just as many as you > can fetch.
Well, let me explain more about my case. Say there is only one entry to list all content of the website: http://a.com/search?city=YourCity. (We take it as search engine on the website of course.) If I input "YourCity"'s value as "NewYork", then it will list all content related with NewYork, and there are many pages. And the pagination URL we expect is http://a.com/search?city=NewYork&pg=1 for the first page http://a.com/search?city=NewYork&pg=2 for the second page. Nutch crawler is not "trapped" in this case. But if we change the website design, it will be "trapped". After the request "http://a.com/search?city=NewYork" is sent, the website store the query parameters into cookie/session first, then redirect to another page, say http://a.com/next.html, and response it. Also the pagination URLs are changed. The query parameters are discarded, since all of them can be retrieved from cookie/session. The final pagination URL will be http://a.com/search?pg=1 for the first page http://a.com/search?pg=2 for the second page. If nutch crawler extract URL from HTML document, it definitely cannot get the desired content in parsing, right? Regards /Jack > People who are technically able to put search engine traps should be > technically able to put robots.txt, of course with both sides not > obeying the rules, it's a bit of a mess lately and everyone are paying > the price. > > I've encountered cases where spam was the issue and not search engine > traps. There's this website that has mod rewrite or something like that > setup so ANY RANDOM link you can type on his page is valid and it will > show you bunch of unrelated random advertisements. This is static page > by the way. Now, if I had 100mbps my fetcher would run over his website > without blinking, being limited to 2 the effect is noticeable. No matter > how many times I run the fetcher, the number of instructions left wasn't > decreasing ;) I've encountered cases like this, and instead of manually > typing regex to clean them off (which takes time) I'd strongly prefer an > automated solution if possible. > > Regards, > EM > > > -- Keep Discovering ... ... http://www.jroller.com/page/jmars ------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
