If we add a depth field to ScheduledURL, then controlling the depth of a crawl
is simple:
/**
* Limits a crawl to a fixed depth. Seeds are depth 0.
*/
public class DepthFLFilter implements ScopeFilter<FetchListScope.Input> {
private int max;
public synchronized int filter(FetchListScope.Input input) {
return input.parent.depth < max ? ALLOW : REJECT;
}
public void setMax(int max) {
this.max = max;
}
}
On Sat, 27 Aug 2005 13:19:18 -0400, Kelvin Tan wrote:
> Hey Michael, did you use the nutch-84 segment location as the
> argument for the respective nutch commands, e.g..
>
> bin/nutch updatedb db <path_to_segment>
>
> If intending to integrate with webdb, you'll need to ensure the
> directory structure of the segment output is what Nutch expects,
> which means
> db/segments/<segment_name>
>
> I haven't tried running Nutch with the index created, but when I
> open the index in Luke, everything looks correct. Let me know if
> you still have problems.
>
> To customize how domains are crawled, you'll want to write a
> ScopeFilter. Take a look at SameParentHostFLFilter for an example.
> When I have some time later today, I'll see if I can hack something
> quick to limit crawling by depth..
>
> k
>
> On Sat, 27 Aug 2005 09:01:28 -0700 (PDT), Michael Ji wrote:
>
>> Hi there,
>>
>> I installed Nutch-84 patch in Nutch 07 and run patch test script
>> successfully with my seeds.txt.
>>
>> It created /segment/ with sub-directories of "content",
>> "fetcher", "parse_data" and "parse_text".
>>
>> Followings are the issues I met and concerning:
>>
>> 1) Indexing
>>
>> Then, I run nutch/index for this segment successfully. But there
>> is no result (hits) returned in searching after I launch tomcat
>> box.
>>
>> 2) Domain control
>>
>> As I understood, this patch is for control domain crawling. Seems
>> we can define the fetching depth for both domain site and
>> outlinking site by ourself. If so, where these parameters I can
>> input?
>>
>> 3) Refetching
>>
>> Based on the fetched data, I tried several things, such as,
>> running nutch/updatedb, nutch/gengerate, nutch/fetcher. Seems
>> not working.
>>
>> Is there a scenario that I can adopt this patch for refetching
>> purpose?
>>
>> thanks,
>>
>> Michael Ji,
>>
>>
>> ____________________________________________________ Start your
>> day with Yahoo! - make it your home page
>> http://www.yahoo.com/r/hs
-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
