Hi Kelvin:
I started to dig into the code and data structure of
OC;
Just curious questions:
1) Where OC forms a fetchlist? I didn't see it in
segment/ of OC created.
2) In OC, FetchList is organized in such a way of URLs
sequence per host. Then, what if there are too many
hosts, saying ten thousand. How about I/O performance
concern? Will it exceed the system open-file
limitation?
thanks,
Michael Ji
--- Kelvin Tan <[EMAIL PROTECTED]> wrote:
> If we add a depth field to ScheduledURL, then
> controlling the depth of a crawl is simple:
>
> /**
> * Limits a crawl to a fixed depth. Seeds are depth
> 0.
> */
> public class DepthFLFilter implements
> ScopeFilter<FetchListScope.Input> {
> private int max;
>
> public synchronized int
> filter(FetchListScope.Input input) {
> return input.parent.depth < max ? ALLOW :
> REJECT;
> }
>
> public void setMax(int max) {
> this.max = max;
> }
> }
>
> On Sat, 27 Aug 2005 13:19:18 -0400, Kelvin Tan
> wrote:
> > Hey Michael, did you use the nutch-84 segment
> location as the
> > argument for the respective nutch commands, e.g..
> >
> > bin/nutch updatedb db <path_to_segment>
> >
> > If intending to integrate with webdb, you'll need
> to ensure the
> > directory structure of the segment output is what
> Nutch expects,
> > which means
> > db/segments/<segment_name>
> >
> > I haven't tried running Nutch with the index
> created, but when I
> > open the index in Luke, everything looks correct.
> Let me know if
> > you still have problems.
> >
> > To customize how domains are crawled, you'll want
> to write a
> > ScopeFilter. Take a look at SameParentHostFLFilter
> for an example.
> > When I have some time later today, I'll see if I
> can hack something
> > quick to limit crawling by depth..
> >
> > k
> >
> > On Sat, 27 Aug 2005 09:01:28 -0700 (PDT), Michael
> Ji wrote:
> >
> >> Hi there,
> >>
> >> I installed Nutch-84 patch in Nutch 07 and run
> patch test script
> >> successfully with my seeds.txt.
> >>
> >> It created /segment/ with sub-directories of
> "content",
> >> "fetcher", "parse_data" and "parse_text".
> >>
> >> Followings are the issues I met and concerning:
> >>
> >> 1) Indexing
> >>
> >> Then, I run nutch/index for this segment
> successfully. But there
> >> is no result (hits) returned in searching after
> I launch tomcat
> >> box.
> >>
> >> 2) Domain control
> >>
> >> As I understood, this patch is for control domain
> crawling. Seems
> >> we can define the fetching depth for both domain
> site and
> >> outlinking site by ourself. If so, where these
> parameters I can
> >> input?
> >>
> >> 3) Refetching
> >>
> >> Based on the fetched data, I tried several
> things, such as,
> >> running nutch/updatedb, nutch/gengerate,
> nutch/fetcher. Seems
> >> not working.
> >>
> >> Is there a scenario that I can adopt this patch
> for refetching
> >> purpose?
> >>
> >> thanks,
> >>
> >> Michael Ji,
> >>
> >>
>
>> ____________________________________________________
> Start your
> >> day with Yahoo! - make it your home page
> >> http://www.yahoo.com/r/hs
>
>
>
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
