Hey Stefan:
The page is (old) ranging notes I kept looking at what it would take to integrate nutch and Heritrix. Hooking up Heritrix as the nutch crawler wouldn't be that easy. In its natural habitat, Heritrix commands the crawl keeping all state and figuring what to fetch next. To work with nutch, Heritrix would have to capitulate and take all instruction from nutch and write fetched content into nutch segments rather than as ARCs; A completly different mode of operation. Though we'd love to see/help it happen, we're not actively working on such an integration.
Slightly related, since we here at the Internet Archive need to be able to search in ARCs, we are actively looking at feeding the ARCs through nutch.
St.Ack
Stefan Groschupf wrote:
Hi there,
great, I notice something is going on regarding heritrix and nutch integration.
http://crawler.archive.org/cgi-bin/wiki.pl?NutchSearchingArcs
I'm a bit confused by this page since it looks like the actually work is focused to make arcs (heritrix file format) search able but not to use heritrix as crawler.
Wouldn't make it sense to make to have a crawler extension point and have the nutch crawler and a heritrix crawler plugin?
I haven't that much experience with heritrix, but looks like this crawler is very powerful.
Can someone involved in the integration project give any status comment and tell us the long term goal of the integration?
Thank you very much!
Stefan
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
