Just a question ?
Will the new HBase implementation allow more sophisticated crawling
strategies than the current score based.

Give you a few  example of what I'd like to do :
Define different crawling frequency for different set of URLs, say
weekly for some url, monthly or more for others.

Select URLs to re-crawl based on attributes previously extracted.Just
one example: recrawl urls that contained a certain keyword (or set of)

Select URLs that have not yet been crawled, at the frontier of the
crawl therefore

2010/4/7, Doğacan Güney <doga...@gmail.com>:
> Hey everyone,
> On Tue, Apr 6, 2010 at 20:23, Andrzej Bialecki <a...@getopt.org> wrote:
>> On 2010-04-06 15:43, Julien Nioche wrote:
>>> Hi guys,
>>> I gather that we'll jump straight to  2.0 after 1.1 and that 2.0 will be
>>> based on what is currently referred to as NutchBase. Shall we create a
>>> branch for 2.0 in the Nutch SVN repository and have a label accordingly
>>> for
>>> JIRA so that we can file issues / feature requests on 2.0? Do you think
>>> that
>>> the current NutchBase could be used as a basis for the 2.0 branch?
>> I'm not sure what is the status of the nutchbase - it's missed a lot of
>> fixes and changes in trunk since it's been last touched ...
> I know... But I still intend to finish it, I just need to schedule
> some time for it.
> My vote would be to go with nutchbase.
>>> Talking about features, what else would we add apart from :
>>> * support for HBase : via ORM or not (see
>>> NUTCH-808<https://issues.apache.org/jira/browse/NUTCH-808>
>>> )
>> This IMHO is promising, this could open the doors to small-to-medium
>> installations that are currently too cumbersome to handle.
> Yeah, there is already a simple ORM within nutchbase that is
> avro-based and should
> be generic enough to also support MySQL, cassandra and berkeleydb. But
> any good ORM will
> be a very good addition.
>>> * plugin cleanup : Tika only for parsing - get rid of everything else?
>> Basically, yes - keep only stuff like HtmlParseFilters (probably with a
>> different API) so that we can post-process the DOM created in Tika from
>> whatever original format.
>> Also, the goal of the crawler-commons project is to provide APIs and
>> implementations of stuff that is needed for every open source crawler
>> project, like: robots handling, url filtering and url normalization, URL
>> state management, perhaps deduplication. We should coordinate our
>> efforts, and share code freely so that other projects (bixo, heritrix,
>> droids) may contribute to this shared pool of functionality, much like
>> Tika does for the common need of parsing complex formats.
>>> * remove index / search and delegate to SOLR
>> +1 - we may still keep a thin abstract layer to allow other
>> indexing/search backends, but the current mess of indexing/query filters
>> and competing indexing frameworks (lucene, fields, solr) should go away.
>> We should go directly from DOM to a NutchDocument, and stop there.
> Agreed. I would like to add support for katta and other indexing
> backends at some point but
> NutchDocument should be our canonical representation. The rest should
> be up to indexing backends.
>> Regarding search - currently the search API is too low-level, with the
>> custom text and query analysis chains. This needlessly introduces the
>> (in)famous Nutch Query classes and Nutch query syntax limitations, We
>> should get rid of it and simply leave this part of the processing to the
>> search backend. Probably we will use the SolrCloud branch that supports
>> sharding and global IDF.
>>> * new functionalities e.g. sitemap support, canonical tag etc...
>> Plus a better handling of redirects, detecting duplicated sites,
>> detection of spam cliques, tools to manage the webgraph, etc.
>>> I suppose that http://wiki.apache.org/nutch/Nutch2Architecture needs an
>>> update?
>> Definitely. :)
>> --
>> Best regards,
>> Andrzej Bialecki     <><
>>  ___. ___ ___ ___ _ _   __________________________________
>> [__ || __|__/|__||\/|  Information Retrieval, Semantic Web
>> ___|||__||  \|  ||  |  Embedded Unix, System Integration
>> http://www.sigram.com  Contact: info at sigram dot com
> --
> Doğacan Güney


Reply via email to