Hey everyone,

On Tue, Apr 6, 2010 at 20:23, Andrzej Bialecki <a...@getopt.org> wrote:
> On 2010-04-06 15:43, Julien Nioche wrote:
>> Hi guys,
>> I gather that we'll jump straight to  2.0 after 1.1 and that 2.0 will be
>> based on what is currently referred to as NutchBase. Shall we create a
>> branch for 2.0 in the Nutch SVN repository and have a label accordingly for
>> JIRA so that we can file issues / feature requests on 2.0? Do you think that
>> the current NutchBase could be used as a basis for the 2.0 branch?
> I'm not sure what is the status of the nutchbase - it's missed a lot of
> fixes and changes in trunk since it's been last touched ...

I know... But I still intend to finish it, I just need to schedule
some time for it.

My vote would be to go with nutchbase.

>> Talking about features, what else would we add apart from :
>> * support for HBase : via ORM or not (see
>> NUTCH-808<https://issues.apache.org/jira/browse/NUTCH-808>
>> )
> This IMHO is promising, this could open the doors to small-to-medium
> installations that are currently too cumbersome to handle.

Yeah, there is already a simple ORM within nutchbase that is
avro-based and should
be generic enough to also support MySQL, cassandra and berkeleydb. But
any good ORM will
be a very good addition.

>> * plugin cleanup : Tika only for parsing - get rid of everything else?
> Basically, yes - keep only stuff like HtmlParseFilters (probably with a
> different API) so that we can post-process the DOM created in Tika from
> whatever original format.
> Also, the goal of the crawler-commons project is to provide APIs and
> implementations of stuff that is needed for every open source crawler
> project, like: robots handling, url filtering and url normalization, URL
> state management, perhaps deduplication. We should coordinate our
> efforts, and share code freely so that other projects (bixo, heritrix,
> droids) may contribute to this shared pool of functionality, much like
> Tika does for the common need of parsing complex formats.
>> * remove index / search and delegate to SOLR
> +1 - we may still keep a thin abstract layer to allow other
> indexing/search backends, but the current mess of indexing/query filters
> and competing indexing frameworks (lucene, fields, solr) should go away.
> We should go directly from DOM to a NutchDocument, and stop there.

Agreed. I would like to add support for katta and other indexing
backends at some point but
NutchDocument should be our canonical representation. The rest should
be up to indexing backends.

> Regarding search - currently the search API is too low-level, with the
> custom text and query analysis chains. This needlessly introduces the
> (in)famous Nutch Query classes and Nutch query syntax limitations, We
> should get rid of it and simply leave this part of the processing to the
> search backend. Probably we will use the SolrCloud branch that supports
> sharding and global IDF.
>> * new functionalities e.g. sitemap support, canonical tag etc...
> Plus a better handling of redirects, detecting duplicated sites,
> detection of spam cliques, tools to manage the webgraph, etc.
>> I suppose that http://wiki.apache.org/nutch/Nutch2Architecture needs an
>> update?
> Definitely. :)
> --
> Best regards,
> Andrzej Bialecki     <><
>  ___. ___ ___ ___ _ _   __________________________________
> [__ || __|__/|__||\/|  Information Retrieval, Semantic Web
> ___|||__||  \|  ||  |  Embedded Unix, System Integration
> http://www.sigram.com  Contact: info at sigram dot com

Doğacan Güney

Reply via email to