I am using Nutch for more than four years now, as a vertical search engine,
having indexed, some times, over one million pages. On the other hand, I
dont know nothing about programming and some specialized aplications. Words
like solr and others are like aliens for me. I am just interested in a
search engine that someone can, really, use and not an application that
serve as a base for developping sophisticated models.
So, what I, personally want for the future of Nutch is that it does not turn
in such a complicated aplication that just some very skilled people can use.
So I hope that Nutch keeps, allways, an eye on the real users, that want it
for plain searching.
Thanks
----- Original Message -----
From: "Dennis Kubes" <ku...@apache.org>
To: <nutch-user@lucene.apache.org>
Sent: Friday, March 13, 2009 9:19 PM
Subject: The Future of Nutch
With the release of Nutch 1.0 I think it is a good time to begin a
discussion about the future of Nutch. Here are some things to consider
and would love to here everyones views on this
Nutch's original intention was as a large-scale www search engine. That
is a very specific goal. Only a few people and organizations actually
use it on that level. (I just happen to be one of them as most of my
work focuses on large scale web search as opposed to vertical search).
Many, perhaps most, people using Nutch these days are either using parts
of Nutch, such as the crawler, or are targeting towards vertical or
intranet type search engines. This can be seen in how many people have
already started using the Solr integration features. So while Nutch was
originally intended as a www search, IMO most people aren't using it for
that purpose.
Since there are different purposes for different users, would it be good
to consider moving Nutch to a top level apache project out from under
the Lucene umbrella? This would then allow the creation of nutch
sub-projects, such as nutch-solr, nutch-hbase. Thoughts?
Many parts of Nutch have also been implemented in other projects. For
example, Tika for the parsers, Droids for the Crawler. In begs the
question what is Nutch's core features going forward. When I think
about search (again my perspective is large scale), I think crawling or
acquisition of data, parsing, analysis, indexing, deployment, and
searching. I personally think that there is much room for improvement
in crawling and especially analysis. Nutch shouldn't just be about the
shell but also the brains.
And one of the biggest things I see is many newcomers to nutch have a
very hard time getting started. Part of this is understanding mapreduce
mentality, part is documentation, part is there is only so much time
some of us have to answer questions so some questions go unanswered on
the lists. How might this be improved going forward?
Any other thoughts also welcome. Really I want to start a discussion
about where everyone thinks we are with the state of Nutch and its future.
Dennis
--------------------------------------------------------------------------------
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.0.237 / Virus Database: 270.11.13/2001 - Release Date: 03/14/09
06:54:00
