Hello,
Comments inlined.
----- Original Message ----
From: Dennis Kubes <ku...@apache.org>
To: nutch-user@lucene.apache.org
Sent: Friday, March 13, 2009 8:19:37 PM
With the release of Nutch 1.0 I think it is a good time to begin a discussion
about the future of Nutch. Here are some things to consider and would love to
here everyones views on this
Nutch's original intention was as a large-scale www search engine. That is a
very specific goal. Only a few people and organizations actually use it on that
level. (I just happen to be one of them as most of my work focuses on large
scale web search as opposed to vertical search).
Yes, there are fewer parties doing large scale web crawling. Still, as there
is no alternative fetcher+parser+indexer+searcher capable of handling large
scale deployments like Nutch (or maybe Heritrix has the same scaling
capabilities?), I think Nutch's ability to perform web-wide crawls, etc. should
be preserved.
Many, perhaps most, people
using Nutch these days are either using parts of Nutch, such as the crawler, or
are targeting towards vertical or intranet type search engines. This can be
seen in how many people have already started using the Solr integration
features. So while Nutch was originally intended as a www search, IMO most
people aren't using it for that purpose.
That's my experience, too. I think we can have both under the same Nutch roof.
Since there are different purposes for different users, would it be good to
consider moving Nutch to a top level apache project out from under the Lucene
umbrella? This would then allow the creation of nutch sub-projects, such as
nutch-solr, nutch-hbase. Thoughts?
I disagree, at least in the near term. There is nothing preventing those sub-projects
existing under Nutch today. Both Solr and Lucene have the contrib area where similar
sub-projects live. I think it's not a matter of being a TLP, but rather attracting
enough developer interest, then user interest, and then contributor interest, so that
these sub-projects can be created, maintained, advanced. Right now, Solr gets a TON of
attention, as does Lucene. Nutch gets the least developer attention, and for some reason
the nutch-user subscribers "feel" a bit different from solr-user or java-user
subscribers.
Many parts of Nutch have also been implemented in other projects. For example,
Tika for the parsers, Droids for the Crawler. In begs the question what is
Nutch's core features going forward. When I think about search (again my
perspective is large scale), I think crawling or acquisition of data, parsing,
analysis, indexing, deployment, and searching. I personally think that there is
much room for improvement in crawling and especially analysis. Nutch shouldn't
just be about the shell but also the brains.
My feeling has long been that indexing and searching should be outsourced to
Solr, parsing to Tika, and that the fetcher should probably be replaced with
Droids. I say probably because I'm not very familiar with Droids just yet.
Nutch should, I think, then be an application built with all those components
combined (is that what you mean by the shell?), and then apply its knowledge of
either web-wide scale trickery, or vertical SE trickery, or ... I think that's
where the brains are needed, to tie it all together, while still making certain
pieces swappable and more easily digestible by potential new contributors and
developers, as well as users. I know plugins do some of that already, but it
seems like there might still be more in the fore than there should/could be...
And one of the biggest things I see is many newcomers to nutch have a very hard
time getting started. Part of this is understanding mapreduce mentality, part
is documentation, part is there is only so much time some of us have to answer
questions so some questions go unanswered on the lists. How might this be
improved going forward?
I am not 100% sure, but I think it's a bit of all of the above. Lucene has been around
for 10 years and from day one had people answer questions from the most basic ones to the
trickiest ones. It's the same with Solr today. Nutch has the least active and the
smallest developer base, so questions don't get answered. Again, people on this list
also tend to have a different "style" of asking questions - no hellos, no thank
yous, and so on, which doesn't help.
I think the existence of a book on Lucene helped Lucene, but Solr doesn't yet
have a book, yet it still has a healthy developer and user community. I think
that's because Solr is simply more needed by more people than Nutch is.
Any other thoughts also welcome. Really I want to start a discussion about
where everyone thinks we are with the state of Nutch and its future.
I think it's good you started this discussion. My opinion about what needs to
be done with Nutch is above. I think it needs to stay with Hadoop. I think it
should remain under Lucene for now. Once and iff it develops those
sub-projects and we all feel it's better for it to be TLP, then I think we can
bring this up again.
Otis
