Hello, Comments inlined.
----- Original Message ---- > From: Dennis Kubes <ku...@apache.org> > To: nutch-user@lucene.apache.org > Sent: Friday, March 13, 2009 8:19:37 PM > > With the release of Nutch 1.0 I think it is a good time to begin a discussion > about the future of Nutch. Here are some things to consider and would love > to > here everyones views on this > > Nutch's original intention was as a large-scale www search engine. That is a > very specific goal. Only a few people and organizations actually use it on > that > level. (I just happen to be one of them as most of my work focuses on large > scale web search as opposed to vertical search). Yes, there are fewer parties doing large scale web crawling. Still, as there is no alternative fetcher+parser+indexer+searcher capable of handling large scale deployments like Nutch (or maybe Heritrix has the same scaling capabilities?), I think Nutch's ability to perform web-wide crawls, etc. should be preserved. > Many, perhaps most, people > using Nutch these days are either using parts of Nutch, such as the crawler, > or > are targeting towards vertical or intranet type search engines. This can be > seen in how many people have already started using the Solr integration > features. So while Nutch was originally intended as a www search, IMO most > people aren't using it for that purpose. That's my experience, too. I think we can have both under the same Nutch roof. > Since there are different purposes for different users, would it be good to > consider moving Nutch to a top level apache project out from under the Lucene > umbrella? This would then allow the creation of nutch sub-projects, such as > nutch-solr, nutch-hbase. Thoughts? I disagree, at least in the near term. There is nothing preventing those sub-projects existing under Nutch today. Both Solr and Lucene have the contrib area where similar sub-projects live. I think it's not a matter of being a TLP, but rather attracting enough developer interest, then user interest, and then contributor interest, so that these sub-projects can be created, maintained, advanced. Right now, Solr gets a TON of attention, as does Lucene. Nutch gets the least developer attention, and for some reason the nutch-user subscribers "feel" a bit different from solr-user or java-user subscribers. > Many parts of Nutch have also been implemented in other projects. For > example, > Tika for the parsers, Droids for the Crawler. In begs the question what is > Nutch's core features going forward. When I think about search (again my > perspective is large scale), I think crawling or acquisition of data, > parsing, > analysis, indexing, deployment, and searching. I personally think that there > is > much room for improvement in crawling and especially analysis. Nutch > shouldn't > just be about the shell but also the brains. My feeling has long been that indexing and searching should be outsourced to Solr, parsing to Tika, and that the fetcher should probably be replaced with Droids. I say probably because I'm not very familiar with Droids just yet. Nutch should, I think, then be an application built with all those components combined (is that what you mean by the shell?), and then apply its knowledge of either web-wide scale trickery, or vertical SE trickery, or ... I think that's where the brains are needed, to tie it all together, while still making certain pieces swappable and more easily digestible by potential new contributors and developers, as well as users. I know plugins do some of that already, but it seems like there might still be more in the fore than there should/could be... > And one of the biggest things I see is many newcomers to nutch have a very > hard > time getting started. Part of this is understanding mapreduce mentality, > part > is documentation, part is there is only so much time some of us have to > answer > questions so some questions go unanswered on the lists. How might this be > improved going forward? I am not 100% sure, but I think it's a bit of all of the above. Lucene has been around for 10 years and from day one had people answer questions from the most basic ones to the trickiest ones. It's the same with Solr today. Nutch has the least active and the smallest developer base, so questions don't get answered. Again, people on this list also tend to have a different "style" of asking questions - no hellos, no thank yous, and so on, which doesn't help. I think the existence of a book on Lucene helped Lucene, but Solr doesn't yet have a book, yet it still has a healthy developer and user community. I think that's because Solr is simply more needed by more people than Nutch is. > Any other thoughts also welcome. Really I want to start a discussion about > where everyone thinks we are with the state of Nutch and its future. I think it's good you started this discussion. My opinion about what needs to be done with Nutch is above. I think it needs to stay with Hadoop. I think it should remain under Lucene for now. Once and iff it develops those sub-projects and we all feel it's better for it to be TLP, then I think we can bring this up again. Otis
