Otis Gospodnetic wrote:
I suppose the first thing to do would be describe the requirements for this
shard management. I imagine you have very specific functionality in mind from
your Wikia Search experience. Mind putting your ideas on the Wiki? I think it
would be very good to share this with [EMAIL PROTECTED] early on, so we can
come up with something general that fits both Nutch and Solr. It might turn
out that this calls for a separate Lucene project, but we'll see that once the
real discussion starts.
I completely agree. This would be better as a shared project. I will
put my current thoughts down on the Nutch wiki, unless there is already
a discussion going somewhere?
Dennis
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
From: Dennis Kubes <[EMAIL PROTECTED]>
To: [email protected]
Sent: Sunday, April 13, 2008 5:44:32 PM
Subject: Re: Next Generation Nutch
Otis Gospodnetic wrote:
Hello,
A few quick comments. I don't know how much you track Solr, but the mention of
shards makes me think of SOLR-303 and DistributedSearch page on Solr Wiki. You'll
want to check those out. In short, Solr has the notion of shards and distributed
search, kind of like Nutch with its RPC framework and searchers. *That* is one
big duplication of work, IMHO. As far as the indexing+searching+shards go, I
think one direction worth looking at carefully would be the gentle Nutch->Solr
relationship -- using Solr to do indexing and searching. Shard management doesn't
exist in either project yet, but I think it would be ideal to come up with a
common management mechanism, if possible.
In thinking about a new nutch I always thought that shard management is
absolutely necessary but it never felt right in terms of where it
belongs. If we are saying that nutch is small tools strung together to
produce different types of search indexes, shard management isn't really
a tool. It is more of something after that is needed. And yes both
Nutch and Solr as well as other people using lucene indexes need some
type of distributed index management system and I don't want to
duplicate this work. Perhaps this is a good proposal for a separate
lucene sub-project. Hey we could even call it shard ;)
Dennis
I think this addresses your "... but Nutch would need to improve
on the search server and shard management side of things to be able to
scale to the billion page level. So the next generation of Nutch I
think should focus on web scale search." statement.
I know of a well-known, large corporation evaluating Solr (and its dist. search
in particular) to handle 1-2B docs and 100 QPS.
I don't fully follow the part about getting rid of plugins, spring, etc., so I
can't comment.
Regarding the webapp - perhaps Solr and SolrJ could be used here. Solr itself
is a webapp, and it contains various ResponseWriters that can output XML, JSON,
pure Ruby, Python, even binary responses (in JIRA).
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
From: Dennis Kubes <[EMAIL PROTECTED]>
To: [email protected]
Sent: Friday, April 11, 2008 5:59:41 PM
Subject: Next Generation Nutch
I have been thinking about a next generation Nutch for a while now, had
some talks with some of the other committers, and have gotten around to
putting some thoughts / requirements down on paper. I wanted to run
these by the community and get feedback. This message will be a bit
long so please bear with me.
First let me define that I think that the purpose of Nutch is to be a
web search engine. When I say that I mean to specifically exclude
enterprise search. By web search I am talking about general or vertical
search engines in the 1M-20B document range. I am excluding things such
as database centric search and possibly even local filesystem search.
IMO Solr is a very capable enterprise search product and could handle
local filesystem search (if it doesn't already) and Nutch shouldn't try
to overlap functionality. I think it should be able to interact, maybe
share indexes yes, but not overlap purpose. I think that Nutch should
be designed to handle large datasets, meaning it has the ability to
scale to billions, perhaps 10s of billions of pages. Hadoop already
gives us this capability for processing but Nutch would need to improve
on the search server and shard management side of things to be able to
scale to the billion page level. So the next generation of Nutch I
think should focus on web scale search.
After working with Hadoop and MapReduce for the last couple of years I
find it interesting just how similar development of MapReduce programs
seem to be to the linux/unix philosophy of small programs chained
together to accomplish big things. So going forward I see this as a
healthy overall general architecture. Nutch would have many small tools
that would be linked through data structures. We already do this to
some extent in the current version of Nutch, an example of which would
be the tools that generate and act on CrawlDatum objects (CrawlDb,
UpdateDb, etc.). I would like to keep that idea of tools and data
structures wth the tools are chained together perhaps only by shell or
management scripts, in different pipelines acting on the data
structures. When I say data structure I don't mean binary map or
sequence files. These may be a standard way to store these objects but
Hadoop allows any input / output formats whether that be to HBase, a
relational database, a local filesytem. I think we should be open to
have those data structures stored however is best for the user through
different hadoop formats. So a general overall architecture of tools
and data structures and pipelines of these tools.
I currently see five or six distinct phases to a web search engine.
They are; Acquire, Parse, Analyze, Index, Search, and Shard Management.
Ok shard management might not be so much a phase as a functionality.
Acquire is simply the acquisition of the document be it PDF, HTML, or
images. This would usually be the crawler phase. Parse is parsing that
content into useful and standard data structures. I do believe that
parsing should be separate and distinct from crawling. If you crawl 50%
of 5M pages and the crawler dies, you should still be able to use that
50% you crawled. Analyze is what we do with the content once it is
parsed into a standard structure we can use. This could be anything
from a better link analysis to natural language processing, language
identification, and machine learning. The analysis phase should
probably have an ever expanding set of tools for different purposes.
These tools would create specialized data structures of their own.
Eventually through all the analysis we come up with a score for a given
piece of content. That could be a document or a field. Indexing is the
process of taking the analysis scores and content and creating the
indexes for searching. Searching is concerned with the searching of the
indexes. This should be doable from command line, web based, or other
ways. Shard management is concerned with the deployment and management
of large number of indexes.
I think the next generation of nutch should allow the changing of
different tools in any of these areas. What this means is the ability
to have different components such as web crawlers (as long as the end
data structure is the same), for example Fetcher, Fetcher2, Grub,
Heretrix, or even specialized crawlers. And different components for
different analysis types. I don't see a lot of cross-cutting concerns
here. And where there is, url normalization for example, I think it can
be handled better through dependency injection.
Which brings me to three. I think it is time to get rid of the plugin
framework. I want to keep the functionality of the various plugins but
I think a dependency injection framework, such as spring, creating the
components needed for logic inside of tools is a much cleaner way to
proceed. This would allow much better unit and mock testing of tool and
logic functionality. It would allow Nutch to run on a non "nutchified"
Hadoop cluster, meaning just a plain old hadoop cluster. We could have
core jars and contrib jars and a contrib directory which is pulled from
by shell scripts when submitting jobs to Hadoop. With the
multiple-resources functionality in Hadoop it would be a simple matter
of creating the correct command lines for the job to run.
And that brings me to separation of data and presentation. Currently
the Nutch website is one monolithic jsp application with plugins. I
think the next generation should segment that out into xml / json feeds
and a separate front end that uses those feeds. Again this would make
it much easier to create web applications using nutch.
And of course I think that shard management, a la Hadoop master and
slave style, is a big requirement as well. I also think a full test
suite with mock objects and local and MiniMR and MiniDFS cluster testing
is important as is better documentation and tutorials (maybe even a book
:)). So up to this point I have created MapReduce jobs that use spring
for dependency injection and it is simple and works well. The above is
the direction I would like to head down but I would also like to see
what everyone else is thinking.
Dennis
