map/reduce will be a suitable approach for indexing large doc
collections. but I don't know is it suitable for retrieval. you can see
*Nutch* for the distributed searching.
under the hadoop/contrib directory , there is a *Index* package. It may
be helpful :)
Matt Wood 写道:
Hello all,
I was wondering if someone in the know could tell me about the current
state of play with building and searching large indices with hadoop?
Some background: I work on the human genome project, and we're
currently setting up a new facility based around the next generation
of DNA sequencing. We're currently producing around 50Tb of data a
week, some of which we would like to provide fast access to via an index.
Having read up on hadoop, it appears that it could play a central part
in our infrastructure, and that others have tried (and succeeded) in
building a distributed indexing and retrieval system with hadoop. I'd
be interested if anyone could point me in the right direction to more
information or examples of such a system. Yahoo! (with webmap) seems
to be close to the sort of thing we would need.
Would map/reduce be a suitable approach for indexing _and_ retrieval,
or just indexing? Would Solr/Lucene be a good fit? Any help or
pointers to more information would be much appreciated!
If you would like any more details, I'd be more than happy to supply
them!
Many thanks,
~ Matt
-------------
Matt Wood
Sequencing Informatics // Production Software
www.sanger.ac.uk
