According to a hadoop tutorial (http://wiki.apache.org/nutch/
NutchHadoopTutorial) on wiki,
"you don't want to search using DFS, you want to search using local
filesystems. Once the index has been created on the DFS you can
use the hadoop copyToLocal command to move it to the local file
system as such" ... "Understand that at this point we are not using
the DFS or MapReduce to do the searching, all of it is on a local
machine".
So my understanding is that hadoop is only good for batch index
building, and is not proper for incremental index building and
search. Is this true?
The reason I am asking is that when I read the article ACM article by
Mike Cafarella and Doug Cutting, to me it sounded like the concern
was to make the index structures fit in the primary memory, not the
entire crawled database. Did I miss understand the ACM article?
- Hadoop distributed search. Trey Spiva
-
