so let me make sure I understand. what this guy did is that he made an XML file from his local backup of wikipedia but he didn't crawl it?maybe I don't need to crawl it, either, since the XML file can include the "id" field which is where Solr keeps URLs, right?
What I want to be able to do is submit a search to Solr, get back an answer set as a file using wt=csv, use a shell script to wget the documents in the answer set and then process them various ways. I already have this working on test data, I just need to be able to include the wiki data in the search & research results so the shell script can go get them too. On Mon, Oct 10, 2011 at 10:32 AM, Markus Jelsma <markus.jel...@openindex.io>wrote: > That's something different. Indexing to Solr from a local backup of > wikipedia > is much and much quicker as you don't have to go through the whole crawldb > and > push all data to a reducer and finally to Solr. > > On Monday 10 October 2011 16:28:02 Fred Zimmerman wrote: > > OK, that sounds good. Tell me about the indexing. I came across an > > article where someone had indexed about 10% of a wikipedia clone > > > > http://h3x.no/2011/05/10/guide-solr-performance-tuning > > > > who with a much bigger machine and a *lot* of tuning was able to reduce > > time required from 168m to 16min for the 600,000 records. > > > > Fred > > > > > > > > On Mon, Oct 10, 2011 at 10:15 AM, Markus Jelsma > > > > <markus.jel...@openindex.io>wrote: > > > Hi, > > > > > > Based on our experience i would recommend running Nutch on a Hadoop > > > pseudo- cluster with a bit more memory and at least 4 CPU cores. Fetch > > > and parse of those url's wont' be a problem but updating the crawldb > and > > > generating fetch > > > lists is going to be a problem. > > > > > > Are you also indexing? Then that will also be a very costly process. > > > > > > Cheers > > > > > > On Saturday 08 October 2011 19:29:49 Fred Zimmerman wrote: > > > > HI, > > > > > > > > I am looking for advice on how to configure Nutch (and Solr) to crawl > a > > > > private Wikipedia mirror. > > > > > > > > - It is my mirror on an intranet so I do not need to be polite to > > > > > > > > myself. - I need to complete this 11 million page crawl as fast as I > > > > reasonably can. > > > > > > > > - Both crawler and mirror are 1.7GB machines dedicated to this > task. > > > > - I only need to crawl internal links (not external). > > > > - Eventually I will need to update the crawl but a monthly update > > > > will > > > > > > > > be sufficient. > > > > > > > > Any advice (and sample config files) would be much appreciated! > > > > > > > > Fred > > > > > > -- > > -- > Markus Jelsma - CTO - Openindex > http://www.linkedin.com/in/markus17 > 050-8536620 / 06-50258350 >
