is the mapreduce working yet? I would also like to test it. -J ----- Original Message ----- From: "Piotr Kosiorowski" <[EMAIL PROTECTED]> To: <nutch-dev@lucene.apache.org> Sent: Friday, August 05, 2005 8:06 AM Subject: Re: Fw: Re: near-term plan
> I am not sure what you exactly did in this test but I understand you > were using jar file prepared by me (it was nutch from trunk + ndfs > patches). As these patches were applied by Andrzej some time ago - we > can assume you were using NDFS code from trunk. > Because a lot of work went into mapreduce branch it woul dbe good to > test it with mapreduce branch code. > Regards > Piotr > > On 8/5/05, webmaster <[EMAIL PROTECTED]> wrote: > > > > ---------- Forwarded Message ----------- > > From: "webmaster" <[EMAIL PROTECTED]> > > To: nutch-dev@lucene.apache.org > > Sent: Thu, 4 Aug 2005 19:42:53 -0500 > > Subject: Re: near-term plan > > > > I was using a nightly build that Pitor had given me the nutch-nightly.jar > > (actually it was nutch-dev0.7.jar or something of that nature) I tested it on > > the windows platform, I had 5 machines running it, 2 at 100 mbit both quad p3 > > xeon, 1 pentium 4 3ghz hyperthreading, 1 amd athlon xp 2600+ and 1 Athlon 64 > > 3500+. all have 1gb or more of ram. now I have my big server and if you have > > worked on ndfs since the begining of july I'll test it again, my big server's > > HD array is very fast 200+mbytes a sec, so it will be able to fully saturate > > gigabit better. anyway the p4 and the 2 amd machines are hooked into the > > switch at gigabit and the 2 xeons are hooked into my other switch at 100mbit, > > but it has a gigabit uplink to my gigabit switch, so both xeons would > > constantly be saturated at 11mbytes a sec. while the p4 was able to reach > > higher speeds of 50-60mbytes a sec with its internal raid 0 array (dual 120gb > > drives) my main pc (athlon 64 3500+) was the namenode and a datanode and also > > the ndfs client, I could not get nutch to work properly with ndfs, it was > > setup correctly, it "kinda" worked but would crash out the namenode when I > > was trying to fetch segments in the ndfs filesystem or index them, or do much > > of anything. so I copied all my segment directories, indexes, > > content.wtahever it was 1.8gb and some dvd images onto ndfs. my primary > > machine and nutch runs off 10000rpm disks raid 0 (2x36gb raptors) they can > > output about 120mbytes a sec sustained so here is what I found out ( in > > windows) if I dont start a datanode on the namenode with the conf pointing to > > 127.0.0.1 instead of its outside ip the namenode will not copy data to the > > other machines, instead if I'm running datanode on the namenode data will > > replicate from the datanode to the other 3 datanodes, I tried this a hundred > > ways to try and make it work with an independant namenode without luck. but > > the way I saw data go across my network was I would put data into ndfs the > > namenode would request a datanode and find the internal datanode, copy data > > to it only then after that the datanode would still be coping data from my > > other hd's into chunks on the raid array, while copying it would replicate to > > the p4 via gigabit at 50-60mbytes a sec, then it would replicate from the p4 > > to the xeons kinda like alternating them as I only had replication at default > > 2 and i had about 100gbytes to copy in so the copy would finish onto the > > internal raid array fairly quickly then it finished replication to the p4 and > > the xeons got a little bit of data, but not near as much as the p4, my guess > > is it only needs 2 copies and the first copy was datanode on the internal > > machine, the second was the p4 datanode. the xeons only had a smaller > > connection so they didnt recieve as many chunks as fast as the p4 could, and > > the p4 had enough space for all the data so it worked out, I should of put > > replication to 4. the amd athlon xp 1900+ was running linux suse 9.3 and it > > would crash the namenode on windows if I connected it as a datanode. so that > > one didnt get tested, but I was able to put out 50-60 mbytes a sec to 1 > > machine, but it would not replicate data to multiple machines at the same > > time it seemed. I would of thought it would of output to the xeons at the > > same time as the p4, give the xeons 20% of the data and the p4 80% or > > something of that nature, but it could be that they just arent fast enough to > > request data before the p4 was recieving its 32mb chunks every 1/2 second? > > The good news cpu usage was only at 50% on my amd 3500+ that was while it was > > copying data to the internal datanode from the ndfs client from another > > internal HD running the namenode and running the datanode internally. does it > > now work with a separate namenode? I'm getting ready to run nutch in linux > > full time, if I can ever get the damn driver for my highpoint 2220 raid card > > to work with suse, any suse, the drivers dont work with dual core cpu's or > > something??? they are working on it, now I'm stuck with fedora 4 untill they > > fix it. so its not ready for testing yet. I'll let you know when I can test > > it in a full linux environment. > > wow that was a long one!!! > > -Jay > > ------- End of Forwarded Message ------- > > > > > > -- > > Pound Web Hosting www.poundwebhosting.com > > (607)-435-3048 > > > >
