Hey Zlatin, That makes sense. No apologies necessary, was a very useful exercise.
Thanks, Eli On Thu, Jan 21, 2010 at 11:41 AM, <zlatin.balev...@barclayscapital.com> wrote: > > > Alright, the problem was caused by me setting the frequency of a block > report to 30 seconds. The idea behind that was to create more load on > the Namenode, but I didn't notice that those block reports were taking > increasing amounts of time to generate. During that time, a lock was > held which I'm guessing didn't allow the reporting datanode to perform > its functions. > > On my hardware, with 100,000 blocks the report takes over 7 seconds. So > every datanode was unavailable for 7 out of every 30 seconds. Changing > the interval to a more reasonable value restored the insertion speed to > linear. > > Apologies for creating this confusion, nevertheless it was a useful > thing to learn. > > Regards, > Zlatin > > -----Original Message----- > From: Eli Collins [mailto:e...@cloudera.com] > Sent: Thursday, January 21, 2010 2:02 PM > To: hdfs-user@hadoop.apache.org > Subject: Re: Exponential performance decay - possible lead > >> >> The messages are of the following: >> >> 2010-01-18 14:51:25,694 WARN org.apache.hadoop.hdfs.StateChange: >> BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request >> received for blk_-5804440919363539694_1026 on ip.removed:port.removed >> size 1024 > > This is odd, you should't be getting this warning, I don't see it when > running your benchmark on my cluster. Are there other relevant/warnings > errors in the NN or DN logs? > > Thanks, > Eli > _______________________________________________ > > This e-mail may contain information that is confidential, privileged or > otherwise protected from disclosure. If you are not an intended recipient of > this e-mail, do not duplicate or redistribute it by any means. Please delete > it and any attachments and notify the sender that you have received it in > error. Unless specifically indicated, this e-mail is not an offer to buy or > sell or a solicitation to buy or sell any securities, investment products or > other financial product or service, an official confirmation of any > transaction, or an official statement of Barclays. Any views or opinions > presented are solely those of the author and do not necessarily represent > those of Barclays. This e-mail is subject to terms available at the following > link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent > to the foregoing. Barclays Capital is the investment banking division of > Barclays Bank PLC, a company registered in England (number 1026167) with its > registered office at 1 Churchill Place, London, E14 5HP. This email may > relate to or be sent from other members of the Barclays Group. > _______________________________________________ >
