My 2c: if it is not possible to move the i/o operations listFiles() and length() outside the lock on FSVolumeSet, maybe set a flag that a block report is in progress so that the rest of the datanode doesn't just hang. Thanks, Zlatin
________________________________ From: Dhruba Borthakur [mailto:dhr...@gmail.com] Sent: Thursday, January 21, 2010 3:38 PM To: hdfs-user@hadoop.apache.org Subject: Re: Exponential performance decay - mystery solved Some of this delay in generating block reports might be mitigated via http://issues.apache.org/jira/browse/HDFS-854 thanks, dhruba On Thu, Jan 21, 2010 at 11:41 AM, <zlatin.balev...@barclayscapital.com> wrote: Alright, the problem was caused by me setting the frequency of a block report to 30 seconds. The idea behind that was to create more load on the Namenode, but I didn't notice that those block reports were taking increasing amounts of time to generate. During that time, a lock was held which I'm guessing didn't allow the reporting datanode to perform its functions. On my hardware, with 100,000 blocks the report takes over 7 seconds. So every datanode was unavailable for 7 out of every 30 seconds. Changing the interval to a more reasonable value restored the insertion speed to linear. Apologies for creating this confusion, nevertheless it was a useful thing to learn. Regards, Zlatin -----Original Message----- From: Eli Collins [mailto:e...@cloudera.com] Sent: Thursday, January 21, 2010 2:02 PM To: hdfs-user@hadoop.apache.org Subject: Re: Exponential performance decay - possible lead > > The messages are of the following: > > 2010-01-18 14:51:25,694 WARN org.apache.hadoop.hdfs.StateChange: > BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request > received for blk_-5804440919363539694_1026 on ip.removed:port.removed > size 1024 This is odd, you should't be getting this warning, I don't see it when running your benchmark on my cluster. Are there other relevant/warnings errors in the NN or DN logs? Thanks, Eli _______________________________________________ This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. _______________________________________________ -- Connect to me at http://www.facebook.com/dhruba _______________________________________________ This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. _______________________________________________
