In the first few tests I created total of 24 files. In the test with the last graph 88 files before the break and 16 after. The files were
#!/bin/bash HDFS_URI=$1 INSERT_NO=$2 for i in $(seq 1 8) do path/to/hadoop/bin/hadoop fs -put bigData hdfs://$HDFS_URI/bigData$INSERT_NO-$i > put$i.log 2>&1 & done And I invoke it manually, increasing $2 each run. -----Original Message----- From: Eli Collins [mailto:e...@cloudera.com] Sent: Thursday, January 14, 2010 9:13 PM To: hdfs-user@hadoop.apache.org Subject: Re: Exponential performance decay when inserting large number of blocks On Thu, Jan 14, 2010 at 3:53 PM, <zlatin.balev...@barclayscapital.com> wrote: > And the results after several hour wait are the same. The cluster was > absolutely idle between the insertions. Attached is a graph. How many files are you creating? Can you post the script you're using to drive the test? Thanks, Eli _______________________________________________ This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. _______________________________________________
