Zlotin, nevermind, just noticed that you're measuring the dfs insert time 1k block are way too small anyway
On Wed, Jan 13, 2010 at 6:08 PM, alex kamil <alex.ka...@gmail.com> wrote: > Zlatin, > i dont know what is the nature of the job that you are running but it looks > like you are hitting an io bottleneck, try to multiple the size of input > data and test with 3-5GB at least, by changing the block size in this case > you can increase the number of maps and scale your app. (Round the block > size to a multiple of 512) > i attached some results from my experiments with KNN sclaing on 4 nodes > cluster, it didnt scale linearly but it wasn't too bad, it sometimes depends > on the algorithm, with other algos, like PCA/SVD the scaling was linear > also note that sometimes the bottleneck in in Reduce step, i added a > Combiner between mapper and reducer in my code and it affected scalability > significantly. > > let me know if you have any questions > Alex Kamil > ak2...@columbia.edu > > > On Wed, Jan 13, 2010 at 5:35 PM, Dhruba Borthakur <dhr...@gmail.com>wrote: > >> Another thing to observe is the rate of IO on the datanodes. Maybe u can >> do a sar/iostat on the datanodes and see if the ddatanode devices show an >> increase in activity while inserting the last lot of blocks. One posiblity >> is that the OS cache on the datanodes cached most of the data from the first >> few runs, but when more and more data started arriving on the datanode it >> triggered more flushing of OS buffers. (on the datanode). >> >> thanks, >> dhruba >> >> >> >> On Wed, Jan 13, 2010 at 2:18 PM, Todd Lipcon <t...@cloudera.com> wrote: >> >>> Hey Zlatin, >>> >>> Thanks for the explanation and the additional data. I'm a bit busy today >>> but will try to go through the data and reproduce the results later this >>> week. >>> >>> -Todd >>> >>> >>> On Wed, Jan 13, 2010 at 2:07 PM, <zlatin.balev...@barclayscapital.com>wrote: >>> >>>> Todd, >>>> >>>> I used a shell script that launched 8 instances of the bin/hadoop fs >>>> -put utility. After all 8 processes were done and I verified though the >>>> web >>>> ui that the files were inserted, I re-launched the script manually again. >>>> That is why you'll notice that in the metrics there are two short periods >>>> without any activity (I edited those out from the graph). There were >>>> occasional NotReplicatedYet exceptions in the logs of those processes, but >>>> they were occurring at constant rate. >>>> >>>> I did not run a profiler, but that will eventually be the next step. >>>> I'm attaching the metrics from the namenode and one of the datanodes from >>>> the experiment with 4 datanodes. They were recorded every 10 seconds. >>>> Heap >>>> size for all processes is 2GB, and while there was occasional CPU usage on >>>> the Namenode it was never 100%. (and there are plenty of cores). >>>> >>>> Ultimately the block size will be much larger than the default as the >>>> total data will be in the 2^(well over 50) range. With this test I am >>>> trying to determine if there are any bottlenecks at the NameNode >>>> component. >>>> >>>> Best Regards, >>>> Zlatin Balevsky >>>> >>>> ------------------------------ >>>> *From:* Todd Lipcon [mailto:t...@cloudera.com] >>>> *Sent:* Wednesday, January 13, 2010 4:34 PM >>>> *To:* hdfs-user@hadoop.apache.org >>>> *Subject:* Re: Exponential performance decay when inserting large >>>> number of blocks >>>> >>>> Also, if you have the program you used to do the insertions, and could >>>> attach it, I'd be interested in trying to replicate this on a test cluster. >>>> If you can't redistribute it, I can start from scratch, but would be easier >>>> to run yours. >>>> >>>> Thanks >>>> -Todd >>>> >>>> On Wed, Jan 13, 2010 at 1:31 PM, Todd Lipcon <t...@cloudera.com> wrote: >>>> >>>>> Hi Zlatin, >>>>> >>>>> This is a very interesting test you've run, and certainly not expected >>>>> results. I know of many clusters happily chugging along with millions of >>>>> blocks, so problems at 400K are very strange. By any chance were you able >>>>> to >>>>> collect profiling information from the NameNode while running this test? >>>>> >>>>> That said, I hope you've set the block size to 1KB for the purpose of >>>>> this test and not because you expect to run that in production. >>>>> Recommended >>>>> block sizes are at least 64MB and often 128MB or 256MB for larger >>>>> clusters. >>>>> >>>>> Thanks >>>>> -Todd >>>>> >>>>> On Wed, Jan 13, 2010 at 1:21 PM, >>>>> <zlatin.balev...@barclayscapital.com>wrote: >>>>> >>>>>> Greetings, >>>>>> >>>>>> I am testing how HDFS scales with very large number of blocks. I did >>>>>> the following setup: >>>>>> >>>>>> Set the default blocks size to 1KB >>>>>> Started 8 insert processes, each inserting a 16MB file >>>>>> Repeated the insert 3 times, keeping the already inserted files in >>>>>> HDFS >>>>>> Repeated the entire experiment on one cluster with 4 and another with >>>>>> 11 >>>>>> identical datanodes (allocated through HOD) >>>>>> >>>>>> Results: >>>>>> The first 128MB (2^18 blocks) insert finished in 5 minutes. The >>>>>> second >>>>>> in 12 minutes. The third didn't finish within 1 hour. The 11-node >>>>>> cluster was marginally faster. >>>>>> >>>>>> Throughout this I was storing all available metrics. There were no >>>>>> signs of insufficient memory on any of the nodes; CPU usage and >>>>>> garbage >>>>>> collections were constant throughout. If anyone is interested I can >>>>>> provide the recorded metrics. I've attached a chart that looks >>>>>> clearly >>>>>> logarithmic. >>>>>> >>>>>> Can anyone please point to what could be the bottleneck here? I'm >>>>>> evaluating HDFS for usage scenarios requiring 2^(a lot more than 18) >>>>>> blocks. >>>>>> >>>>>> Bes <<insertion_rate_4_and_11_datanodes.JPG>> t Regards, >>>>>> Zlatin Balevsky >>>>>> >>>>>> _______________________________________________ >>>>>> >>>>>> This e-mail may contain information that is confidential, privileged >>>>>> or otherwise protected from disclosure. If you are not an intended >>>>>> recipient >>>>>> of this e-mail, do not duplicate or redistribute it by any means. Please >>>>>> delete it and any attachments and notify the sender that you have >>>>>> received >>>>>> it in error. Unless specifically indicated, this e-mail is not an offer >>>>>> to >>>>>> buy or sell or a solicitation to buy or sell any securities, investment >>>>>> products or other financial product or service, an official confirmation >>>>>> of >>>>>> any transaction, or an official statement of Barclays. Any views or >>>>>> opinions >>>>>> presented are solely those of the author and do not necessarily represent >>>>>> those of Barclays. This e-mail is subject to terms available at the >>>>>> following link: www.barcap.com/emaildisclaimer. By messaging with >>>>>> Barclays you consent to the foregoing. Barclays Capital is the >>>>>> investment >>>>>> banking division of Barclays Bank PLC, a company registered in England >>>>>> (number 1026167) with its registered office at 1 Churchill Place, London, >>>>>> E14 5HP. This email may relate to or be sent from other members of the >>>>>> Barclays Group. >>>>>> _______________________________________________ >>>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> >>>> >>>> >>>> This e-mail may contain information that is confidential, privileged or >>>> otherwise protected from disclosure. If you are not an intended recipient >>>> of >>>> this e-mail, do not duplicate or redistribute it by any means. Please >>>> delete >>>> it and any attachments and notify the sender that you have received it in >>>> error. Unless specifically indicated, this e-mail is not an offer to buy or >>>> sell or a solicitation to buy or sell any securities, investment products >>>> or >>>> other financial product or service, an official confirmation of any >>>> transaction, or an official statement of Barclays. Any views or opinions >>>> presented are solely those of the author and do not necessarily represent >>>> those of Barclays. This e-mail is subject to terms available at the >>>> following link: www.barcap.com/emaildisclaimer. By messaging with >>>> Barclays you consent to the foregoing. Barclays Capital is the >>>> investment banking division of Barclays Bank PLC, a company registered in >>>> England (number 1026167) with its registered office at 1 Churchill Place, >>>> London, E14 5HP. This email may relate to or be sent from other >>>> members of the Barclays Group.** >>>> >>>> _______________________________________________ >>>> >>> >>> >> >> >> -- >> Connect to me at http://www.facebook.com/dhruba >> > >
