Hi Tom, I got the following error "Client.InvalidParameterValue: The requested instance type does not agree with the architecture specified in the AMI manifest." During meantime, I built a private AMI from the 64 bit version of fedora with hadoop version 14.4.
What do I have to do to take advantages of the extra CPUS and the Ram (I only increased the mapred.child.java.opts)? My map/reduce code is comparable to wordcount (7413862 combine input pairs, 122475 reduce input pairs) and doesn't make a lot of calculations, but still takes over 30 seconds to complete, while the cpu is displayed to be idle about 99% of the time in top. (If I don't collect pairs in the map function, the job finished in 3 seconds (just to test how long it takes for my RecordReader to deliver the key/value pairs)). Also increasing the mappers dramatically reduces performances. (The above is measured with 2 mappers only on one x1.large instance). Any ideas on what causes the performance to be so low? Tom White wrote: > > Hi Thibaut, > > Do you know why the existing hadoop images don't work with larger > instance types? What's the error message you're getting? > > It should be relatively easy to change the launch-hadoop-cluster > script to specify the instance type. Also, there's some work to be > done to configure Hadoop on larger images to take advantage of the > extra CPUs and memory. > > BTW I just published a 0.15.1 AMI. > > Tom > > On 11/12/2007, Thibaut Britz <[EMAIL PROTECTED]> wrote: >> >> Hi, >> >> The current public images only work on the smaller instances. >> It would be very helpful (save me some time) if someone would be so kind >> create or publish their hadoop image. >> >> Thibaut >> >> -- >> View this message in context: >> http://www.nabble.com/public-Amazon-EC2-hadoop-images-for-larger-instances-%28m1.large-and-m1.xlarge%29-tp14276807p14276807.html >> Sent from the Hadoop Users mailing list archive at Nabble.com. >> >> > > > -- > Blog: http://problemsworthyofattack.blogspot.com/ > > -- View this message in context: http://www.nabble.com/public-Amazon-EC2-hadoop-images-for-larger-instances-%28m1.large-and-m1.xlarge%29-tp14276807p14296045.html Sent from the Hadoop Users mailing list archive at Nabble.com.
