Dear Andrei and everyone I think I have found the cause of my problem. I used an image of Ubuntu to build my hadoop cluster and the image is a 'ebs' image. The 'ebs' type caused this problem. If the image is changed to a 'instance store' type one, everything works fine. The 'Configured Capicity' became 3.91TB. Maybe the prepare_all_disk.sh script doesn't work well in an ebs instance.
Best regards, Jingchen LIU 2012/10/11 刘景琛 <[email protected]> > Dear Andrei > > Thank you for reply! I still have a question: > I launched a cluster of 13 c1.medium instances, and access the Namenode > web UI (http://NAMENODE_PUBLIC_DNS:50070/dfshealth.jsp). > It shows that the 'Configured Capicity' is 82.49GB. > I logged into the master instance, run the command 'df -h', it shows the > size of /dev/sda1 is 7.9 GB and the size of /dev/sdb is 335GB. > If the /data0 is linked to /mnt, shouldn't the 'Configured Capicity' be > about 335*12 GB? > > Thank you very much! > > Best regards > Jingchen Liu > > 2012/10/10 Andrei Savu <[email protected]> > >> /data0 is just a symlink to /mnt >> >> You should have enough space. See the following file for details: >> >> https://github.com/apache/whirr/blob/trunk/services/hadoop/src/main/resources/functions/prepare_all_disks.sh >> >> -- Andrei Savu / axemblr.com >> >> >> On Wed, Oct 10, 2012 at 2:15 PM, 刘景琛 <[email protected]> wrote: >> >>> Dear Whirr Developers and Users >>> >>> I'm using Whirr to run a hadoop cluster on AWS EC2, and I met a problem. >>> After launching a hadoop cluster, I logged into the master instance >>> using putty, get to the path /usr/local/hadoop/conf, >>> and checked the hdfs-site.xml. I found that the 'dfs.data.dir' was set >>> to '/data0/hadoop/hdfs/data'. >>> >>> The /data0 folder is in the instance's /dev/sda1 device, which is very >>> small. If the hadoop store the HDFS data here, the space will not be enough >>> for my big data. >>> The EC2 instances have another device /dev/sdb, which is much bigger and >>> has been mounted to /mnt. >>> So I think the 'dfs.data.dir' should be set to a folder under /mnt (and >>> also some other properties such as 'hadoop.tmp.dir', 'mapred.local.dir' >>> etc.) >>> Could anyone tell me how to do this using Whirr? >>> >>> Thank you very much! >>> >>> Best regards, >>> >>> Jingchen LIU >>> >> >> >
