We have a cluster and the workers have different memory. The problem we faced is that we first use spark 0.8 ec2 script to create 1 master and some slaves using m1.large instances. Each worker has 7.5G memory and spark use about 6G memory. Everything looks good. However, when we manually added some m3.2xlarge instances to our cluster, we found that new instances use also 6G memory while they have 30G memory. The memory usage information is from master:8080 such as "27.7 GB (6.0 B Used)"
A workaround is to start 4 workers on m3.2xlarge so that each worker uses 6G ram. We want to know is there a better way to set executor memory for each worker? -- Regards, Jyun-Fan Tsai
