How are you launching the cluster, and how are you submitting the job to it? Can you list any Spark configuration parameters you provide?
On Mon, Oct 20, 2014 at 12:53 PM, Daniel Mahler <dmah...@gmail.com> wrote: > > I am launching EC2 clusters using the spark-ec2 scripts. > My understanding is that this configures spark to use the available > resources. > I can see that spark will use the available memory on larger istance types. > However I have never seen spark running at more than 400% (using 100% on 4 > cores) > on machines with many more cores. > Am I misunderstanding the docs? Is it just that high end ec2 instances get > I/O starved when running spark? It would be strange if that consistently > produced a 400% hard limit though. > > thanks > Daniel >
