Weiwei, Please see this documentation for configuring Spark and other apps on EMR 4.x: http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-configure-apps.html This documentation about what has changed between 3.x and 4.x should also be helpful: http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-release-differences.html
~ Jonathan On Fri, Feb 26, 2016 at 6:38 PM Weiwei Zhang <wzhan...@dons.usfca.edu> wrote: > Hi there, > > I am trying to configure memory for spark using AWS CLI. However, I got > the following message: > > *A client error (ValidationException) occurred when calling the RunJobFlow > operation: Cannot specify args for application 'Spark' when release label > is used.* > > In the aws 'create-cluster' command, I have '--release-label emr-4.0.0 > --applications Name=Hadoop > Name=Spark,Args=[-d,num-executors=4,spark.executor.memory=3000M,spark.driver.memory=4000M]' > and it seems like I cannot specify args when there is '--release-label'. > How do I get around this? > > I also tried using a JSON configuration file saved in a S3 bucket and add > "--configurations http://path/bucket/config.json"; to the command but it > gave me an 403 error (access denied). But when I did "aws s3 ls > (s3://bucket)" I could see that bucket and the config.json in the bucket. > > Please advise. Thank you very much. > > Best Regards, > Vivian >
