Hi, keep in mind that you're going to have a bad time if your secret key contains a "/" This is due to old and stupid hadoop bug: https://issues.apache.org/jira/browse/HADOOP-3733
Best way is to regenerate the key so it does not include a "/" /Raf Akhil Das wrote: > Try the following: > > 1. Set the access key and secret key in the sparkContext: > > sparkContext.set(" > > AWS_ACCESS_KEY_ID",yourAccessKey) > > sparkContext.set(" > > AWS_SECRET_ACCESS_KEY",yourSecretKey) > > > 2. Set the access key and secret key in the environment before > starting your application: > > > > export > > AWS_ACCESS_KEY_ID=<your access> > > export > > AWS_SECRET_ACCESS_KEY=<your secret> > > > 3. Set the access key and secret key inside the hadoop configurations > > val hadoopConf=sparkContext.hadoopConfiguration; > > hadoopConf.set("fs.s3.impl", > "org.apache.hadoop.fs.s3native.NativeS3FileSystem") > > hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey) > > hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey) > > > 4. You can also try: > > val lines = > > s > > parkContext.textFile("s3n://yourAccessKey:yourSecretKey@<yourBucket>/path/") > > > Thanks > Best Regards > > On Mon, Oct 13, 2014 at 11:33 PM, Ranga <sra...@gmail.com > <mailto:sra...@gmail.com>> wrote: > > Hi > > I am trying to access files/buckets in S3 and encountering a > permissions issue. The buckets are configured to authenticate > using an IAMRole provider. > I have set the KeyId and Secret using environment variables > (AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still > unable to access the S3 buckets. > > Before setting the access key and secret the error was: > "java.lang.IllegalArgumentException: AWS Access Key ID and Secret > Access Key must be specified as the username or password > (respectively) of a s3n URL, or by setting the > fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties > (respectively)." > > After setting the access key and secret, the error is: "The AWS > Access Key Id you provided does not exist in our records." > > The id/secret being set are the right values. This makes me > believe that something else ("token", etc.) needs to be set as well. > Any help is appreciated. > > > - Ranga > >