> On 30 Dec 2015, at 19:31, KOSTIANTYN Kudriavtsev > <kudryavtsev.konstan...@gmail.com> wrote: > > Hi Jerry, > > I want to run different jobs on different S3 buckets - different AWS creds - > on the same instances. Could you shed some light if it's possible to achieve > with hdfs-site? > > Thank you, > Konstantin Kudryavtsev >
The Hadoop s3a client doesn't have much (anything?) in the way for multiple logins. It'd be possible to do it by hand (create a Hadoop Configuration object, fill with the credential, and set "fs.s3a.impl.disable.cache"= true to make sure you weren't getting an existing version. I don't know how you'd hook that up to spark jobs. maybe try setting the credentials and that fs.s3a.impl.disable.cache flag in your spark context to see if together they get picked up --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
