I am running Spark on Amazon EMR and writing data to an S3 bucket. However, the data is read from an S3 bucket in a separate AWS account. Setting the fs.s3a.access.key and fs.s3a.secret.key values is sufficient to get access to the other account (using the s3a protocol), however I then won't have access to the S3 bucket in the EMR cluster's AWS account.
Is there any way for Spark to access S3 buckets in multiple accounts? If not, is there any best practice for how to work around this? -- Daniel Siegmann Senior Software Engineer *SecurityScorecard Inc.* 214 W 29th Street, 5th Floor New York, NY 10001
