Re: Reading from and writing to different S3 buckets in spark

2016-10-12 Thread Mridul Muralidharan
If using RDD's, you can use saveAsHadoopFile or saveAsNewAPIHadoopFile with the conf passed in which overrides the keys you need. For example, you can do : val saveConf = new Configuration(sc.hadoopConfiguration) // configure saveConf with overridden s3 config rdd.saveAsNewAPIHadoopFile(..., conf

Re: Reading from and writing to different S3 buckets in spark

2016-10-12 Thread Steve Loughran
On 12 Oct 2016, at 10:49, Aseem Bansal > wrote: Hi I want to read CSV from one bucket, do some processing and write to a different bucket. I know the way to set S3 credentials using jssc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId",

Reading from and writing to different S3 buckets in spark

2016-10-12 Thread Aseem Bansal
Hi I want to read CSV from one bucket, do some processing and write to a different bucket. I know the way to set S3 credentials using jssc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", YOUR_ACCESS_KEY) jssc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", YOUR_SECRET_KEY) But the