If using RDD's, you can use saveAsHadoopFile or saveAsNewAPIHadoopFile
with the conf passed in which overrides the keys you need.
For example, you can do :
val saveConf = new Configuration(sc.hadoopConfiguration)
// configure saveConf with overridden s3 config
rdd.saveAsNewAPIHadoopFile(..., conf
On 12 Oct 2016, at 10:49, Aseem Bansal
mailto:asmbans...@gmail.com>> wrote:
Hi
I want to read CSV from one bucket, do some processing and write to a different
bucket. I know the way to set S3 credentials using
jssc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", YOUR_ACCESS_KEY)
jssc.hadoo
Hi
I want to read CSV from one bucket, do some processing and write to a
different bucket. I know the way to set S3 credentials using
jssc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", YOUR_ACCESS_KEY)
jssc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", YOUR_SECRET_KEY)
But the prob