> On 23 Jul 2015, at 10:47, Greg Anderson
> wrote:
>
> So when I go to ~/ephemeral-hdfs/bin/hadoop and check its version, it says
> Hadoop 2.0.0-cdh4.2.0. If I run pyspark and use the s3a address, things
> should work, right? What am I missing? And thanks so much for the help so
> far!
n
...@hortonworks.com]
Sent: Thursday, July 23, 2015 11:37 AM
To: Ewan Leith
Cc: Greg Anderson; user@spark.apache.org
Subject: Re: Help accessing protected S3
> On 23 Jul 2015, at 01:50, Ewan Leith wrote:
>
> I think the standard S3 driver used in Spark from the Hadoop project (S3n)
> On 23 Jul 2015, at 01:50, Ewan Leith wrote:
>
> I think the standard S3 driver used in Spark from the Hadoop project (S3n)
> doesn't support IAM role based authentication.
>
> However, S3a should support it. If you're running Hadoop 2.6 via the
> spark-ec2 scripts (I'm not sure what it laun
I think the standard S3 driver used in Spark from the Hadoop project (S3n)
doesn't support IAM role based authentication.
However, S3a should support it. If you're running Hadoop 2.6 via the spark-ec2
scripts (I'm not sure what it launches with by default) try accessing your
bucket via s3a:// U
