Re: Accessing s3a files from Spark

Hi, I am sorry, I do read this https://wiki.apache.org/hadoop/AmazonS3 which mentions about s3:// being deprecated. From what I read using s3a is the preferred way to go. Ofcourse, I have been using it for writing data from SPARK but not for reading yet. Let me try that and come back. Regards,

Re: Accessing s3a files from Spark

How do I use it? I'm accessing s3a from Spark's textFile API. On Tue, May 31, 2016 at 7:16 AM, Deepak Sharma wrote: > Hi Mayuresh > Instead of s3a , have you tried the https:// uri for the same s3 bucket? > > HTH > Deepak > > On Tue, May 31, 2016 at 4:41 PM, Mayuresh

Re: Accessing s3a files from Spark

On Tue, May 31, 2016 at 7:05 AM, Gourav Sengupta wrote: > Hi, > > And on another note, is it required to use s3a? Why not use s3:// only? I > prefer to use s3a:// only while writing files to S3 from EMR > ​Does Spark support s3://? I am using s3a over s3n because I

Re: Accessing s3a files from Spark

Hi Mayuresh Instead of s3a , have you tried the https:// uri for the same s3 bucket? HTH Deepak On Tue, May 31, 2016 at 4:41 PM, Mayuresh Kunjir wrote: > > > On Tue, May 31, 2016 at 5:29 AM, Steve Loughran > wrote: > >> which s3 endpoint? >> >> >

Re: Accessing s3a files from Spark

On Tue, May 31, 2016 at 5:29 AM, Steve Loughran wrote: > which s3 endpoint? > > ​I have tried both s3.amazonaws.com and s3-external-1.amazonaws.com​. > > > On 29 May 2016, at 22:55, Mayuresh Kunjir wrote: > > I'm running into permission issues

Re: Accessing s3a files from Spark

Hi, And on another note, is it required to use s3a? Why not use s3:// only? I prefer to use s3a:// only while writing files to S3 from EMR. Regards, Gourav Sengupta On Tue, May 31, 2016 at 12:04 PM, Gourav Sengupta wrote: > Hi, > > Is your spark cluster running in

Re: Accessing s3a files from Spark

Hi, Is your spark cluster running in EMR or via self created SPARK cluster using EC2 or from a local cluster behind firewall? What is the SPARK version you are using? Regards, Gourav Sengupta On Sun, May 29, 2016 at 10:55 PM, Mayuresh Kunjir wrote: > I'm running into

Re: Accessing s3a files from Spark

which s3 endpoint? On 29 May 2016, at 22:55, Mayuresh Kunjir > wrote: I'm running into permission issues while accessing data in S3 bucket stored using s3a file system from a local Spark cluster. Has anyone found success with this? My setup

Re: Accessing s3a files from Spark

On Sun, May 29, 2016 at 7:49 PM, Ted Yu wrote: > Have you seen this thread ? > > > http://search-hadoop.com/m/q3RTthWU8o1MbFC2=Re+Forbidded+Error+Code+403 > > ​ Thanks for the pointer. I have followed the thread, got no success though. I am trying out the Spark branch

Re: Accessing s3a files from Spark

Have you seen this thread ? http://search-hadoop.com/m/q3RTthWU8o1MbFC2=Re+Forbidded+Error+Code+403 On Sun, May 29, 2016 at 2:55 PM, Mayuresh Kunjir wrote: > I'm running into permission issues while accessing data in S3 bucket > stored using s3a file system from a local

Accessing s3a files from Spark

I'm running into permission issues while accessing data in S3 bucket stored using s3a file system from a local Spark cluster. Has anyone found success with this? My setup is: - Spark 1.6.1 compiled against Hadoop 2.7.2 - aws-java-sdk-1.7.4.jar and hadoop-aws-2.7.2.jar in the classpath - Spark's