Try this way:
val data = sc.textFile("s3n://ACCESS_KEY:SECRET_KEY@mybucket/temp/")
Thanks
Best Regards
On Mon, Jun 29, 2015 at 11:59 PM, didi <did...@gmail.com> wrote:
> Hi
>
> *Cant read text file from s3 to create RDD
> *
>
> after setting the configuration
> val hadoopConf=sparkContext.hadoopConfiguration;
> hadoopConf.set("fs.s3.impl",
> "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
> hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
> hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
>
> 1. running the following
> val hfile = sc.textFile("s3n://mybucket/temp/")
>
> I get the error
>
> Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: S3 HEAD request failed for '/temp' -
> ResponseCode=400, ResponseMessage=Bad Request
>
> 2. running the following
> val hfile = sc.textFile("s3n://mybucket/*.txt")
>
> I get the error
>
> Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Error
> Message: <?xml version="1.0"
> encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>*The
> authorization mechanism you have provided is not supported. Please use
>
> AWS4-HMAC-SHA256*.</Message><RequestId>C2174C316DEC91CB</RequestId><HostId>3oPZfZoPZUbvzXJdVaUGl9N0oI1buMx+A/wJiisx7uZ0bpnTkwsaT6i0fhYhjY97JDWBX1x/2Y8=</HostId></Error>
>
> I read it has to do something with the v4 signature??? isn't it supported
> by
> the sdk??
>
> 3. running the following
>
> val hfile = sc.textFile("s3n://mybucket")
>
> get the error
> Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: S3 HEAD request failed for
> '/user%2Fdidi' - ResponseCode=400, ResponseMessage=Bad Request
>
> what does the user has to do here??? i am using key & secret !!!
>
> How can i simply create RDD from text file on S3
>
> Thanks
>
> Didi
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/s3-bucket-access-read-file-tp23536.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
