Hi
*Cant read text file from s3 to create RDD
*
after setting the configuration
val hadoopConf=sparkContext.hadoopConfiguration;
hadoopConf.set("fs.s3.impl",
"org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
1. running the following
val hfile = sc.textFile("s3n://mybucket/temp/")
I get the error
Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception:
org.jets3t.service.S3ServiceException: S3 HEAD request failed for '/temp' -
ResponseCode=400, ResponseMessage=Bad Request
2. running the following
val hfile = sc.textFile("s3n://mybucket/*.txt")
I get the error
Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception:
org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Error
Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>*The
authorization mechanism you have provided is not supported. Please use
AWS4-HMAC-SHA256*.</Message><RequestId>C2174C316DEC91CB</RequestId><HostId>3oPZfZoPZUbvzXJdVaUGl9N0oI1buMx+A/wJiisx7uZ0bpnTkwsaT6i0fhYhjY97JDWBX1x/2Y8=</HostId></Error>
I read it has to do something with the v4 signature??? isn't it supported by
the sdk??
3. running the following
val hfile = sc.textFile("s3n://mybucket")
get the error
Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception:
org.jets3t.service.S3ServiceException: S3 HEAD request failed for
'/user%2Fdidi' - ResponseCode=400, ResponseMessage=Bad Request
what does the user has to do here??? i am using key & secret !!!
How can i simply create RDD from text file on S3
Thanks
Didi
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/s3-bucket-access-read-file-tp23536.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
