Ashic Mahtab created SPARK-16363:
------------------------------------
Summary: Spark-submit doesn't work with IAM Roles
Key: SPARK-16363
URL: https://issues.apache.org/jira/browse/SPARK-16363
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 1.6.2
Environment: Spark Stand-Alone with EC2 instances configured with IAM
Roles.
Reporter: Ashic Mahtab
When running Spark Stand-alone in EC2 boxes,
spark-submit --master spark://master-ip:7077 --class Foo
--deploy-mode cluster --verbose s3://bucket/dir/foo/jar
fails to find the jar even if AWS IAM roles are configured to allow the EC2
boxes (that are running Spark master, and workers) access to the file in S3.
The exception is provided below. It's asking us to set keys, etc. when the
boxes are configured via IAM roles.
16/07/04 11:44:09 ERROR ClientEndpoint: Exception from cluster was:
java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
must be specified as the username or password (respectively) of a s3 URL, or by
setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties
(respectively).
java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
must be specified as the username or password (respectively) of a s3 URL, or by
setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties
(respectively).
at
org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:66)
at
org.apache.hadoop.fs.s3.Jets3tFileSystemStore.initialize(Jets3tFileSystemStore.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at com.sun.proxy.$Proxy5.initialize(Unknown Source)
at org.apache.hadoop.fs.s3.S3FileSystem.initialize(S3FileSystem.java:77)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1686)
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:598)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:395)
at
org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:150)
at
org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:79)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
