Ondrej Kokes created SPARK-32035:
------------------------------------
Summary: Inconsistent AWS environment variables in documentation
Key: SPARK-32035
URL: https://issues.apache.org/jira/browse/SPARK-32035
Project: Spark
Issue Type: Bug
Components: Documentation
Affects Versions: 3.0.0, 2.4.6
Reporter: Ondrej Kokes
Looking at the actual Scala code, the environment variables used to log into
AWS are:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN
These are the same that AWS uses in their libraries.
However, looking through the Spark documentation and comments, I see that these
are not denoted correctly across the board:
docs/cloud-integration.md
106:1. `spark-submit` reads the `AWS_ACCESS_KEY`, `AWS_SECRET_KEY` *<-- both
different*
107:and `AWS_SESSION_TOKEN` environment variables and sets the associated
authentication options
docs/streaming-kinesis-integration.md
232:- Set up the environment variables `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_KEY` with your AWS credentials. *<-- secret key different*
external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
34: $ export AWS_ACCESS_KEY_ID=<your-access-key>
35: $ export AWS_SECRET_KEY=<your-secret-key> *<-- different*
48: Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY *<-- secret
key different*
core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
438: val keyId = System.getenv("AWS_ACCESS_KEY_ID")
439: val accessKey = System.getenv("AWS_SECRET_ACCESS_KEY")
448: val sessionToken = System.getenv("AWS_SESSION_TOKEN")
external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
53: * $ export AWS_ACCESS_KEY_ID=<your-access-key>
54: * $ export AWS_SECRET_KEY=<your-secret-key> *<-- different*
65: * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY *<-- secret
key different*
external/kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java
59: * $ export AWS_ACCESS_KEY_ID=[your-access-key]
60: * $ export AWS_SECRET_KEY=<your-secret-key> *<-- different*
71: * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY *<-- secret
key different*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
