Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/5439#discussion_r28390431
--- Diff:
extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala
---
@@ -82,15 +82,19 @@ private[kinesis] class KinesisReceiver(
var workerId: String = null
/*
- * This impl uses the DefaultAWSCredentialsProviderChain and searches
for credentials
- * in the following order of precedence:
+ * This impl uses the DefaultAWSCredentialsProviderChain unless it's
provided by constructor
+ * and searches for credentials in the following order of precedence:
* Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
* Java System Properties - aws.accessKeyId and aws.secretKey
* Credential profiles file at the default location (~/.aws/credentials)
shared by all
* AWS SDKs and the AWS CLI
* Instance profile credentials delivered through the Amazon EC2
metadata service
*/
- var credentialsProvider: AWSCredentialsProvider = null
+ var credentialsProvider: AWSCredentialsProvider = (credentials map { cr
=> new AWSCredentialsProvider {
--- End diff --
i seem to remember having problems when constructing the
AWSCredentialsProvider outside of the onStart() method due to this hierarchy
(including DefaultAWSCredentialsProviderChain) not being Serializable.
i assume this works, but wondering what may have changed.
also, is there a simpler way to do this in Scala? i feel like this could
be simplified, but maybe i'm missing something. seems like a lot of syntax for
something so simple.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
