Github user soulcutter commented on a diff in the pull request:
https://github.com/apache/spark/pull/11215#discussion_r54432870
--- Diff:
extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala
---
@@ -221,13 +237,51 @@ private[kinesis] object KinesisTestUtils {
case Failure(e) =>
throw new Exception(
s"""
- |Kinesis tests enabled using environment variable
$envVarNameForEnablingTests
- |but could not find AWS credentials. Please follow
instructions in AWS documentation
- |to set the credentials in your system such that the
DefaultAWSCredentialsProviderChain
- |can find the credentials.
+ |Kinesis tests that actually send data has been enabled by
setting the environment
+ |variable $envVarNameForEnablingTests to 1. This will create
Kinesis Streams and
+ |DynamoDB tables in AWS. Please be aware that this may incur
some AWS costs.
+ |By default, the tests use the endpoint URL
$defaultEndpointUrl to create Kinesis streams.
+ |To change this endpoint URL to a different region, you can
set the environment variable
+ |$endVarNameForEndpoint to the desired endpoint URL
+ |(e.g.
$endVarNameForEndpoint="https://kinesis.us-west-2.amazonaws.com";).
""".stripMargin)
}
}
+
+ def getAWSProfileCredentials(profile: String): AWSCredentials = {
+ Try { new ProfileCredentialsProvider(profile).getCredentials() } match
{
+ case Success(cred) => {
+ if (profile.equals(KinesisTestUtils.DYNAMODB_PROFILE)) {
+ KinesisTestUtils.cleanUpProfileDynamoTables = true;
+ }
+ cred
+ }
+ case Failure(e) =>
+ println(
+ s"""
+ |Could not load the AWS credentials for profile $profile.
+ |
+ |Credentialpool tests verify the ability to uses Kinesis
stream, Dynamo DB and CloudWatch Metrics in
+ |separate AWS accounts. This requires a separate set of AWS
credentials to be stored in system with the profile
+ |names "dynamoDB" and "cloudWatch" along with the default
credentials. The test assumes that the Kinesis stream
+ |in the In absence of this only the underlying API will be
tested.
--- End diff --
This sentence looks like it got mangled during editing somehow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
