GitHub user kaklakariada opened a pull request:
https://github.com/apache/spark/pull/11215
[SPARK-10969] [Streaming] [Kinesis] Allow specifying separate credentials
for Kinesis and DynamoDB
`KinesisUtils.createStream()` allows specifying only one set of AWS
credentials that will be used by Amazon Kinesis Client Library (KCL) for
accessing Kinesis, DynamoDB and CloudWatch.
#### Motivation
In a scenario where one needs to read from a Kinesis Stream owned by a
different AWS account the user usually has minimal rights (i.e. only read from
the stream). In this case creating the DynamoDB table in KCL will fail. Instead
the DynamoDB table should be created in your own AWS account.
#### Proposal
This PR adds a new serializable class `AWSCredentialPool` containing
credentials for Kinesis, DynamoDB and CloudWatch that is passed as a parameter
to `KinesisUtils.createStream()`. These credentials are then passed to the
constructor of `KinesisClientLibConfiguration`.
@tdas @cfregly please review if this PR is OK.
This contribution is @ajayrammohan and my original work and we license the
work to the project under the project's open source license.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ajayrammohan/spark SPARK-10969
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11215.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11215
----
commit 517dfb4f00405b967c244b2d75756fc33f02021f
Author: Ajay Rammohan <[email protected]>
Date: 2016-02-10T13:06:30Z
SPARK-10969- Added new credentials Pool class to handle multiple AWS
credentials
commit d50ce058f24e938eb2d4c733dffc517bbb410efa
Author: Ajay Rammohan <[email protected]>
Date: 2016-02-10T13:06:58Z
SPARK-10969- Changed interface to handle the credential Pool as parameter
commit 027452253e472180b2cd6d01bc641b6e15e33d5e
Author: kaklakariada <[email protected]>
Date: 2016-02-10T14:01:54Z
SPARK-10969 fix scala compilation error
commit f3093ebcaed4919803f6cbb1dbfb6c58d1310ec0
Author: kaklakariada <[email protected]>
Date: 2016-02-10T15:26:31Z
SPARK-10969 pass AWSCredentialsPool to KinesisReceiver and pass AWS
credentials to KinesisClientLibConfiguration
commit 0e9db8c398d8f6a60a98d226139704a835c16326
Author: Ajay Rammohan <[email protected]>
Date: 2016-02-11T07:53:52Z
SPARK-10969- adapted javadoc
commit 44cdc8e6d349402755f7e5bbf41a8ffb77702218
Author: Ajay Rammohan <[email protected]>
Date: 2016-02-12T07:46:31Z
SPARK-10969- added new test case to test new api
commit 552831412ff10268aeccd6ede37b08c4d5ebfa07
Author: Ajay Rammohan <[email protected]>
Date: 2016-02-12T07:47:45Z
SPARK-10969- added new methods to get profile based credentials and cleanup
tables created in separate accounts
commit 892f9c99a86fd00bdbd17e2de47370f765db0ab9
Author: Ajay Rammohan <[email protected]>
Date: 2016-02-15T07:29:13Z
SPARK-10969 - Adapted DynamoDB cleanup. Now checks for tables in multiple
accounts.
commit d7c29f9cd4654986d720236155960ec7497aa1ef
Author: kaklakariada <[email protected]>
Date: 2016-02-15T14:25:34Z
SPARK-10969 - Allow passing a credential pool from Java, add Java unit tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
