GitHub user addisonj opened a pull request:
https://github.com/apache/spark/pull/11245
[SPARK-13367] [Streaming] Refactor KinesusUtils to specify more KCL options
This patch refactors the KinesisUtils and adds new APIs such that it is
easy to pass more configuration options into the KCL, such as using a
different DynamoDBClient with a different endpoint.
The core of this refactoring/change is to introduce the `KinesisConfig`
class which is intended to encapsulate all configuration concerns.
Currently, it doesn't do much more than allow for setting the DynamoDB
endpoint, but it is a good place for where other options could be
contained without changing underlying APIs. It could also be inherited
to override behavior for more options without requiring any API changes
(such as overriding `buildKCLConfig` ti allow for more configuration
changes)
This also introduce a new external API for creating a KinesisRDD, which
takes a `KinesisConfig` object which may be a more manageable API going
forward.
Docs are still lacking for the new class as well as some basic unit
specs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/addisonj/spark kinesis_config
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11245.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11245
----
commit 32e66fb2b6c8828bdc8b4f3f810df456d09cf580
Author: Addison Higham <[email protected]>
Date: 2016-02-16T23:45:13Z
[SPARK-13367] Refactor KinesusUtils to specify more KCL options
This patch refactors the KinesisUtils and adds new APIs such that it is
easy to pass more configuration options into the KCL, such as using a
different DynamoDBClient with a different endpoint.
The core of this refactoring/change is to introduce the `KinesisConfig`
class which is intended to encapsulate all configuration concerns.
Currently, it doesn't do much more than allow for setting the DynamoDB
endpoint, but it is a good place for where other options could be
contained without changing underlying APIs. It could also be inherited
to override behavior for more options without requiring any API changes
(such as overriding `buildKCLConfig` ti allow for more configuration
changes)
This also introduce a new external API for creating a KinesisRDD, which
takes a `KinesisConfig` object which may be a more manageable API going
forward.
Docs are still lacking for the new class as well as some basic unit
specs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
