[
https://issues.apache.org/jira/browse/FLINK-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247492#comment-15247492
]
ASF GitHub Bot commented on FLINK-3229:
---------------------------------------
GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/1911
[FLINK-3229] Flink streaming consumer for AWS Kinesis
I've been using this consumer for a while in off-production environments.
I understand we should have good test coverage for each PR, but since
Kinesis is a hosted service, reliable integration tests are hard to pull off.
To speed up merging Kinesis connector for the next release, I'm submitting the
consumer now for some early reviews.
On the other hand, since @rmetzger is submitting a separate PR for Kinesis
producer, I'd like to postpone writing more tests for the consumer, as well as
corresponding modification to the document until both the consumer and producer
are in place.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-3229
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1911.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1911
----
commit 0e7c4eccca132e0fcf35262b36229e681c77530e
Author: Gordon Tai <gor...@vm5.com>
Date: 2016-04-16T03:33:17Z
[FLINK-3229] Initial working version for FlinkKinesisConsumer.
commit 6d46a6d09c346e490053a2f6319616a5178dab4e
Author: Gordon Tai <gor...@vm5.com>
Date: 2016-04-17T09:34:55Z
[FLINK-3229] Change access level of `assignShards` and
`validatePropertiesConfig` to protected for testing purposes
commit c92b491fce8b3c35b409bc6f308d25ce52835027
Author: Gordon Tai <gor...@vm5.com>
Date: 2016-04-17T11:12:00Z
[FLINK-3229] Fix coding stype violations regarding leading spaces in
indentations
commit bc9f771c0f36618ce07772cfdefe7d87a35800fb
Author: Gordon Tai <gor...@vm5.com>
Date: 2016-04-19T09:56:57Z
[FLINK-3229] Change scope of flink-streaming-java module to provided
commit fc454efaac5dc7969ad4834b892f26799cfe5a33
Author: Gordon Tai <gor...@vm5.com>
Date: 2016-04-19T09:58:46Z
[FLINK-3229] Basic unit test for stable shard-to-consumer assignment
----
> Kinesis streaming consumer with integration of Flink's checkpointing mechanics
> ------------------------------------------------------------------------------
>
> Key: FLINK-3229
> URL: https://issues.apache.org/jira/browse/FLINK-3229
> Project: Flink
> Issue Type: Sub-task
> Components: Streaming Connectors
> Affects Versions: 1.0.0
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Tzu-Li (Gordon) Tai
>
> Opening a sub-task to implement data source consumer for Kinesis streaming
> connector (https://issues.apache.org/jira/browser/FLINK-3211).
> An example of the planned user API for Flink Kinesis Consumer:
> {code}
> Properties kinesisConfig = new Properties();
> config.put(KinesisConfigConstants.CONFIG_AWS_REGION, "us-east-1");
> config.put(KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_TYPE,
> "BASIC");
> config.put(
> KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID,
> "aws_access_key_id_here");
> config.put(
> KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY,
> "aws_secret_key_here");
> config.put(KinesisConfigConstants.CONFIG_STREAM_INIT_POSITION_TYPE,
> "LATEST"); // or TRIM_HORIZON
> DataStream<T> kinesisRecords = env.addSource(new FlinkKinesisConsumer<>(
> "kinesis_stream_name",
> new SimpleStringSchema(),
> kinesisConfig));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
