GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/2822
[FLINK-5075] [kinesis] Make Kinesis consumer fail-proof to incorrect
Kinesalite API behaviour
A user reported that when tested against Kinesalite (a widely used mock
Kinesis implementation), the connector was incorrectly determining already seen
shards as newly discovered.
The problem was that the connector was using the API
`describeStream(streamName, exclusiveStartIShardId)` to fetch shards of a
stream after the provided `exclusiveStartShardId` (given as the last id of the
latest shard we've already discovered), and Kinesalite behaves differently for
this from the official Kinesis API.
For example, if the current complete shard list is [shard-0, shard-1,
shard-2, shard-3] for "test-stream",
then `describeStream("test-stream", "shard-1")` should return: [shard-2,
shard-3].
Kinesalite, however, incorrectly returns the whole list.
I've manually tested this change against Kinesalite, and shard discovery is
working normally again.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-5075
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2822.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2822
----
commit cbef7eabeae645a1cd3533d3274cdb7491b1a779
Author: Tzu-Li (Gordon) Tai <[email protected]>
Date: 2016-11-17T06:24:24Z
[FLINK-5075] [kinesis] Make connector fail-proof to incorrect Kinesalite
API behaviour
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
