GitHub user pluppens opened a pull request:
https://github.com/apache/flink/pull/5337
[FLINK-8484][flink-kinesis-connector] Ensure a Kinesis consumer snapshot
restoration is able to handle recently closed shards
FLINK-8484: ensure that a state change in the StreamShardMetadata other
than `StreamShardMetadata.shardId` or `StreamShardMetadata.streamName` does not
result in the shard not being able to be restored. This handles the corner case
where a shard might have been closed (ending sequence number set to not-null)
since the last savepoint or checkpoint when a job is restarted from a snapshot
state.
## Brief change log
- Created a new method to perform the sequence number lookup
- Ensure that a lookup for a given existing Kinesis shard does not rely on
equals(), but rather checks for equality on the stream name and shard id only
## Verifying this change
This change added tests and can be verified as follows:
- A new unit test was added in `FlinkKinesisConsumerTest` called
`testFindSequenceNumberToRestoreFrom()` which tests the lookup mechanism
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pluppens/flink master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5337.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5337
----
commit 5c756390002a2e1c00c7368bea3e1135b7722a20
Author: Philip Luppens <philip.luppens@...>
Date: 2018-01-23T08:00:23Z
FLINK-8484: ensure that a state change in the StreamShardMetadata other
than `StreamShardMetadata.shardId` or `StreamShardMetadata.streamName` does not
result in the shard not being able to be restored. This handles the corner case
where a shard might have been closed (ending sequence number set to not-null)
since the last savepoint or checkpoint when a job is restarted from a snapshot
state.
----
---
