Repository: flink Updated Branches: refs/heads/master a8e85a2d5 -> 00d1ad86a
[FLINK-4523] [kinesis] Add documentation for start position configuration This closes #2916. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/00d1ad86 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/00d1ad86 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/00d1ad86 Branch: refs/heads/master Commit: 00d1ad86a021911f25b0a0aa6e095267d51af1f4 Parents: 8d8a5ab Author: Tzu-Li (Gordon) Tai <[email protected]> Authored: Mon Jan 9 18:04:12 2017 +0800 Committer: Tzu-Li (Gordon) Tai <[email protected]> Committed: Tue Jan 24 14:20:07 2017 +0800 ---------------------------------------------------------------------- docs/dev/connectors/kinesis.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flink/blob/00d1ad86/docs/dev/connectors/kinesis.md ---------------------------------------------------------------------- diff --git a/docs/dev/connectors/kinesis.md b/docs/dev/connectors/kinesis.md index d95fe21..59f3d61 100644 --- a/docs/dev/connectors/kinesis.md +++ b/docs/dev/connectors/kinesis.md @@ -113,6 +113,18 @@ configured to enable checkpointing, so that the new shards due to resharding can Kinesis consumer after the job is restored. This is a temporary limitation that will be resolved in future versions. Please see [FLINK-4341](https://issues.apache.org/jira/browse/FLINK-4341) for more detail. +#### Configuring Starting Position + +The Flink Kinesis Consumer currently provides the following options to configure where to start reading Kinesis streams, simply by setting `ConsumerConfigConstants.STREAM_INITIAL_POSITION` to +one of the following values in the provided configuration properties (the naming of the options identically follows [the namings used by the AWS Kinesis Streams service](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html#API_GetShardIterator_RequestSyntax)): + +- `LATEST`: read all shards of all streams starting from the latest record. +- `TRIM_HORIZON`: read all shards of all streams starting from the earliest record possible (data may be trimmed by Kinesis depending on the retention settings). +- `AT_TIMESTAMP`: read all shards of all streams starting from a specified timestamp. The timestamp must also be specified in the configuration +properties by providing a value for `ConsumerConfigConstants.STREAM_INITIAL_TIMESTAMP`, either in the date pattern +`yyyy-MM-dd'T'HH:mm:ss.SSSXXX` (for example, `2016-04-04T19:58:46.480-00:00`), or a non-negative double value representing the number of seconds +that has elapsed since the Unix epoch (for example, `1459799926.480`). + #### Fault Tolerance for Exactly-Once User-Defined State Update Semantics With Flink's checkpointing enabled, the Flink Kinesis Consumer will consume records from shards in Kinesis streams and
