[
https://issues.apache.org/jira/browse/BEAM-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166076#comment-16166076
]
ASF GitHub Bot commented on BEAM-2467:
--------------------------------------
GitHub user pawel-kaczmarczyk opened a pull request:
https://github.com/apache/beam/pull/3851
[BEAM-2467] Kinesis source watermark based on approximateArrivalTimestamp
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [X] Make sure there is a [JIRA
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the
change (usually before you start working on it). Trivial changes like typos do
not require a JIRA issue. Your pull request should address just this issue,
without pulling in other changes.
- [X] Each commit in the pull request should have a meaningful subject
line and body.
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue.
- [X] Write a pull request description that is detailed enough to
understand what the pull request does, how, and why.
- [X] Run `mvn clean verify` to make sure basic checks pass. A more
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
---
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ocadotechnology/incubator-beam
kinesis_watermark
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3851.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3851
----
commit 7757ca61fb4b392469ebfd0d378cd7a61f46ae48
Author: Pawel Kaczmarczyk <[email protected]>
Date: 2017-09-14T10:52:16Z
[BEAM-2467] Kinesis source watermark based on approximateArrivalTimestamp
----
> KinesisIO watermark based on approximateArrivalTimestamp
> --------------------------------------------------------
>
> Key: BEAM-2467
> URL: https://issues.apache.org/jira/browse/BEAM-2467
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-extensions
> Reporter: Paweł Kaczmarczyk
> Assignee: Paweł Kaczmarczyk
>
> In Kinesis we can start reading the stream at some point in the past during
> the retention period (up to 7 days). With current approach for setting
> record's timestamp and watermark (both are always set to current time, i.e.
> Instant.now()), we can't observe the actual position in the stream.
> So the idea is to change this behaviour and set the record timestamp based on
> the
> [ApproximateArrivalTimestamp|http://docs.aws.amazon.com/kinesis/latest/APIReference/API_Record.html#Streams-Type-Record-ApproximateArrivalTimestamp].
> Watermark will be set accordingly to the last read record's timestamp.
> ApproximateArrivalTimestamp is still some approximation and may result in
> having records with out-of-order timestamp's which in turn may result in some
> events marked as late. This however should not be a frequent issue and even
> if it happens it should be a matter of milliseconds or seconds so can be
> handled even with a tiny allowedLateness setting
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
