[jira] [Commented] (FLINK-3379) Refactor TimestampExtractor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150325#comment-15150325
 ] 

ASF GitHub Bot commented on FLINK-3379:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1646


> Refactor TimestampExtractor
> ---
>
> Key: FLINK-3379
> URL: https://issues.apache.org/jira/browse/FLINK-3379
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 0.10.1
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Critical
> Fix For: 1.0.0
>
>
> Based on a lot of user feedback, the current {{TimestampExtractor}} seems 
> very confusing. It implements simultaneously two modes of generating 
> watermarks:
>   - Each record that passes through can decide to cause a watermark.
>   - The timestamp extractor can define a certain watermark timestamp which is 
> periodically picked up by the system and triggers a watermark (if larger than 
> the previous watermark).
> Figuring out how these modes interplay, and how to define the methods to only 
> use one mode has been quite an obstacle for several users. We should break 
> this class into two different classes, one per mode of generating watermarks, 
> to make it easier to understand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3379) Refactor TimestampExtractor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150183#comment-15150183
 ] 

ASF GitHub Bot commented on FLINK-3379:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1646#issuecomment-185116590
  
I found a minor test flakeyness. Fixing that and merging this...


> Refactor TimestampExtractor
> ---
>
> Key: FLINK-3379
> URL: https://issues.apache.org/jira/browse/FLINK-3379
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 0.10.1
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Critical
> Fix For: 1.0.0
>
>
> Based on a lot of user feedback, the current {{TimestampExtractor}} seems 
> very confusing. It implements simultaneously two modes of generating 
> watermarks:
>   - Each record that passes through can decide to cause a watermark.
>   - The timestamp extractor can define a certain watermark timestamp which is 
> periodically picked up by the system and triggers a watermark (if larger than 
> the previous watermark).
> Figuring out how these modes interplay, and how to define the methods to only 
> use one mode has been quite an obstacle for several users. We should break 
> this class into two different classes, one per mode of generating watermarks, 
> to make it easier to understand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3379) Refactor TimestampExtractor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150059#comment-15150059
 ] 

ASF GitHub Bot commented on FLINK-3379:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1646#issuecomment-185083361
  
+1 This should go into 1.0


> Refactor TimestampExtractor
> ---
>
> Key: FLINK-3379
> URL: https://issues.apache.org/jira/browse/FLINK-3379
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 0.10.1
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Critical
> Fix For: 1.0.0
>
>
> Based on a lot of user feedback, the current {{TimestampExtractor}} seems 
> very confusing. It implements simultaneously two modes of generating 
> watermarks:
>   - Each record that passes through can decide to cause a watermark.
>   - The timestamp extractor can define a certain watermark timestamp which is 
> periodically picked up by the system and triggers a watermark (if larger than 
> the previous watermark).
> Figuring out how these modes interplay, and how to define the methods to only 
> use one mode has been quite an obstacle for several users. We should break 
> this class into two different classes, one per mode of generating watermarks, 
> to make it easier to understand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3379) Refactor TimestampExtractor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149103#comment-15149103
 ] 

ASF GitHub Bot commented on FLINK-3379:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/1646

[FLINK-3379] [FLINK-3415] [streaming] Refactor TimestampExtractor

This pull request divides the TimestampExtractor into two separate classes
 - one class handled periodic watermarks
 - the other class handled watermarks triggered by individual elements

This also makes sure that any timestamp assigner / watermark generators 
cannot generate
negative watermarks.

This also adds tests for the operators that run the timestamp extractors.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink 
refactor_extractors

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1646.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1646


commit 1e3c4597dd1785430735950eef2b7b8c2154faaf
Author: Stephan Ewen 
Date:   2016-02-16T09:37:59Z

[FLINK-3379] [FLINK-3415] [streaming] Refactor TimestampExtractor into two 
separate classes

 - one class handled periodic watermarks
 - the other class handled watermarks triggered by elements

This also makes sure that any timestamp assigner / watermark generators 
cannot generate
negative watermarks




> Refactor TimestampExtractor
> ---
>
> Key: FLINK-3379
> URL: https://issues.apache.org/jira/browse/FLINK-3379
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 0.10.1
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Critical
> Fix For: 1.0.0
>
>
> Based on a lot of user feedback, the current {{TimestampExtractor}} seems 
> very confusing. It implements simultaneously two modes of generating 
> watermarks:
>   - Each record that passes through can decide to cause a watermark.
>   - The timestamp extractor can define a certain watermark timestamp which is 
> periodically picked up by the system and triggers a watermark (if larger than 
> the previous watermark).
> Figuring out how these modes interplay, and how to define the methods to only 
> use one mode has been quite an obstacle for several users. We should break 
> this class into two different classes, one per mode of generating watermarks, 
> to make it easier to understand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)