[jira] [Updated] (FLINK-19807) Add documentation for Flink's "pre-flight phase" and "cluster phase"
[ https://issues.apache.org/jira/browse/FLINK-19807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manas Kale updated FLINK-19807: --- Description: A result of [this|https://lists.apache.org/thread.html/r496e1c312d4675afd32388f3c1f993e2f6d1c369c98580fd7c436457%40%3Cuser.flink.apache.org%3E] thread. Basically, care has to be taken when dealing with static variables when running in a minicluster in your IDE v/s running on a cluster. As such, Flink's "pre-flight phase" and "cluster phase" should be explained on the website along with a warning about this non-obvious behaviour. was: A result of [this|https://lists.apache.org/thread.html/r496e1c312d4675afd32388f3c1f993e2f6d1c369c98580fd7c436457%40%3Cuser.flink.apache.org%3E] thread. Basically, care has to be taken when dealing with static variables when running in a minicluster in your IDE v/s running on a cluster. As such, Flink's "pre-flight phase" and "cluster phase" should be explained on the website along with a warning about this behaviour. > Add documentation for Flink's "pre-flight phase" and "cluster phase" > > > Key: FLINK-19807 > URL: https://issues.apache.org/jira/browse/FLINK-19807 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Manas Kale >Priority: Major > > A result of > [this|https://lists.apache.org/thread.html/r496e1c312d4675afd32388f3c1f993e2f6d1c369c98580fd7c436457%40%3Cuser.flink.apache.org%3E] > thread. > Basically, care has to be taken when dealing with static variables when > running in a minicluster in your IDE v/s running on a cluster. > As such, Flink's "pre-flight phase" and "cluster phase" should be explained > on the website along with a warning about this non-obvious behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19807) Add documentation for Flink's "pre-flight phase" and "cluster phase"
[ https://issues.apache.org/jira/browse/FLINK-19807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manas Kale updated FLINK-19807: --- Description: A result of [this|https://lists.apache.org/thread.html/r496e1c312d4675afd32388f3c1f993e2f6d1c369c98580fd7c436457%40%3Cuser.flink.apache.org%3E] thread. Basically, care has to be taken when dealing with static variables when running in a minicluster in your IDE v/s running on a cluster. As such, Flink's "pre-flight phase" and "cluster phase" should be explained on the website along with a warning about this behaviour. was: A result of [this|https://lists.apache.org/thread.html/r496e1c312d4675afd32388f3c1f993e2f6d1c369c98580fd7c436457%40%3Cuser.flink.apache.org%3E] thread. Basically, care has to be taken when dealing with static variables - the behaviour might be different when running in a minicluster in your IDE v/s running on a cluster. As such, Flink's "pre-flight phase" and "cluster phase" should be explained on the website along with a warning about this behaviour. > Add documentation for Flink's "pre-flight phase" and "cluster phase" > > > Key: FLINK-19807 > URL: https://issues.apache.org/jira/browse/FLINK-19807 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Manas Kale >Priority: Major > > A result of > [this|https://lists.apache.org/thread.html/r496e1c312d4675afd32388f3c1f993e2f6d1c369c98580fd7c436457%40%3Cuser.flink.apache.org%3E] > thread. > Basically, care has to be taken when dealing with static variables when > running in a minicluster in your IDE v/s running on a cluster. > As such, Flink's "pre-flight phase" and "cluster phase" should be explained > on the website along with a warning about this behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19807) Add documentation for Flink's "pre-flight phase" and "cluster phase"
[ https://issues.apache.org/jira/browse/FLINK-19807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manas Kale updated FLINK-19807: --- Description: A result of [this|https://lists.apache.org/thread.html/r496e1c312d4675afd32388f3c1f993e2f6d1c369c98580fd7c436457%40%3Cuser.flink.apache.org%3E] thread. Basically, care has to be taken when dealing with static variables - the behaviour might be different when running in a minicluster in your IDE v/s running on a cluster. As such, Flink's "pre-flight phase" and "cluster phase" should be explained on the website along with a warning about this behaviour. was: This is a result of [this|https://lists.apache.org/thread.html/r496e1c312d4675afd32388f3c1f993e2f6d1c369c98580fd7c436457%40%3Cuser.flink.apache.org%3E] thread. Basically, care has to be taken when dealing with static variables - the behaviour might be different when running in a minicluster in your IDE v/s running on a cluster. As such, Flink's "pre-flight phase" and "cluster phase" should be explained on the website along with a warning about this behaviour. > Add documentation for Flink's "pre-flight phase" and "cluster phase" > > > Key: FLINK-19807 > URL: https://issues.apache.org/jira/browse/FLINK-19807 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Manas Kale >Priority: Major > > A result of > [this|https://lists.apache.org/thread.html/r496e1c312d4675afd32388f3c1f993e2f6d1c369c98580fd7c436457%40%3Cuser.flink.apache.org%3E] > thread. > Basically, care has to be taken when dealing with static variables - the > behaviour might be different when running in a minicluster in your IDE v/s > running on a cluster. > As such, Flink's "pre-flight phase" and "cluster phase" should be explained > on the website along with a warning about this behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19807) Add documentation for Flink's "pre-flight phase" and "cluster phase"
Manas Kale created FLINK-19807: -- Summary: Add documentation for Flink's "pre-flight phase" and "cluster phase" Key: FLINK-19807 URL: https://issues.apache.org/jira/browse/FLINK-19807 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Manas Kale This is a result of [this|https://lists.apache.org/thread.html/r496e1c312d4675afd32388f3c1f993e2f6d1c369c98580fd7c436457%40%3Cuser.flink.apache.org%3E] thread. Basically, care has to be taken when dealing with static variables - the behaviour might be different when running in a minicluster in your IDE v/s running on a cluster. As such, Flink's "pre-flight phase" and "cluster phase" should be explained on the website along with a warning about this behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16039) [Discuss] Add API method to get last element in session window
[ https://issues.apache.org/jira/browse/FLINK-16039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067332#comment-17067332 ] Manas Kale edited comment on FLINK-16039 at 3/26/20, 3:49 AM: -- Hi all, I've started a discussion on the mailing archive [here][[https://lists.apache.org/thread.html/rf44c033e20d165105a3adf1d36c1e3857231e7266aa844d4fc4b6e1f%40%3Cdev.flink.apache.org%3E]]. was (Author: manas96): Hi all, I've started a discussion on the mailing archive [here|[https://lists.apache.org/thread.html/rf44c033e20d165105a3adf1d36c1e3857231e7266aa844d4fc4b6e1f%40%3Cdev.flink.apache.org%3E]]. > [Discuss] Add API method to get last element in session window > -- > > Key: FLINK-16039 > URL: https://issues.apache.org/jira/browse/FLINK-16039 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.10.0 >Reporter: Manas Kale >Priority: Minor > > Consider the events : > [1, event], [2, event] > where first element is event timestamp in seconds and second element is event > code/name. > Also consider that an Event time session window with inactivityGap = 2 > seconds is acting on above stream. > When the first event arrives, a session window should be created that is > [1,1]. > When the second event arrives, a new session window should be created that is > [2,2]. Since this falls within firstWindowTimestamp+inactivityGap, it should > be merged into session window [1,2] and [2,2] should be deleted. > This is my understanding of how session windows are created. *Please correct > me if wrong.* > However, Flink does not follow such a definition of windows semantically. If > I call the getEnd() method of the TimeWindow() class, I get back _timestamp > + inactivityGap_. > For the above example, after processing the first element, I would get 1 + 2 > = 3 seconds as the window "end". > The actual window end should be the timestamp 1, which is the last event in > the session window. > A solution would be to change the "end" definition of all windows, but I > suppose this would be breaking and would need some debate. > Therefore, I propose an intermediate solution : add a new API method that > keeps track of the last element added in the session window. > If there is agreement on this, I would like to start drafting a change > document and implement this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16039) [Discuss] Add API method to get last element in session window
[ https://issues.apache.org/jira/browse/FLINK-16039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067332#comment-17067332 ] Manas Kale edited comment on FLINK-16039 at 3/26/20, 3:49 AM: -- Hi all, I've started a discussion on the mailing archive [here|[https://lists.apache.org/thread.html/rf44c033e20d165105a3adf1d36c1e3857231e7266aa844d4fc4b6e1f%40%3Cdev.flink.apache.org%3E]] was (Author: manas96): Hi all, I've started a discussion on the mailing archive [here][[https://lists.apache.org/thread.html/rf44c033e20d165105a3adf1d36c1e3857231e7266aa844d4fc4b6e1f%40%3Cdev.flink.apache.org%3E]]. > [Discuss] Add API method to get last element in session window > -- > > Key: FLINK-16039 > URL: https://issues.apache.org/jira/browse/FLINK-16039 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.10.0 >Reporter: Manas Kale >Priority: Minor > > Consider the events : > [1, event], [2, event] > where first element is event timestamp in seconds and second element is event > code/name. > Also consider that an Event time session window with inactivityGap = 2 > seconds is acting on above stream. > When the first event arrives, a session window should be created that is > [1,1]. > When the second event arrives, a new session window should be created that is > [2,2]. Since this falls within firstWindowTimestamp+inactivityGap, it should > be merged into session window [1,2] and [2,2] should be deleted. > This is my understanding of how session windows are created. *Please correct > me if wrong.* > However, Flink does not follow such a definition of windows semantically. If > I call the getEnd() method of the TimeWindow() class, I get back _timestamp > + inactivityGap_. > For the above example, after processing the first element, I would get 1 + 2 > = 3 seconds as the window "end". > The actual window end should be the timestamp 1, which is the last event in > the session window. > A solution would be to change the "end" definition of all windows, but I > suppose this would be breaking and would need some debate. > Therefore, I propose an intermediate solution : add a new API method that > keeps track of the last element added in the session window. > If there is agreement on this, I would like to start drafting a change > document and implement this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16039) [Discuss] Add API method to get last element in session window
[ https://issues.apache.org/jira/browse/FLINK-16039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067332#comment-17067332 ] Manas Kale commented on FLINK-16039: Hi all, I've started a discussion on the mailing archive [here|[https://lists.apache.org/thread.html/rf44c033e20d165105a3adf1d36c1e3857231e7266aa844d4fc4b6e1f%40%3Cdev.flink.apache.org%3E]]. > [Discuss] Add API method to get last element in session window > -- > > Key: FLINK-16039 > URL: https://issues.apache.org/jira/browse/FLINK-16039 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.10.0 >Reporter: Manas Kale >Priority: Minor > > Consider the events : > [1, event], [2, event] > where first element is event timestamp in seconds and second element is event > code/name. > Also consider that an Event time session window with inactivityGap = 2 > seconds is acting on above stream. > When the first event arrives, a session window should be created that is > [1,1]. > When the second event arrives, a new session window should be created that is > [2,2]. Since this falls within firstWindowTimestamp+inactivityGap, it should > be merged into session window [1,2] and [2,2] should be deleted. > This is my understanding of how session windows are created. *Please correct > me if wrong.* > However, Flink does not follow such a definition of windows semantically. If > I call the getEnd() method of the TimeWindow() class, I get back _timestamp > + inactivityGap_. > For the above example, after processing the first element, I would get 1 + 2 > = 3 seconds as the window "end". > The actual window end should be the timestamp 1, which is the last event in > the session window. > A solution would be to change the "end" definition of all windows, but I > suppose this would be breaking and would need some debate. > Therefore, I propose an intermediate solution : add a new API method that > keeps track of the last element added in the session window. > If there is agreement on this, I would like to start drafting a change > document and implement this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6726) Allow setting Timers in ProcessWindowFunction
[
https://issues.apache.org/jira/browse/FLINK-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038084#comment-17038084
]
Manas Kale commented on FLINK-6726:
---
Hi, is there any progress on this issue? I have a similar use case.
> Allow setting Timers in ProcessWindowFunction
> -
>
> Key: FLINK-6726
> URL: https://issues.apache.org/jira/browse/FLINK-6726
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Vishnu Viswanath
>Assignee: Vishnu Viswanath
>Priority: Minor
>
> Allow registration of timers in ProcessWindowFunction.
> {code}
> public abstract void registerEventTimeTimer(long time);
> public abstract void registerProcessingTimeTimer(long time);
> {code}
> This is based on one of the use case that I have, where I need to register an
> EventTimeTimer that will clean the elements in the Window State based on some
> condition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16039) [Discussion] Add API method to get last element in session window
[ https://issues.apache.org/jira/browse/FLINK-16039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manas Kale updated FLINK-16039: --- Summary: [Discussion] Add API method to get last element in session window (was: Add API method to get last element in session window) > [Discussion] Add API method to get last element in session window > - > > Key: FLINK-16039 > URL: https://issues.apache.org/jira/browse/FLINK-16039 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.10.0 >Reporter: Manas Kale >Priority: Minor > > Consider the events : > [1, event], [2, event] > where first element is event timestamp in seconds and second element is event > code/name. > Also consider that an Event time session window with inactivityGap = 2 > seconds is acting on above stream. > When the first event arrives, a session window should be created that is > [1,1]. > When the second event arrives, a new session window should be created that is > [2,2]. Since this falls within firstWindowTimestamp+inactivityGap, it should > be merged into session window [1,2] and [2,2] should be deleted. > This is my understanding of how session windows are created. *Please correct > me if wrong.* > However, Flink does not follow such a definition of windows semantically. If > I call the getEnd() method of the TimeWindow() class, I get back _timestamp > + inactivityGap_. > For the above example, after processing the first element, I would get 1 + 2 > = 3 seconds as the window "end". > The actual window end should be the timestamp 1, which is the last event in > the session window. > A solution would be to change the "end" definition of all windows, but I > suppose this would be breaking and would need some debate. > Therefore, I propose an intermediate solution : add a new API method that > keeps track of the last element added in the session window. > If there is agreement on this, I would like to start drafting a change > document and implement this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16039) [Discuss] Add API method to get last element in session window
[ https://issues.apache.org/jira/browse/FLINK-16039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manas Kale updated FLINK-16039: --- Summary: [Discuss] Add API method to get last element in session window (was: [Discussion] Add API method to get last element in session window) > [Discuss] Add API method to get last element in session window > -- > > Key: FLINK-16039 > URL: https://issues.apache.org/jira/browse/FLINK-16039 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.10.0 >Reporter: Manas Kale >Priority: Minor > > Consider the events : > [1, event], [2, event] > where first element is event timestamp in seconds and second element is event > code/name. > Also consider that an Event time session window with inactivityGap = 2 > seconds is acting on above stream. > When the first event arrives, a session window should be created that is > [1,1]. > When the second event arrives, a new session window should be created that is > [2,2]. Since this falls within firstWindowTimestamp+inactivityGap, it should > be merged into session window [1,2] and [2,2] should be deleted. > This is my understanding of how session windows are created. *Please correct > me if wrong.* > However, Flink does not follow such a definition of windows semantically. If > I call the getEnd() method of the TimeWindow() class, I get back _timestamp > + inactivityGap_. > For the above example, after processing the first element, I would get 1 + 2 > = 3 seconds as the window "end". > The actual window end should be the timestamp 1, which is the last event in > the session window. > A solution would be to change the "end" definition of all windows, but I > suppose this would be breaking and would need some debate. > Therefore, I propose an intermediate solution : add a new API method that > keeps track of the last element added in the session window. > If there is agreement on this, I would like to start drafting a change > document and implement this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16039) Add API method to get last element in session window
Manas Kale created FLINK-16039: -- Summary: Add API method to get last element in session window Key: FLINK-16039 URL: https://issues.apache.org/jira/browse/FLINK-16039 Project: Flink Issue Type: Improvement Components: API / DataStream Affects Versions: 1.10.0 Reporter: Manas Kale Consider the events : [1, event], [2, event] where first element is event timestamp in seconds and second element is event code/name. Also consider that an Event time session window with inactivityGap = 2 seconds is acting on above stream. When the first event arrives, a session window should be created that is [1,1]. When the second event arrives, a new session window should be created that is [2,2]. Since this falls within firstWindowTimestamp+inactivityGap, it should be merged into session window [1,2] and [2,2] should be deleted. This is my understanding of how session windows are created. *Please correct me if wrong.* However, Flink does not follow such a definition of windows semantically. If I call the getEnd() method of the TimeWindow() class, I get back _timestamp + inactivityGap_. For the above example, after processing the first element, I would get 1 + 2 = 3 seconds as the window "end". The actual window end should be the timestamp 1, which is the last event in the session window. A solution would be to change the "end" definition of all windows, but I suppose this would be breaking and would need some debate. Therefore, I propose an intermediate solution : add a new API method that keeps track of the last element added in the session window. If there is agreement on this, I would like to start drafting a change document and implement this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
