Re: ConnectedStream keyby issues

Hi, The State will never be automatically GC’ed. You have to do it in the onTimer() callback, as mentioned earlier. Best, Aljoscha > On 19. May 2017, at 10:39, gaurav wrote: > > Hello > > I am little confused on when the state will be gc. For example, > > Example 1:

Re: ConnectedStream keyby issues

Even if you increase the operator parallelism, you can still use the state operation. On Fri, May 19, 2017 at 7:47 PM Tarek khal wrote: > If I increase the parallelism operator, I risk losing shared state solution > or it has nothing to do. > And if it's going to

Re: ConnectedStream keyby issues

If I increase the parallelism operator, I risk losing shared state solution or it has nothing to do. And if it's going to be an advantage, is it limited to what? I am new with this framework I find difficulty in some notions. Best Regards, -- View this message in context:

Re: ConnectedStream keyby issues

Jason's solution is right, l'm just clarifying the mistake in the explanation. Tarek khal 于2017年5月19日周五 下午7:11写道： > Hello Renjie, > > Yes, the parallelism is 1. what should i do pls ? > > Regards, > > > > -- > View this message in context: >

Re: ConnectedStream keyby issues

Hello Renjie, Yes, the parallelism is 1. what should i do pls ? Regards, -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ConnectedStream-keyby-issues-tp12999p13226.html Sent from the Apache Flink User Mailing List archive. mailing list

Re: ConnectedStream keyby issues

@Jason I think there's a mistake in your explanation since each task in the task manager has its own copy of an operator instance, so the tuple may not be shared. State is a great solution but I think that's not the root cause. @Tarek What's the parallelism of your data stream? I think the reason

Re: ConnectedStream keyby issues

Hello I am little confused on when the state will be gc. For example, Example 1: Class abc extends RichProcessFunction,Tuple<>> { public void processElement(..) { if(timer never set) {

Re: ConnectedStream keyby issues

Yes, that looks right. > On 10. May 2017, at 14:56, yunfan123 wrote: > > In upstairs example, it seems I should clear the state in onTimer function in > order to free resource like follows: > public void onTimer(long l, OnTimerContext onTimerContext, >

Re: ConnectedStream keyby issues

In upstairs example, it seems I should clear the state in onTimer function in order to free resource like follows: public void onTimer(long l, OnTimerContext onTimerContext, Collector> collector) throws Exception { if (state.value() != null) {

Re: ConnectedStream keyby issues

private static class MergeFunction extends RichProcessFunction, Tuple2> { private ValueState> state; @Override public void open(Configuration parameters) throws Exception { state =

Re: ConnectedStream keyby issues

Hi, Yes, if you use ProcessFunction or CoProcessFunction you can (and should) set a timer to clean up state. Best, Aljoscha > On 7. May 2017, at 09:28, yunfan123 wrote: > > But what happened if some data can't be merged forever ? > The state will be saved forever?

Re: ConnectedStream keyby issues

But what happened if some data can't be merged forever ? The state will be saved forever? Can I set a timeout? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ConnectedStream-keyby-issues-tp12999p13037.html Sent from the Apache Flink User

Re: ConnectedStream keyby issues

Hi Jason, Thank you very much for your help, it solves my problem. Best regards, -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ConnectedStream-keyby-issues-tp12999p13003.html Sent from the Apache Flink User Mailing List archive. mailing

Re: ConnectedStream keyby issues

I think the issue is that t2 is not registered to keyed state, so it is being shared across all of the keys on that taskmanager. Take a look at this article: https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/state.html#using-managed-keyed-state Basically you need to change