BTW open() enables me only to initialize data to my objects. It still impossible to access to previous records’ data unless I use state’ tools.
That’s why I said that Flink’ state functionality is futile. Because when you use primitives or any serializable object then you don’t need to use open() and StateValue and so on. Indeed I then used only simple data in my program then everything worked fine and it been seem to me as if all the state’ tools are useless. I think this should be explained better in the documentation. Ziv. From: Tzu-Li (Gordon) Tai [via Apache Flink Mailing List archive.] [mailto:ml+s1008284n19650...@n3.nabble.com] Sent: 26 September 2017 14:40 To: ziv Subject: RE: is flink' states functionality futile? Hi Ziv, if I mark it as transient, then its value erased and I get null pointer exception. on the other hand, leave it without 'transient', it's a compilation error - implementation of function is not serializable. For unserializable class fields that you need to use for processing, you need to make it transient, and only initialize it in the open() method. To override the open() method you will need to extend the Rich variant of the operator you are implementing. For example, the RichFlatMapFunction is the Rich variant of the FlatMapFunction. Does that answer your question? Cheers, Gordon On 26 September 2017 at 11:58:43 AM, Meri Ziv ([hidden email]</user/SendEmail.jtp?type=node&node=19650&i=0>) wrote: hi Tzu-Li, sorry I have to reopen it but I'm still having troubles with using data from previous records. you said: “If that’s the case, and you don’t care about losing state on failures and don’t care about at-least-once / exactly-once, then yes you don’t have to use the managed state APIs in Flink. You can just have ordinary fields to achieve this since Flink’s streaming operators are basically long-running processes that continuously process events in the stream and manipulates its state.” look that if one of my fields is of unserializable type, then: if I mark it as transient, then its value erased and I get null pointer exception. on the other hand, leave it without 'transient', it's a compilation error - implementation of function is not serializable. how you explain that? Best, Ziv. From: Tzu-Li (Gordon) Tai [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=19650&i=1>] Sent: 28 July 2017 09:12 To: [hidden email]</user/SendEmail.jtp?type=node&node=19650&i=2>; Meri Ziv Subject: RE: is flink' states functionality futile? Here it is (sorry about that): [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/stream_checkpointing.html On 26 July 2017 at 4:15:54 PM, ziv ([hidden email]</user/SendEmail.jtp?type=node&node=19650&i=3><mailto:[hidden email]</user/SendEmail.jtp?type=node&node=19650&i=4>>) wrote: Hi, what is the reference you wanted to link in [1]? From: Tzu-Li (Gordon) Tai [via Apache Flink Mailing List archive.] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=19650&i=5>]<mailto:[mailto:[hidden email]</user/SendEmail.jtp?type=node&node=19650&i=6>]> Sent: 26 July 2017 05:44 To: Meri Ziv Subject: Re: is flink' states functionality futile? <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html> it's a program that sinks messages only after enough items accumulated in the buffe Yes, that’s another important aspect of how Flink’s checkpointing is used to achieve at-least-once / exactly-once delivery to external systems. For more details on how this works, I recommend taking at the docs here [1]. Now, assume I'm not bothered of recovering failures and only want the simplest way to implement a program that remembers data from the last run in the stream By “the last run in the stream”, you mean the history of the stream so far, correct? If that’s the case, and you don’t care about losing state on failures and don’t care about at-least-once / exactly-once, then yes you don’t have to use the managed state APIs in Flink. You can just have ordinary fields to achieve this since Flink’s streaming operators are basically long-running processes that continuously process events in the stream and manipulates its state. Cheers, Gordon On 23 July 2017 at 10:28:37 PM, ziv ([hidden email]</user/SendEmail.jtp?type=node&node=18936&i=0>) wrote: Ok, Let me see if I understand you correctly. You actually state that flink' states functionality is introduced only to handle recovering from failures. Let's take the example given in 1.3 documentary - https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html> it's a program that sinks messages only after enough items accumulated in the buffer. Now, assume I'm not bothered of recovering failures and only want the simplest way to implement a program that remembers data from the last run in the stream, then, according to you, I may not use none of the elements associated with flink' states - ListState snapshotState initializeState restoreState and the program still functions correctly? -- View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/is-flink-states-functionality-futile-tp18867p18879.html Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com. ________________________________ If you reply to this email, your message will be added to the discussion below: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/is-flink-states-functionality-futile-tp18867p18936.html To start a new topic under Apache Flink Mailing List archive., email [hidden email]</user/SendEmail.jtp?type=node&node=19650&i=7><mailto:[hidden email]</user/SendEmail.jtp?type=node&node=19650&i=8><mailto:[hidden email]</user/SendEmail.jtp?type=node&node=19650&i=9>%3cmailto:[hidden email]</user/SendEmail.jtp?type=node&node=19650&i=10>>> To unsubscribe from is flink' states functionality futile?, click here< NAML<http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> The information in this e-mail transmission contains proprietary and business sensitive information. Unauthorized interception of this e-mail may constitute a violation of law. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. You are also asked to contact the sender by reply email and immediately destroy all copies of the original message. -- View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/is-flink-states-functionality-futile-tp18867p18939.html Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com. The information in this e-mail transmission contains proprietary and business sensitive information. Unauthorized interception of this e-mail may constitute a violation of law. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. You are also asked to contact the sender by reply email and immediately destroy all copies of the original message. ________________________________ If you reply to this email, your message will be added to the discussion below: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/is-flink-states-functionality-futile-tp18867p19650.html To start a new topic under Apache Flink Mailing List archive., email ml+s1008284n1...@n3.nabble.com<mailto:ml+s1008284n1...@n3.nabble.com> To unsubscribe from is flink' states functionality futile?, click here<http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=18867&code=eml2bWVyaUBnbWFpbC5jb218MTg4Njd8MTk1NTY5MzMz>. NAML<http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
