[jira] [Commented] (FLINK-8522) DefaultOperatorStateBackend writes data in checkpoint that is never read.
[ https://issues.apache.org/jira/browse/FLINK-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345395#comment-16345395 ] Kostas Kloudas commented on FLINK-8522: --- There is a commit for this here: [https://github.com/apache/flink/pull/5230] It is the first commit of that PR. > DefaultOperatorStateBackend writes data in checkpoint that is never read. > - > > Key: FLINK-8522 > URL: https://issues.apache.org/jira/browse/FLINK-8522 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Affects Versions: 1.4.0 >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas >Priority: Major > Fix For: 1.4.1 > > > In the DefaultOperatorStateBackend at line 283 we write in the checkpoint an > int declaring the number of the operator states that we include in the > checkpoint. > This number is never read when restoring and this can lead to confusion and > problems with backwards compatibility and/or extension of the types of state > we support (e.g. broadcast state). > There are two easy solutions, either remove the line and do not write the > size, or make sure that we also read this number when restoring and simply > ignore it. > I would go for the first one, i.e. remove the line. What do you think > [~richtesn] and [~tzulitai] ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8522) DefaultOperatorStateBackend writes data in checkpoint that is never read.
[ https://issues.apache.org/jira/browse/FLINK-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344764#comment-16344764 ] Kostas Kloudas commented on FLINK-8522: --- @[~tzulitai] : I think that if the person that extends the code in future versions by adding data after the MetaInfo, also implements the migration policy correctly (i.e. bump the VERSION), then we will have no problem, as when he restores from the savepoint, the correct `restore()` method will be called, and it will not try to read the newly added data. Given this, I will go on and simply remove the line. > DefaultOperatorStateBackend writes data in checkpoint that is never read. > - > > Key: FLINK-8522 > URL: https://issues.apache.org/jira/browse/FLINK-8522 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Affects Versions: 1.4.0 >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas >Priority: Major > Fix For: 1.4.1 > > > In the DefaultOperatorStateBackend at line 283 we write in the checkpoint an > int declaring the number of the operator states that we include in the > checkpoint. > This number is never read when restoring and this can lead to confusion and > problems with backwards compatibility and/or extension of the types of state > we support (e.g. broadcast state). > There are two easy solutions, either remove the line and do not write the > size, or make sure that we also read this number when restoring and simply > ignore it. > I would go for the first one, i.e. remove the line. What do you think > [~richtesn] and [~tzulitai] ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8522) DefaultOperatorStateBackend writes data in checkpoint that is never read.
[ https://issues.apache.org/jira/browse/FLINK-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343622#comment-16343622 ] Stefan Richter commented on FLINK-8522: --- I don't think that we ever need to read the int, it was never read as far as I can see because the code already starts seeking to read positions. That is why the int never had any effect, even if it was there but not consumed. But you should double check. > DefaultOperatorStateBackend writes data in checkpoint that is never read. > - > > Key: FLINK-8522 > URL: https://issues.apache.org/jira/browse/FLINK-8522 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Affects Versions: 1.4.0 >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas >Priority: Major > Fix For: 1.4.1 > > > In the DefaultOperatorStateBackend at line 283 we write in the checkpoint an > int declaring the number of the operator states that we include in the > checkpoint. > This number is never read when restoring and this can lead to confusion and > problems with backwards compatibility and/or extension of the types of state > we support (e.g. broadcast state). > There are two easy solutions, either remove the line and do not write the > size, or make sure that we also read this number when restoring and simply > ignore it. > I would go for the first one, i.e. remove the line. What do you think > [~richtesn] and [~tzulitai] ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8522) DefaultOperatorStateBackend writes data in checkpoint that is never read.
[ https://issues.apache.org/jira/browse/FLINK-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343584#comment-16343584 ] Tzu-Li (Gordon) Tai commented on FLINK-8522: Don't we need to do both options? i.e., 1. We don't want to write that int anymore. 2. Also, for previous savepoint versions, we want to make sure to read (but ignore) that int, so that additions we add elsewhere is not affected by it. > DefaultOperatorStateBackend writes data in checkpoint that is never read. > - > > Key: FLINK-8522 > URL: https://issues.apache.org/jira/browse/FLINK-8522 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Affects Versions: 1.4.0 >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas >Priority: Major > Fix For: 1.4.1 > > > In the DefaultOperatorStateBackend at line 283 we write in the checkpoint an > int declaring the number of the operator states that we include in the > checkpoint. > This number is never read when restoring and this can lead to confusion and > problems with backwards compatibility and/or extension of the types of state > we support (e.g. broadcast state). > There are two easy solutions, either remove the line and do not write the > size, or make sure that we also read this number when restoring and simply > ignore it. > I would go for the first one, i.e. remove the line. What do you think > [~richtesn] and [~tzulitai] ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8522) DefaultOperatorStateBackend writes data in checkpoint that is never read.
[ https://issues.apache.org/jira/browse/FLINK-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343580#comment-16343580 ] Stefan Richter commented on FLINK-8522: --- Option 1 looks good for me. > DefaultOperatorStateBackend writes data in checkpoint that is never read. > - > > Key: FLINK-8522 > URL: https://issues.apache.org/jira/browse/FLINK-8522 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Affects Versions: 1.4.0 >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas >Priority: Major > Fix For: 1.4.1 > > > In the DefaultOperatorStateBackend at line 283 we write in the checkpoint an > int declaring the number of the operator states that we include in the > checkpoint. > This number is never read when restoring and this can lead to confusion and > problems with backwards compatibility and/or extension of the types of state > we support (e.g. broadcast state). > There are two easy solutions, either remove the line and do not write the > size, or make sure that we also read this number when restoring and simply > ignore it. > I would go for the first one, i.e. remove the line. What do you think > [~richtesn] and [~tzulitai] ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
