[jira] [Updated] (FLINK-6763) Inefficient PojoSerializerConfigSnapshot serialization format
[ https://issues.apache.org/jira/browse/FLINK-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-6763: - Fix Version/s: (was: 1.7.0) > Inefficient PojoSerializerConfigSnapshot serialization format > - > > Key: FLINK-6763 > URL: https://issues.apache.org/jira/browse/FLINK-6763 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing, Type Serialization System >Affects Versions: 1.3.0, 1.4.0 >Reporter: Till Rohrmann >Assignee: Tzu-Li (Gordon) Tai >Priority: Major > Fix For: 1.8.0 > > > The {{PojoSerializerConfigSnapshot}} stores for each serializer the beginning > offset and ending offset in the serialization stream. This information is > also written if the serializer serialization is supposed to be ignored. The > beginning and ending offsets are stored as a sequence of integers at the > beginning of the serialization stream. We store this information to skip > broken serializers. > I think we don't need both offsets. Instead I would suggest to write the > length of the serialized serializer first into the serialization stream and > then the serialized serializer. This can be done in > {{TypeSerializerSerializationUtil.writeSerializer}}. When reading the > serializer via {{TypeSerializerSerializationUtil.tryReadSerializer}}, we can > try to deserialize the serializer. If this operation fails, then we can skip > the number of serialized serializer because we know how long it was. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-6763) Inefficient PojoSerializerConfigSnapshot serialization format
[ https://issues.apache.org/jira/browse/FLINK-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-6763: - Fix Version/s: 1.8.0 > Inefficient PojoSerializerConfigSnapshot serialization format > - > > Key: FLINK-6763 > URL: https://issues.apache.org/jira/browse/FLINK-6763 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing, Type Serialization System >Affects Versions: 1.3.0, 1.4.0 >Reporter: Till Rohrmann >Assignee: Tzu-Li (Gordon) Tai >Priority: Major > Fix For: 1.7.0, 1.8.0 > > > The {{PojoSerializerConfigSnapshot}} stores for each serializer the beginning > offset and ending offset in the serialization stream. This information is > also written if the serializer serialization is supposed to be ignored. The > beginning and ending offsets are stored as a sequence of integers at the > beginning of the serialization stream. We store this information to skip > broken serializers. > I think we don't need both offsets. Instead I would suggest to write the > length of the serialized serializer first into the serialization stream and > then the serialized serializer. This can be done in > {{TypeSerializerSerializationUtil.writeSerializer}}. When reading the > serializer via {{TypeSerializerSerializationUtil.tryReadSerializer}}, we can > try to deserialize the serializer. If this operation fails, then we can skip > the number of serialized serializer because we know how long it was. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-6763) Inefficient PojoSerializerConfigSnapshot serialization format
[ https://issues.apache.org/jira/browse/FLINK-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-6763: - Fix Version/s: (was: 1.6.0) 1.7.0 > Inefficient PojoSerializerConfigSnapshot serialization format > - > > Key: FLINK-6763 > URL: https://issues.apache.org/jira/browse/FLINK-6763 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing, Type Serialization System >Affects Versions: 1.3.0, 1.4.0 >Reporter: Till Rohrmann >Assignee: Tzu-Li (Gordon) Tai >Priority: Major > Fix For: 1.7.0 > > > The {{PojoSerializerConfigSnapshot}} stores for each serializer the beginning > offset and ending offset in the serialization stream. This information is > also written if the serializer serialization is supposed to be ignored. The > beginning and ending offsets are stored as a sequence of integers at the > beginning of the serialization stream. We store this information to skip > broken serializers. > I think we don't need both offsets. Instead I would suggest to write the > length of the serialized serializer first into the serialization stream and > then the serialized serializer. This can be done in > {{TypeSerializerSerializationUtil.writeSerializer}}. When reading the > serializer via {{TypeSerializerSerializationUtil.tryReadSerializer}}, we can > try to deserialize the serializer. If this operation fails, then we can skip > the number of serialized serializer because we know how long it was. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-6763) Inefficient PojoSerializerConfigSnapshot serialization format
[ https://issues.apache.org/jira/browse/FLINK-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-6763: Priority: Major (was: Blocker) > Inefficient PojoSerializerConfigSnapshot serialization format > - > > Key: FLINK-6763 > URL: https://issues.apache.org/jira/browse/FLINK-6763 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing, Type Serialization System >Affects Versions: 1.3.0, 1.4.0 >Reporter: Till Rohrmann >Assignee: Tzu-Li (Gordon) Tai >Priority: Major > Fix For: 1.6.0 > > > The {{PojoSerializerConfigSnapshot}} stores for each serializer the beginning > offset and ending offset in the serialization stream. This information is > also written if the serializer serialization is supposed to be ignored. The > beginning and ending offsets are stored as a sequence of integers at the > beginning of the serialization stream. We store this information to skip > broken serializers. > I think we don't need both offsets. Instead I would suggest to write the > length of the serialized serializer first into the serialization stream and > then the serialized serializer. This can be done in > {{TypeSerializerSerializationUtil.writeSerializer}}. When reading the > serializer via {{TypeSerializerSerializationUtil.tryReadSerializer}}, we can > try to deserialize the serializer. If this operation fails, then we can skip > the number of serialized serializer because we know how long it was. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-6763) Inefficient PojoSerializerConfigSnapshot serialization format
[ https://issues.apache.org/jira/browse/FLINK-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai updated FLINK-6763: --- Fix Version/s: (was: 1.5.0) 1.6.0 > Inefficient PojoSerializerConfigSnapshot serialization format > - > > Key: FLINK-6763 > URL: https://issues.apache.org/jira/browse/FLINK-6763 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing, Type Serialization System >Affects Versions: 1.3.0, 1.4.0 >Reporter: Till Rohrmann >Assignee: Tzu-Li (Gordon) Tai >Priority: Blocker > Fix For: 1.6.0 > > > The {{PojoSerializerConfigSnapshot}} stores for each serializer the beginning > offset and ending offset in the serialization stream. This information is > also written if the serializer serialization is supposed to be ignored. The > beginning and ending offsets are stored as a sequence of integers at the > beginning of the serialization stream. We store this information to skip > broken serializers. > I think we don't need both offsets. Instead I would suggest to write the > length of the serialized serializer first into the serialization stream and > then the serialized serializer. This can be done in > {{TypeSerializerSerializationUtil.writeSerializer}}. When reading the > serializer via {{TypeSerializerSerializationUtil.tryReadSerializer}}, we can > try to deserialize the serializer. If this operation fails, then we can skip > the number of serialized serializer because we know how long it was. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-6763) Inefficient PojoSerializerConfigSnapshot serialization format
[ https://issues.apache.org/jira/browse/FLINK-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai updated FLINK-6763: --- Priority: Blocker (was: Major) > Inefficient PojoSerializerConfigSnapshot serialization format > - > > Key: FLINK-6763 > URL: https://issues.apache.org/jira/browse/FLINK-6763 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing, Type Serialization System >Affects Versions: 1.3.0, 1.4.0 >Reporter: Till Rohrmann >Assignee: Tzu-Li (Gordon) Tai >Priority: Blocker > Fix For: 1.5.0 > > > The {{PojoSerializerConfigSnapshot}} stores for each serializer the beginning > offset and ending offset in the serialization stream. This information is > also written if the serializer serialization is supposed to be ignored. The > beginning and ending offsets are stored as a sequence of integers at the > beginning of the serialization stream. We store this information to skip > broken serializers. > I think we don't need both offsets. Instead I would suggest to write the > length of the serialized serializer first into the serialization stream and > then the serialized serializer. This can be done in > {{TypeSerializerSerializationUtil.writeSerializer}}. When reading the > serializer via {{TypeSerializerSerializationUtil.tryReadSerializer}}, we can > try to deserialize the serializer. If this operation fails, then we can skip > the number of serialized serializer because we know how long it was. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-6763) Inefficient PojoSerializerConfigSnapshot serialization format
[ https://issues.apache.org/jira/browse/FLINK-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai updated FLINK-6763: --- Fix Version/s: 1.5.0 > Inefficient PojoSerializerConfigSnapshot serialization format > - > > Key: FLINK-6763 > URL: https://issues.apache.org/jira/browse/FLINK-6763 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing, Type Serialization System >Affects Versions: 1.3.0, 1.4.0 >Reporter: Till Rohrmann >Assignee: Tzu-Li (Gordon) Tai > Fix For: 1.5.0 > > > The {{PojoSerializerConfigSnapshot}} stores for each serializer the beginning > offset and ending offset in the serialization stream. This information is > also written if the serializer serialization is supposed to be ignored. The > beginning and ending offsets are stored as a sequence of integers at the > beginning of the serialization stream. We store this information to skip > broken serializers. > I think we don't need both offsets. Instead I would suggest to write the > length of the serialized serializer first into the serialization stream and > then the serialized serializer. This can be done in > {{TypeSerializerSerializationUtil.writeSerializer}}. When reading the > serializer via {{TypeSerializerSerializationUtil.tryReadSerializer}}, we can > try to deserialize the serializer. If this operation fails, then we can skip > the number of serialized serializer because we know how long it was. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-6763) Inefficient PojoSerializerConfigSnapshot serialization format
[ https://issues.apache.org/jira/browse/FLINK-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai updated FLINK-6763: --- Labels: (was: flink-rel-1.3.1-blockers) > Inefficient PojoSerializerConfigSnapshot serialization format > - > > Key: FLINK-6763 > URL: https://issues.apache.org/jira/browse/FLINK-6763 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing, Type Serialization System >Affects Versions: 1.3.0, 1.4.0 >Reporter: Till Rohrmann >Assignee: Tzu-Li (Gordon) Tai > > The {{PojoSerializerConfigSnapshot}} stores for each serializer the beginning > offset and ending offset in the serialization stream. This information is > also written if the serializer serialization is supposed to be ignored. The > beginning and ending offsets are stored as a sequence of integers at the > beginning of the serialization stream. We store this information to skip > broken serializers. > I think we don't need both offsets. Instead I would suggest to write the > length of the serialized serializer first into the serialization stream and > then the serialized serializer. This can be done in > {{TypeSerializerSerializationUtil.writeSerializer}}. When reading the > serializer via {{TypeSerializerSerializationUtil.tryReadSerializer}}, we can > try to deserialize the serializer. If this operation fails, then we can skip > the number of serialized serializer because we know how long it was. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (FLINK-6763) Inefficient PojoSerializerConfigSnapshot serialization format
[ https://issues.apache.org/jira/browse/FLINK-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-6763: -- Labels: flink-rel-1.3.1-blockers (was: ) > Inefficient PojoSerializerConfigSnapshot serialization format > - > > Key: FLINK-6763 > URL: https://issues.apache.org/jira/browse/FLINK-6763 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing, Type Serialization System >Affects Versions: 1.3.0, 1.4.0 >Reporter: Till Rohrmann >Assignee: Tzu-Li (Gordon) Tai > Labels: flink-rel-1.3.1-blockers > > The {{PojoSerializerConfigSnapshot}} stores for each serializer the beginning > offset and ending offset in the serialization stream. This information is > also written if the serializer serialization is supposed to be ignored. The > beginning and ending offsets are stored as a sequence of integers at the > beginning of the serialization stream. We store this information to skip > broken serializers. > I think we don't need both offsets. Instead I would suggest to write the > length of the serialized serializer first into the serialization stream and > then the serialized serializer. This can be done in > {{TypeSerializerSerializationUtil.writeSerializer}}. When reading the > serializer via {{TypeSerializerSerializationUtil.tryReadSerializer}}, we can > try to deserialize the serializer. If this operation fails, then we can skip > the number of serialized serializer because we know how long it was. -- This message was sent by Atlassian JIRA (v6.3.15#6346)