[
https://issues.apache.org/jira/browse/IGNITE-28939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Vinogradov updated IGNITE-28939:
--------------------------------------
Description:
*Wire format change. Do this before 2.19 is released - after that it would
break rolling upgrade.*
h3. Goal
Remove the question "which marshaller" for fields that hold internal classes.
h3. Why
Some fields are sent as blobs only because an internal class was never made a
{{Message}}. Binary gives no schema for them anyway:
{{BinaryUtils#isCustomJavaSerialization}} checks the whole class hierarchy, so
{{Externalizable}} classes fall back to {{OptimizedMarshaller}} inside binary.
One of these fields is also a mismatch: {{BinaryMetadata}} uses binary in
{{BinaryMetadataVersionInfo}} and jdk in {{MetadataUpdateProposedMessage}}.
h3. How
Convert to normal message fields:
* {{BinaryMetadataVersionInfo#metadata}} and
{{MetadataUpdateProposedMessage#metadata}} - {{BinaryMetadata}} has simple
fields: type id, type name, field map, schemas, flags;
* {{ChangeGlobalStateMessage#baselineTopology}} - {{BaselineTopology}};
* {{GridJobSiblingsResponse#siblings}} - the implementation is the internal
{{GridJobSiblingImpl}};
* {{StoredCacheData#qryEntities}} and {{DynamicCacheChangeRequest#schema}} -
{{QueryEntityMessage}} already exists.
Fields that really hold user classes stay blobs: node attributes, service
affinity key, query entity default values, entry processors, invoke arguments,
query filters, task results, message topic, cache configuration, plugin data
and {{Throwable}}.
h3. On-disk compatibility
Three of these types also live on disk, so the conversion must not change how
they are stored:
* {{BinaryMetadata}} - written by {{BinaryMetadataFileStore#writeMetadata}};
the format comes from its
{{Externalizable}} methods;
* {{BaselineTopology}} - kept in the metastore under
{{METASTORE_CURR_BLT_KEY}}; {{MetaStorage}} marshals values with jdk;
* {{QueryEntity}} inside {{StoredCacheData}} - {{cache_data.dat}} files and
snapshots, jdk again.
Rules that keep the stored format byte-identical:
# keep the explicit {{serialVersionUID}}. All of these classes declare one, so
adding the {{Message}} interface does
not change the computed id. Without an explicit id, adding an interface changes
it and old files stop loading.
# make new wire companion fields {{transient}}, the way {{ccfgBytes}} and
{{qryEntitiesBytes}} are done in
{{StoredCacheData}}. Non-transient fields are what the disk format is made of.
# do not add or remove non-transient fields, and do not touch
{{writeExternal}}/{{readExternal}}. For
{{BinaryMetadata}} the {{Externalizable}} methods *are* the on-disk format:
adding {{Message}} on top is fine,
dropping {{Externalizable}} is not.
Precedent: {{StoredCacheData}}, {{CacheConfigurationEnrichment}} and
{{SnapshotMetadata}} are already both
{{Serializable}} and {{Message}}, and that coexists with the disk today.
Verification for this ticket: read a {{cache_data.dat}} and a binary metadata
file written by the previous build,
and restore a snapshot taken by it.
h3. Expected result
Blob fields: 24 now, about 19 after. The {{BinaryMetadata}} mismatch is gone.
These fields no longer depend on a marshaller, and usually become smaller on
the wire.
h3. How to verify
Binary metadata tests, node join tests, cluster state change tests, dynamic
cache start with query entities, job siblings tests. Compare message size
before and after.
was:
*Wire format change. Do this before 2.19 is released - after that it would
break rolling upgrade.*
h3. Goal
Remove the question "which marshaller" for fields that hold internal classes.
h3. Why
Some fields are sent as blobs only because an internal class was never made a
{{Message}}. Binary gives no schema for them anyway:
{{BinaryUtils#isCustomJavaSerialization}} checks the whole class hierarchy, so
{{Externalizable}} classes fall back to {{OptimizedMarshaller}} inside binary.
One of these fields is also a mismatch: {{BinaryMetadata}} uses binary in
{{BinaryMetadataVersionInfo}} and jdk in {{MetadataUpdateProposedMessage}}.
h3. How
Convert to normal message fields:
* {{BinaryMetadataVersionInfo#metadata}} and
{{MetadataUpdateProposedMessage#metadata}} - {{BinaryMetadata}} has simple
fields: type id, type name, field map, schemas, flags;
* {{ChangeGlobalStateMessage#baselineTopology}} - {{BaselineTopology}};
* {{GridJobSiblingsResponse#siblings}} - the implementation is the internal
{{GridJobSiblingImpl}};
* {{StoredCacheData#qryEntities}} and {{DynamicCacheChangeRequest#schema}} -
{{QueryEntityMessage}} already exists.
Fields that really hold user classes stay blobs: node attributes, service
affinity key, query entity default values, entry processors, invoke arguments,
query filters, task results, message topic, cache configuration, plugin data
and {{Throwable}}.
h3. Expected result
Blob fields: 24 now, about 19 after. The {{BinaryMetadata}} mismatch is gone.
These fields no longer depend on a marshaller, and usually become smaller on
the wire.
h3. How to verify
Binary metadata tests, node join tests, cluster state change tests, dynamic
cache start with query entities, job siblings tests. Compare message size
before and after.
> Turn blob fields of internal types into normal message fields
> -------------------------------------------------------------
>
> Key: IGNITE-28939
> URL: https://issues.apache.org/jira/browse/IGNITE-28939
> Project: Ignite
> Issue Type: Sub-task
> Components: messaging
> Reporter: Anton Vinogradov
> Assignee: Anton Vinogradov
> Priority: Major
> Labels: compatibility
> Fix For: 2.19
>
>
> *Wire format change. Do this before 2.19 is released - after that it would
> break rolling upgrade.*
> h3. Goal
> Remove the question "which marshaller" for fields that hold internal classes.
> h3. Why
> Some fields are sent as blobs only because an internal class was never made a
> {{Message}}. Binary gives no schema for them anyway:
> {{BinaryUtils#isCustomJavaSerialization}} checks the whole class hierarchy,
> so {{Externalizable}} classes fall back to {{OptimizedMarshaller}} inside
> binary. One of these fields is also a mismatch: {{BinaryMetadata}} uses
> binary in {{BinaryMetadataVersionInfo}} and jdk in
> {{MetadataUpdateProposedMessage}}.
> h3. How
> Convert to normal message fields:
> * {{BinaryMetadataVersionInfo#metadata}} and
> {{MetadataUpdateProposedMessage#metadata}} - {{BinaryMetadata}} has simple
> fields: type id, type name, field map, schemas, flags;
> * {{ChangeGlobalStateMessage#baselineTopology}} - {{BaselineTopology}};
> * {{GridJobSiblingsResponse#siblings}} - the implementation is the internal
> {{GridJobSiblingImpl}};
> * {{StoredCacheData#qryEntities}} and {{DynamicCacheChangeRequest#schema}} -
> {{QueryEntityMessage}} already exists.
> Fields that really hold user classes stay blobs: node attributes, service
> affinity key, query entity default values, entry processors, invoke
> arguments, query filters, task results, message topic, cache configuration,
> plugin data and {{Throwable}}.
> h3. On-disk compatibility
> Three of these types also live on disk, so the conversion must not change how
> they are stored:
> * {{BinaryMetadata}} - written by {{BinaryMetadataFileStore#writeMetadata}};
> the format comes from its
> {{Externalizable}} methods;
> * {{BaselineTopology}} - kept in the metastore under
> {{METASTORE_CURR_BLT_KEY}}; {{MetaStorage}} marshals values with jdk;
> * {{QueryEntity}} inside {{StoredCacheData}} - {{cache_data.dat}} files and
> snapshots, jdk again.
> Rules that keep the stored format byte-identical:
> # keep the explicit {{serialVersionUID}}. All of these classes declare one,
> so adding the {{Message}} interface does
> not change the computed id. Without an explicit id, adding an interface
> changes it and old files stop loading.
> # make new wire companion fields {{transient}}, the way {{ccfgBytes}} and
> {{qryEntitiesBytes}} are done in
> {{StoredCacheData}}. Non-transient fields are what the disk format is made of.
> # do not add or remove non-transient fields, and do not touch
> {{writeExternal}}/{{readExternal}}. For
> {{BinaryMetadata}} the {{Externalizable}} methods *are* the on-disk format:
> adding {{Message}} on top is fine,
> dropping {{Externalizable}} is not.
> Precedent: {{StoredCacheData}}, {{CacheConfigurationEnrichment}} and
> {{SnapshotMetadata}} are already both
> {{Serializable}} and {{Message}}, and that coexists with the disk today.
> Verification for this ticket: read a {{cache_data.dat}} and a binary metadata
> file written by the previous build,
> and restore a snapshot taken by it.
> h3. Expected result
> Blob fields: 24 now, about 19 after. The {{BinaryMetadata}} mismatch is gone.
> These fields no longer depend on a marshaller, and usually become smaller on
> the wire.
> h3. How to verify
> Binary metadata tests, node join tests, cluster state change tests, dynamic
> cache start with query entities, job siblings tests. Compare message size
> before and after.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
