[
https://issues.apache.org/jira/browse/MAPREDUCE-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chaoli updated MAPREDUCE-7290:
------------------------------
Description:
Recently, we need to add a property in ShuffleHeader class. And there were a
problem that client&server may not compatible because of
serialization/deserialization when we were rolling update NodeManager instance.
Therefore, we made an improvement to help shuffle header version upgrade
smoothly.
In HeaderVersion class, shuffle client uses *targetVersion(eg 1.2.0)* to
negotiate with shuffle server. In that case, shuffle server has three cases:
# shuffle server does not have the newest code which is able to support
version chosen, and we have to use *defaultVersion(eg 1.0.0)* instead*.*
# shuffle server can support older version(eg 1.1.0), then it can reply to
shuffle client with *compatibleVersion(eg 1.1.0),* and finally, client and
server can use the same version.
# shuffle server can support older version(eg 1.3.0), then it can reply to
shuffle client with *compatibleVersion(eg 1.2.0),* and finally, client and
server can use the same version.
Then, client and server can serialize&deserialize property using the available
version.
was:
Recently, we need to add a property in ShuffleHeader class. And there were a
problem that client&server may not compatible because of
serialization/deserialization when we were rolling update NodeManager instance.
Therefore, we made an improvement to help shuffle header version upgrade
smoothly.
In HeaderVersion class, shuffle client uses *targetVersion(eg 1.2.0)* to
negotiate with shuffle server. In that case, shuffle server has three cases:
# shuffle server does not have the newest code which is able to support
version chosen, and we have to use *defaultVersion(eg 1.0.0)* instead*.*
# shuffle server can support older version(eg 1.1.0), then it can reply to
shuffle client with *compatibleVersion(eg 1.1.0),* and finally, client and
server can use the same version.
# shuffle server can support older version(eg 1.3.0), then it can reply to
shuffle client with *compatibleVersion(eg 1.2.0),* and finally, client and
server can use the same version.
Then, client and server can serialize&deserialize property using the available
version.
```
/**
* unified hard code header version for new properties.
* @throws IOException
*/
private void writeByVersion(DataOutput out) throws IOException {
if (headerVersion == null ||
headerVersion.compareTo(DEFAULT_HEADER_VERSION_INSTANCE) == 0) {
return;
}
// if current version larger then target version,
// we should serialize the properties owned by the version in order.
if (headerVersion.compareTo(HEADER_VERSION_INSTANCE_V1_1) >= 0) {
// here write version properties
}
}
```
> ShuffleHeader should be compatible between client&server when rolling update
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-7290
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7290
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: chaoli
> Priority: Major
> Attachments:
> 0001-MAPREDUCE-7290-ShuffleHeader-should-be-compatible-be.patch
>
>
> Recently, we need to add a property in ShuffleHeader class. And there were a
> problem that client&server may not compatible because of
> serialization/deserialization when we were rolling update NodeManager
> instance.
> Therefore, we made an improvement to help shuffle header version upgrade
> smoothly.
> In HeaderVersion class, shuffle client uses *targetVersion(eg 1.2.0)* to
> negotiate with shuffle server. In that case, shuffle server has three cases:
> # shuffle server does not have the newest code which is able to support
> version chosen, and we have to use *defaultVersion(eg 1.0.0)* instead*.*
> # shuffle server can support older version(eg 1.1.0), then it can reply to
> shuffle client with *compatibleVersion(eg 1.1.0),* and finally, client and
> server can use the same version.
> # shuffle server can support older version(eg 1.3.0), then it can reply to
> shuffle client with *compatibleVersion(eg 1.2.0),* and finally, client and
> server can use the same version.
> Then, client and server can serialize&deserialize property using the
> available version.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
