[
https://issues.apache.org/jira/browse/MAPREDUCE-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated MAPREDUCE-7290:
---------------------------------------
===Bulk update===
I am planning to cut the branch for Hadoop 3.3.1 release, and this jira targets
3.3.1 currently. Please take the time to review the patch, or push out of 3.3.1
if you think it can't be finished in the next few weeks.
> ShuffleHeader should be compatible between client&server when rolling update
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-7290
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7290
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 2.6.0
> Reporter: chaoli
> Priority: Major
> Fix For: 2.9.0, 3.3.0
>
> Attachments:
> 0001-MAPREDUCE-7290-ShuffleHeader-should-be-compatible-be.patch,
> 0001-MAPREDUCE-7290-ShuffleHeader-should-be-compatible-be.patch
>
>
> Recently, we need to add a property in ShuffleHeader class. And there were a
> problem that client&server may not compatible because of
> serialization/deserialization when we were rolling update NodeManager
> instance.
> To solve the problem, we made an improvement to help shuffle header version
> upgrade smoothly.
> In HeaderVersion class, shuffle client uses *targetVersion(eg 1.2.0)* to
> negotiate with shuffle server. In that case, shuffle server has three cases:
> # shuffle server does not have the newest code which is able to support
> version chosen, and we have to use *defaultVersion(eg 1.0.0)* instead*.*
> # shuffle server can support older version(eg 1.1.0), then it can reply to
> shuffle client with *compatibleVersion(eg 1.1.0),* and finally, client and
> server can use the same version.
> # shuffle server can support older version(eg 1.3.0), then it can reply to
> shuffle client with *compatibleVersion(eg 1.2.0),* and finally, client and
> server can use the same version.
> Then, client and server can serialize&deserialize property using the
> available version.
>
> {code:java}
> /**
> * unified hard code header version for new properties.
> * @throws IOException
> */
> private void writeByVersion(DataOutput out) throws IOException {
> if (headerVersion == null ||
> headerVersion.compareTo(DEFAULT_HEADER_VERSION_INSTANCE) == 0) {
> return;
> }
> // if current version larger then target version,
> // we should serialize the properties owned by the version in order.
> if (headerVersion.compareTo(HEADER_VERSION_INSTANCE_V1_1) >= 0) {
> // here write version properties
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
