* For YARN-6616, for branch-2.8 and below, it was only committed to 2.7.8/2.8.6 which have not been released (as I understand). Perhaps we can revert YARN-6616 from branch-2.7 and branch-2.8. - This seems reasonable. Since we haven't released anything, it should be no issue to change the 2.7/2.8 protobuf field to have the same value as 2.9+
* For YARN-6050, there's a bit here: https://developers.google.com/protocol-buffers/docs/proto that says "optional is compatible with repeated", so I think we should be OK there. - Optional is compatible with repeatable over the wire such that protobuf won't blow up, but does that actually mean that it's compatible in this case? If it's expecting an optional and gets a repeated, it's going to drop everything except for the last value. I don't know enough about YARN-6050 to say if this will be ok or not. * For YARN-7813, it's in 2.8.4 so it seems upgrading from 2.8.4 or 2.8.5 to a 2.9+ version will be an issue. One option could be to move the intraQueuePreemptionDisabled field from id 12 to id 13 in branch-2.8, then users would upgrade from 2.8.4/2.8.5 to 2.8.6 (someone would have to release this), then upgrade from 2.8.6 to 2.9+. - I'm ok with this, but it should be noted that the upgrade from 2.8.4/2.8.5 to 2.8.6 (or 2.9+) would not be compatible for a rolling upgrade. So this would cause some pain to anybody with clusters on those versions. Eric On Tue, Sep 24, 2019 at 2:42 PM Jonathan Hung <jyhung2...@gmail.com> wrote: > Sorry, let me edit my first point. We can just create addendums for > YARN-6616 in branch-2.7 and branch-2.8 to edit the submitTime field to the > correct id 28. We don’t need to revert YARN-6616 from these branches > completely. > > Jonathan > > ________________________________ > From: Jonathan Hung <jyhung2...@gmail.com> > Sent: Tuesday, September 24, 2019 11:38 AM > To: Eric Badger > Cc: Hadoop Common; yarn-dev; mapreduce-dev; Hdfs-dev > Subject: Re: Incompatible changes between branch-2.8 and branch-2.9 > > Hi Eric, thanks for the investigation. > > * For YARN-6616, for branch-2.8 and below, it was only committed to > 2.7.8/2.8.6 which have not been released (as I understand). Perhaps we can > revert YARN-6616 from branch-2.7 and branch-2.8. > * For YARN-6050, there's a bit here: > https://developers.google.com/protocol-buffers/docs/proto that says > "optional is compatible with repeated", so I think we should be OK there. > * For YARN-7813, it's in 2.8.4 so it seems upgrading from 2.8.4 or > 2.8.5 to a 2.9+ version will be an issue. One option could be to move the > intraQueuePreemptionDisabled field from id 12 to id 13 in branch-2.8, then > users would upgrade from 2.8.4/2.8.5 to 2.8.6 (someone would have to > release this), then upgrade from 2.8.6 to 2.9+. > > Jonathan Hung > > > On Tue, Sep 24, 2019 at 9:23 AM Eric Badger <ebad...@verizonmedia.com.invalid> > wrote: > We (Verizon Media) are currently moving towards upgrading our clusters from > our internal fork of branch-2.8 to an internal fork of branch-2. During > this process, we have found multiple incompatible changes in protobufs > between branch-2.8 and branch-2. These incompatibilities were all > introduced between branch-2.8 and branch-2.9. I did a git diff over all > .proto files across the branch-2.8 and branch-2.9 and found 3 instances of > incompatibilities from 3 separate commits. All of the incompatibilities are > in yarn_protos.proto > > > I would like to discuss how to fix these incompatible changes. Otherwise, > rolling upgrades will not be supported between branch-2.8 (or below) and > branch-2.9 (or beyond). We could revert the incompatible changes, but then > the new releases would be incompatible with the releases that have these > incompatible changes. If we do nothing, then rolling upgrades won't work > between 2.8- and 2.9+. > > > Thanks, > > > Eric > > > ------------------------------------------------------------------- > > > git diff branch-2.8..branch-2.9 $(find . -name '*\.proto') > > > https://issues.apache.org/jira/browse/YARN-6616 > > - Trunk patch (applied through branch-2.9) differs from branch-2.8 patch > > @@ -211,7 +245,20 @@ message ApplicationReportProto { > > optional PriorityProto priority = 23; > > optional string appNodeLabelExpression = 24; > > optional string amNodeLabelExpression = 25; > > - optional int64 submitTime = 26; > > + repeated AppTimeoutsMapProto appTimeouts = 26; > > + optional int64 launchTime = 27; > > + optional int64 submitTime = 28; > > > https://issues.apache.org/jira/browse/YARN-6050 > > - Trunk and branch-2 patches both change the protobuf type in the same > way. > > @@ -356,7 +416,22 @@ message ApplicationSubmissionContextProto { > > optional LogAggregationContextProto log_aggregation_context = 14; > > optional ReservationIdProto reservation_id = 15; > > optional string node_label_expression = 16; > > - optional ResourceRequestProto am_container_resource_request = 17; > > + repeated ResourceRequestProto am_container_resource_request = 17; > > + repeated ApplicationTimeoutMapProto application_timeouts = 18; > > > https://issues.apache.org/jira/browse/YARN-7813 > > - Trunk (applied through branch-3.1) and branch-3.0 (applied through > branch-2.9) patches differ from branch-2.8 patch > > @@ -425,7 +501,21 @@ message QueueInfoProto { > > optional string defaultNodeLabelExpression = 9; > > optional QueueStatisticsProto queueStatistics = 10; > > optional bool preemptionDisabled = 11; > > - optional bool intraQueuePreemptionDisabled = 12; > > + repeated QueueConfigurationsMapProto queueConfigurationsMap = 12; > > + optional bool intraQueuePreemptionDisabled = 13; >
