If the patch is not complicated and not destabilizing, I have no concerns.
On Fri, Nov 13, 2015 at 6:55 AM, Sunil Govind <[email protected]>
wrote:
> Hi Vinod and Karthik
>
> YARN-3849 was trying fix an irregularity in getting used queue resource for
> ProportionalPremptionPolicy. Its not very complicated patch. And it fixes a
> critical issue in preemption policy when used with DRC.
>
> In PCPP, earlier to get the used capacity, total resource were multiplied
> with absoluteCapacity per queue. Instead, we could get this resource
> information direct with an api
> {{curQueue.getQueueResourceUsage().getUsed(partition)}}
> w/o using any more extra calculation. While using DRC we found that if we
> do totalResource*absoluteCapacity, it may normalize the total resource
> (either memory or vcore). To avoid, we could directly get the used capacity
> per-queue per partition.
> Rest all patch is to have DRC to support in test code of
> ProportionalPremptionPolicy.
>
> + Wangda. Kindly share your thoughts.
>
> Thank You
> Sunil
>
> On Thu, Nov 12, 2015 at 10:51 PM Karthik Kambatla <[email protected]>
> wrote:
>
> > If YARN-3849 is a complicated patch, are we comfortable including it in
> > 2.7.3? If it is not de-stabilizing, I am fine with it. Otherwise, it
> might
> > make more sense to not include it in later 2.7.x.
> >
> > With my Cloudera hat on, inclusion/exclusion in 2.7.3 doesn't really
> > matter.
> >
> > With my Apache hat on, the only way we inspire confidence to our users is
> > increasingly stable maintenance releases. I know I have been harping on
> > this a lot.
> >
> > On Tue, Nov 3, 2015 at 11:37 AM, Vinod Vavilapalli <
> > [email protected]>
> > wrote:
> >
> > > YARN-3849 looks like a bit of a non-trivial change this late for
> 2.7.2, I
> > > requested Wangda offline to punt it for 2.7.3.
> > >
> > > Thanks
> > > +Vinod
> > >
> > > > On Nov 2, 2015, at 12:43 AM, Sunil Govind <[email protected]>
> > > wrote:
> > > >
> > > > Thank you.
> > > > I will help to backport to 2.7.2.
> > > >
> > > > Thank you
> > > > Sunil
> > > >
> > > > On Mon, Nov 2, 2015 at 2:10 PM Wangda Tan <[email protected]>
> wrote:
> > > >
> > > >> +1 to back port it to 2.7.2, marked to 2.7.2-candidate.
> > > >>
> > > >> On Mon, Nov 2, 2015 at 12:37 AM, Sunil Govind <
> [email protected]
> > >
> > > >> wrote:
> > > >>
> > > >>> Thank you Wangda. Sorry to pitch in late here.
> > > >>>
> > > >>> I feel YARN-3849 is also a good candidate for 2.7.2. This s a bug
> fix
> > > for
> > > >>> DRC and preemption.
> > > >>>
> > > >>> Thank You
> > > >>> Sunil
> > > >>>
> > > >>>
> > > >>> On Thu, Oct 29, 2015 at 12:26 PM Naganarasimha G R (Naga) <
> > > >>> [email protected]> wrote:
> > > >>>
> > > >>>> Thanks for sharing this important viewpoint.
> > > >>>>
> > > >>>> This sub list of NodeLabels jiras what i have selected is doing
> > > minimal
> > > >>>> modifications to the core code but tries to increase the usability
> > of
> > > >>>> NodeLabels and fix some bugs or add missing necessary features
> > > >>>> Other additional features which were done for NodeLabels like
> > > >>> Distributed
> > > >>>> Scheduling or Delegated Centralized are all not included.
> > > >>>> May be Wangda could be better judge to further scrutinize the list
> > and
> > > >>>> select from them or even add to them
> > > >>>> My intention here is to only make the Node Labels more usable as
> > there
> > > >>> has
> > > >>>> been long delay for 2.8 and not heard of any approximate dates for
> > it.
> > > >>>>
> > > >>>> Regards,
> > > >>>> + Naga
> > > >>>>
> > > >>>>
> > > >>>> ________________________________________
> > > >>>> From: Karthik Kambatla [[email protected]]
> > > >>>> Sent: Thursday, October 29, 2015 04:04
> > > >>>> To: [email protected]
> > > >>>> Cc: Tsuyoshi Ozawa; Vinod Vavilapalli; [email protected]
> ;
> > > >>>> [email protected]; Vinod Kumar Vavilapalli; Wangda Tan
> > > >>>> Subject: Re: 2.7.2 release plan
> > > >>>>
> > > >>>> I would like for us to make sure later maintenance releases are
> more
> > > >>> stable
> > > >>>> than previous ones. IMO, increasing stability is more important
> than
> > > >> the
> > > >>>> timing of a release.
> > > >>>>
> > > >>>> Will adding all the patches in 2.7.3 reduce the stability going
> from
> > > >>> 2.7.2
> > > >>>> to 2.7.3? If yes, can we just leave them for 2.8.0?
> > > >>>>
> > > >>>> On Wed, Oct 28, 2015 at 12:07 PM, Naganarasimha G R (Naga) <
> > > >>>> [email protected]> wrote:
> > > >>>>
> > > >>>>> Yes Vinod & Tsuyoshi,
> > > >>>>>
> > > >>>>> Within a week merging them would be difficult. I can start
> > > >> backporting
> > > >>>>> them after 2.7.2 so that it can be ported to 2.7.3 faster, also
> > > >> shall i
> > > >>>>> apply 2.7.3-candidate labels to them ?
> > > >>>>>
> > > >>>>> + Naga
> > > >>>>> ______________________________
> > > >>>>> From: Tsuyoshi Ozawa [[email protected]]
> > > >>>>> Sent: Wednesday, October 28, 2015 23:13
> > > >>>>> To: Vinod Vavilapalli
> > > >>>>> Cc: [email protected]; [email protected];
> > > >>>>> [email protected]; Vinod Kumar Vavilapalli; Wangda
> Tan;
> > > >>>>> Tsuyoshi Ozawa; Naganarasimha G R (Naga)
> > > >>>>> Subject: Re: 2.7.2 release plan
> > > >>>>>
> > > >>>>> Vinod,
> > > >>>>>
> > > >>>>> Thank you for taking care of this. I've checked the list of
> > changes.
> > > >>>>> As a result, I agree that we don't have enough time to backport
> > these
> > > >>>>> changes into 2.7.2 by this weekend. For a fast move, it's better
> > > >>>>> suggestion to me to backport these tickets into 2.7.3.
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> - Tsuyoshi
> > > >>>>>
> > > >>>>> On Thu, Oct 29, 2015 at 2:19 AM, Vinod Vavilapalli
> > > >>>>> <[email protected]> wrote:
> > > >>>>>> Tsuyoshi / Wangda / Naga,
> > > >>>>>>
> > > >>>>>> This looks too big of a list to me if we have to cut an RC by
> this
> > > >>>>> weekend per my plan.
> > > >>>>>>
> > > >>>>>> I’d suggest a fast move on things you think are low risk enough
> > and
> > > >>>> punt
> > > >>>>> everything else for next release.
> > > >>>>>>
> > > >>>>>> Thanks
> > > >>>>>> +Vinod
> > > >>>>>>
> > > >>>>>>> On Oct 28, 2015, at 3:08 AM, Naganarasimha G R (Naga) <
> > > >>>>> [email protected]> wrote:
> > > >>>>>>>
> > > >>>>>>> Thanks Tsuyoshi,
> > > >>>>>>> If required even i can pitch in :)
> > > >>>>>>> Additional to this we added the support in Mapreduce for labels
> > in
> > > >>>>> MAPREDUCE-6304,
> > > >>>>>>>
> > > >>>>>>> Regards,
> > > >>>>>>> + Naga
> > > >>>>>>> ________________________________________
> > > >>>>>>> From: Tsuyoshi Ozawa [[email protected]]
> > > >>>>>>> Sent: Wednesday, October 28, 2015 14:28
> > > >>>>>>> To: [email protected]
> > > >>>>>>> Cc: [email protected]; [email protected];
> > > >> Vinod
> > > >>>>> Kumar Vavilapalli; Wangda tan
> > > >>>>>>> Subject: Re: 2.7.2 release plan
> > > >>>>>>>
> > > >>>>>>> Thank you for reporting, Naganarasimha.
> > > >>>>>>> Vinod and Wangda, I will help you to backport these changes.
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> - Tsuyoshi
> > > >>>>>>>
> > > >>>>>>> On Wed, Oct 28, 2015 at 2:57 PM, Naganarasimha G R (Naga)
> > > >>>>>>> <[email protected]> wrote:
> > > >>>>>>>> Hi Vinod, & Wangda
> > > >>>>>>>>
> > > >>>>>>>> I think it would be good to backport, following jira's related
> > to
> > > >>>>> NodeLabels as it will improve debug ability and usability of
> > > >> NodeLabels
> > > >>>>>>>> --------------------------------
> > > >>>>>>>> Key Summary
> > > >>>>>>>> --------------------------------
> > > >>>>>>>> YARN-4215 YARN-2492 RMNodeLabels Manager Need to verify
> > and
> > > >>>>> replace node labels for the only modified Node Label Mappings in
> > the
> > > >>>> request
> > > >>>>>>>> YARN-4162 YARN-2492 CapacityScheduler: Add resource
> usage
> > > >> by
> > > >>>>> partition and queue capacity by partition to REST API
> > > >>>>>>>> YARN-4140 YARN-2492 RM container allocation delayed
> incase
> > > >> of
> > > >>>>> app submitted to Nodelabel partition
> > > >>>>>>>> YARN-3717 YARN-2492 Expose app/am/queue's
> > > >>> node-label-expression
> > > >>>>> to RM web UI / CLI / REST-API
> > > >>>>>>>> YARN-3647 YARN-2492 RMWebServices api's should use
> updated
> > > >>> api
> > > >>>>> from CommonNodeLabelsManager to get NodeLabel object
> > > >>>>>>>> YARN-3593 YARN-2492 Add label-type and Improve
> > > >>>>> "DEFAULT_PARTITION" in Node Labels Page
> > > >>>>>>>> YARN-3583 YARN-2492 Support of NodeLabel object instead
> of
> > > >>>> plain
> > > >>>>> String in YarnClient side.
> > > >>>>>>>> YARN-3581 YARN-2492 Deprecate
> > -directlyAccessNodeLabelStore
> > > >>> in
> > > >>>>> RMAdminCLI
> > > >>>>>>>> YARN-3579 YARN-2492 CommonNodeLabelsManager should
> support
> > > >>>>> NodeLabel instead of string label name when getting
> > > >>>>> node-to-label/label-to-label mappings
> > > >>>>>>>> YARN-3565 YARN-2492
> > > >>>>> NodeHeartbeatRequest/RegisterNodeManagerRequest should use
> > NodeLabel
> > > >>>> object
> > > >>>>> instead of String
> > > >>>>>>>> YARN-3521 YARN-2492 Support return structured NodeLabel
> > > >>> objects
> > > >>>>> in REST API
> > > >>>>>>>> YARN-3362 YARN-2492 Add node label usage in RM
> > > >>>> CapacityScheduler
> > > >>>>> web UI
> > > >>>>>>>> YARN-3326 YARN-2492 Support RESTful API for
> > > >> getLabelsToNodes
> > > >>>>>>>> YARN-3216 YARN-2492 Max-AM-Resource-Percentage should
> > > >> respect
> > > >>>>> node labels
> > > >>>>>>>> YARN-3136 YARN-3091 getTransferredContainers can be a
> > > >>>> bottleneck
> > > >>>>> during AM registration
> > > >>>>>>>>
> > > >>>>>>>> Please inform if any support is required to backport them to
> > > >> 2.7.2
> > > >>>>>>>>
> > > >>>>>>>> Regards,
> > > >>>>>>>> + Naga
> > > >>>>>>>> ________________________________________
> > > >>>>>>>> From: Kihwal Lee [[email protected]]
> > > >>>>>>>> Sent: Tuesday, October 27, 2015 20:42
> > > >>>>>>>> To: [email protected]; [email protected]
> > > >>>>>>>> Cc: Chris Nauroth; [email protected];
> > > >>>>> [email protected]; Vinod Kumar Vavilapalli; Ming
> Ma
> > > >>>>>>>> Subject: Re: 2.7.2 release plan
> > > >>>>>>>>
> > > >>>>>>>> I think we need HDFS-8950 and HDFS-7725 in 2.7.2.It should be
> > > >> easy
> > > >>>> to
> > > >>>>> backport/cherry-pick HDFS-7725. For HDFS-8950, it will be nice if
> > > >> Ming
> > > >>>> can
> > > >>>>> chime in.
> > > >>>>>>>> Kihwal
> > > >>>>>>>>
> > > >>>>>>>> From: Tsuyoshi Ozawa <[email protected]>
> > > >>>>>>>> To: "[email protected]" <
> > [email protected]
> > > >>>
> > > >>>>>>>> Cc: Chris Nauroth <[email protected]>; "
> > > >>>>> [email protected]" <[email protected]>; "
> > > >>>>> [email protected]" <[email protected]>; "
> > > >>>>> [email protected]" <
> [email protected]
> > >;
> > > >>>> Vinod
> > > >>>>> Kumar Vavilapalli <[email protected]>
> > > >>>>>>>> Sent: Tuesday, October 27, 2015 2:39 AM
> > > >>>>>>>> Subject: Re: 2.7.2 release plan
> > > >>>>>>>>
> > > >>>>>>>> Vinod and Chris,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for your reply. I'll do also backport not only bug
> fixes
> > > >> but
> > > >>>>>>>> also documentations(I think 2.7.2 includes them). It helps
> users
> > > >> a
> > > >>>> lot.
> > > >>>>>>>>
> > > >>>>>>>> Best,
> > > >>>>>>>> - Tsuyoshi
> > > >>>>>>>>
> > > >>>>>>>> On Tuesday, 27 October 2015, Vinod Vavilapalli <
> > > >>>>> [email protected]>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> +1.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks
> > > >>>>>>>>> +Vinod
> > > >>>>>>>>>
> > > >>>>>>>>>> On Jul 16, 2015, at 8:18 AM, Chris Nauroth <
> > > >>>> [email protected]
> > > >>>>>>>>> <javascript:;>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> I'd be comfortable with inclusion of any doc-only patch in
> > > >> minor
> > > >>>>>>>>> releases.
> > > >>>>>>>>>> There is a lot of value to end users in pushing
> documentation
> > > >>> fixes
> > > >>>>> as
> > > >>>>>>>>>> quickly as possible, and they don't bear the same risk of
> > > >>>>> regressions or
> > > >>>>>>>>>> incompatibilities as code changes.
> > > >>>>>>>>>>
> > > >>>>>>>>>> --Chris Nauroth
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On 7/16/15, 12:38 AM, "Tsuyoshi Ozawa" <[email protected]
> > > >>>>> <javascript:;>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> thank you for starting the discussion about 2.7.2 release.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> The focus obviously is to have blocker issues [2],
> bug-fixes
> > > >>> and
> > > >>>>> *no*
> > > >>>>>>>>>>> features / improvements.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I've committed YARN-3170, which is an improvement of
> > > >>>> documentation.
> > > >>>>> I
> > > >>>>>>>>>>> thought documentation pages which can be fit into
> branch-2.7
> > > >> can
> > > >>>> be
> > > >>>>>>>>>>> included easily. Should I revert it?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>> I need help from all committers in automatically
> > > >>>>>>>>>>> merging in any patch that fits the above criterion into
> 2.7.2
> > > >>>>> instead of
> > > >>>>>>>>>>> only on trunk or 2.8.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Sure, I'll try my best.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> That way we can include not only blocker but also critical
> > > >> bug
> > > >>>>> fixes to
> > > >>>>>>>>>>>> 2.7.2 release.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> As Vinod mentioned, we should also apply major bug fixes
> into
> > > >>>>>>>>> branch-2.7.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks,
> > > >>>>>>>>>>> - Tsuyoshi
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Thu, Jul 16, 2015 at 3:52 PM, Akira AJISAKA
> > > >>>>>>>>>>> <[email protected] <javascript:;>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>>>>> Thanks Vinod for starting 2.7.2 release plan.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> The focus obviously is to have blocker issues [2],
> > bug-fixes
> > > >>> and
> > > >>>>> *no*
> > > >>>>>>>>>>>>> features / improvements.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Can we adopt the plan as Karthik mentioned in "Additional
> > > >>>>> maintenance
> > > >>>>>>>>>>>> releases for Hadoop 2.y versions" thread? That way we can
> > > >>> include
> > > >>>>> not
> > > >>>>>>>>>>>> only
> > > >>>>>>>>>>>> blocker but also critical bug fixes to 2.7.2 release.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> In addition, branch-2.7 is a special case. (2.7.1 is the
> > > >> first
> > > >>>>> stable
> > > >>>>>>>>>>>> release) Therefore I'm thinking we can include major bug
> > > >> fixes
> > > >>> as
> > > >>>>> well.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>> Akira
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On 7/16/15 04:13, Vinod Kumar Vavilapalli wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks everyone for the push on 2.7.1! Branch-2.7 is now
> > > >> open
> > > >>>> for
> > > >>>>>>>>>>>>> commits
> > > >>>>>>>>>>>>> to a 2.7.2 release. JIRA also now has a 2.7.2 version for
> > > >> all
> > > >>>> the
> > > >>>>>>>>>>>>> sub-projects.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Continuing the previous 2.7.1 thread on steady
> maintenance
> > > >>>>> releases
> > > >>>>>>>>>>>>> [1],
> > > >>>>>>>>>>>>> we
> > > >>>>>>>>>>>>> should follow up 2.7.1 with a 2.7.2 within 4 weeks.
> Earlier
> > > >> I
> > > >>>>> tried a
> > > >>>>>>>>>>>>> 2-3
> > > >>>>>>>>>>>>> week cycle for 2.7.1, but it seems to be impractical
> given
> > > >> the
> > > >>>>>>>>>>>>> community
> > > >>>>>>>>>>>>> size. So, I propose we target a release by the end for 4
> > > >> weeks
> > > >>>>> from
> > > >>>>>>>>>>>>> now,
> > > >>>>>>>>>>>>> starting the release close-down within 2-3 weeks.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> The focus obviously is to have blocker issues [2],
> > bug-fixes
> > > >>> and
> > > >>>>> *no*
> > > >>>>>>>>>>>>> features / improvements. I need help from all committers
> in
> > > >>>>>>>>>>>>> automatically
> > > >>>>>>>>>>>>> merging in any patch that fits the above criterion into
> > > >> 2.7.2
> > > >>>>> instead
> > > >>>>>>>>>>>>> of
> > > >>>>>>>>>>>>> only on trunk or 2.8.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thoughts?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> +Vinod
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> [1] A 2.7.1 release to follow up 2.7.0
> > > >>>>>>>>>>>>> http://markmail.org/message/zwzze6cqqgwq4rmw
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> [2] 2.7.2 release blockers:
> > > >>>>>>>>>>>>> https://issues.apache.org/jira/issues/?filter=12332867
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> > >
> >
>
