Hi All As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary patches are pulled into branch-3.2.
Thank You - Sunil On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sun...@apache.org> wrote: > Hi All > > We are now down to the last Blocker and HADOOP-15407 is merged to trunk. > Thanks for the support. > > *Plan for RC* > 3.2 branch cut and reset trunk : *25th Tuesday* > RC0 for 3.2: *28th Friday* > > Thank You > Sunil > > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sun...@apache.org> wrote: > >> Hi All >> >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for >> helping in this. I am following up on these tickets, once its closed we >> will cut the 3.2 branch. >> >> Thanks >> Sunil Govindan >> >> >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sun...@apache.org> wrote: >> >>> Hi All, >>> >>> Inline with the original 3.2 communication proposal dated 17th July >>> 2018, I would like to provide more updates. >>> >>> We are approaching previously proposed code freeze date (September 14, >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing >>> trunk to 3.3 if there are no issues. >>> >>> *Current Release Plan:* >>> Feature freeze date : all features to merge by September 7, 2018. >>> Code freeze date : blockers/critical only, no improvements and >>> blocker/critical bug-fixes September 14, 2018. >>> Release date: September 28, 2018 >>> >>> If any critical/blocker tickets which are targeted to 3.2.0, we need to >>> backport to 3.2 post branch cut. >>> >>> Here's an updated 3.2.0 feature status: >>> >>> 1. Merged & Completed features: >>> >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning >>> workloads Initial cut. >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity >>> Scheduler. >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API >>> and CLI. >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works. >>> >>> 2. Features close to finish: >>> >>> - (Steve) S3Guard Phase III. Close to commit. >>> - (Steve) S3a phase V. Close to commit. >>> - (Steve) Support Windows Azure Storage. Close to commit. >>> >>> 3. Tentative/Cancelled features for 3.2: >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from >>> ATSv2. Patch in progress. >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to >>> be done before Aug 2018. >>> - (Eric) YARN-7129: Application Catalog for YARN applications. >>> Challenging as more discussions are on-going. >>> >>> *Summary of 3.2.0 issues status:* >>> 19 Blocker and Critical issues [1] are open, I am following up with >>> owners to get status on each of them to get in by Code Freeze date. >>> >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER >>> BY priority DESC >>> >>> Thanks, >>> Sunil >>> >>> >>> >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sun...@apache.org> wrote: >>> >>>> Hi All, >>>> >>>> Inline with earlier communication dated 17th July 2018, I would like to >>>> provide some updates. >>>> >>>> We are approaching previously proposed code freeze date (Aug 31). >>>> >>>> One of the critical feature Node Attributes feature merge >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit more >>>> time. With regard to this, suggesting to push the feature/code freeze for 2 >>>> more weeks to accommodate these jiras too. >>>> >>>> Proposing Updated changes in plan inline with this: >>>> Feature freeze date : all features to merge by September 7, 2018. >>>> Code freeze date : blockers/critical only, no improvements and >>>> blocker/critical bug-fixes September 14, 2018. >>>> Release date: September 28, 2018 >>>> >>>> If any features in branch which are targeted to 3.2.0, please reply to >>>> this email thread. >>>> >>>> *Here's an updated 3.2.0 feature status:* >>>> >>>> 1. Merged & Completed features: >>>> >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning >>>> workloads Initial cut. >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity >>>> Scheduler. >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service >>>> API and CLI. >>>> >>>> 2. Features close to finish: >>>> >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote >>>> Ongoing. >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from >>>> ATSv2. Patch in progress. >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works. >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure >>>> Storage. In progress. >>>> >>>> 3. Tentative features: >>>> >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to >>>> be done before Aug 2018. >>>> - (Eric) YARN-7129: Application Catalog for YARN applications. >>>> Challenging as more discussions are on-going. >>>> >>>> *Summary of 3.2.0 issues status:* >>>> >>>> 26 Blocker and Critical issues [1] are open, I am following up with >>>> owners to get status on each of them to get in by Code Freeze date. >>>> >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" = >>>> 3.2.0 ORDER BY priority DESC >>>> >>>> Thanks, >>>> Sunil >>>> >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sun...@apache.org> wrote: >>>> >>>>> Hi All, >>>>> >>>>> Thanks for the feedbacks. Inline with earlier communication dated 17th >>>>> July 2018, I would like to provide some updates. >>>>> >>>>> We are approaching previously proposed feature freeze date (Aug 21, >>>>> about 7 days from today). >>>>> If any features in branch which are targeted to 3.2.0, please reply to >>>>> this email thread. >>>>> Steve has mentioned about the s3 features which will come close to >>>>> Code Freeze Date (Aug 31st). >>>>> >>>>> *Here's an updated 3.2.0 feature status:* >>>>> >>>>> 1. Merged & Completed features: >>>>> >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning >>>>> workloads Initial cut. >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier >>>>> >>>>> 2. Features close to finish: >>>>> >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major >>>>> patches are all in, only one last >>>>> patch is in review state. >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity >>>>> Scheduler. Close to commit. >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service >>>>> API and CLI. 2 patches are pending >>>>> which will be closed by Feature freeze date. >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from >>>>> ATSv2. Patch in progress. >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement >>>>> works. >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure >>>>> Storage. In progress. >>>>> >>>>> 3. Tentative features: >>>>> >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging >>>>> to be done before Aug 2018. >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications. >>>>> Challenging as more discussions are on-going. >>>>> >>>>> *Summary of 3.2.0 issues status:* >>>>> >>>>> 39 Blocker and Critical issues [1] are open, I am checking with owners >>>>> to get status on each of them to get in by Code Freeze date. >>>>> >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" = >>>>> 3.2.0 ORDER BY priority DESC >>>>> >>>>> Thanks, >>>>> Sunil >>>>> >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sun...@apache.org> wrote: >>>>> >>>>>> Thanks Subru for the thoughts. >>>>>> One of the main reason for a major release is to push out critical >>>>>> features with a faster cadence to the users. If we are pulling more and >>>>>> more different types of features to a minor release, that branch will >>>>>> become more destabilized and it may be tough to say that 3.1.2 is stable >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in >>>>>> subsequent minor release. >>>>>> For few companies, it makes sense to push out these new features >>>>>> faster to make a reach to the users. Adding to the point to the >>>>>> backporting >>>>>> issues, I agree that its a pain and we can workaround that with some git >>>>>> scripts. If we can make such scripts available to committers, backport >>>>>> will >>>>>> be seem-less across branches and we can achieve the faster release >>>>>> cadence >>>>>> also. >>>>>> >>>>>> Thoughts? >>>>>> >>>>>> - Sunil >>>>>> >>>>>> >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am >>>>>>> generally >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we >>>>>>> already >>>>>>> have too many branches to be maintained. I already see many commits >>>>>>> are in >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has >>>>>>> commits >>>>>>> which are absent in 3.0 etc. >>>>>>> >>>>>>> Additionally AFAIK 3.x has not been deployed in any major production >>>>>>> setting so the cost of adding features should be minimal. >>>>>>> >>>>>>> Thoughts? >>>>>>> >>>>>>> -Subru >>>>>>> >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sun...@apache.org> wrote: >>>>>>> >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts. >>>>>>> > >>>>>>> > Yes, important changes and features are much needed, hence we will >>>>>>> be >>>>>>> > keeping the door open for them as possible. Also considering few >>>>>>> more >>>>>>> > offline requests from other folks, I think extending the timeframe >>>>>>> by >>>>>>> > couple of weeks makes sense (including a second RC buffer) and >>>>>>> this should >>>>>>> > ideally help us to ship this by September itself. >>>>>>> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well) >>>>>>> > >>>>>>> > - Feature freeze date : all features to merge by August 21, 2018. >>>>>>> > >>>>>>> > - Code freeze date : blockers/critical only, no improvements and >>>>>>> non >>>>>>> > blocker/critical >>>>>>> > >>>>>>> > bug-fixes August 31, 2018. >>>>>>> > >>>>>>> > - Release date: September 15, 2018 >>>>>>> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your >>>>>>> questions. >>>>>>> > >>>>>>> > Thanks >>>>>>> > Sunil >>>>>>> > >>>>>>> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheele...@gmail.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for >>>>>>> that. >>>>>>> > > >>>>>>> > > To concerns from Steve, >>>>>>> > > >>>>>>> > > It is a good idea to keep the door open to get important changes >>>>>>> / >>>>>>> > > features in before cutoff. I would prefer to keep the proposed >>>>>>> release >>>>>>> > date >>>>>>> > > to make sure things can happen earlier instead of last minute >>>>>>> and we all >>>>>>> > > know that releases are always get delayed :). I'm also fine if >>>>>>> we want >>>>>>> > get >>>>>>> > > another several weeks time. >>>>>>> > > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before >>>>>>> thanksgiving. >>>>>>> > > Do you think is it good or too early / late? >>>>>>> > > >>>>>>> > > Eric, >>>>>>> > > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can >>>>>>> get merged >>>>>>> > > in time, we probably not need the YARN-8220. >>>>>>> > > >>>>>>> > > Sunil, >>>>>>> > > >>>>>>> > > Could u update >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/ >>>>>>> > Roadmap >>>>>>> > > with proposed plan as well? We can fill feature list first >>>>>>> before getting >>>>>>> > > consensus of time. >>>>>>> > > >>>>>>> > > Thanks, >>>>>>> > > Wangda >>>>>>> > > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri >>>>>>> <fab...@cloudera.com.invalid >>>>>>> > > >>>>>>> > > wrote: >>>>>>> > > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran < >>>>>>> ste...@hortonworks.com> >>>>>>> > >> wrote: >>>>>>> > >> >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sun...@apache.org<mailto: >>>>>>> > >> > sun...@apache.org>> wrote: >>>>>>> > >> > >>>>>>> > >> > I would also would like to take this opportunity to come up >>>>>>> with a >>>>>>> > >> detailed >>>>>>> > >> > plan. >>>>>>> > >> > >>>>>>> > >> > - Feature freeze date : all features should be merged by >>>>>>> August 10, >>>>>>> > >> 2018. >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > <snip> >>>>>>> > >> >>>>>>> > >> > >>>>>>> > >> > Please let me know if I missed any features targeted to 3.2 >>>>>>> per this >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard. >>>>>>> > >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226 >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220 >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen >>>>>>> 2 >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407 >>>>>>> > >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing work >>>>>>> on it, but >>>>>>> > >> if >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely >>>>>>> reviews on >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546 >>>>>>> > >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 & >>>>>>> S3Guard >>>>>>> > >> JIRAs >>>>>>> > >> > to see which of those are blocking. Then there are some >>>>>>> pressing >>>>>>> > "guave, >>>>>>> > >> > java 9 prep" >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> I can help with this part if you like. >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release manager >>>>>>> of 3.2.0 >>>>>>> > >> > release. >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > well volunteered! >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> Yes, thank you for stepping up. >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> > >>>>>>> > >> > I think this raises a good q: what timetable should we have >>>>>>> for the >>>>>>> > >> 3.2. & >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the >>>>>>> outline >>>>>>> > >> time >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less >>>>>>> concern about >>>>>>> > >> > things not making the 3.2 dealine >>>>>>> > >> > >>>>>>> > >> > -Steve >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> Good idea to mitigate the short deadline. >>>>>>> > >> >>>>>>> > >> -AF >>>>>>> > >> >>>>>>> > > >>>>>>> > >>>>>>> >>>>>>
