Re: [DISCUSS] Looking to Apache Hadoop 3.1 release

[Vinod Kumar Vavilapalli]

Thanks for starting this thread, Wangda!

+1 for establishing a faster cadence now itself.

One word of caution though. The same I expressed while we were trying to do 
both 2.8 and 3.0 releases at the same time. Please try avoiding concurrent 
releases and splitting community bandwidth - it's not that we cannot do 
multiple releases in parallel, it's mainly that we will not be able to give our 
best on both.

Thanks
+Vinod

> On Sep 6, 2017, at 11:13 AM, Wangda Tan  wrote:
> 
> Hi all,
> 
> As we discussed on [1], there were proposals from Steve / Vinod etc to have
> a faster cadence of releases and to start thinking of a Hadoop 3.1 release
> earlier than March 2018 as is currently proposed.
> 
> I think this is a good idea. I'd like to start the process sooner, and
> establish timeline etc so that we can be ready when 3.0.0 GA is out. With
> this we can also establish faster cadence for future Hadoop 3.x releases.
> 
> To this end, I propose to target Hadoop 3.1.0 for a release by mid Jan
> 2018. (About 4.5 months from now and 2.5 months after 3.0-GA, instead of
> 6.5 months from now).
> 
> I'd also want to take this opportunity to come up with a more elaborate
> release plan to avoid some of the confusion we had with 3.0 beta. General
> proposal for the timeline (per this other proposal [2])
> - Feature freeze date - all features should be merged by Dec 15, 2017.
> - Code freeze date - blockers/critical only, no more improvements and non
> blocker/critical bug-fixes: Jan 1, 2018.
> - Release date: Jan 15, 2018
> 
> Following is a list of features on my radar which could be candidates for a
> 3.1 release:
> - YARN-5734, Dynamic scheduler queue configuration. (Owner: Jonathan Hung)
> - YARN-5881, Add absolute resource configuration to CapacityScheduler.
> (Owner: Sunil)
> - YARN-5673, Container-executor rewrite for better security, extensibility
> and portability. (Owner Varun Vasudev)
> - YARN-6223, GPU isolation. (Owner: Wangda)
> 
> And from email [3] mentioned by Andrew, there’re several other HDFS
> features want to be released with 3.1 as well, assuming they fit the
> timelines:
> - Storage Policy Satisfier
> - HDFS tiered storage
> 
> Please let me know if I missed any features targeted to 3.1 per this
> timeline.
> 
> And I want to volunteer myself as release manager of 3.1.0 release. Please
> let me know if you have any suggestions/concerns.
> 
> Thanks,
> Wangda Tan
> 
> [1] http://markmail.org/message/hwar5f5ap654ck5o?q=
> Branch+merges+and+3%2E0%2E0-beta1+scope
> [2] http://markmail.org/message/hwar5f5ap654ck5o?q=Branch+
> merges+and+3%2E0%2E0-beta1+scope#query:Branch%20merges%
> 20and%203.0.0-beta1%20scope+page:1+mid:2hqqkhl2dymcikf5+state:results
> [3] http://markmail.org/message/h35obzqrh3ag6dgn?q=Branch+merge
> s+and+3%2E0%2E0-beta1+scope


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Looking to Apache Hadoop 3.1 release

[Wangda Tan]

Thanks for all your valuable feedbacks.

Regarding to security issues for alpha features: I completely agree with
Larry: ideally, all alpha features should be disabled by default.

Steve/Arun/Haibo: could you please comment about feature's rough merge plan
and status (like alpha/beta).

I will wait several days to see if there are any other features wanted to
be added to 3.1 before creating initial release scope/plan on confluent.

Best,
Wangda


On Thu, Sep 7, 2017 at 12:14 PM, Haibo Chen  wrote:

> Thanks Wangda for initiating 3.1.0 release efforts. One YARN feature I'd
> like to add to 3.1.0 is YARN Oversubscription (YARN-1011)
>
> Best,
> Haibo
>
> On Wed, Sep 6, 2017 at 11:13 AM, Wangda Tan  wrote:
>
>> Hi all,
>>
>> As we discussed on [1], there were proposals from Steve / Vinod etc to
>> have
>> a faster cadence of releases and to start thinking of a Hadoop 3.1 release
>> earlier than March 2018 as is currently proposed.
>>
>> I think this is a good idea. I'd like to start the process sooner, and
>> establish timeline etc so that we can be ready when 3.0.0 GA is out. With
>> this we can also establish faster cadence for future Hadoop 3.x releases.
>>
>> To this end, I propose to target Hadoop 3.1.0 for a release by mid Jan
>> 2018. (About 4.5 months from now and 2.5 months after 3.0-GA, instead of
>> 6.5 months from now).
>>
>> I'd also want to take this opportunity to come up with a more elaborate
>> release plan to avoid some of the confusion we had with 3.0 beta. General
>> proposal for the timeline (per this other proposal [2])
>>  - Feature freeze date - all features should be merged by Dec 15, 2017.
>>  - Code freeze date - blockers/critical only, no more improvements and non
>> blocker/critical bug-fixes: Jan 1, 2018.
>>  - Release date: Jan 15, 2018
>>
>> Following is a list of features on my radar which could be candidates for
>> a
>> 3.1 release:
>> - YARN-5734, Dynamic scheduler queue configuration. (Owner: Jonathan Hung)
>> - YARN-5881, Add absolute resource configuration to CapacityScheduler.
>> (Owner: Sunil)
>> - YARN-5673, Container-executor rewrite for better security, extensibility
>> and portability. (Owner Varun Vasudev)
>> - YARN-6223, GPU isolation. (Owner: Wangda)
>>
>> And from email [3] mentioned by Andrew, there’re several other HDFS
>> features want to be released with 3.1 as well, assuming they fit the
>> timelines:
>> - Storage Policy Satisfier
>> - HDFS tiered storage
>>
>> Please let me know if I missed any features targeted to 3.1 per this
>> timeline.
>>
>> And I want to volunteer myself as release manager of 3.1.0 release. Please
>> let me know if you have any suggestions/concerns.
>>
>> Thanks,
>> Wangda Tan
>>
>> [1] http://markmail.org/message/hwar5f5ap654ck5o?q=
>> Branch+merges+and+3%2E0%2E0-beta1+scope
>> [2] http://markmail.org/message/hwar5f5ap654ck5o?q=Branch+
>> merges+and+3%2E0%2E0-beta1+scope#query:Branch%20merges%
>> 20and%203.0.0-beta1%20scope+page:1+mid:2hqqkhl2dymcikf5+state:results
>> [3] http://markmail.org/message/h35obzqrh3ag6dgn?q=Branch+merge
>> s+and+3%2E0%2E0-beta1+scope
>>
>
>

Re: [DISCUSS] Looking to Apache Hadoop 3.1 release

[larry mccay]

Hi Wangda -

Thank you for starting this conversation!

+1000 for a faster release cadence.
Quicker releases make turning around security fixes so much easier.

When we consider alpha features, let’s please ensure that they are not
delivered in a state that has known security issues and also make sure that
they are disabled by default. IMO - it is not a feature - alpha or
otherwise - unless it has some reasonable assurance of being secure. Please
don't see this as calling out any particular feature. I just think we need
to be very explicit about security expectations. Maybe this is already well
understood.

Thank you for this proposed plan and for volunteering!

—larry

On Wed, Sep 6, 2017 at 7:22 PM, Anu Engineer 
wrote:

> Hi Wangda,
>
> We are planning to start the Ozone merge discussion by the end of this
> month. I am hopeful that it will be merged pretty soon after that.
> Please add Ozone to the list of features that are being tracked for Apache
> Hadoop 3.1.
>
> We would love to release Ozone as an alpha feature in Hadoop 3.1.
>
> Thanks
> Anu
>
>
> On 9/6/17, 2:26 PM, "Arun Suresh"  wrote:
>
> >Thanks for starting this Wangda.
> >
> >I would also like to add:
> >- YARN-5972: Support Pausing/Freezing of opportunistic containers
> >
> >Cheers
> >-Arun
> >
> >On Wed, Sep 6, 2017 at 1:49 PM, Steve Loughran 
> >wrote:
> >
> >>
> >> > On 6 Sep 2017, at 19:13, Wangda Tan  wrote:
> >> >
> >> > Hi all,
> >> >
> >> > As we discussed on [1], there were proposals from Steve / Vinod etc to
> >> have
> >> > a faster cadence of releases and to start thinking of a Hadoop 3.1
> >> release
> >> > earlier than March 2018 as is currently proposed.
> >> >
> >> > I think this is a good idea. I'd like to start the process sooner, and
> >> > establish timeline etc so that we can be ready when 3.0.0 GA is out.
> With
> >> > this we can also establish faster cadence for future Hadoop 3.x
> releases.
> >> >
> >> > To this end, I propose to target Hadoop 3.1.0 for a release by mid Jan
> >> > 2018. (About 4.5 months from now and 2.5 months after 3.0-GA, instead
> of
> >> > 6.5 months from now).
> >> >
> >> > I'd also want to take this opportunity to come up with a more
> elaborate
> >> > release plan to avoid some of the confusion we had with 3.0 beta.
> General
> >> > proposal for the timeline (per this other proposal [2])
> >> > - Feature freeze date - all features should be merged by Dec 15, 2017.
> >> > - Code freeze date - blockers/critical only, no more improvements and
> non
> >> > blocker/critical bug-fixes: Jan 1, 2018.
> >> > - Release date: Jan 15, 2018
> >> >
> >> > Following is a list of features on my radar which could be candidates
> >> for a
> >> > 3.1 release:
> >> > - YARN-5734, Dynamic scheduler queue configuration. (Owner: Jonathan
> >> Hung)
> >> > - YARN-5881, Add absolute resource configuration to CapacityScheduler.
> >> > (Owner: Sunil)
> >> > - YARN-5673, Container-executor rewrite for better security,
> >> extensibility
> >> > and portability. (Owner Varun Vasudev)
> >> > - YARN-6223, GPU isolation. (Owner: Wangda)
> >> >
> >> > And from email [3] mentioned by Andrew, there’re several other HDFS
> >> > features want to be released with 3.1 as well, assuming they fit the
> >> > timelines:
> >> > - Storage Policy Satisfier
> >> > - HDFS tiered storage
> >> >
> >> > Please let me know if I missed any features targeted to 3.1 per this
> >> > timeline.
> >>
> >>
> >> HADOOP-13786 : S3Guard committer, which also adds resilience to failures
> >> talking to S3 (we barely have any today),
> >>
> >> >
> >> > And I want to volunteer myself as release manager of 3.1.0 release.
> >> Please
> >> > let me know if you have any suggestions/concerns.
> >>
> >> well volunteered :)
> >>
> >> >
> >> > Thanks,
> >> > Wangda Tan
> >> >
> >> > [1] http://markmail.org/message/hwar5f5ap654ck5o?q=
> >> > Branch+merges+and+3%2E0%2E0-beta1+scope
> >> > [2] http://markmail.org/message/hwar5f5ap654ck5o?q=Branch+
> >> > merges+and+3%2E0%2E0-beta1+scope#query:Branch%20merges%
> >> > 20and%203.0.0-beta1%20scope+page:1+mid:2hqqkhl2dymcikf5+state:results
> >> > [3] http://markmail.org/message/h35obzqrh3ag6dgn?q=Branch+merge
> >> > s+and+3%2E0%2E0-beta1+scope
>

Re: [DISCUSS] Looking to Apache Hadoop 3.1 release

[Anu Engineer]

Hi Wangda,

We are planning to start the Ozone merge discussion by the end of this month. I 
am hopeful that it will be merged pretty soon after that. 
Please add Ozone to the list of features that are being tracked for Apache 
Hadoop 3.1. 

We would love to release Ozone as an alpha feature in Hadoop 3.1.

Thanks
Anu


On 9/6/17, 2:26 PM, "Arun Suresh"  wrote:

>Thanks for starting this Wangda.
>
>I would also like to add:
>- YARN-5972: Support Pausing/Freezing of opportunistic containers
>
>Cheers
>-Arun
>
>On Wed, Sep 6, 2017 at 1:49 PM, Steve Loughran 
>wrote:
>
>>
>> > On 6 Sep 2017, at 19:13, Wangda Tan  wrote:
>> >
>> > Hi all,
>> >
>> > As we discussed on [1], there were proposals from Steve / Vinod etc to
>> have
>> > a faster cadence of releases and to start thinking of a Hadoop 3.1
>> release
>> > earlier than March 2018 as is currently proposed.
>> >
>> > I think this is a good idea. I'd like to start the process sooner, and
>> > establish timeline etc so that we can be ready when 3.0.0 GA is out. With
>> > this we can also establish faster cadence for future Hadoop 3.x releases.
>> >
>> > To this end, I propose to target Hadoop 3.1.0 for a release by mid Jan
>> > 2018. (About 4.5 months from now and 2.5 months after 3.0-GA, instead of
>> > 6.5 months from now).
>> >
>> > I'd also want to take this opportunity to come up with a more elaborate
>> > release plan to avoid some of the confusion we had with 3.0 beta. General
>> > proposal for the timeline (per this other proposal [2])
>> > - Feature freeze date - all features should be merged by Dec 15, 2017.
>> > - Code freeze date - blockers/critical only, no more improvements and non
>> > blocker/critical bug-fixes: Jan 1, 2018.
>> > - Release date: Jan 15, 2018
>> >
>> > Following is a list of features on my radar which could be candidates
>> for a
>> > 3.1 release:
>> > - YARN-5734, Dynamic scheduler queue configuration. (Owner: Jonathan
>> Hung)
>> > - YARN-5881, Add absolute resource configuration to CapacityScheduler.
>> > (Owner: Sunil)
>> > - YARN-5673, Container-executor rewrite for better security,
>> extensibility
>> > and portability. (Owner Varun Vasudev)
>> > - YARN-6223, GPU isolation. (Owner: Wangda)
>> >
>> > And from email [3] mentioned by Andrew, there’re several other HDFS
>> > features want to be released with 3.1 as well, assuming they fit the
>> > timelines:
>> > - Storage Policy Satisfier
>> > - HDFS tiered storage
>> >
>> > Please let me know if I missed any features targeted to 3.1 per this
>> > timeline.
>>
>>
>> HADOOP-13786 : S3Guard committer, which also adds resilience to failures
>> talking to S3 (we barely have any today),
>>
>> >
>> > And I want to volunteer myself as release manager of 3.1.0 release.
>> Please
>> > let me know if you have any suggestions/concerns.
>>
>> well volunteered :)
>>
>> >
>> > Thanks,
>> > Wangda Tan
>> >
>> > [1] http://markmail.org/message/hwar5f5ap654ck5o?q=
>> > Branch+merges+and+3%2E0%2E0-beta1+scope
>> > [2] http://markmail.org/message/hwar5f5ap654ck5o?q=Branch+
>> > merges+and+3%2E0%2E0-beta1+scope#query:Branch%20merges%
>> > 20and%203.0.0-beta1%20scope+page:1+mid:2hqqkhl2dymcikf5+state:results
>> > [3] http://markmail.org/message/h35obzqrh3ag6dgn?q=Branch+merge
>> > s+and+3%2E0%2E0-beta1+scope

Re: [DISCUSS] Looking to Apache Hadoop 3.1 release

[Arun Suresh]

Thanks for starting this Wangda.

I would also like to add:
- YARN-5972: Support Pausing/Freezing of opportunistic containers

Cheers
-Arun

On Wed, Sep 6, 2017 at 1:49 PM, Steve Loughran 
wrote:

>
> > On 6 Sep 2017, at 19:13, Wangda Tan  wrote:
> >
> > Hi all,
> >
> > As we discussed on [1], there were proposals from Steve / Vinod etc to
> have
> > a faster cadence of releases and to start thinking of a Hadoop 3.1
> release
> > earlier than March 2018 as is currently proposed.
> >
> > I think this is a good idea. I'd like to start the process sooner, and
> > establish timeline etc so that we can be ready when 3.0.0 GA is out. With
> > this we can also establish faster cadence for future Hadoop 3.x releases.
> >
> > To this end, I propose to target Hadoop 3.1.0 for a release by mid Jan
> > 2018. (About 4.5 months from now and 2.5 months after 3.0-GA, instead of
> > 6.5 months from now).
> >
> > I'd also want to take this opportunity to come up with a more elaborate
> > release plan to avoid some of the confusion we had with 3.0 beta. General
> > proposal for the timeline (per this other proposal [2])
> > - Feature freeze date - all features should be merged by Dec 15, 2017.
> > - Code freeze date - blockers/critical only, no more improvements and non
> > blocker/critical bug-fixes: Jan 1, 2018.
> > - Release date: Jan 15, 2018
> >
> > Following is a list of features on my radar which could be candidates
> for a
> > 3.1 release:
> > - YARN-5734, Dynamic scheduler queue configuration. (Owner: Jonathan
> Hung)
> > - YARN-5881, Add absolute resource configuration to CapacityScheduler.
> > (Owner: Sunil)
> > - YARN-5673, Container-executor rewrite for better security,
> extensibility
> > and portability. (Owner Varun Vasudev)
> > - YARN-6223, GPU isolation. (Owner: Wangda)
> >
> > And from email [3] mentioned by Andrew, there’re several other HDFS
> > features want to be released with 3.1 as well, assuming they fit the
> > timelines:
> > - Storage Policy Satisfier
> > - HDFS tiered storage
> >
> > Please let me know if I missed any features targeted to 3.1 per this
> > timeline.
>
>
> HADOOP-13786 : S3Guard committer, which also adds resilience to failures
> talking to S3 (we barely have any today),
>
> >
> > And I want to volunteer myself as release manager of 3.1.0 release.
> Please
> > let me know if you have any suggestions/concerns.
>
> well volunteered :)
>
> >
> > Thanks,
> > Wangda Tan
> >
> > [1] http://markmail.org/message/hwar5f5ap654ck5o?q=
> > Branch+merges+and+3%2E0%2E0-beta1+scope
> > [2] http://markmail.org/message/hwar5f5ap654ck5o?q=Branch+
> > merges+and+3%2E0%2E0-beta1+scope#query:Branch%20merges%
> > 20and%203.0.0-beta1%20scope+page:1+mid:2hqqkhl2dymcikf5+state:results
> > [3] http://markmail.org/message/h35obzqrh3ag6dgn?q=Branch+merge
> > s+and+3%2E0%2E0-beta1+scope
>
>

Re: [DISCUSS] Looking to Apache Hadoop 3.1 release

[Steve Loughran]

> On 6 Sep 2017, at 19:13, Wangda Tan  wrote:
> 
> Hi all,
> 
> As we discussed on [1], there were proposals from Steve / Vinod etc to have
> a faster cadence of releases and to start thinking of a Hadoop 3.1 release
> earlier than March 2018 as is currently proposed.
> 
> I think this is a good idea. I'd like to start the process sooner, and
> establish timeline etc so that we can be ready when 3.0.0 GA is out. With
> this we can also establish faster cadence for future Hadoop 3.x releases.
> 
> To this end, I propose to target Hadoop 3.1.0 for a release by mid Jan
> 2018. (About 4.5 months from now and 2.5 months after 3.0-GA, instead of
> 6.5 months from now).
> 
> I'd also want to take this opportunity to come up with a more elaborate
> release plan to avoid some of the confusion we had with 3.0 beta. General
> proposal for the timeline (per this other proposal [2])
> - Feature freeze date - all features should be merged by Dec 15, 2017.
> - Code freeze date - blockers/critical only, no more improvements and non
> blocker/critical bug-fixes: Jan 1, 2018.
> - Release date: Jan 15, 2018
> 
> Following is a list of features on my radar which could be candidates for a
> 3.1 release:
> - YARN-5734, Dynamic scheduler queue configuration. (Owner: Jonathan Hung)
> - YARN-5881, Add absolute resource configuration to CapacityScheduler.
> (Owner: Sunil)
> - YARN-5673, Container-executor rewrite for better security, extensibility
> and portability. (Owner Varun Vasudev)
> - YARN-6223, GPU isolation. (Owner: Wangda)
> 
> And from email [3] mentioned by Andrew, there’re several other HDFS
> features want to be released with 3.1 as well, assuming they fit the
> timelines:
> - Storage Policy Satisfier
> - HDFS tiered storage
> 
> Please let me know if I missed any features targeted to 3.1 per this
> timeline.


HADOOP-13786 : S3Guard committer, which also adds resilience to failures 
talking to S3 (we barely have any today),

> 
> And I want to volunteer myself as release manager of 3.1.0 release. Please
> let me know if you have any suggestions/concerns.

well volunteered :)

> 
> Thanks,
> Wangda Tan
> 
> [1] http://markmail.org/message/hwar5f5ap654ck5o?q=
> Branch+merges+and+3%2E0%2E0-beta1+scope
> [2] http://markmail.org/message/hwar5f5ap654ck5o?q=Branch+
> merges+and+3%2E0%2E0-beta1+scope#query:Branch%20merges%
> 20and%203.0.0-beta1%20scope+page:1+mid:2hqqkhl2dymcikf5+state:results
> [3] http://markmail.org/message/h35obzqrh3ag6dgn?q=Branch+merge
> s+and+3%2E0%2E0-beta1+scope

[DISCUSS] Looking to Apache Hadoop 3.1 release

[Wangda Tan]

Hi all,

As we discussed on [1], there were proposals from Steve / Vinod etc to have
a faster cadence of releases and to start thinking of a Hadoop 3.1 release
earlier than March 2018 as is currently proposed.

I think this is a good idea. I'd like to start the process sooner, and
establish timeline etc so that we can be ready when 3.0.0 GA is out. With
this we can also establish faster cadence for future Hadoop 3.x releases.

To this end, I propose to target Hadoop 3.1.0 for a release by mid Jan
2018. (About 4.5 months from now and 2.5 months after 3.0-GA, instead of
6.5 months from now).

I'd also want to take this opportunity to come up with a more elaborate
release plan to avoid some of the confusion we had with 3.0 beta. General
proposal for the timeline (per this other proposal [2])
 - Feature freeze date - all features should be merged by Dec 15, 2017.
 - Code freeze date - blockers/critical only, no more improvements and non
blocker/critical bug-fixes: Jan 1, 2018.
 - Release date: Jan 15, 2018

Following is a list of features on my radar which could be candidates for a
3.1 release:
- YARN-5734, Dynamic scheduler queue configuration. (Owner: Jonathan Hung)
- YARN-5881, Add absolute resource configuration to CapacityScheduler.
(Owner: Sunil)
- YARN-5673, Container-executor rewrite for better security, extensibility
and portability. (Owner Varun Vasudev)
- YARN-6223, GPU isolation. (Owner: Wangda)

And from email [3] mentioned by Andrew, there’re several other HDFS
features want to be released with 3.1 as well, assuming they fit the
timelines:
- Storage Policy Satisfier
- HDFS tiered storage

Please let me know if I missed any features targeted to 3.1 per this
timeline.

And I want to volunteer myself as release manager of 3.1.0 release. Please
let me know if you have any suggestions/concerns.

Thanks,
Wangda Tan

[1] http://markmail.org/message/hwar5f5ap654ck5o?q=
Branch+merges+and+3%2E0%2E0-beta1+scope
[2] http://markmail.org/message/hwar5f5ap654ck5o?q=Branch+
merges+and+3%2E0%2E0-beta1+scope#query:Branch%20merges%
20and%203.0.0-beta1%20scope+page:1+mid:2hqqkhl2dymcikf5+state:results
[3] http://markmail.org/message/h35obzqrh3ag6dgn?q=Branch+merge
s+and+3%2E0%2E0-beta1+scope