Re: Branch merges and 3.0.0-beta1 scope

[Vrushali C]

bq. I don't think it's a lot to ask that feature leads shoot an email to
the release manager of their target version. DISCUSS emails right before a
proposed merge VOTE are way too late, it ends up being a fire drill where
we need to scramble on many fronts.

Yes, I have been thinking about this too. Based on what I have observed,
[DISCUSS] is more of a late-stage communication typically just like 8-10
days before the [VOTE] goes out. Most the leads send out [DISCUSS] emails
when the feature is extremely close to completion. So it seems to me that
there is an inherent hesitation in sending out [DISCUSS] emails when things
are not close to completion, so the Release Manager does not hear about
these early on.

Perhaps we want to add another step like [HEADS-UP] which goes out to at
least the Release Manager and known stakeholders or perhaps even the entire
dev community well before [DISCUSS].

The  [HEADS-UP] could be a very simple note that basically says we are
still working on this but we wish to aim for so-and-so release. It could go
out as soon as release idea is floated but perhaps no later than 7-8 weeks
before planned release date. [HEADS-UP] indicates that despite
contributors' best efforts, there exists a possibility that that feature
may not finish in time to make it into the release but at least the RM is
aware of the intentions and can track these.

Perhaps a time frame like the following may be good to follow:
[HEADS-UP]: typically at least 2 months before release date or as early as
anyone wants to communicate to RM.
[DISCUSS]: approx 4-6 weeks before release date
[VOTE]: closes before 2 (or 3?) weeks before release date

While I do think some timeframes like these should become the norm, we may
not want to be too rigidly pedantic as well. It may be a good idea to keep
ourselves open to making exceptions on a per case basis.

thanks
Vrushali



On Tue, Aug 29, 2017 at 2:25 PM, Andrew Wang 
wrote:

> Hi Vinod,
>
> On Fri, Aug 25, 2017 at 2:42 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> > From a release management perspective, it's *extremely* reasonable to
>> block the inclusion of new features a month from the planned release date.
>> A typical software development lifecycle includes weeks of feature freeze
>> and weeks of code freeze. It is no knock on any developer or any feature to
>> say that we should not include something in 3.0.0.
>>
>>
>> We have never followed the ‘typical' lifecycle that I am guessing you are
>> referring to. If we are, you'll need to publish some of the following: a
>> feature freeze date, blockers-criticals-only-from-now date,
>> testing-finish date, documentation-finish date, final release date and so
>> on.
>>
>
> We discussed this as part of the 3.0 alpha/beta/GA plan. The point of the
> extended alpha/beta process was to release on a schedule. Things that
> weren't ready could be merged for the next alpha. I also advertised alpha4
> as feature complete and beta1 as code complete so we could quickly move on
> to GA.
>
>
>> What we do with Apache releases typically is instead we say ‘this' is
>> roughly when we want to release, and roughly what features must land and
>> let the rest figure out itself.
>>
>> We did this too. We defined the original scope for 3.0.0 GA way back when
> we started the 3.0.0 release process. I've been writing status updates on
> the wiki and tracking targeted features and release blockers throughout.
>
> The target versions of this recent batch of features were not discussed
> with me, the release manager, until just recently. After some discussion, I
> think we've arrived at a release plan that everyone's happy with. But, I
> want to be clear that late-breaking inclusion of additional scope should be
> considered the exception rather than the norm. Merging code so close to
> release means less time for testing and validation, which means lower
> quality releases.
>
> I don't think it's a lot to ask that feature leads shoot an email to the
> release manager of their target version. DISCUSS emails right before a
> proposed merge VOTE are way too late, it ends up being a fire drill where
> we need to scramble on many fronts.
>
>
>> Neither is right or wrong. If we want to change the process, we should
>> communicate as such.
>>
>> Proposing a feature freeze date on the fly is only going to confuse
>> people.
>>
>
>> > I've been very open and clear about the goals, schedule, and scope of
>> 3.0.0 over the last year plus. The point of the extended alpha process was
>> to get all our features in during alpha, and the alpha merge window has
>> been open for a year. I'm unmoved by arguments about how long a feature has
>> been worked on. None of these were not part of the original 3.0.0 scope,
>> and our users have been waiting even longer for big-ticket 3.0 items like
>> JDK8 and HDFS EC that were part of the discussed scope.
>>
>>
>> Except our schedule is so fluid (not due to the release 

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Hi Vinod,

On Fri, Aug 25, 2017 at 2:42 PM, Vinod Kumar Vavilapalli  wrote:

> > From a release management perspective, it's *extremely* reasonable to
> block the inclusion of new features a month from the planned release date.
> A typical software development lifecycle includes weeks of feature freeze
> and weeks of code freeze. It is no knock on any developer or any feature to
> say that we should not include something in 3.0.0.
>
>
> We have never followed the ‘typical' lifecycle that I am guessing you are
> referring to. If we are, you'll need to publish some of the following: a
> feature freeze date, blockers-criticals-only-from-now date,
> testing-finish date, documentation-finish date, final release date and so
> on.
>

We discussed this as part of the 3.0 alpha/beta/GA plan. The point of the
extended alpha/beta process was to release on a schedule. Things that
weren't ready could be merged for the next alpha. I also advertised alpha4
as feature complete and beta1 as code complete so we could quickly move on
to GA.


> What we do with Apache releases typically is instead we say ‘this' is
> roughly when we want to release, and roughly what features must land and
> let the rest figure out itself.
>
> We did this too. We defined the original scope for 3.0.0 GA way back when
we started the 3.0.0 release process. I've been writing status updates on
the wiki and tracking targeted features and release blockers throughout.

The target versions of this recent batch of features were not discussed
with me, the release manager, until just recently. After some discussion, I
think we've arrived at a release plan that everyone's happy with. But, I
want to be clear that late-breaking inclusion of additional scope should be
considered the exception rather than the norm. Merging code so close to
release means less time for testing and validation, which means lower
quality releases.

I don't think it's a lot to ask that feature leads shoot an email to the
release manager of their target version. DISCUSS emails right before a
proposed merge VOTE are way too late, it ends up being a fire drill where
we need to scramble on many fronts.


> Neither is right or wrong. If we want to change the process, we should
> communicate as such.
>
> Proposing a feature freeze date on the fly is only going to confuse
> people.
>

> > I've been very open and clear about the goals, schedule, and scope of
> 3.0.0 over the last year plus. The point of the extended alpha process was
> to get all our features in during alpha, and the alpha merge window has
> been open for a year. I'm unmoved by arguments about how long a feature has
> been worked on. None of these were not part of the original 3.0.0 scope,
> and our users have been waiting even longer for big-ticket 3.0 items like
> JDK8 and HDFS EC that were part of the discussed scope.
>
>
> Except our schedule is so fluid (not due to the release management process
> to be fair) that it is hard for folks to plan their features. IIRC, our
> schedule was a GA release beginning of this year. Again, this is not a
> critique of 3.0 release process - I have myself done enough releases to
> know that sticking to a date and herding the crowd has been an extremely
> hard job.
>
>
Schedules have been fluid because we don't know when features are getting
in, and there's an unwillingness to bump features to the next release. The
goal of the 3.x alphas and betas was to break out of this release
anti-pattern, and release on a schedule.

There have been schedule delays during the 3.x alphas, but I'm still proud
that we released 4 alphas in 10 months. I'm doing my best to stick to our
published schedule, and add a beta and GA to that list by EOY.

Best,
Andrew

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Jonathan, thanks for the heads up. I don't have much familiarity with YARN,
but gave the PBs and pom changes a look, and left a few small comments on
the umbrella JIRA.

This seems like a smaller change than some of the other branch merges we're
discussing, but I'm again reticent about adding scope if we can avoid it.

In your mind, is this truly a "must-have" for 3.0? It looks compatible, and
thus something we could add in a minor release like 2.9 or 3.1.

Best,
Andrew

On Fri, Aug 25, 2017 at 12:31 PM, Jonathan Hung 
wrote:

> Hi Andrew,
>
> Thanks for starting the discussion - we have a feature YARN-5734 for API
> based scheduler configuration that I feel is pretty close to merge (also "a
> few weeks"). It's almost completely code and API additions and we were
> careful to design it so that it's compatible (feature is also turned off by
> default). Hoping to get this in before 3.0.0-GA. Just wanted to send this
> note so that we are not caught off guard by this feature.
>
> Thanks!
>
>
> Jonathan Hung
>
> On Fri, Aug 25, 2017 at 11:06 AM, Wangda Tan  wrote:
>
>> Resource profile is similar to TSv2, the feature is:
>> - Alpha feature, we will not freeze new added APIs. And all added APIs are
>> explicitly marked to @Unstable.
>> - Allow rolling upgrade from branch-2.
>> - Touched existing code, but we have, and will continue tests to make sure
>> changes are safe.
>>
>> Discussed with Andrew offline, we decided to not put this to beta1 since
>> beta1 is not far away. But we want to put it before GA if sufficient tests
>> are done.
>>
>> Thanks,
>> Wangda
>>
>>
>>
>> On Fri, Aug 25, 2017 at 10:54 AM, Rohith Sharma K S <
>> rohithsharm...@apache.org> wrote:
>>
>> > On 25 August 2017 at 22:39, Andrew Wang 
>> wrote:
>> >
>> > > Hi Rohith,
>> > >
>> > > Given that we're advertising TSv2 as an alpha feature, I think we're
>> > > allowed to break compatibility. Let's make sure this is clear in the
>> > > release notes and documentation.
>> > >
>> >
>> > > That said, with TSv2 phase 2, is the API going to be frozen? The
>> umbrella
>> > > JIRA refers to "TSv2 alpha2" which indicated to me it was still
>> > alpha-level
>> > > quality and stability.
>> > >
>> > YES, We have decided to freeze API's. I do not think we make any
>> > compatibility break in future.
>> >
>> >
>> >
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> >
>>
>
>

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Here's a summary of some 1-on-1 conversations I had with contributors of
the different features I'm tracking.

Storage Policy Satisfier (Uma Gangumalla)
* Target version: 3.1.0, maybe 3.0.0 GA
* We're resolving some design discussions on JIRA. Plan is to do some MVP
work on the API to get this into 3.1, and if we're happy with the second
phase, consider for 3.0 GA.

YARN native services (Jian He)
* Target version: 3.0.0-beta1 as an alpha feature
* I was convinced that this is very separate from the core. I'll get
someone from Cloudera to run it through our integration tests to verify it
doesn't break anything downstream, then happy to merge.

Resource profiles (Wangda Tan)
* Target version: 3.0.0 GA
* Already provided update above, we're going to test it a lot and target
for GA.

HDFS router-based federation (Chris Douglas)
* Target version: 3.0.0 GA
* This is like YARN federation, very separate and doesn't add new APIs, run
in production.
* If it passes our internal integration testing, I'm fine putting this in
late.

HDFS tiered storage (Chris Douglas):
* Target version: 3.1.0
* This touches some core stuff, and the write path is still being worked
on. Still somewhat useful with just the read path. Targeting at 3.1.0 gives
enough time to wrap this up.

TSv2 phase 2 (Vrushali C)
* Target version: 3.0.0-beta1
* This is more like "beta" in terms of readiness, Twitter is planning to
roll it out to production.
* I double checked with Haibo, and he successfully ran it through our
internal integration testing.

Thanks to everyone for meeting with me on short notice, and being very
reasonable about target versions and quality bars. If I mischaracterized
any of our discussions, please reach out or comment.

The branching and versioning discussion is still proceeding. I'd ask that
the pending merge VOTEs watch this carefully; I'm hoping to resolve the
discussion and branch before the VOTEs close, but let's make sure the
branches and versions are ready before doing the actual merges.

Thanks,
Andrew

On Fri, Aug 25, 2017 at 11:06 AM, Wangda Tan  wrote:

> Resource profile is similar to TSv2, the feature is:
> - Alpha feature, we will not freeze new added APIs. And all added APIs are
> explicitly marked to @Unstable.
> - Allow rolling upgrade from branch-2.
> - Touched existing code, but we have, and will continue tests to make sure
> changes are safe.
>
> Discussed with Andrew offline, we decided to not put this to beta1 since
> beta1 is not far away. But we want to put it before GA if sufficient tests
> are done.
>
> Thanks,
> Wangda
>
>
>
> On Fri, Aug 25, 2017 at 10:54 AM, Rohith Sharma K S <
> rohithsharm...@apache.org> wrote:
>
>> On 25 August 2017 at 22:39, Andrew Wang  wrote:
>>
>> > Hi Rohith,
>> >
>> > Given that we're advertising TSv2 as an alpha feature, I think we're
>> > allowed to break compatibility. Let's make sure this is clear in the
>> > release notes and documentation.
>> >
>>
>> > That said, with TSv2 phase 2, is the API going to be frozen? The
>> umbrella
>> > JIRA refers to "TSv2 alpha2" which indicated to me it was still
>> alpha-level
>> > quality and stability.
>> >
>> YES, We have decided to freeze API's. I do not think we make any
>> compatibility break in future.
>>
>>
>>
>> >
>> > Best,
>> > Andrew
>> >
>>
>
>

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

> From a release management perspective, it's *extremely* reasonable to block 
> the inclusion of new features a month from the planned release date. A 
> typical software development lifecycle includes weeks of feature freeze and 
> weeks of code freeze. It is no knock on any developer or any feature to say 
> that we should not include something in 3.0.0.


We have never followed the ‘typical' lifecycle that I am guessing you are 
referring to. If we are, you'll need to publish some of the following: a 
feature freeze date, blockers-criticals-only-from-now date, testing-finish 
date, documentation-finish date, final release date and so on.

What we do with Apache releases typically is instead we say ‘this' is roughly 
when we want to release, and roughly what features must land and let the rest 
figure out itself.

Neither is right or wrong. If we want to change the process, we should 
communicate as such.

Proposing a feature freeze date on the fly is only going to confuse people.


> I've been very open and clear about the goals, schedule, and scope of 3.0.0 
> over the last year plus. The point of the extended alpha process was to get 
> all our features in during alpha, and the alpha merge window has been open 
> for a year. I'm unmoved by arguments about how long a feature has been worked 
> on. None of these were not part of the original 3.0.0 scope, and our users 
> have been waiting even longer for big-ticket 3.0 items like JDK8 and HDFS EC 
> that were part of the discussed scope.


Except our schedule is so fluid (not due to the release management process to 
be fair) that it is hard for folks to plan their features. IIRC, our schedule 
was a GA release beginning of this year. Again, this is not a critique of 3.0 
release process - I have myself done enough releases to know that sticking to a 
date and herding the crowd has been an extremely hard job.

The only way out of this is get 3.0 out and stick to 1-2 month minor releases. 
May be we start that by planning a 3.1 right now and push one in January 
assuming 3.0 GA happens in November. Cadence is a habit.


> I see that two VOTEs have gone out since I was out. I still plan to follow 
> the proposal in my original email. This means I'll cut branch-3 and 
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open up 
> development for Hadoop 3.1.0 and 4.0.0.
> 
> I'm reaching out to the lead contributor of each of these features 
> individually to discuss. We need to close on this quickly, and email is too 
> low bandwidth at this stage.


My only concern in all of this is if some of these branch contributors decide 
that they don’t want and then proceed to having a parallel release and cause 
pains for everyone. This is what I tried avoiding parallel 2.9 and 2.10 offline 
and that’s what I am trying to do here. For now, I don’t have a horse in this 
race - that’s between you and the branch-contributors in question for now. If 
you can reach an agreement, we are all good for it.

+Vinod



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Jonathan Hung]

Hi Andrew,

Thanks for starting the discussion - we have a feature YARN-5734 for API
based scheduler configuration that I feel is pretty close to merge (also "a
few weeks"). It's almost completely code and API additions and we were
careful to design it so that it's compatible (feature is also turned off by
default). Hoping to get this in before 3.0.0-GA. Just wanted to send this
note so that we are not caught off guard by this feature.

Thanks!


Jonathan Hung

On Fri, Aug 25, 2017 at 11:06 AM, Wangda Tan  wrote:

> Resource profile is similar to TSv2, the feature is:
> - Alpha feature, we will not freeze new added APIs. And all added APIs are
> explicitly marked to @Unstable.
> - Allow rolling upgrade from branch-2.
> - Touched existing code, but we have, and will continue tests to make sure
> changes are safe.
>
> Discussed with Andrew offline, we decided to not put this to beta1 since
> beta1 is not far away. But we want to put it before GA if sufficient tests
> are done.
>
> Thanks,
> Wangda
>
>
>
> On Fri, Aug 25, 2017 at 10:54 AM, Rohith Sharma K S <
> rohithsharm...@apache.org> wrote:
>
> > On 25 August 2017 at 22:39, Andrew Wang 
> wrote:
> >
> > > Hi Rohith,
> > >
> > > Given that we're advertising TSv2 as an alpha feature, I think we're
> > > allowed to break compatibility. Let's make sure this is clear in the
> > > release notes and documentation.
> > >
> >
> > > That said, with TSv2 phase 2, is the API going to be frozen? The
> umbrella
> > > JIRA refers to "TSv2 alpha2" which indicated to me it was still
> > alpha-level
> > > quality and stability.
> > >
> > YES, We have decided to freeze API's. I do not think we make any
> > compatibility break in future.
> >
> >
> >
> > >
> > > Best,
> > > Andrew
> > >
> >
>

Re: Branch merges and 3.0.0-beta1 scope

[Wangda Tan]

Resource profile is similar to TSv2, the feature is:
- Alpha feature, we will not freeze new added APIs. And all added APIs are
explicitly marked to @Unstable.
- Allow rolling upgrade from branch-2.
- Touched existing code, but we have, and will continue tests to make sure
changes are safe.

Discussed with Andrew offline, we decided to not put this to beta1 since
beta1 is not far away. But we want to put it before GA if sufficient tests
are done.

Thanks,
Wangda



On Fri, Aug 25, 2017 at 10:54 AM, Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> On 25 August 2017 at 22:39, Andrew Wang  wrote:
>
> > Hi Rohith,
> >
> > Given that we're advertising TSv2 as an alpha feature, I think we're
> > allowed to break compatibility. Let's make sure this is clear in the
> > release notes and documentation.
> >
>
> > That said, with TSv2 phase 2, is the API going to be frozen? The umbrella
> > JIRA refers to "TSv2 alpha2" which indicated to me it was still
> alpha-level
> > quality and stability.
> >
> YES, We have decided to freeze API's. I do not think we make any
> compatibility break in future.
>
>
>
> >
> > Best,
> > Andrew
> >
>

Re: Branch merges and 3.0.0-beta1 scope

[Rohith Sharma K S]

On 25 August 2017 at 22:39, Andrew Wang  wrote:

> Hi Rohith,
>
> Given that we're advertising TSv2 as an alpha feature, I think we're
> allowed to break compatibility. Let's make sure this is clear in the
> release notes and documentation.
>

> That said, with TSv2 phase 2, is the API going to be frozen? The umbrella
> JIRA refers to "TSv2 alpha2" which indicated to me it was still alpha-level
> quality and stability.
>
YES, We have decided to freeze API's. I do not think we make any
compatibility break in future.



>
> Best,
> Andrew
>

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Hi Jason,

I agree with this proposal. I'll start another email thread spelling this
out, and gather additional feedback.

Best,
Andrew

On Fri, Aug 25, 2017 at 6:27 AM, Jason Lowe  wrote:

> Andrew Wang wrote:
>
>
>> This means I'll cut branch-3 and
>> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
>> up development for Hadoop 3.1.0 and 4.0.0.
>
>
> I can see a need for branch-3.0, but please do not create branch-3.  Doing
> so will relegate trunk back to the "patch purgatory" branch, a place where
> patches won't see a release for years.  Unless something is imminently
> going in that will break backwards compatibility and warrant a new 4.x
> release, I don't see the need to distinguish trunk from the 3.x line.
> Leaving trunk as the 3.x line means less branches to commit patches through
> and more testing of every patch since trunk would remain an active area for
> testing and releasing.  If we separate trunk and branch-3 then it's almost
> certain only-trunk patches will start to accumulate and never get any
> "real" testing until someone eventually decides it's time to go to Hadoop
> 4.x.  Looking back at trunk-as-3.x for an example, patches committed there
> in the early days after branch-2 was cut didn't see a release for almost 6
> years.
>
> My apologies if I've missed a feature that is just going to miss the 3.0
> release and will break compatibility when it goes in.  If so then we need
> to cut branch-3, but if not then here's my plea to hold off until we do
> need it.
>
> Jason
>
>
> On Thu, Aug 24, 2017 at 3:33 PM, Andrew Wang 
> wrote:
>
>> Glad to see the discussion continued in my absence :)
>>
>> From a release management perspective, it's *extremely* reasonable to
>> block
>> the inclusion of new features a month from the planned release date. A
>> typical software development lifecycle includes weeks of feature freeze
>> and
>> weeks of code freeze. It is no knock on any developer or any feature to
>> say
>> that we should not include something in 3.0.0.
>>
>> I've been very open and clear about the goals, schedule, and scope of
>> 3.0.0
>> over the last year plus. The point of the extended alpha process was to
>> get
>> all our features in during alpha, and the alpha merge window has been open
>> for a year. I'm unmoved by arguments about how long a feature has been
>> worked on. None of these were not part of the original 3.0.0 scope, and
>> our
>> users have been waiting even longer for big-ticket 3.0 items like JDK8 and
>> HDFS EC that were part of the discussed scope.
>>
>> I see that two VOTEs have gone out since I was out. I still plan to follow
>> the proposal in my original email. This means I'll cut branch-3 and
>> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
>> up development for Hadoop 3.1.0 and 4.0.0.
>>
>> I'm reaching out to the lead contributor of each of these features
>> individually to discuss. We need to close on this quickly, and email is
>> too
>> low bandwidth at this stage.
>>
>> Best,
>> Andrew
>>
>
>

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Hi Rohith,

Given that we're advertising TSv2 as an alpha feature, I think we're
allowed to break compatibility. Let's make sure this is clear in the
release notes and documentation.

That said, with TSv2 phase 2, is the API going to be frozen? The umbrella
JIRA refers to "TSv2 alpha2" which indicated to me it was still alpha-level
quality and stability.

Best,
Andrew

On Thu, Aug 24, 2017 at 11:47 PM, Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> Hi Andrew
>
> Thanks for update on release plan!
>
> I would like to discuss specifically regarding compatibility of releases.
> What is the compatibility to be maintained for GA if we don't merge to
> beta1 release? IIUC, till now all the releases were alpha where
> compatibility was not that important. All the public interfaces are
> subjected to modifications. Once we release beta, compatibility would be a
> matter.
> During this gap i.e between beta-GA release, should we maintain
> compatibility ?
> If my understanding is right then TSv2 have to be merged with beta1
> release. In TSv2 phase-2, we have compatibility changes from phase-1.
>
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 25 August 2017 at 02:03, Andrew Wang  wrote:
>
> > Glad to see the discussion continued in my absence :)
> >
> > From a release management perspective, it's *extremely* reasonable to
> block
> > the inclusion of new features a month from the planned release date. A
> > typical software development lifecycle includes weeks of feature freeze
> and
> > weeks of code freeze. It is no knock on any developer or any feature to
> say
> > that we should not include something in 3.0.0.
> >
> > I've been very open and clear about the goals, schedule, and scope of
> 3.0.0
> > over the last year plus. The point of the extended alpha process was to
> get
> > all our features in during alpha, and the alpha merge window has been
> open
> > for a year. I'm unmoved by arguments about how long a feature has been
> > worked on. None of these were not part of the original 3.0.0 scope, and
> our
> > users have been waiting even longer for big-ticket 3.0 items like JDK8
> and
> > HDFS EC that were part of the discussed scope.
> >
> > I see that two VOTEs have gone out since I was out. I still plan to
> follow
> > the proposal in my original email. This means I'll cut branch-3 and
> > branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will
> open
> > up development for Hadoop 3.1.0 and 4.0.0.
> >
> > I'm reaching out to the lead contributor of each of these features
> > individually to discuss. We need to close on this quickly, and email is
> too
> > low bandwidth at this stage.
> >
> > Best,
> > Andrew
> >
>

Re: Branch merges and 3.0.0-beta1 scope

[Jason Lowe]

Andrew Wang wrote:


> This means I'll cut branch-3 and
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
> up development for Hadoop 3.1.0 and 4.0.0.


I can see a need for branch-3.0, but please do not create branch-3.  Doing
so will relegate trunk back to the "patch purgatory" branch, a place where
patches won't see a release for years.  Unless something is imminently
going in that will break backwards compatibility and warrant a new 4.x
release, I don't see the need to distinguish trunk from the 3.x line.
Leaving trunk as the 3.x line means less branches to commit patches through
and more testing of every patch since trunk would remain an active area for
testing and releasing.  If we separate trunk and branch-3 then it's almost
certain only-trunk patches will start to accumulate and never get any
"real" testing until someone eventually decides it's time to go to Hadoop
4.x.  Looking back at trunk-as-3.x for an example, patches committed there
in the early days after branch-2 was cut didn't see a release for almost 6
years.

My apologies if I've missed a feature that is just going to miss the 3.0
release and will break compatibility when it goes in.  If so then we need
to cut branch-3, but if not then here's my plea to hold off until we do
need it.

Jason


On Thu, Aug 24, 2017 at 3:33 PM, Andrew Wang 
wrote:

> Glad to see the discussion continued in my absence :)
>
> From a release management perspective, it's *extremely* reasonable to block
> the inclusion of new features a month from the planned release date. A
> typical software development lifecycle includes weeks of feature freeze and
> weeks of code freeze. It is no knock on any developer or any feature to say
> that we should not include something in 3.0.0.
>
> I've been very open and clear about the goals, schedule, and scope of 3.0.0
> over the last year plus. The point of the extended alpha process was to get
> all our features in during alpha, and the alpha merge window has been open
> for a year. I'm unmoved by arguments about how long a feature has been
> worked on. None of these were not part of the original 3.0.0 scope, and our
> users have been waiting even longer for big-ticket 3.0 items like JDK8 and
> HDFS EC that were part of the discussed scope.
>
> I see that two VOTEs have gone out since I was out. I still plan to follow
> the proposal in my original email. This means I'll cut branch-3 and
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
> up development for Hadoop 3.1.0 and 4.0.0.
>
> I'm reaching out to the lead contributor of each of these features
> individually to discuss. We need to close on this quickly, and email is too
> low bandwidth at this stage.
>
> Best,
> Andrew
>

Re: Branch merges and 3.0.0-beta1 scope

[Rohith Sharma K S]

Hi Andrew

Thanks for update on release plan!

I would like to discuss specifically regarding compatibility of releases.
What is the compatibility to be maintained for GA if we don't merge to
beta1 release? IIUC, till now all the releases were alpha where
compatibility was not that important. All the public interfaces are
subjected to modifications. Once we release beta, compatibility would be a
matter.
During this gap i.e between beta-GA release, should we maintain
compatibility ?
If my understanding is right then TSv2 have to be merged with beta1
release. In TSv2 phase-2, we have compatibility changes from phase-1.


Thanks & Regards
Rohith Sharma K S

On 25 August 2017 at 02:03, Andrew Wang  wrote:

> Glad to see the discussion continued in my absence :)
>
> From a release management perspective, it's *extremely* reasonable to block
> the inclusion of new features a month from the planned release date. A
> typical software development lifecycle includes weeks of feature freeze and
> weeks of code freeze. It is no knock on any developer or any feature to say
> that we should not include something in 3.0.0.
>
> I've been very open and clear about the goals, schedule, and scope of 3.0.0
> over the last year plus. The point of the extended alpha process was to get
> all our features in during alpha, and the alpha merge window has been open
> for a year. I'm unmoved by arguments about how long a feature has been
> worked on. None of these were not part of the original 3.0.0 scope, and our
> users have been waiting even longer for big-ticket 3.0 items like JDK8 and
> HDFS EC that were part of the discussed scope.
>
> I see that two VOTEs have gone out since I was out. I still plan to follow
> the proposal in my original email. This means I'll cut branch-3 and
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
> up development for Hadoop 3.1.0 and 4.0.0.
>
> I'm reaching out to the lead contributor of each of these features
> individually to discuss. We need to close on this quickly, and email is too
> low bandwidth at this stage.
>
> Best,
> Andrew
>

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Glad to see the discussion continued in my absence :)

>From a release management perspective, it's *extremely* reasonable to block
the inclusion of new features a month from the planned release date. A
typical software development lifecycle includes weeks of feature freeze and
weeks of code freeze. It is no knock on any developer or any feature to say
that we should not include something in 3.0.0.

I've been very open and clear about the goals, schedule, and scope of 3.0.0
over the last year plus. The point of the extended alpha process was to get
all our features in during alpha, and the alpha merge window has been open
for a year. I'm unmoved by arguments about how long a feature has been
worked on. None of these were not part of the original 3.0.0 scope, and our
users have been waiting even longer for big-ticket 3.0 items like JDK8 and
HDFS EC that were part of the discussed scope.

I see that two VOTEs have gone out since I was out. I still plan to follow
the proposal in my original email. This means I'll cut branch-3 and
branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
up development for Hadoop 3.1.0 and 4.0.0.

I'm reaching out to the lead contributor of each of these features
individually to discuss. We need to close on this quickly, and email is too
low bandwidth at this stage.

Best,
Andrew

Re: Branch merges and 3.0.0-beta1 scope

[Ravi Prakash]

Also, when people +1 a merge, can they please describe if they did testing
/ use the feature in addition to what is already described in the thread?

On Wed, Aug 23, 2017 at 11:18 AM, Vrushali Channapattan <
vrushalic2...@gmail.com> wrote:

> For timeline service v2, we have completed all subtasks under YARN-5355
> [1].
>
> We initiated a merge-to-trunk vote [2] yesterday.
>
> thanks
> Vrushali
> [1] https://issues.apache.org/jira/browse/YARN-5355
> [2]
> http://mail-archives.apache.org/mod_mbox/hadoop-common-
> dev/201708.mbox/%3CCAE=b_fbLT2J+Ezb4wqdN_UwBiG1Sd5kpqGaw+9Br__zou5yNTQ@
> mail.gmail.com%3E
>
>
> On Wed, Aug 23, 2017 at 11:12 AM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
> > Agreed. I was very clearly not advocating for rushing in features. If you
> > have followed my past emails, I have only strongly advocated features be
> > worked in branches and get merged when they are in a reasonable state.
> >
> > Each branch contributor group should look at their readiness and merge
> > stuff in assuming that the branch reached a satisfactory state. That’s
> it.
> >
> > From release management perspective, blocking features just because we
> are
> > a month close to the deadline is not reasonable. Let the branch
> > contributors rationalize, make this decision and the rest of us can
> support
> > them in making the decision.
> >
> > +Vinod
> >
> > > At this point, there have been three planned alphas from September 2016
> > until July 2017 to "get in features".  While a couple of upcoming
> features
> > are "a few weeks" away, I think all of us are aware how predictable
> > software development schedules can be.  I think we can also all agree
> that
> > rushing just to meet a release deadline isn't the best practice when it
> > comes to software development either.
> > >
> > > Andrew has been very clear about his goals at each step and I think
> > Wangda's willingness to not rush in resource types was an appropriate
> > response.  I'm sympathetic to the goals of getting in a feature for 3.0,
> > but it might be a good idea for each project that is a "few weeks away"
> to
> > seriously look at the readiness compared to the features which have been
> > testing for 6+ months already.
> > >
> > > -Ray
> >
> >
>

Re: Branch merges and 3.0.0-beta1 scope

[Vrushali Channapattan]

For timeline service v2, we have completed all subtasks under YARN-5355
[1].

We initiated a merge-to-trunk vote [2] yesterday.

thanks
Vrushali
[1] https://issues.apache.org/jira/browse/YARN-5355
[2]
http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201708.mbox/%3CCAE=b_fblt2j+ezb4wqdn_uwbig1sd5kpqgaw+9br__zou5y...@mail.gmail.com%3E


On Wed, Aug 23, 2017 at 11:12 AM, Vinod Kumar Vavilapalli <
vino...@apache.org> wrote:

> Agreed. I was very clearly not advocating for rushing in features. If you
> have followed my past emails, I have only strongly advocated features be
> worked in branches and get merged when they are in a reasonable state.
>
> Each branch contributor group should look at their readiness and merge
> stuff in assuming that the branch reached a satisfactory state. That’s it.
>
> From release management perspective, blocking features just because we are
> a month close to the deadline is not reasonable. Let the branch
> contributors rationalize, make this decision and the rest of us can support
> them in making the decision.
>
> +Vinod
>
> > At this point, there have been three planned alphas from September 2016
> until July 2017 to "get in features".  While a couple of upcoming features
> are "a few weeks" away, I think all of us are aware how predictable
> software development schedules can be.  I think we can also all agree that
> rushing just to meet a release deadline isn't the best practice when it
> comes to software development either.
> >
> > Andrew has been very clear about his goals at each step and I think
> Wangda's willingness to not rush in resource types was an appropriate
> response.  I'm sympathetic to the goals of getting in a feature for 3.0,
> but it might be a good idea for each project that is a "few weeks away" to
> seriously look at the readiness compared to the features which have been
> testing for 6+ months already.
> >
> > -Ray
>
>

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Agreed. I was very clearly not advocating for rushing in features. If you have 
followed my past emails, I have only strongly advocated features be worked in 
branches and get merged when they are in a reasonable state.

Each branch contributor group should look at their readiness and merge stuff in 
assuming that the branch reached a satisfactory state. That’s it.

From release management perspective, blocking features just because we are a 
month close to the deadline is not reasonable. Let the branch contributors 
rationalize, make this decision and the rest of us can support them in making 
the decision.

+Vinod

> At this point, there have been three planned alphas from September 2016 until 
> July 2017 to "get in features".  While a couple of upcoming features are "a 
> few weeks" away, I think all of us are aware how predictable software 
> development schedules can be.  I think we can also all agree that rushing 
> just to meet a release deadline isn't the best practice when it comes to 
> software development either.
> 
> Andrew has been very clear about his goals at each step and I think Wangda's 
> willingness to not rush in resource types was an appropriate response.  I'm 
> sympathetic to the goals of getting in a feature for 3.0, but it might be a 
> good idea for each project that is a "few weeks away" to seriously look at 
> the readiness compared to the features which have been testing for 6+ months 
> already.
> 
> -Ray

Re: Branch merges and 3.0.0-beta1 scope

[Allen Wittenauer]

We should avoid turning this into a replay of Apache Hadoop 2.6.0 (and 
to a lesser degree, 2.7.0 and 2.8.0) where a bunch of last minute 
“experimental” features derail stability for a significantly long period of 
time.
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Ray Chiang]

On 8/22/17 3:20 AM, Steve Loughran wrote:


On 21 Aug 2017, at 22:22, Vinod Kumar Vavilapalli  wrote:

Steve,

You can be strict & ruthless about the timelines. Anything that doesn’t get in 
by mid-September, as was originally planned, can move to the next release - whether 
it is feature work on branches or feature work on trunk.

The problem I see here is that code & branches being worked on for a year are 
now (apparently) close to being done and we are telling them to hold for 7 more 
months - this is not a reasonable ask..

If you are advocating for a 3.1 plan, I’m sure one of these branch ‘owners’ can 
volunteer. But this is how you get competing releases and split bandwidth.

As for compatibility / testing etc, it seems like there is a belief that the 
current ‘scoped’ features are all tested well in these areas and so adding more 
is going to hurt the release. There is no way this is the reality, trunk has so 
many features that have been landing for years, the only way we can 
collectively attempt towards making this stable is by getting as many parties 
together as possible, each verifying stuff that they need. Not by excluding 
specific features.


If everyone is confident & its coming together, it does make sense. I think 
those of us (myself included) who are merging stuff in do have to recognise that we 
really need to follow it through by being responsive to any problem -and with the 
release manager having the right to pull things out if its felt to be significantly 
threatening the stability of the final 3.0 release.

I think we should also consider making the 3.0 beta the feature freeze; after that fixes 
on the features go in, but nothing else of significance, otherwise the value of the beta 
"test this code more broadly" is diminoshed
At this point, there have been three planned alphas from September 2016 
until July 2017 to "get in features".  While a couple of upcoming 
features are "a few weeks" away, I think all of us are aware how 
predictable software development schedules can be.  I think we can also 
all agree that rushing just to meet a release deadline isn't the best 
practice when it comes to software development either.


Andrew has been very clear about his goals at each step and I think 
Wangda's willingness to not rush in resource types was an appropriate 
response.  I'm sympathetic to the goals of getting in a feature for 3.0, 
but it might be a good idea for each project that is a "few weeks away" 
to seriously look at the readiness compared to the features which have 
been testing for 6+ months already.


-Ray


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Steve Loughran]

> On 21 Aug 2017, at 22:22, Vinod Kumar Vavilapalli  wrote:
> 
> Steve,
> 
> You can be strict & ruthless about the timelines. Anything that doesn’t get 
> in by mid-September, as was originally planned, can move to the next release 
> - whether it is feature work on branches or feature work on trunk.
> 
> The problem I see here is that code & branches being worked on for a year are 
> now (apparently) close to being done and we are telling them to hold for 7 
> more months - this is not a reasonable ask..
> 
> If you are advocating for a 3.1 plan, I’m sure one of these branch ‘owners’ 
> can volunteer. But this is how you get competing releases and split bandwidth.
> 
> As for compatibility / testing etc, it seems like there is a belief that the 
> current ‘scoped’ features are all tested well in these areas and so adding 
> more is going to hurt the release. There is no way this is the reality, trunk 
> has so many features that have been landing for years, the only way we can 
> collectively attempt towards making this stable is by getting as many parties 
> together as possible, each verifying stuff that they need. Not by excluding 
> specific features.
> 

If everyone is confident & its coming together, it does make sense. I think 
those of us (myself included) who are merging stuff in do have to recognise 
that we really need to follow it through by being responsive to any problem 
-and with the release manager having the right to pull things out if its felt 
to be significantly threatening the stability of the final 3.0 release.

I think we should also consider making the 3.0 beta the feature freeze; after 
that fixes on the features go in, but nothing else of significance, otherwise 
the value of the beta "test this code more broadly" is diminoshed

-steve
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

RE: Branch merges and 3.0.0-beta1 scope

[Brahma Reddy Battula]

IMHO, when we propose any feature branch merge, we might need to consider 
following aspects

1)Use cases
2) Pluggable ---> if it's not pluggable , give the reason
3) API Compatibility
4) Impact---> when it's enable
5) Performance
6) Stability
7) Test Sufficiency
8) Documentation

Above points helps to validate the feature quality at first glance and based on 
that further consideration to merge can be done.


--Brahma Reddy Battula

-Original Message-
From: Wangda Tan [mailto:wheele...@gmail.com] 
Sent: 22 August 2017 06:27
To: Vinod Kumar Vavilapalli
Cc: Steve Loughran; Andrew Wang; common-...@hadoop.apache.org; 
hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Branch merges and 3.0.0-beta1 scope

Andrew,

Thanks for your help to pushing this release.

Echoing what Vinod said, all contributors in these branches are putting months 
to years of time working on these features, we don't have to decide excluded 
features now since we have 25 days till 3.0-beta1 planned release time.

The best approach to stabilize feature is to let people try that, instead of 
waiting for feature becomes perfect. For features which can be turned off, I 
think we should consider to bring it in if it is end-to-end ready. I will try 
best to help merge efforts of YARN-3926 branch to trunk before Sep 15, and I'm 
OK with moving to the next release train if we fail to merge the feature before 
release date.

Thanks,
Wangda


On Mon, Aug 21, 2017 at 2:22 PM, Vinod Kumar Vavilapalli <vino...@apache.org
> wrote:

> Steve,
>
> You can be strict & ruthless about the timelines. Anything that 
> doesn’t get in by mid-September, as was originally planned, can move 
> to the next release - whether it is feature work on branches or feature work 
> on trunk.
>
> The problem I see here is that code & branches being worked on for a 
> year are now (apparently) close to being done and we are telling them 
> to hold for 7 more months - this is not a reasonable ask..
>
> If you are advocating for a 3.1 plan, I’m sure one of these branch 
> ‘owners’ can volunteer. But this is how you get competing releases and 
> split bandwidth.
>
> As for compatibility / testing etc, it seems like there is a belief 
> that the current ‘scoped’ features are all tested well in these areas 
> and so adding more is going to hurt the release. There is no way this 
> is the reality, trunk has so many features that have been landing for 
> years, the only way we can collectively attempt towards making this 
> stable is by getting as many parties together as possible, each 
> verifying stuff that they need. Not by excluding specific features.
>
> +Vinod
>
> > This is one of those curse-of-cadence things: The higher your 
> > release
> cadence, the less pressure to get "everything in". With a slower 
> cadence, more pressure to get stuff in, more pressure to hold up the 
> release, slows the cadence, gets even more stuff in, etc. etc.
> >
> > - Andrew has been working on the release for months, we all need to
> appreciate how much hard work that is and has been, especially for 
> what is going to be a major release.
> >
> > - We know that things will be unstable in 3.0; Andrew's concern is 
> > about
> making sure that the newest, unstablest (?) features can at least be 
> bypassed if there are problems. I we should also call out in the 
> release notes what we think are the unstable bits where people need to 
> use caution
> (example: S3Guard in "authoritative" mode)
> >
> > - Anything related to wire compatibility has been problematic in the
> past; I think it's essential that whatever packets get sent around are 
> going to be stable, so changes there need to be in, or at least the 
> payloads set up ready for the features. Same for new public APIs.
> >
> > - As fpr the rest, I don't know. I think being strict about it and
> ruthless in assessing the feature's stability & consequences of 
> postponing the feature until a Hadoop 3.1 release in Jan/Feb, with a 
> plan to ship then and follow up with a 3.2 in the summer.
> >
> > Then: start planning that 3.1 release. Maybe I should put my hand up 
> > as
> release manager for that one. Then everyone would realise how amenable 
> Andrew is being today.
> >
> >
> > One other thing: alongside the big branches, there's the eternal 
> > backlog
> of small patches. We should organise spending a few days updating, 
> reviewing & merging them in
> >
> > -Steve
> >
> >
> > 
> > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 
> >  hdfs-dev-unsubscr...@hadoop.apache.org>
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> <mailto:hdfs-dev-h...@hadoop.apache.org>
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Wangda Tan]

Andrew,

Thanks for your help to pushing this release.

Echoing what Vinod said, all contributors in these branches are putting
months to years of time working on these features, we don't have to decide
excluded features now since we have 25 days till 3.0-beta1 planned release
time.

The best approach to stabilize feature is to let people try that, instead
of waiting for feature becomes perfect. For features which can be turned
off, I think we should consider to bring it in if it is end-to-end ready. I
will try best to help merge efforts of YARN-3926 branch to trunk before Sep
15, and I'm OK with moving to the next release train if we fail to merge
the feature before release date.

Thanks,
Wangda


On Mon, Aug 21, 2017 at 2:22 PM, Vinod Kumar Vavilapalli  wrote:

> Steve,
>
> You can be strict & ruthless about the timelines. Anything that doesn’t
> get in by mid-September, as was originally planned, can move to the next
> release - whether it is feature work on branches or feature work on trunk.
>
> The problem I see here is that code & branches being worked on for a year
> are now (apparently) close to being done and we are telling them to hold
> for 7 more months - this is not a reasonable ask..
>
> If you are advocating for a 3.1 plan, I’m sure one of these branch
> ‘owners’ can volunteer. But this is how you get competing releases and
> split bandwidth.
>
> As for compatibility / testing etc, it seems like there is a belief that
> the current ‘scoped’ features are all tested well in these areas and so
> adding more is going to hurt the release. There is no way this is the
> reality, trunk has so many features that have been landing for years, the
> only way we can collectively attempt towards making this stable is by
> getting as many parties together as possible, each verifying stuff that
> they need. Not by excluding specific features.
>
> +Vinod
>
> > This is one of those curse-of-cadence things: The higher your release
> cadence, the less pressure to get "everything in". With a slower cadence,
> more pressure to get stuff in, more pressure to hold up the release, slows
> the cadence, gets even more stuff in, etc. etc.
> >
> > - Andrew has been working on the release for months, we all need to
> appreciate how much hard work that is and has been, especially for what is
> going to be a major release.
> >
> > - We know that things will be unstable in 3.0; Andrew's concern is about
> making sure that the newest, unstablest (?) features can at least be
> bypassed if there are problems. I we should also call out in the release
> notes what we think are the unstable bits where people need to use caution
> (example: S3Guard in "authoritative" mode)
> >
> > - Anything related to wire compatibility has been problematic in the
> past; I think it's essential that whatever packets get sent around are
> going to be stable, so changes there need to be in, or at least the
> payloads set up ready for the features. Same for new public APIs.
> >
> > - As fpr the rest, I don't know. I think being strict about it and
> ruthless in assessing the feature's stability & consequences of postponing
> the feature until a Hadoop 3.1 release in Jan/Feb, with a plan to ship then
> and follow up with a 3.2 in the summer.
> >
> > Then: start planning that 3.1 release. Maybe I should put my hand up as
> release manager for that one. Then everyone would realise how amenable
> Andrew is being today.
> >
> >
> > One other thing: alongside the big branches, there's the eternal backlog
> of small patches. We should organise spending a few days updating,
> reviewing & merging them in
> >
> > -Steve
> >
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org  hdfs-dev-unsubscr...@hadoop.apache.org>
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 
>

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Steve,

You can be strict & ruthless about the timelines. Anything that doesn’t get in 
by mid-September, as was originally planned, can move to the next release - 
whether it is feature work on branches or feature work on trunk.

The problem I see here is that code & branches being worked on for a year are 
now (apparently) close to being done and we are telling them to hold for 7 more 
months - this is not a reasonable ask..

If you are advocating for a 3.1 plan, I’m sure one of these branch ‘owners’ can 
volunteer. But this is how you get competing releases and split bandwidth.

As for compatibility / testing etc, it seems like there is a belief that the 
current ‘scoped’ features are all tested well in these areas and so adding more 
is going to hurt the release. There is no way this is the reality, trunk has so 
many features that have been landing for years, the only way we can 
collectively attempt towards making this stable is by getting as many parties 
together as possible, each verifying stuff that they need. Not by excluding 
specific features.

+Vinod

> This is one of those curse-of-cadence things: The higher your release 
> cadence, the less pressure to get "everything in". With a slower cadence, 
> more pressure to get stuff in, more pressure to hold up the release, slows 
> the cadence, gets even more stuff in, etc. etc.
> 
> - Andrew has been working on the release for months, we all need to 
> appreciate how much hard work that is and has been, especially for what is 
> going to be a major release.
> 
> - We know that things will be unstable in 3.0; Andrew's concern is about 
> making sure that the newest, unstablest (?) features can at least be bypassed 
> if there are problems. I we should also call out in the release notes what we 
> think are the unstable bits where people need to use caution (example: 
> S3Guard in "authoritative" mode)
> 
> - Anything related to wire compatibility has been problematic in the past; I 
> think it's essential that whatever packets get sent around are going to be 
> stable, so changes there need to be in, or at least the payloads set up ready 
> for the features. Same for new public APIs.
> 
> - As fpr the rest, I don't know. I think being strict about it and ruthless 
> in assessing the feature's stability & consequences of postponing the feature 
> until a Hadoop 3.1 release in Jan/Feb, with a plan to ship then and follow up 
> with a 3.2 in the summer.
> 
> Then: start planning that 3.1 release. Maybe I should put my hand up as 
> release manager for that one. Then everyone would realise how amenable Andrew 
> is being today.
> 
> 
> One other thing: alongside the big branches, there's the eternal backlog of 
> small patches. We should organise spending a few days updating, reviewing & 
> merging them in
> 
> -Steve
> 
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 
> 
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org 
> 

Re: Branch merges and 3.0.0-beta1 scope

[Steve Loughran]

> On 19 Aug 2017, at 02:22, Andrew Wang  wrote:
> 
> * 3.0.0-beta1 includes S3Guard and YARN federation. Target date remains
> mid-Sept.
> * 3.0.0-GA includes TSv2 alpha2. Target date remains early November.
> * Everything else waits for 3.1, approximately March 2018.
> 


probably time to create those releases in JIRA, so people can start explicitly 
targeting them.


> On 19 Aug 2017, at 06:49, Vinod Kumar Vavilapalli  wrote:
> 
> Andrew,
> 
> Each of the branches below have been created more than a year ago (!) and 
> have been consistently worked upon and are now finally seeing the light of 
> the day. When they are "few weeks” away, pushing them out by 7 *more* months 
> just doesn’t make sense.
> 
> While I deeply appreciate the push for the dates, we shouldn’t be putting 
> moratorium on features like this till the proposed date is due. As a release 
> manager, it’s good to say that we will release a specific version as soon as 
> so-and-so features are in, but let’s not put exclusions like this.
> 
> I propose that, as we have done with every release so far, (and more 
> specifically the way we did 2.x alphas, betas, GA back in the day), we float 
> the date, let the individual branch contributors decide their fate. As long 
> as of course, they meet the timelines and the usual branch merge bar, testing 
> / compatibility / impact on rest of the code-base etc.
> 
> Anything short of that, we will just be incentivizing contributors away from 
> doing work in branches and towards putting stuff directly into trunk.
> 
> +Vinod


This is one of those curse-of-cadence things: The higher your release cadence, 
the less pressure to get "everything in". With a slower cadence, more pressure 
to get stuff in, more pressure to hold up the release, slows the cadence, gets 
even more stuff in, etc. etc.

- Andrew has been working on the release for months, we all need to appreciate 
how much hard work that is and has been, especially for what is going to be a 
major release.

- We know that things will be unstable in 3.0; Andrew's concern is about making 
sure that the newest, unstablest (?) features can at least be bypassed if there 
are problems. I we should also call out in the release notes what we think are 
the unstable bits where people need to use caution (example: S3Guard in 
"authoritative" mode)

- Anything related to wire compatibility has been problematic in the past; I 
think it's essential that whatever packets get sent around are going to be 
stable, so changes there need to be in, or at least the payloads set up ready 
for the features. Same for new public APIs.

- As fpr the rest, I don't know. I think being strict about it and ruthless in 
assessing the feature's stability & consequences of postponing the feature 
until a Hadoop 3.1 release in Jan/Feb, with a plan to ship then and follow up 
with a 3.2 in the summer.

Then: start planning that 3.1 release. Maybe I should put my hand up as release 
manager for that one. Then everyone would realise how amenable Andrew is being 
today.


One other thing: alongside the big branches, there's the eternal backlog of 
small patches. We should organise spending a few days updating, reviewing & 
merging them in

-Steve

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Andrew,

Each of the branches below have been created more than a year ago (!) and have 
been consistently worked upon and are now finally seeing the light of the day. 
When they are "few weeks” away, pushing them out by 7 *more* months just 
doesn’t make sense.

While I deeply appreciate the push for the dates, we shouldn’t be putting 
moratorium on features like this till the proposed date is due. As a release 
manager, it’s good to say that we will release a specific version as soon as 
so-and-so features are in, but let’s not put exclusions like this.

I propose that, as we have done with every release so far, (and more 
specifically the way we did 2.x alphas, betas, GA back in the day), we float 
the date, let the individual branch contributors decide their fate. As long as 
of course, they meet the timelines and the usual branch merge bar, testing / 
compatibility / impact on rest of the code-base etc.

Anything short of that, we will just be incentivizing contributors away from 
doing work in branches and towards putting stuff directly into trunk.

+Vinod

> On Aug 18, 2017, at 6:22 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> As you might have seen, we've had a number of branch merges floated this
> past week targeted for 3.0.0-beta1, which is planned for about a month from
> now.
> 
> In total, I'm currently tracking these branches:
> 
> YARN-2915: YARN federation (recently merged)
> HADOOP-13345: S3Guard (currently being voted on)
> YARN-5355: TSv2 alpha2 ("few weeks")
> YARN-5079: Native services ("few weeks")
> YARN-3926: Resource profiles ("few weeks")
> 
> We should effectively be in code freeze (only blockers/criticals), so the
> volume of merge proposals at this point came as a surprise. Despite our
> best efforts as software engineers, big code movement always comes with
> risk.
> 
> Since we started the 3.0 release series close to a year ago, I'm also loath
> to increase the scope. The main motivation for 3.0 was to deliver JDK8 and
> HDFS EC, and our users deserve a production-quality release with these
> features. We've also been good about the release cadence thus far in 3.x,
> so a 3.1 isn't that far out.
> 
> Here's my proposal:
> 
> * 3.0.0-beta1 includes S3Guard and YARN federation. Target date remains
> mid-Sept.
> * 3.0.0-GA includes TSv2 alpha2. Target date remains early November.
> * Everything else waits for 3.1, approximately March 2018.
> 
> My rationale for inclusion and exclusion is as follows:
> 
> Inclusion:
> 
> * YARN federation has been run in production, does not touch existing code,
> adds no new APIs, and is off by default.
> * S3Guard has been run in production and is off by default.
> * The first iteration of TSv2 was shipped in 3.0.0-alpha1, so we're
> committed to this for 3.0.0 GA. It's off by default and adds no impact.
> 
> Exclusion:
> 
> * The primary reason for exclusion is to maintain the planned release
> schedule. Native services and resource profiles are still a few weeks from
> being ready for merge.
> * A reminder that 3.1 is only another 3 months after 3.0 given our release
> cadence thus far. If there's demand, we could even do a 3.1 immediately
> following 3.0.
> 
> I'm happy to talk with the contributors of each of these features to
> understand their timelines and requirements, with the caveat that I'll be
> out through Wednesday next week. Please reach out.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Hi folks,

As you might have seen, we've had a number of branch merges floated this
past week targeted for 3.0.0-beta1, which is planned for about a month from
now.

In total, I'm currently tracking these branches:

YARN-2915: YARN federation (recently merged)
HADOOP-13345: S3Guard (currently being voted on)
YARN-5355: TSv2 alpha2 ("few weeks")
YARN-5079: Native services ("few weeks")
YARN-3926: Resource profiles ("few weeks")

We should effectively be in code freeze (only blockers/criticals), so the
volume of merge proposals at this point came as a surprise. Despite our
best efforts as software engineers, big code movement always comes with
risk.

Since we started the 3.0 release series close to a year ago, I'm also loath
to increase the scope. The main motivation for 3.0 was to deliver JDK8 and
HDFS EC, and our users deserve a production-quality release with these
features. We've also been good about the release cadence thus far in 3.x,
so a 3.1 isn't that far out.

Here's my proposal:

* 3.0.0-beta1 includes S3Guard and YARN federation. Target date remains
mid-Sept.
* 3.0.0-GA includes TSv2 alpha2. Target date remains early November.
* Everything else waits for 3.1, approximately March 2018.

My rationale for inclusion and exclusion is as follows:

Inclusion:

* YARN federation has been run in production, does not touch existing code,
adds no new APIs, and is off by default.
* S3Guard has been run in production and is off by default.
* The first iteration of TSv2 was shipped in 3.0.0-alpha1, so we're
committed to this for 3.0.0 GA. It's off by default and adds no impact.

Exclusion:

* The primary reason for exclusion is to maintain the planned release
schedule. Native services and resource profiles are still a few weeks from
being ready for merge.
* A reminder that 3.1 is only another 3 months after 3.0 given our release
cadence thus far. If there's demand, we could even do a 3.1 immediately
following 3.0.

I'm happy to talk with the contributors of each of these features to
understand their timelines and requirements, with the caveat that I'll be
out through Wednesday next week. Please reach out.

Best,
Andrew