Re: Branch merges and 3.0.0-beta1 scope

[Vrushali C]

bq. I don't think it's a lot to ask that feature leads shoot an email to
the release manager of their target version. DISCUSS emails right before a
proposed merge VOTE are way too late, it ends up being a fire drill where
we need to scramble on many fronts.

Yes, I have been thinking about this too. Based on what I have observed,
[DISCUSS] is more of a late-stage communication typically just like 8-10
days before the [VOTE] goes out. Most the leads send out [DISCUSS] emails
when the feature is extremely close to completion. So it seems to me that
there is an inherent hesitation in sending out [DISCUSS] emails when things
are not close to completion, so the Release Manager does not hear about
these early on.

Perhaps we want to add another step like [HEADS-UP] which goes out to at
least the Release Manager and known stakeholders or perhaps even the entire
dev community well before [DISCUSS].

The  [HEADS-UP] could be a very simple note that basically says we are
still working on this but we wish to aim for so-and-so release. It could go
out as soon as release idea is floated but perhaps no later than 7-8 weeks
before planned release date. [HEADS-UP] indicates that despite
contributors' best efforts, there exists a possibility that that feature
may not finish in time to make it into the release but at least the RM is
aware of the intentions and can track these.

Perhaps a time frame like the following may be good to follow:
[HEADS-UP]: typically at least 2 months before release date or as early as
anyone wants to communicate to RM.
[DISCUSS]: approx 4-6 weeks before release date
[VOTE]: closes before 2 (or 3?) weeks before release date

While I do think some timeframes like these should become the norm, we may
not want to be too rigidly pedantic as well. It may be a good idea to keep
ourselves open to making exceptions on a per case basis.

thanks
Vrushali



On Tue, Aug 29, 2017 at 2:25 PM, Andrew Wang 
wrote:

> Hi Vinod,
>
> On Fri, Aug 25, 2017 at 2:42 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> > From a release management perspective, it's *extremely* reasonable to
>> block the inclusion of new features a month from the planned release date.
>> A typical software development lifecycle includes weeks of feature freeze
>> and weeks of code freeze. It is no knock on any developer or any feature to
>> say that we should not include something in 3.0.0.
>>
>>
>> We have never followed the ‘typical' lifecycle that I am guessing you are
>> referring to. If we are, you'll need to publish some of the following: a
>> feature freeze date, blockers-criticals-only-from-now date,
>> testing-finish date, documentation-finish date, final release date and so
>> on.
>>
>
> We discussed this as part of the 3.0 alpha/beta/GA plan. The point of the
> extended alpha/beta process was to release on a schedule. Things that
> weren't ready could be merged for the next alpha. I also advertised alpha4
> as feature complete and beta1 as code complete so we could quickly move on
> to GA.
>
>
>> What we do with Apache releases typically is instead we say ‘this' is
>> roughly when we want to release, and roughly what features must land and
>> let the rest figure out itself.
>>
>> We did this too. We defined the original scope for 3.0.0 GA way back when
> we started the 3.0.0 release process. I've been writing status updates on
> the wiki and tracking targeted features and release blockers throughout.
>
> The target versions of this recent batch of features were not discussed
> with me, the release manager, until just recently. After some discussion, I
> think we've arrived at a release plan that everyone's happy with. But, I
> want to be clear that late-breaking inclusion of additional scope should be
> considered the exception rather than the norm. Merging code so close to
> release means less time for testing and validation, which means lower
> quality releases.
>
> I don't think it's a lot to ask that feature leads shoot an email to the
> release manager of their target version. DISCUSS emails right before a
> proposed merge VOTE are way too late, it ends up being a fire drill where
> we need to scramble on many fronts.
>
>
>> Neither is right or wrong. If we want to change the process, we should
>> communicate as such.
>>
>> Proposing a feature freeze date on the fly is only going to confuse
>> people.
>>
>
>> > I've been very open and clear about the goals, schedule, and scope of
>> 3.0.0 over the last year plus. The point of the extended alpha process was
>> to get all our features in during alpha, and the alpha merge window has
>> been open for a year. I'm unmoved by arguments about how long a feature has
>> been worked on. None of these were not part of the original 3.0.0 scope,
>> and our users have been waiting even longer for big-ticket 3.0 items like
>> JDK8 and HDFS EC that were part of the discussed scope.
>>
>>
>> Except our schedule is so fluid (not due to the release 

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Hi Vinod,

On Fri, Aug 25, 2017 at 2:42 PM, Vinod Kumar Vavilapalli  wrote:

> > From a release management perspective, it's *extremely* reasonable to
> block the inclusion of new features a month from the planned release date.
> A typical software development lifecycle includes weeks of feature freeze
> and weeks of code freeze. It is no knock on any developer or any feature to
> say that we should not include something in 3.0.0.
>
>
> We have never followed the ‘typical' lifecycle that I am guessing you are
> referring to. If we are, you'll need to publish some of the following: a
> feature freeze date, blockers-criticals-only-from-now date,
> testing-finish date, documentation-finish date, final release date and so
> on.
>

We discussed this as part of the 3.0 alpha/beta/GA plan. The point of the
extended alpha/beta process was to release on a schedule. Things that
weren't ready could be merged for the next alpha. I also advertised alpha4
as feature complete and beta1 as code complete so we could quickly move on
to GA.


> What we do with Apache releases typically is instead we say ‘this' is
> roughly when we want to release, and roughly what features must land and
> let the rest figure out itself.
>
> We did this too. We defined the original scope for 3.0.0 GA way back when
we started the 3.0.0 release process. I've been writing status updates on
the wiki and tracking targeted features and release blockers throughout.

The target versions of this recent batch of features were not discussed
with me, the release manager, until just recently. After some discussion, I
think we've arrived at a release plan that everyone's happy with. But, I
want to be clear that late-breaking inclusion of additional scope should be
considered the exception rather than the norm. Merging code so close to
release means less time for testing and validation, which means lower
quality releases.

I don't think it's a lot to ask that feature leads shoot an email to the
release manager of their target version. DISCUSS emails right before a
proposed merge VOTE are way too late, it ends up being a fire drill where
we need to scramble on many fronts.


> Neither is right or wrong. If we want to change the process, we should
> communicate as such.
>
> Proposing a feature freeze date on the fly is only going to confuse
> people.
>

> > I've been very open and clear about the goals, schedule, and scope of
> 3.0.0 over the last year plus. The point of the extended alpha process was
> to get all our features in during alpha, and the alpha merge window has
> been open for a year. I'm unmoved by arguments about how long a feature has
> been worked on. None of these were not part of the original 3.0.0 scope,
> and our users have been waiting even longer for big-ticket 3.0 items like
> JDK8 and HDFS EC that were part of the discussed scope.
>
>
> Except our schedule is so fluid (not due to the release management process
> to be fair) that it is hard for folks to plan their features. IIRC, our
> schedule was a GA release beginning of this year. Again, this is not a
> critique of 3.0 release process - I have myself done enough releases to
> know that sticking to a date and herding the crowd has been an extremely
> hard job.
>
>
Schedules have been fluid because we don't know when features are getting
in, and there's an unwillingness to bump features to the next release. The
goal of the 3.x alphas and betas was to break out of this release
anti-pattern, and release on a schedule.

There have been schedule delays during the 3.x alphas, but I'm still proud
that we released 4 alphas in 10 months. I'm doing my best to stick to our
published schedule, and add a beta and GA to that list by EOY.

Best,
Andrew

Re: [DISCUSS] Branches and versions for Hadoop 3

[Andrew Wang]

Hi Subru,

Basically we're amending the proposal from the original email in the chain
to also immediately create the branch-3.0.0-beta1 release branch. As
described in my 2017-08-25 wiki update, we're gating the merge of these two
features to branch-3.0 on additional testing,  but this keeps 3.0.0 open
for development.

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

For completeness, here's what our branches and versions would look like:

trunk: 3.1.0-SNAPSHOT
branch-3.0: 3.0.0-SNAPSHOT
branch-3.0.0-beta1: 3.0.0-beta1-SNAPSHOT
branch-2 and etc: remain as is

Best,
Andrew

On Tue, Aug 29, 2017 at 12:21 PM, Subramaniam V K 
wrote:

> Andrew,
>
> First up thanks for tirelessly pushing on 3.0 release.
>
> I am confused about your comment on creating 2 branches as my
> understanding of Jason's (and Vinod's) comments are that we defer creating
> branch-3?
>
> IMHO, we should consider creating branch-3 (necessary but not sufficient)
> only when we have:
>
>1. a significant incompatible change.
>2. a new feature that cannot be turned off without affecting core
>components.
>
> In summary, I feel we should follow a lazy rather than eager approach
> towards creating mainline branches.
>
> Thanks,
> Subru
>
>
>
> On Tue, Aug 29, 2017 at 11:45 AM, Wangda Tan  wrote:
>
>> Gotcha, make sense, so I will hold commit until you cut the two branches
>> and TSv2 get committed.
>>
>> Thanks,
>> Wangda
>>
>> On Tue, Aug 29, 2017 at 11:25 AM, Andrew Wang 
>> wrote:
>>
>> > Hi Wangda,
>> >
>> > I'll cut two branches: branch-3.0 (3.0.0-SNAPSHOT) and
>> branch-3.0.0-beta1
>> > (3.0.0-beta1-SNAPSHOT). This way we can merge GA features to branch-3.0
>> but
>> > not branch-3.0.0-beta1.
>> >
>> > Best,
>> > Andrew
>> >
>> > On Tue, Aug 29, 2017 at 11:18 AM, Wangda Tan 
>> wrote:
>> >
>> >> Vrushali,
>> >>
>> >> Sure we can wait TSv2 merged before merge resource profile branch.
>> >>
>> >> Andrew,
>> >>
>> >> My understanding is you're going to cut branch-3.0 for 3.0-beta1, and
>> the
>> >> same branch (branch-3.0) will be used for 3.0-GA as well. So my
>> question
>> >> is, there're several features (TSv2, resource profile, YARN-5734) are
>> >> targeted to merge to 3.0-GA but not 3.0-beta1, which branch we should
>> >> commit to, and when we can commit? Also, similar to 3.0.0-alpha1 to 4,
>> you
>> >> will cut branch-3.0.0-beta1, correct?
>> >>
>> >> Thanks,
>> >> Wangda
>> >>
>> >>
>> >> On Tue, Aug 29, 2017 at 11:05 AM, Andrew Wang <
>> andrew.w...@cloudera.com>
>> >> wrote:
>> >>
>> >>> Sure. Ping me when the TSv2 goes in, and I can take care of branching.
>> >>>
>> >>> We're still waiting on the native services and S3Guard merges, but I
>> >>> don't want to hold branching to the last minute.
>> >>>
>> >>> On Tue, Aug 29, 2017 at 10:51 AM, Vrushali C > >
>> >>> wrote:
>> >>>
>>  Hi Andrew,
>>  As Rohith mentioned, if you are good with it, from the TSv2 side, we
>>  are ready to go for merge tonight itself (Pacific time)  right after
>> the
>>  voting period ends. Varun Saxena has been diligently rebasing up
>> until now
>>  so most likely our merge should be reasonably straightforward.
>> 
>>  @Wangda: your resource profile vote ends tomorrow, could we please
>>  coordinate our merges?
>> 
>>  thanks
>>  Vrushali
>> 
>> 
>>  On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
>>  rohithsharm...@apache.org> wrote:
>> 
>> > On 29 August 2017 at 06:24, Andrew Wang 
>> > wrote:
>> >
>> > > So far I've seen no -1's to the branching proposal, so I plan to
>> > execute
>> > > this tomorrow unless there's further feedback.
>> > >
>> > For on going branch merge threads i.e TSv2, voting will be closing
>> > tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
>> > branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait
>> > for
>> > couple of more days before creating branch-3.0 so that TSv2 branch
>> > merge
>> > would be done directly to trunk?
>> >
>> >
>> >
>> > >
>> > > Regarding the above discussion, I think Jason and I have
>> essentially
>> > the
>> > > same opinion.
>> > >
>> > > I hope that keeping trunk a release branch means a higher bar for
>> > merges
>> > > and code review in general. In the past, I've seen some patches
>> > committed
>> > > to trunk-only as a way of passing responsibility to a future user
>> or
>> > > reviewer. That doesn't help anyone; patches should be committed
>> with
>> > the
>> > > intent of running them in production.
>> > >
>> > > I'd also like to repeat the above thanks to the many, many
>> > contributors
>> > > who've helped with release improvements. Allen's work on
>> > create-release 

Re: [DISCUSS] Branches and versions for Hadoop 3

[Subramaniam V K]

Andrew,

First up thanks for tirelessly pushing on 3.0 release.

I am confused about your comment on creating 2 branches as my understanding
of Jason's (and Vinod's) comments are that we defer creating branch-3?

IMHO, we should consider creating branch-3 (necessary but not sufficient)
only when we have:

   1. a significant incompatible change.
   2. a new feature that cannot be turned off without affecting core
   components.

In summary, I feel we should follow a lazy rather than eager approach
towards creating mainline branches.

Thanks,
Subru



On Tue, Aug 29, 2017 at 11:45 AM, Wangda Tan  wrote:

> Gotcha, make sense, so I will hold commit until you cut the two branches
> and TSv2 get committed.
>
> Thanks,
> Wangda
>
> On Tue, Aug 29, 2017 at 11:25 AM, Andrew Wang 
> wrote:
>
> > Hi Wangda,
> >
> > I'll cut two branches: branch-3.0 (3.0.0-SNAPSHOT) and branch-3.0.0-beta1
> > (3.0.0-beta1-SNAPSHOT). This way we can merge GA features to branch-3.0
> but
> > not branch-3.0.0-beta1.
> >
> > Best,
> > Andrew
> >
> > On Tue, Aug 29, 2017 at 11:18 AM, Wangda Tan 
> wrote:
> >
> >> Vrushali,
> >>
> >> Sure we can wait TSv2 merged before merge resource profile branch.
> >>
> >> Andrew,
> >>
> >> My understanding is you're going to cut branch-3.0 for 3.0-beta1, and
> the
> >> same branch (branch-3.0) will be used for 3.0-GA as well. So my question
> >> is, there're several features (TSv2, resource profile, YARN-5734) are
> >> targeted to merge to 3.0-GA but not 3.0-beta1, which branch we should
> >> commit to, and when we can commit? Also, similar to 3.0.0-alpha1 to 4,
> you
> >> will cut branch-3.0.0-beta1, correct?
> >>
> >> Thanks,
> >> Wangda
> >>
> >>
> >> On Tue, Aug 29, 2017 at 11:05 AM, Andrew Wang  >
> >> wrote:
> >>
> >>> Sure. Ping me when the TSv2 goes in, and I can take care of branching.
> >>>
> >>> We're still waiting on the native services and S3Guard merges, but I
> >>> don't want to hold branching to the last minute.
> >>>
> >>> On Tue, Aug 29, 2017 at 10:51 AM, Vrushali C 
> >>> wrote:
> >>>
>  Hi Andrew,
>  As Rohith mentioned, if you are good with it, from the TSv2 side, we
>  are ready to go for merge tonight itself (Pacific time)  right after
> the
>  voting period ends. Varun Saxena has been diligently rebasing up
> until now
>  so most likely our merge should be reasonably straightforward.
> 
>  @Wangda: your resource profile vote ends tomorrow, could we please
>  coordinate our merges?
> 
>  thanks
>  Vrushali
> 
> 
>  On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
>  rohithsharm...@apache.org> wrote:
> 
> > On 29 August 2017 at 06:24, Andrew Wang 
> > wrote:
> >
> > > So far I've seen no -1's to the branching proposal, so I plan to
> > execute
> > > this tomorrow unless there's further feedback.
> > >
> > For on going branch merge threads i.e TSv2, voting will be closing
> > tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
> > branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait
> > for
> > couple of more days before creating branch-3.0 so that TSv2 branch
> > merge
> > would be done directly to trunk?
> >
> >
> >
> > >
> > > Regarding the above discussion, I think Jason and I have
> essentially
> > the
> > > same opinion.
> > >
> > > I hope that keeping trunk a release branch means a higher bar for
> > merges
> > > and code review in general. In the past, I've seen some patches
> > committed
> > > to trunk-only as a way of passing responsibility to a future user
> or
> > > reviewer. That doesn't help anyone; patches should be committed
> with
> > the
> > > intent of running them in production.
> > >
> > > I'd also like to repeat the above thanks to the many, many
> > contributors
> > > who've helped with release improvements. Allen's work on
> > create-release and
> > > automated changes and release notes were essential, as was Xiao's
> > work on
> > > LICENSE and NOTICE files. I'm also looking forward to Marton's site
> > > improvements, which addresses one of the remaining sore spots in
> the
> > > release process.
> > >
> > > Things have gotten smoother with each alpha we've done over the
> last
> > year,
> > > and it's a testament to everyone's work that we have a good
> > probability of
> > > shipping beta and GA later this year.
> > >
> > > Cheers,
> > > Andrew
> > >
> > >
> >
> 
> 
> >>>
> >>
> >
>

Re: [DISCUSS] Branches and versions for Hadoop 3

[Wangda Tan]

Gotcha, make sense, so I will hold commit until you cut the two branches
and TSv2 get committed.

Thanks,
Wangda

On Tue, Aug 29, 2017 at 11:25 AM, Andrew Wang 
wrote:

> Hi Wangda,
>
> I'll cut two branches: branch-3.0 (3.0.0-SNAPSHOT) and branch-3.0.0-beta1
> (3.0.0-beta1-SNAPSHOT). This way we can merge GA features to branch-3.0 but
> not branch-3.0.0-beta1.
>
> Best,
> Andrew
>
> On Tue, Aug 29, 2017 at 11:18 AM, Wangda Tan  wrote:
>
>> Vrushali,
>>
>> Sure we can wait TSv2 merged before merge resource profile branch.
>>
>> Andrew,
>>
>> My understanding is you're going to cut branch-3.0 for 3.0-beta1, and the
>> same branch (branch-3.0) will be used for 3.0-GA as well. So my question
>> is, there're several features (TSv2, resource profile, YARN-5734) are
>> targeted to merge to 3.0-GA but not 3.0-beta1, which branch we should
>> commit to, and when we can commit? Also, similar to 3.0.0-alpha1 to 4, you
>> will cut branch-3.0.0-beta1, correct?
>>
>> Thanks,
>> Wangda
>>
>>
>> On Tue, Aug 29, 2017 at 11:05 AM, Andrew Wang 
>> wrote:
>>
>>> Sure. Ping me when the TSv2 goes in, and I can take care of branching.
>>>
>>> We're still waiting on the native services and S3Guard merges, but I
>>> don't want to hold branching to the last minute.
>>>
>>> On Tue, Aug 29, 2017 at 10:51 AM, Vrushali C 
>>> wrote:
>>>
 Hi Andrew,
 As Rohith mentioned, if you are good with it, from the TSv2 side, we
 are ready to go for merge tonight itself (Pacific time)  right after the
 voting period ends. Varun Saxena has been diligently rebasing up until now
 so most likely our merge should be reasonably straightforward.

 @Wangda: your resource profile vote ends tomorrow, could we please
 coordinate our merges?

 thanks
 Vrushali


 On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
 rohithsharm...@apache.org> wrote:

> On 29 August 2017 at 06:24, Andrew Wang 
> wrote:
>
> > So far I've seen no -1's to the branching proposal, so I plan to
> execute
> > this tomorrow unless there's further feedback.
> >
> For on going branch merge threads i.e TSv2, voting will be closing
> tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
> branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait
> for
> couple of more days before creating branch-3.0 so that TSv2 branch
> merge
> would be done directly to trunk?
>
>
>
> >
> > Regarding the above discussion, I think Jason and I have essentially
> the
> > same opinion.
> >
> > I hope that keeping trunk a release branch means a higher bar for
> merges
> > and code review in general. In the past, I've seen some patches
> committed
> > to trunk-only as a way of passing responsibility to a future user or
> > reviewer. That doesn't help anyone; patches should be committed with
> the
> > intent of running them in production.
> >
> > I'd also like to repeat the above thanks to the many, many
> contributors
> > who've helped with release improvements. Allen's work on
> create-release and
> > automated changes and release notes were essential, as was Xiao's
> work on
> > LICENSE and NOTICE files. I'm also looking forward to Marton's site
> > improvements, which addresses one of the remaining sore spots in the
> > release process.
> >
> > Things have gotten smoother with each alpha we've done over the last
> year,
> > and it's a testament to everyone's work that we have a good
> probability of
> > shipping beta and GA later this year.
> >
> > Cheers,
> > Andrew
> >
> >
>


>>>
>>
>

Re: [DISCUSS] Branches and versions for Hadoop 3

[Andrew Wang]

Hi Wangda,

I'll cut two branches: branch-3.0 (3.0.0-SNAPSHOT) and branch-3.0.0-beta1
(3.0.0-beta1-SNAPSHOT). This way we can merge GA features to branch-3.0 but
not branch-3.0.0-beta1.

Best,
Andrew

On Tue, Aug 29, 2017 at 11:18 AM, Wangda Tan  wrote:

> Vrushali,
>
> Sure we can wait TSv2 merged before merge resource profile branch.
>
> Andrew,
>
> My understanding is you're going to cut branch-3.0 for 3.0-beta1, and the
> same branch (branch-3.0) will be used for 3.0-GA as well. So my question
> is, there're several features (TSv2, resource profile, YARN-5734) are
> targeted to merge to 3.0-GA but not 3.0-beta1, which branch we should
> commit to, and when we can commit? Also, similar to 3.0.0-alpha1 to 4, you
> will cut branch-3.0.0-beta1, correct?
>
> Thanks,
> Wangda
>
>
> On Tue, Aug 29, 2017 at 11:05 AM, Andrew Wang 
> wrote:
>
>> Sure. Ping me when the TSv2 goes in, and I can take care of branching.
>>
>> We're still waiting on the native services and S3Guard merges, but I
>> don't want to hold branching to the last minute.
>>
>> On Tue, Aug 29, 2017 at 10:51 AM, Vrushali C 
>> wrote:
>>
>>> Hi Andrew,
>>> As Rohith mentioned, if you are good with it, from the TSv2 side, we are
>>> ready to go for merge tonight itself (Pacific time)  right after the voting
>>> period ends. Varun Saxena has been diligently rebasing up until now so most
>>> likely our merge should be reasonably straightforward.
>>>
>>> @Wangda: your resource profile vote ends tomorrow, could we please
>>> coordinate our merges?
>>>
>>> thanks
>>> Vrushali
>>>
>>>
>>> On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
>>> rohithsharm...@apache.org> wrote:
>>>
 On 29 August 2017 at 06:24, Andrew Wang 
 wrote:

 > So far I've seen no -1's to the branching proposal, so I plan to
 execute
 > this tomorrow unless there's further feedback.
 >
 For on going branch merge threads i.e TSv2, voting will be closing
 tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
 branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait for
 couple of more days before creating branch-3.0 so that TSv2 branch merge
 would be done directly to trunk?



 >
 > Regarding the above discussion, I think Jason and I have essentially
 the
 > same opinion.
 >
 > I hope that keeping trunk a release branch means a higher bar for
 merges
 > and code review in general. In the past, I've seen some patches
 committed
 > to trunk-only as a way of passing responsibility to a future user or
 > reviewer. That doesn't help anyone; patches should be committed with
 the
 > intent of running them in production.
 >
 > I'd also like to repeat the above thanks to the many, many
 contributors
 > who've helped with release improvements. Allen's work on
 create-release and
 > automated changes and release notes were essential, as was Xiao's
 work on
 > LICENSE and NOTICE files. I'm also looking forward to Marton's site
 > improvements, which addresses one of the remaining sore spots in the
 > release process.
 >
 > Things have gotten smoother with each alpha we've done over the last
 year,
 > and it's a testament to everyone's work that we have a good
 probability of
 > shipping beta and GA later this year.
 >
 > Cheers,
 > Andrew
 >
 >

>>>
>>>
>>
>

Re: [DISCUSS] Branches and versions for Hadoop 3

[Wangda Tan]

Vrushali,

Sure we can wait TSv2 merged before merge resource profile branch.

Andrew,

My understanding is you're going to cut branch-3.0 for 3.0-beta1, and the
same branch (branch-3.0) will be used for 3.0-GA as well. So my question
is, there're several features (TSv2, resource profile, YARN-5734) are
targeted to merge to 3.0-GA but not 3.0-beta1, which branch we should
commit to, and when we can commit? Also, similar to 3.0.0-alpha1 to 4, you
will cut branch-3.0.0-beta1, correct?

Thanks,
Wangda


On Tue, Aug 29, 2017 at 11:05 AM, Andrew Wang 
wrote:

> Sure. Ping me when the TSv2 goes in, and I can take care of branching.
>
> We're still waiting on the native services and S3Guard merges, but I don't
> want to hold branching to the last minute.
>
> On Tue, Aug 29, 2017 at 10:51 AM, Vrushali C 
> wrote:
>
>> Hi Andrew,
>> As Rohith mentioned, if you are good with it, from the TSv2 side, we are
>> ready to go for merge tonight itself (Pacific time)  right after the voting
>> period ends. Varun Saxena has been diligently rebasing up until now so most
>> likely our merge should be reasonably straightforward.
>>
>> @Wangda: your resource profile vote ends tomorrow, could we please
>> coordinate our merges?
>>
>> thanks
>> Vrushali
>>
>>
>> On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
>> rohithsharm...@apache.org> wrote:
>>
>>> On 29 August 2017 at 06:24, Andrew Wang 
>>> wrote:
>>>
>>> > So far I've seen no -1's to the branching proposal, so I plan to
>>> execute
>>> > this tomorrow unless there's further feedback.
>>> >
>>> For on going branch merge threads i.e TSv2, voting will be closing
>>> tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
>>> branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait for
>>> couple of more days before creating branch-3.0 so that TSv2 branch merge
>>> would be done directly to trunk?
>>>
>>>
>>>
>>> >
>>> > Regarding the above discussion, I think Jason and I have essentially
>>> the
>>> > same opinion.
>>> >
>>> > I hope that keeping trunk a release branch means a higher bar for
>>> merges
>>> > and code review in general. In the past, I've seen some patches
>>> committed
>>> > to trunk-only as a way of passing responsibility to a future user or
>>> > reviewer. That doesn't help anyone; patches should be committed with
>>> the
>>> > intent of running them in production.
>>> >
>>> > I'd also like to repeat the above thanks to the many, many contributors
>>> > who've helped with release improvements. Allen's work on
>>> create-release and
>>> > automated changes and release notes were essential, as was Xiao's work
>>> on
>>> > LICENSE and NOTICE files. I'm also looking forward to Marton's site
>>> > improvements, which addresses one of the remaining sore spots in the
>>> > release process.
>>> >
>>> > Things have gotten smoother with each alpha we've done over the last
>>> year,
>>> > and it's a testament to everyone's work that we have a good
>>> probability of
>>> > shipping beta and GA later this year.
>>> >
>>> > Cheers,
>>> > Andrew
>>> >
>>> >
>>>
>>
>>
>

Re: [DISCUSS] Branches and versions for Hadoop 3

[Andrew Wang]

Sure. Ping me when the TSv2 goes in, and I can take care of branching.

We're still waiting on the native services and S3Guard merges, but I don't
want to hold branching to the last minute.

On Tue, Aug 29, 2017 at 10:51 AM, Vrushali C 
wrote:

> Hi Andrew,
> As Rohith mentioned, if you are good with it, from the TSv2 side, we are
> ready to go for merge tonight itself (Pacific time)  right after the voting
> period ends. Varun Saxena has been diligently rebasing up until now so most
> likely our merge should be reasonably straightforward.
>
> @Wangda: your resource profile vote ends tomorrow, could we please
> coordinate our merges?
>
> thanks
> Vrushali
>
>
> On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
> rohithsharm...@apache.org> wrote:
>
>> On 29 August 2017 at 06:24, Andrew Wang  wrote:
>>
>> > So far I've seen no -1's to the branching proposal, so I plan to execute
>> > this tomorrow unless there's further feedback.
>> >
>> For on going branch merge threads i.e TSv2, voting will be closing
>> tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
>> branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait for
>> couple of more days before creating branch-3.0 so that TSv2 branch merge
>> would be done directly to trunk?
>>
>>
>>
>> >
>> > Regarding the above discussion, I think Jason and I have essentially the
>> > same opinion.
>> >
>> > I hope that keeping trunk a release branch means a higher bar for merges
>> > and code review in general. In the past, I've seen some patches
>> committed
>> > to trunk-only as a way of passing responsibility to a future user or
>> > reviewer. That doesn't help anyone; patches should be committed with the
>> > intent of running them in production.
>> >
>> > I'd also like to repeat the above thanks to the many, many contributors
>> > who've helped with release improvements. Allen's work on create-release
>> and
>> > automated changes and release notes were essential, as was Xiao's work
>> on
>> > LICENSE and NOTICE files. I'm also looking forward to Marton's site
>> > improvements, which addresses one of the remaining sore spots in the
>> > release process.
>> >
>> > Things have gotten smoother with each alpha we've done over the last
>> year,
>> > and it's a testament to everyone's work that we have a good probability
>> of
>> > shipping beta and GA later this year.
>> >
>> > Cheers,
>> > Andrew
>> >
>> >
>>
>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[varunsax...@apache.org]

+1 (binding).

Kudos to all the team members for their great work!

Being part of the ATSv2 team, I have been involved with either development
or review of most of the JIRAs'.
Tested ATSv2 in both secure and non-secure mode. Also verified that there
is no impact when ATSv2 is turned off.

Regards,
Varun Saxena.

On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
vrushalic2...@gmail.com> wrote:

> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>

Re: [DISCUSS] Branches and versions for Hadoop 3

[Vrushali C]

Hi Andrew,
As Rohith mentioned, if you are good with it, from the TSv2 side, we are
ready to go for merge tonight itself (Pacific time)  right after the voting
period ends. Varun Saxena has been diligently rebasing up until now so most
likely our merge should be reasonably straightforward.

@Wangda: your resource profile vote ends tomorrow, could we please
coordinate our merges?

thanks
Vrushali


On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> On 29 August 2017 at 06:24, Andrew Wang  wrote:
>
> > So far I've seen no -1's to the branching proposal, so I plan to execute
> > this tomorrow unless there's further feedback.
> >
> For on going branch merge threads i.e TSv2, voting will be closing
> tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
> branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait for
> couple of more days before creating branch-3.0 so that TSv2 branch merge
> would be done directly to trunk?
>
>
>
> >
> > Regarding the above discussion, I think Jason and I have essentially the
> > same opinion.
> >
> > I hope that keeping trunk a release branch means a higher bar for merges
> > and code review in general. In the past, I've seen some patches committed
> > to trunk-only as a way of passing responsibility to a future user or
> > reviewer. That doesn't help anyone; patches should be committed with the
> > intent of running them in production.
> >
> > I'd also like to repeat the above thanks to the many, many contributors
> > who've helped with release improvements. Allen's work on create-release
> and
> > automated changes and release notes were essential, as was Xiao's work on
> > LICENSE and NOTICE files. I'm also looking forward to Marton's site
> > improvements, which addresses one of the remaining sore spots in the
> > release process.
> >
> > Things have gotten smoother with each alpha we've done over the last
> year,
> > and it's a testament to everyone's work that we have a good probability
> of
> > shipping beta and GA later this year.
> >
> > Cheers,
> > Andrew
> >
> >
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Chris Trezzo]

+1

I helped deploy and run the ATSv2 aux service on a cluster with ~400 node
managers on it. I also verified that ATSv2 has no significant impact when
disabled.

Nice work everyone!

On Tue, Aug 29, 2017 at 6:17 AM, Aaron Gresch  wrote:

> +1 non-binding
>
> I did some testing with security off running both ATS v1 and v2.
>
> On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
> vrushalic2...@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/

[Aug 28, 2017 6:52:56 AM] (sunilg) YARN-7051. Avoid concurrent modification 
exception in
[Aug 28, 2017 5:09:46 PM] (yufei) YARN-7099. 
ResourceHandlerModule.parseConfiguredCGroupPath only works
[Aug 28, 2017 6:37:35 PM] (wangda) YARN-7112. TestAMRMProxy is failing with 
invalid request. (Jason Lowe
[Aug 28, 2017 10:49:59 PM] (arp) HDFS-12293. DataNode should log file name on 
disk error. Contributed by
[Aug 29, 2017 1:26:51 AM] (junping_du) YARN-7076. yarn application -list 
-appTypes is not working. Contributed




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 
   hadoop.hdfs.TestLeaseRecoveryStriped 
   hadoop.hdfs.TestClientProtocolForPipelineRecovery 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 
   hadoop.hdfs.TestEncryptedTransfer 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 
   hadoop.hdfs.TestFileCreationDelete 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 
   hadoop.yarn.sls.TestReservationSystemInvariants 
   hadoop.yarn.sls.TestSLSRunner 

Timed out junit tests :

   org.apache.hadoop.hdfs.TestWriteReadStripedFile 
   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/diff-compile-javac-root.txt
  [292K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/whitespace-tabs.txt
  [1.2M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [1.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/507/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [28K]

Powered by Apache Yetus 0.6.0-SNAPSHOT 

Re: Apache Hadoop 2.8.2 Release Plan

[Brahma Reddy Battula]

Hi All

Update on 2.8.2 release status
we are down to 3 critical issues ( YARN-6091,YARN-7083,HADOOP-9747),all are 
patch available and closer to commit.
Junping is closing tracking this.

Todo:

1) Update pom.xml ..?  currently it's with 2.8.3
https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21
2) Wiki is 
outdated, need to update the wiki..?
3) As this is going to stable release,are we planing enable Big top for 2.8.2 
testing Or Dynamometer testing (anybody from linked-in can help)..?

@Junping Du,Please correct me,if I am wrong.


--Brahma Reddy Battula

From: Junping Du 
Sent: Monday, August 7, 2017 2:44 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x
Issue Navigator - ASF JIRA
s.apache.org
Linked Applications. Loading… Dashboards




I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping

From: Junping Du 
Sent: Monday, July 24, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping



From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If no other concerns, I will do the branch update in next 30 minutes.


Thanks,


Junping



From: Brahma Reddy Battula 
Sent: Sunday, July 23, 2017 1:50 AM
To: Junping Du; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan



Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are 
missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two 
commits.Hope we'll not miss any commits which present in only in branch-2.8.2.


From: Junping Du 
Sent: Saturday, July 22, 2017 5:57 AM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping


From: Junping Du 
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 

[jira] [Reopened] (MAPREDUCE-6641) TestTaskAttempt fails in trunk

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened MAPREDUCE-6641:
---

Seeing this fail the same way in 2.8 builds as well.  Unfortunately since the 
fix uses lambdas I can't just cherry-pick the fix down to other branches.  
Reopening so Jenkins can comment on a branch-2 version of the patch.

> TestTaskAttempt fails in trunk
> --
>
> Key: MAPREDUCE-6641
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6641
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Tsuyoshi Ozawa
>Assignee: Haibo Chen
> Fix For: 3.0.0-alpha1
>
> Attachments: mapreduce6641.001.patch, mapreduce6641.002.patch, 
> MAPREDUCE-6641-branch-2.002.patch, 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt-output.txt
>
>
> {code}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.917 sec 
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> testMRAppHistoryForTAFailedInAssigned(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>   Time elapsed: 12.732 sec  <<< FAILURE!
> java.lang.AssertionError: No Ta Started JH Event
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testTaskAttemptAssignedKilledHistory(TestTaskAttempt.java:388)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testMRAppHistoryForTAFailedInAssigned(TestTaskAttempt.java:177)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Jason Lowe]

+1 (binding)

I participated in the review for the reader authorization and verified that
ATSv2 has no significant impact when disabled.  Looking forward to seeing
the next increment in functionality in a release.  A big thank you to
everyone involved in this effort!

Jason


On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
vrushalic2...@gmail.com> wrote:

> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>