Re: NEWS Layout

[Joshua Cohen]

+1

On Tue, Feb 2, 2016 at 1:09 PM, Bill Farner  wrote:

> +1
>
> On Tuesday, February 2, 2016, Erb, Stephan 
> wrote:
>
> > Hi everyone,
> >
> > I'd like to propose that we give our NEWS file a little bit more
> > structure. Currently, it is quite cluttered [1].
> >
> > To keep it simple, I'd suggest that we adopt the style from the 0.11
> > Aurora blog post:
> >
> > * New/updated
> > * Deprecations and removals
> >
> > [1] https://github.com/apache/aurora/blob/master/NEWS
> > [2] https://aurora.apache.org/blog/aurora-0-11-0-released/
> >
> >
> > Thoughts?
> >
>

Re: NEWS Layout

[David McLaughlin]

+1

On Tue, Feb 2, 2016 at 11:25 AM, Jake Farrell  wrote:

> sounds good. thanks Stephan
>
> -Jake
>
> On Tue, Feb 2, 2016 at 2:05 PM, Erb, Stephan 
> wrote:
>
> > Hi everyone,
> >
> > I'd like to propose that we give our NEWS file a little bit more
> > structure. Currently, it is quite cluttered [1].
> >
> > To keep it simple, I'd suggest that we adopt the style from the 0.11
> > Aurora blog post:
> >
> > * New/updated
> > * Deprecations and removals
> >
> > [1] https://github.com/apache/aurora/blob/master/NEWS
> > [2] https://aurora.apache.org/blog/aurora-0-11-0-released/
> >
> >
> > Thoughts?
> >
>

Re: NEWS Layout

[Jake Farrell]

sounds good. thanks Stephan

-Jake

On Tue, Feb 2, 2016 at 2:05 PM, Erb, Stephan 
wrote:

> Hi everyone,
>
> I'd like to propose that we give our NEWS file a little bit more
> structure. Currently, it is quite cluttered [1].
>
> To keep it simple, I'd suggest that we adopt the style from the 0.11
> Aurora blog post:
>
> * New/updated
> * Deprecations and removals
>
> [1] https://github.com/apache/aurora/blob/master/NEWS
> [2] https://aurora.apache.org/blog/aurora-0-11-0-released/
>
>
> Thoughts?
>

Re: NEWS Layout

[Erb, Stephan]

Thanks for the quick responses! Now on master: 
https://reviews.apache.org/r/43109/ 

From: Zameer Manji 
Sent: Tuesday, February 2, 2016 8:35 PM
To: dev@aurora.apache.org
Subject: Re: NEWS Layout

+1

On Tue, Feb 2, 2016 at 11:32 AM, John Sirois  wrote:

> +1
>
> On Tue, Feb 2, 2016 at 12:29 PM, Joshua Cohen  wrote:
>
> > +1
> >
> > On Tue, Feb 2, 2016 at 1:09 PM, Bill Farner  wrote:
> >
> > > +1
> > >
> > > On Tuesday, February 2, 2016, Erb, Stephan <
> stephan@blue-yonder.com>
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'd like to propose that we give our NEWS file a little bit more
> > > > structure. Currently, it is quite cluttered [1].
> > > >
> > > > To keep it simple, I'd suggest that we adopt the style from the 0.11
> > > > Aurora blog post:
> > > >
> > > > * New/updated
> > > > * Deprecations and removals
> > > >
> > > > [1] https://github.com/apache/aurora/blob/master/NEWS
> > > > [2] https://aurora.apache.org/blog/aurora-0-11-0-released/
> > > >
> > > >
> > > > Thoughts?
> > > >
> > >
> >
>
>
>
> --
> John Sirois
> 303-512-3301
>
> --
> Zameer Manji
>
> <303-512-3301>

Re: Rollback Testing

[Joshua Cohen]

https://issues.apache.org/jira/browse/AURORA-1608
https://issues.apache.org/jira/browse/AURORA-1609

On Tue, Feb 2, 2016 at 7:42 PM, Joshua Cohen  wrote:

> Ok, I'll file a ticket to at least track the intent. We'll see if I have
> time to work on it or if it just languishes (anyone else interested, feel
> free to pick it up as well of course ;)).
>
> On Mon, Feb 1, 2016 at 4:28 PM, Bill Farner  wrote:
>
>> Yes.  Our nightly packaging builds take advantage of this.
>>
>> On Mon, Feb 1, 2016 at 2:27 PM, Joshua Cohen  wrote:
>>
>> > I know the blocker for e2e tests in CI previously was the ability to run
>> > vagrant. Is running docker from CI doable today?
>> >
>> > On Mon, Feb 1, 2016 at 2:40 PM, Bill Farner  wrote:
>> >
>> > > Definitely a nice thing to have, the big uncertainty will be whether
>> > anyone
>> > > cares enough to see the effort through.
>> > >
>> > > e2e tests in jenkins can be done, but likely only if e2e tests start
>> > using
>> > > docker instead of vagrant.  I am supportive of both, but cannot
>> > personally
>> > > invest the time at the moment.
>> > >
>> > > On Mon, Feb 1, 2016 at 12:14 PM, Joshua Cohen 
>> wrote:
>> > >
>> > > > Another topic that came up today's IRC meeting was possibly adding
>> some
>> > > > sort of automated rollback testing between builds. This is related
>> to
>> > > > https://issues.apache.org/jira/browse/AURORA-1603 which was caused
>> by
>> > an
>> > > > inability to rollback to an earlier commit after discovering
>> > > > https://issues.apache.org/jira/browse/AURORA-1605.
>> > > >
>> > > > I'm not sure what's feasible in the immediate future. To me the most
>> > > > obvious solution would be to run the e2e tests as part of our CI
>> job,
>> > and
>> > > > have the last step of the job be to checkout HEAD^ and
>> rebuild/restart
>> > > the
>> > > > scheduler. This is predicated on actually running the e2e tests as
>> part
>> > > of
>> > > > CI though, which historically has been difficult for us to achieve
>> > (see:
>> > > > https://issues.apache.org/jira/browse/AURORA-127).
>> > > >
>> > > > Curious to hear thoughts on this. It's clearly more beneficial to
>> those
>> > > > deploying from master rather than from official releases (though it
>> > would
>> > > > be great, at least, as part of the release verification process to
>> > ensure
>> > > > we can rollback to the previous release). Do folks think this would
>> be
>> > > > beneficial, or is it overkill?
>> > > >
>> > > > Cheers,
>> > > >
>> > > > Joshua
>> > > >
>> > >
>> >
>>
>
>

Re: Rollback Testing

[Joshua Cohen]

Ok, I'll file a ticket to at least track the intent. We'll see if I have
time to work on it or if it just languishes (anyone else interested, feel
free to pick it up as well of course ;)).

On Mon, Feb 1, 2016 at 4:28 PM, Bill Farner  wrote:

> Yes.  Our nightly packaging builds take advantage of this.
>
> On Mon, Feb 1, 2016 at 2:27 PM, Joshua Cohen  wrote:
>
> > I know the blocker for e2e tests in CI previously was the ability to run
> > vagrant. Is running docker from CI doable today?
> >
> > On Mon, Feb 1, 2016 at 2:40 PM, Bill Farner  wrote:
> >
> > > Definitely a nice thing to have, the big uncertainty will be whether
> > anyone
> > > cares enough to see the effort through.
> > >
> > > e2e tests in jenkins can be done, but likely only if e2e tests start
> > using
> > > docker instead of vagrant.  I am supportive of both, but cannot
> > personally
> > > invest the time at the moment.
> > >
> > > On Mon, Feb 1, 2016 at 12:14 PM, Joshua Cohen 
> wrote:
> > >
> > > > Another topic that came up today's IRC meeting was possibly adding
> some
> > > > sort of automated rollback testing between builds. This is related to
> > > > https://issues.apache.org/jira/browse/AURORA-1603 which was caused
> by
> > an
> > > > inability to rollback to an earlier commit after discovering
> > > > https://issues.apache.org/jira/browse/AURORA-1605.
> > > >
> > > > I'm not sure what's feasible in the immediate future. To me the most
> > > > obvious solution would be to run the e2e tests as part of our CI job,
> > and
> > > > have the last step of the job be to checkout HEAD^ and
> rebuild/restart
> > > the
> > > > scheduler. This is predicated on actually running the e2e tests as
> part
> > > of
> > > > CI though, which historically has been difficult for us to achieve
> > (see:
> > > > https://issues.apache.org/jira/browse/AURORA-127).
> > > >
> > > > Curious to hear thoughts on this. It's clearly more beneficial to
> those
> > > > deploying from master rather than from official releases (though it
> > would
> > > > be great, at least, as part of the release verification process to
> > ensure
> > > > we can rollback to the previous release). Do folks think this would
> be
> > > > beneficial, or is it overkill?
> > > >
> > > > Cheers,
> > > >
> > > > Joshua
> > > >
> > >
> >
>

Re: NEWS Layout

[Bill Farner]

+1

On Tuesday, February 2, 2016, Erb, Stephan 
wrote:

> Hi everyone,
>
> I'd like to propose that we give our NEWS file a little bit more
> structure. Currently, it is quite cluttered [1].
>
> To keep it simple, I'd suggest that we adopt the style from the 0.11
> Aurora blog post:
>
> * New/updated
> * Deprecations and removals
>
> [1] https://github.com/apache/aurora/blob/master/NEWS
> [2] https://aurora.apache.org/blog/aurora-0-11-0-released/
>
>
> Thoughts?
>

Re: NEWS Layout

[John Sirois]

+1

On Tue, Feb 2, 2016 at 12:29 PM, Joshua Cohen  wrote:

> +1
>
> On Tue, Feb 2, 2016 at 1:09 PM, Bill Farner  wrote:
>
> > +1
> >
> > On Tuesday, February 2, 2016, Erb, Stephan 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to propose that we give our NEWS file a little bit more
> > > structure. Currently, it is quite cluttered [1].
> > >
> > > To keep it simple, I'd suggest that we adopt the style from the 0.11
> > > Aurora blog post:
> > >
> > > * New/updated
> > > * Deprecations and removals
> > >
> > > [1] https://github.com/apache/aurora/blob/master/NEWS
> > > [2] https://aurora.apache.org/blog/aurora-0-11-0-released/
> > >
> > >
> > > Thoughts?
> > >
> >
>



-- 
John Sirois
303-512-3301

0.12.0 RC status

[John Sirois]

Although the last blocker raised for the 0.12.0 RC series has been resolved
[1], it looks like resolution of several issues related to rolling back to
0.11.0 are required to cut the next RC:
1. "Scheduler fails to start after rollback":
https://issues.apache.org/jira/browse/AURORA-1603
2. "Add a flag to disable the HTTP redirect to the leader":
https://issues.apache.org/jira/browse/AURORA-1601
3. "Update recovery docs to reflect changes":
https://issues.apache.org/jira/browse/AURORA-1605

These issues fall into 2 classes:
Item 1 above needs to fix the immediate problem of rolling back to 0.11.0;
although there may be more changes to process, tooling and code to support
the problem better going forward.
Items 2 & 3 address tooling & procedure that support rollback.

It looks like Maxim has claimed item 1/AURORA-1603 and Joshua is working
item 2/AURORA-1601.  I assume one of Maxim, Joshua or Zameer will tackle
item 3/AURORA-1605 to update rollback docs with what they learned rolling
back.

If I have any of this wrong, please speak up; otherwise I'll be cutting the
next 0.12.0 RC3 when the above 3 issues are resolved.

[1] "Identity.role is still used in the UI leading to duplicate instances
on job page": https://issues.apache.org/jira/browse/AURORA-1604

Re: [PROPOSAL] Change java thrift code gen

[John Sirois]

To wrap things up - I'm considering this proposal as having failed; so I've
marked the 3 associated RBs as discarded.

I'll be working on https://issues.apache.org/jira/browse/THRIFT-3583 to
introduce an immutable builder-style of java codegen that carries over
thrift annotations to Apache Thrift.  Once that is complete and released,
I'll circle back and re-do an RB like 3/3 (
https://reviews.apache.org/r/42756/) that flips over the codebase to the
new apache thrift gen.

On Thu, Jan 28, 2016 at 10:08 AM, John Sirois  wrote:

>
>
> On Thu, Jan 28, 2016 at 8:26 AM, John Sirois  wrote:
>
>> On Thu, Jan 28, 2016 at 6:45 AM, Jake Farrell 
>> wrote:
>>
>> > +1 to making this apart of Thrift, i'm happy to help shepard this on the
>> > Thrift side and get it in as soon as its ready
>> >
>>
>> I've filed https://issues.apache.org/jira/browse/THRIFT-3583 to use as
>> the
>> basis for discussion of this feature over in the Apache Thrift project.
>> I looked at the problem a bit and noted some challenges.
>>
>
> And started a d...@thrift.apache.org thread here if anyone wants to follow
> along or participate: http://markmail.org/message/mlxyyauyvlvuxjsf
>
>
>>
>>
>> >
>> > -Jake
>> >
>> > On Wed, Jan 27, 2016 at 8:08 PM, Maxim Khutornenko 
>> > wrote:
>> >
>> >> I am +1 to making immutable thrift objects solely based on perf
>> numbers.
>> >>
>> >> My biggest concern though is maintenance of a pretty intricate
>> codebase,
>> >> especially when it comes to upgrading any of the frameworks involved.
>> >> Bill's suggestion to explore paths to make this a part of Apache Thrift
>> >> sounds great to me.
>> >>
>> >> On Wed, Jan 27, 2016 at 5:00 PM, Bill Farner 
>> wrote:
>> >>
>> >> > Tony - this would not be a technical fork so much as a spiritual
>> fork.
>> >> > While we would have our own bugs to work out, the only upstream
>> exposure
>> >> > would be IDL or wire format changes.
>> >> >
>> >> > On Wed, Jan 27, 2016 at 4:58 PM, Tony Dong > >
>> >> > wrote:
>> >> >
>> >> > > Awesome performance numbers! I don't particularly know the
>> logistics
>> >> of
>> >> > > upstreaming a change like this, but optimistically I would suggest
>> >> > > upstreaming it to Apache Thrift if possible.
>> >> > >
>> >> > > As someone that maintains a fork of a thrift compiler(fork of
>> >> scrooge), I
>> >> > > have to say that it's not that fun. There's a lot of custom code
>> that
>> >> > needs
>> >> > > to be maintained and a bunch of work to rebase the code
>> periodically.
>> >> > >
>> >> > >
>> >> > >
>> >> > > On Wed, Jan 27, 2016 at 4:51 PM, Bill Farner 
>> >> wrote:
>> >> > >
>> >> > > > Firstly - thanks for the clean organization and delineation of
>> >> steps in
>> >> > > > this change.  Top notch work!
>> >> > > >
>> >> > > > Some of the performance improvements are very nice; and in a
>> >> > particularly
>> >> > > > hot code path.  I will wager a guess that the majority of the
>> >> savings
>> >> > is
>> >> > > in
>> >> > > > avoiding what amounts to copy constructors between mutable and
>> >> > immutable
>> >> > > > types.  I further wager there are alternative approaches we could
>> >> weigh
>> >> > > to
>> >> > > > achieve those performance improvements.  As an example - you note
>> >> above
>> >> > > > that we could provide a patch to Apache Thrift.  Depending how
>> much
>> >> > > > performance inspires our decision here, it will be prudent to
>> >> evaluate
>> >> > > > alternatives.
>> >> > > >
>> >> > > > I think there are (at least) two major issues worth discussing -
>> >> code
>> >> > > > volume (which you note) and an increase in logical complexity.
>> This
>> >> > will
>> >> > > > leave us with a bifurcation in code generation tooling
>> (custom+swift
>> >> > for
>> >> > > > Java, Apache Thrift for python and js).  It's difficult to
>> quantify
>> >> the
>> >> > > > downside of that, but it seems like an unfortunate state with
>> >> potential
>> >> > > for
>> >> > > > compatibility risks.
>> >> > > >
>> >> > > >
>> >> > > > On Wed, Jan 27, 2016 at 2:45 PM, Zameer Manji > >
>> >> > wrote:
>> >> > > >
>> >> > > > > Some high level comments without looking at the code.
>> >> > > > >
>> >> > > > > I'm in favor from abandoning the thrift generated java code in
>> >> favor
>> >> > of
>> >> > > > > immutable objects. I think it is easier to reason about and
>> will
>> >> > ensure
>> >> > > > we
>> >> > > > > have less errors in our code. If I understand correctly, the
>> >> ProtoBuf
>> >> > > > > format does this by default, so there some precedent for this
>> >> style
>> >> > of
>> >> > > > code
>> >> > > > > generation already.
>> >> > > > >
>> >> > > > > I think using Facebook's swift is the best approach here. I
>> would
>> >> be
>> >> > > > > hesitant to accept any custom code generation that involved us
>> >> > parsing
>> >> > > > > thrift IDL 

Re: 0.12.0 RC status

[Maxim Khutornenko]

+1 to having 1603 and 1601 as blockers. I am planning to work on 1603 today.

As for 1605, I don't believe it's a blocker given that all findings are
already documented in the ticket.

On Tue, Feb 2, 2016 at 7:03 AM, Joshua Cohen  wrote:

> I'd only consider item 1 to be a blocker to 0.12.0, but 2 and 3 should be
> relatively quick so in general this sounds like a reasonable plan of action
> to me.
>
> On Tue, Feb 2, 2016 at 8:52 AM, John Sirois  wrote:
>
> > Although the last blocker raised for the 0.12.0 RC series has been
> resolved
> > [1], it looks like resolution of several issues related to rolling back
> to
> > 0.11.0 are required to cut the next RC:
> > 1. "Scheduler fails to start after rollback":
> > https://issues.apache.org/jira/browse/AURORA-1603
> > 2. "Add a flag to disable the HTTP redirect to the leader":
> > https://issues.apache.org/jira/browse/AURORA-1601
> > 3. "Update recovery docs to reflect changes":
> > https://issues.apache.org/jira/browse/AURORA-1605
> >
> > These issues fall into 2 classes:
> > Item 1 above needs to fix the immediate problem of rolling back to
> 0.11.0;
> > although there may be more changes to process, tooling and code to
> support
> > the problem better going forward.
> > Items 2 & 3 address tooling & procedure that support rollback.
> >
> > It looks like Maxim has claimed item 1/AURORA-1603 and Joshua is working
> > item 2/AURORA-1601.  I assume one of Maxim, Joshua or Zameer will tackle
> > item 3/AURORA-1605 to update rollback docs with what they learned rolling
> > back.
> >
> > If I have any of this wrong, please speak up; otherwise I'll be cutting
> the
> > next 0.12.0 RC3 when the above 3 issues are resolved.
> >
> > [1] "Identity.role is still used in the UI leading to duplicate instances
> > on job page": https://issues.apache.org/jira/browse/AURORA-1604
> >
>