Re: [VOTE] Release Apache Aurora 0.18.1 RC1

[Joshua Cohen]

+1

On Sun, Oct 29, 2017 at 6:30 PM, Bill Farner <wfar...@apache.org> wrote:

> >
> > sha512 signature
>
>
> Thanks, this is now fixed.  The release script runs from the released SHA,
> which pre-dates the inclusion of a sha512 signature.  The verification
> script on master should otherwise work against 0.18.1-rc1, but no
> guarantees.
>
> On Sun, Oct 29, 2017 at 3:20 PM, Joshua Cohen <jco...@apache.org> wrote:
>
> > I'm trying to run the verify-release-candidate script, but getting a 404
> > for the sha512 signature?
> >
> > + download_rc_file apache-aurora-0.18.1-rc1.tar.gz.sha512
> > + download_dist_file 0.18.1-rc1/apache-aurora-0.18.1-rc1.tar.gz.sha512
> > + curl -f -O
> > https://dist.apache.org/repos/dist/dev/aurora/0.18.1-rc1/
> > apache-aurora-0.18.1-rc1.tar.gz.sha512
> >   % Total% Received % Xferd  Average Speed   TimeTime Time
> > Current
> >  Dload  Upload   Total   SpentLeft
> > Speed
> >   0 00 00 0  0  0 --:--:-- --:--:-- --:--:--
> >  0
> > curl: (22) The requested URL returned error: 404 Not Found
> >
> >
> > On Sun, Oct 29, 2017 at 5:03 PM, Bill Farner <wfar...@apache.org> wrote:
> >
> > > +1
> > >
> > > Verified by running ./build-support/release/verify-release-candidate
> > > 0.18.1-rc1
> > >
> > > On Sun, Oct 29, 2017 at 12:05 PM, David McLaughlin <
> > dmclaugh...@apache.org
> > > >
> > > wrote:
> > >
> > > > +1
> > > >
> > > > On Sun, Oct 29, 2017 at 10:32 AM, Bill Farner <wfar...@apache.org>
> > > wrote:
> > > >
> > > > > All,
> > > > >
> > > > > I propose that we accept the following release candidate as the
> > > official
> > > > > Apache Aurora 0.18.1 release.
> > > > >
> > > > > Aurora 0.18.1-rc1 includes the following:
> > > > > ---
> > > > > The RELEASE NOTES for the release are available at:
> > > > > https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> > > > > RELEASE-NOTES.md=rel/0.18.1-rc1
> > > > >
> > > > > The CHANGELOG for the release is available at:
> > > > > https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> > > > > CHANGELOG=rel/0.18.1-rc1
> > > > >
> > > > > The tag used to create the release candidate is:
> > > > > https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=
> > > > > shortlog;h=refs/tags/rel/0.18.1-rc1
> > > > >
> > > > > The release candidate is available at:
> > > > > https://dist.apache.org/repos/dist/dev/aurora/0.18.1-rc1/
> > > > > apache-aurora-0.18.1-rc1.tar.gz
> > > > >
> > > > > The MD5 checksum of the release candidate can be found at:
> > > > > https://dist.apache.org/repos/dist/dev/aurora/0.18.1-rc1/
> > > > > apache-aurora-0.18.1-rc1.tar.gz.md5
> > > > >
> > > > > The signature of the release candidate can be found at:
> > > > > https://dist.apache.org/repos/dist/dev/aurora/0.18.1-rc1/
> > > > > apache-aurora-0.18.1-rc1.tar.gz.asc
> > > > >
> > > > > The GPG key used to sign the release are available at:
> > > > > https://dist.apache.org/repos/dist/dev/aurora/KEYS
> > > > >
> > > > > Please download, verify, and test.
> > > > >
> > > > > The vote will close on Wed Nov  1 10:31:07 PDT 2017
> > > > >
> > > > > [ ] +1 Release this as Apache Aurora 0.18.1
> > > > > [ ] +0
> > > > > [ ] -1 Do not release this as Apache Aurora 0.18.1 because...
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Aurora 0.18.1 RC1

[Joshua Cohen]

I'm trying to run the verify-release-candidate script, but getting a 404
for the sha512 signature?

+ download_rc_file apache-aurora-0.18.1-rc1.tar.gz.sha512
+ download_dist_file 0.18.1-rc1/apache-aurora-0.18.1-rc1.tar.gz.sha512
+ curl -f -O
https://dist.apache.org/repos/dist/dev/aurora/0.18.1-rc1/apache-aurora-0.18.1-rc1.tar.gz.sha512
  % Total% Received % Xferd  Average Speed   TimeTime Time
Current
 Dload  Upload   Total   SpentLeft
Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:--
 0
curl: (22) The requested URL returned error: 404 Not Found


On Sun, Oct 29, 2017 at 5:03 PM, Bill Farner  wrote:

> +1
>
> Verified by running ./build-support/release/verify-release-candidate
> 0.18.1-rc1
>
> On Sun, Oct 29, 2017 at 12:05 PM, David McLaughlin  >
> wrote:
>
> > +1
> >
> > On Sun, Oct 29, 2017 at 10:32 AM, Bill Farner 
> wrote:
> >
> > > All,
> > >
> > > I propose that we accept the following release candidate as the
> official
> > > Apache Aurora 0.18.1 release.
> > >
> > > Aurora 0.18.1-rc1 includes the following:
> > > ---
> > > The RELEASE NOTES for the release are available at:
> > > https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> > > RELEASE-NOTES.md=rel/0.18.1-rc1
> > >
> > > The CHANGELOG for the release is available at:
> > > https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> > > CHANGELOG=rel/0.18.1-rc1
> > >
> > > The tag used to create the release candidate is:
> > > https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=
> > > shortlog;h=refs/tags/rel/0.18.1-rc1
> > >
> > > The release candidate is available at:
> > > https://dist.apache.org/repos/dist/dev/aurora/0.18.1-rc1/
> > > apache-aurora-0.18.1-rc1.tar.gz
> > >
> > > The MD5 checksum of the release candidate can be found at:
> > > https://dist.apache.org/repos/dist/dev/aurora/0.18.1-rc1/
> > > apache-aurora-0.18.1-rc1.tar.gz.md5
> > >
> > > The signature of the release candidate can be found at:
> > > https://dist.apache.org/repos/dist/dev/aurora/0.18.1-rc1/
> > > apache-aurora-0.18.1-rc1.tar.gz.asc
> > >
> > > The GPG key used to sign the release are available at:
> > > https://dist.apache.org/repos/dist/dev/aurora/KEYS
> > >
> > > Please download, verify, and test.
> > >
> > > The vote will close on Wed Nov  1 10:31:07 PDT 2017
> > >
> > > [ ] +1 Release this as Apache Aurora 0.18.1
> > > [ ] +0
> > > [ ] -1 Do not release this as Apache Aurora 0.18.1 because...
> > >
> >
>

Re: Future of storage in Aurora

[Joshua Cohen]

What does this mean in terms of the original goals behind the storage
system refactor? Are we confident that Jordan's work for hot-followers will
alleviate the problems w/ long failovers? I'm definitely in favor of
killing the H2 code if its goals can never be realized and it's just a
maintenance burden, but I'd also like to know what our plans are for
storage in the future.

Also, what does this mean for stores that have never existed as non-H2
(i.e. the job update store). Will converting it have an impact on, e.g.,
storage write-lock contention?

On Sun, Oct 1, 2017 at 5:59 PM, Bill Farner <wfar...@apache.org> wrote:

> I would like to revive this discussion in light of some work i have been
> doing around the storage system.  The fruits of the DB storage system will
> require a lot of additional effort to reach the beneficial outcomes i laid
> out above, and i agree that we should cut our losses.
>
> I plan to introduce patches soon to introduce non-H2 in-memory store
> implementations.  *If anyone disagrees with removing the H2 implementations
> as well, please chime in here.*
>
> Disclaimer - i may propose an alternative for the persistent storage in the
> near future.
>
> On Mon, Apr 3, 2017 at 9:40 AM, Stephan Erb <s...@apache.org> wrote:
>
> > H2 could give us fine granular data access. However, most of our code
> > performs massive joins to reconstruct fully hydrated thrift objects.
> > Most of the time we are then only interested in very few properties of
> > those thrift structs. This applies to internal usage, but also how we
> > use the API.
> >
> > I therefore believe we have to improve and refine our domain model in
> > order to significantly improve the storage situation.
> >
> > I really liked Maxim's proposal from last year, and I think it is worth
> > reconsidering: https://docs.google.com/document/d/1myYX3yuofGr8JIzud98x
> > Xd5mqgpZ8q_RqKBpSff4-WE/edit
> >
> > Best regards,
> > Stephan
> >
> > On Thu, 2017-03-30 at 15:53 -0700, David McLaughlin wrote:
> > > So it sounds like before we make any decisions around removing the
> > > work
> > > done in H2 so far, we should figure out what is remaining to move to
> > > external storage (or if it's even still a goal).
> > >
> > > I may still play around with reviving the in-memory stores, but will
> > > separate that work from any goal to remove the H2 layer. Since it's
> > > motivated by performance, I'd verify there is a benefit before
> > > submitting
> > > any review.
> > >
> > > Thanks all for the feedback.
> > >
> > >
> > > On Thu, Mar 30, 2017 at 12:08 PM, Bill Farner <wfarnerapa...@gmail.co
> > > m>
> > > wrote:
> > >
> > > > Adding some background - there were several motivators to using SQL
> > > > that
> > > > come to mind:
> > > > a) well-understood transaction isolation guarantees leading to a
> > > > simpler
> > > > programming model w.r.t. concurrency
> > > > b) ability to offload storage to a separate system (e.g. Postgres)
> > > > and
> > > > scale it separately
> > > > c) relief of computational burden of performing snapshots and
> > > > backups due
> > > > to (b)
> > > > d) simpler code and operations model due to (b)
> > > > e) schema backwards compatibility guarantees due to persistence-
> > > > friendly
> > > > migration-scripts
> > > > f) straightforward normalization to facilitate sharing of
> > > > otherwise-redundant state (I.e. TaskConfig)
> > > >
> > > > The storage overhaul comes with a huge caveat requiring the
> > > > approach to
> > > > scheduling rounds to change. I concur that the current model is
> > > > hostile to
> > > > offloaded storage, as ~all state must be read every scheduling
> > > > round. If
> > > > that cannot be worked around with lazy state or best-effort
> > > > concurrency
> > > > (I.e. in-memory caching), the approach is indeed flawed.
> > > >
> > > > On Mar 30, 2017, 10:29 AM -0700, Joshua Cohen <jco...@apache.org>,
> > > > wrote:
> > > > > My understanding of the H2-backed stores is that at least part of
> > > > > the
> > > > > original rationale behind them was that they were meant to be an
> > > > > interim
> > > > > point on the way to external SQL-backed stores which should
> > > > > theoretically
&g

Re: Make drain MAX_STATUS_WAIT configurable

[Joshua Cohen]

echoing my +1 from Slack here.

On Tue, Sep 5, 2017 at 2:05 PM, Erb, Stephan 
wrote:

> +1
>
> On 05.09.17, 19:17, "David McLaughlin"  wrote:
>
> +1
>
> On Tue, Sep 5, 2017 at 10:09 AM, Mauricio Garavaglia <
> mauriciogaravag...@gmail.com> wrote:
>
> > Hi folks,
> >
> > The aurora-admin drain command currently has a hardcoded limit of 5
> minutes
> > waiting for a node to be drained, after that timeout it fails.
> >
> > This doesn't work very well when tasks are expected be in killing
> state for
> > more than that, for example, if the scheduler
> transient_task_state_timeout
> > was
> > adjusted.
> >
> > Any objection on making this 5 minutes MAX_STATUS_WAIT in
> aurora-admin
> > drain configurable?
> >
> > https://github.com/apache/aurora/blob/master/src/main/
> > python/apache/aurora/admin/host_maintenance.py#L40
> >
> > Thanks,
> >
> > Mauricio
> >
>
>
>

Re: Redesign of the Aurora UI

[Joshua Cohen]

FB came back w/ their response to relicense React today:
https://code.facebook.com/posts/112130496157735/explaining-react-s-license

tl;dr: they're not relicensing React. So... if we do go down this path,
we'll have to investigate alternatives to React (Preact, Vue, etc.) or
potentially release this as a separate project outside of the ASF that can
serve as a drop in UI replacement.

On Sun, Jul 23, 2017 at 4:48 PM, Erb, Stephan <stephan@blue-yonder.com>
wrote:

> A big +1 from me as well. We have not touched or updated the existing UI
> for quite some time, which is a bad sign for code health.
>
> I would even be OK with a couple of bigger initial code dumps. I am not
> really a web-developer, so a working piece of code to play around with
> would probably be the fastest way to get up to speed with the tech and its
> usage in Aurora.
>
> Thanks a lot for driving this, David!
>
> On 21.07.17, 07:00, "Kai Huang" <texasred2...@hotmail.com> wrote:
>
> David - Sure, let's sync on the work when you are ready.
>
>
> 
> From: David McLaughlin <dmclaugh...@apache.org>
> Sent: Wednesday, July 19, 2017 14:10
> To: dev@aurora.apache.org
> Subject: Re: Redesign of the Aurora UI
>
> Thanks for the feedback!
>
> Joshua - I haven't tried to drop in Preact yet, but I was also
> planning to
> throw away the prototype and starting again when it came to
> upstreaming it,
> so as part of that we can just address incompatibilities as we go. If
> I was
> to guess, then the only significant impact on my prototype would
> probably
> be the reactable plugin I was using (replacement for Angular's
> smart-table). But longer term I do have concerns about moving away from
> what is a constantly improving and healthy ecosystem around React. So
> most
> likely I'll hold off until a decision is made there one way or the
> other
> (which should be within a week).
>
>
> Kai - I'd be happy to coordinate and collaborate on this with others.
> Let
> me try and finish up the CSS/UX of the pages in my prototype and from
> there
> we can sync on who does what. Does that sound good?
>
>
>
>
> On Wed, Jul 19, 2017 at 1:56 PM, Kai Huang <texasred2...@hotmail.com>
> wrote:
>
> > Just a few thoughts as an aurora developer and operator:
> >
> >
> > From my experiences with Aurora users, some persisting complaints
> are:
> >
> >   1.  The current UI is not very intuitive for the users to
> understand the
> > task lifecycle, resource utilization of their job.
> >   2.  Often times users are unaware of the new features/changes in
> Aurora
> > Scheduler/Executor, which leads to a lot of misuse of the system.
> >   3.  Users have preferences on the appearance of the
> scheduler/thermos UI
> > due to special use cases, and ask us to customize for them(or start
> to
> > write their own UI, which is often not recommended).
> >
> >
> > The other major issue I see in the current UI is that it's built on
> an
> > obsolete tech stack(AngularJS) that has all the binaries and
> dependencies
> > in the repo. From a developer's perspective, it's a big burden to
> > maintain/test the code, and make fast iterations on it.
> >
> >
> > Currently the scheduler UI is readonly, and mainly designed for
> debugging
> > purposes. We could have done much better to make the UI more
> friendly to
> > the end user, empower them to discover, understand and use all the
> Aurora
> > features, and give them more insights into the their jobs, or even
> the
> > entire cluster. I love the idea that redesign the UI with a modern
> stack,
> > and more importantly, every single part of the application is a
> module that
> > you could take your own customization.
> >
> >
> > As for the development strategy, I'm in favor of the incremental
> approach
> > that posts one page at a time. The main benefit is that we are
> educating
> > the developers while iterating on it, and this will improve the
> adoption
> > rate in the long term.
> >
> >
> > Overall I'm very interested in this work, and would like to
> collaborate
> > with David to redesign and improve the UI.
> >
> >
> > 
> > From: Joshua Cohen <jco...@apache.org>
> > Sent: Wednesday, July 19, 2017 11:39
> > To: dev@aurora.ap

Re: [VOTE] Release Apache Aurora 0.18.x packages

[Joshua Cohen]

+1, everything looks good to me!

On Sat, Jul 22, 2017 at 1:36 AM, Erb, Stephan 
wrote:

> +1
>
> The verification scripts for all distributions in the test repository have
> passed for me.
>
>
> On 19.07.17, 01:58, "Santhosh Kumar Shanmugham" 
> 
> wrote:
>
> I missed to update one of the bintray links.
>
> It should be https://dl.bintray.com/shanmugh/aurora/
>
> On Tue, Jul 18, 2017 at 3:43 PM, Santhosh Kumar Shanmugham <
> sshanmug...@twitter.com> wrote:
>
> > Kicking off the votes.
> >
> > +1
> >
> > On Tue, Jul 18, 2017 at 3:42 PM, Santhosh Kumar Shanmugham <
> > sshanmug...@twitter.com> wrote:
> >
> >> All,
> >>
> >> I propose that we accept the following artifacts as the official
> deb and
> >> rpm packaging for
> >> Apache Aurora 0.18.x:
> >>
> >> https://dl.bintray.com/sshanmugham/aurora/
> >>
> >> The Aurora deb and rpm packaging includes the following:
> >>
> >> ---
> >>
> >> The branch used to create the packaging is:
> >> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> >> .git;a=tree;hb=refs/heads/0.18.x
> >>
> >> The packages are available at:
> >> https://dl.bintray.com/shanmugh/aurora/
> >>
> >> The GPG keys used to sign the packages are available at:
> >> https://dist.apache.org/repos/dist/release/aurora/KEYS
> >>
> >> Please download, verify, and test. Detailed test instructions are
> >> available here:
> >> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> >> .git;a=tree;f=test;hb=refs/heads/0.18.x
> >>
> >>
> >> The vote will close on Fri Jul 21 14:30:27 PDT 2017
> >>
> >> [ ] +1 Release these as the deb and rpm packages for Apache Aurora
> 0.18.x
> >> [ ] +0
> >> [ ] -1 Do not release these artifacts because...
> >>
> >> 
> >> 
> >>
> >> -Santhosh
> >>
> >
> >
>
>
>

Re: Redesign of the Aurora UI

[Joshua Cohen]

I think this looks great overall! I'm super excited to see the UI get some
love (and to set us up for better iteration on the UI going forward). My
biggest concern, of course, is the current hubbub vis-a-vis Apache and the
BSD-3+Patents license[1]. Have you tried running this against Preact to
confirm compatibility/performance? Also note that other Facebook libraries
have the same license problem (e.g. Immutable, Jest), so unless FB changes
their patent grant clause, I imagine we'd have to find alternatives to
those as well. If only we had landed this a week ago, we could've been
grandfathered in on the license front :(.

As far as the options for landing this, I'll leave that up to more active
reviewers, but my gut says that smaller reviews will make this easier to
parse, especially for those unfamiliar with React. That said, perhaps we
could go with an alternate method for reviewing here, where people review
against your fork directly and only when they're comfortable do you post
the whole patch to reviewboard for what should, by that point, be a rubber
stamp review?

In general, fantastic work, David!

[1] https://issues.apache.org/jira/browse/LEGAL-319

On Wed, Jul 19, 2017 at 12:55 AM, David McLaughlin <dmclaugh...@apache.org>
wrote:

> Hey all,
>
> At Twitter we have had a long-standing desire to be able to put custom
> widgets and other UX enhancements into the Aurora UI. Recent prototype work
> to do this in a clean way has proved fruitful and I'd like to present this
> approach to the community and get feedback on the overall approach.
>
> The basic approach is simple:
>
> 1) Use the node plugin for gradle to bootstrap a modern web development
> build system using webpack and npm.
> 2) Use a modern JS view library like ReactJS (or Preact depending on the
> Facebook+Patents license issues for Apache projects) to have all UI
> components defined as ES6 modules.
> 3) Use webpack to set up a custom plugin directory that modules should be
> loaded from first. This would allow anyone with custom Scheduler builds to
> also drop in custom JS files to replace selected OSS UI components.
> Example, you could drop a "Navigation.js" into the plugin directory, and
> the build system will use that module instead whenever it sees a
>  used by parent components. Basically: it's a hybrid
> plugin/dependency injection (similar to Guice) approach to customization of
> the UI.
>
> The nice thing here is that since every single part of the application is a
> module you could take your customizations to any level of precision that
> you need. You could replace (or add) key pages or components, or just keep
> it as simple as adding some helper text to one of the pages.
>
> Some use cases this could enable:
>
> * A  widget that you can replace with configuration
> parsing that understands special processes and/or custom executors.
> * Integration with your telemetry stacks to add resource utilization
> feedback or other performance indicators into the job page.
> * Adding custom stats tracking widgets for internal analytics.
>
>
> Some points from my prototype work:
>
> 1) Switching to npm for JS asset management allows us to delete the vast
> majority of our 3rdparty assets that we had to copy into the repo. We can
> most likely extend this to remove all of them, including our CSS frameworks
> like Bootstrap.
> 2) ReactJS was proposed before (by Joshua Cohen) and ironically I was one
> of the main voices against. The argument I used at the time was lack of
> familiarity with the stack within the community, as well as concerns about
> the complexity. So what changed? Almost the entire Aurora team at Twitter
> has ReactJS experience now - from working on our internal Continuous
> Delivery system. I also believe that as a view layer, ReactJS is
> conceptually very simple and that most of the complexity in these modern JS
> applications is in a state layer like Redux. The Aurora UI is currently
> read-only and relatively simple, and we can avoid much of the complexity by
> just using component state instead.
> 3) Unit tests are trivial to add as part of this work.
> 4) I'd like to move to modern, maintainable CSS tooling - SASS - as part of
> this.
>
>
> My prototype is (hastily) published here:
>
> https://github.com/DavidMcLaughlin/aurora/tree/js-build
>
>
> Some highlighted changes:
>
> 1) Bootstrapping the modern UI toolchain from gradle:
> https://github.com/DavidMcLaughlin/aurora/blob/js-build/build.gradle#L452-
> L459
>
> 2) The webpack configuration to enable loading from a plugin directory (and
> SASS):
> https://github.com/DavidMcLaughlin/aurora/blob/js-build/webpack.config.js
>
> 3) The Hello, World of the plugin mechanism:
>
> https://github.com/DavidMcLaughlin/aurora/b

Re: Future of storage in Aurora

[Joshua Cohen]

My understanding of the H2-backed stores is that at least part of the
original rationale behind them was that they were meant to be an interim
point on the way to external SQL-backed stores which should theoretically
provide significant benefits w.r.t. to GC (obviously unproven, especially
at scale).

I don't disagree that the H2 stores themselves are problematic (to say the
least); do we have evidence that returning to memory based stores will be
an improvement on that?

On Thu, Mar 30, 2017 at 12:16 PM, David McLaughlin 
wrote:

> Hi all,
>
> I'd like to start a discussion around storage in Aurora.
>
> I think one of the biggest mistakes we made in migrating our storage to H2
> was deleting the memory stores as we moved. We made a pretty big bet that
> we could eventually make H2/relational databases work. I don't think that
> bet has paid off and that we need to revisit the direction we're taking.
>
> My belief is that the current H2/MyBatis approach is untenable for large
> production clusters, at least without changing our current single-master
> architecture. At Twitter we are already having to fight to keep GC
> manageable even without DbTaskStore enabled, so I don't see a path forward
> where we could eventually enable that. So far experiments with H2 off-heap
> storage have provided marginal (if any) gains.
>
> Would anyone object to restoring the in-memory stores and creating new
> implementations for the missing ones (UpdateStore)? I'd even go further and
> propose that we consider in-memory H2 and MyBatis a failed experiment and
> we drop that storage layer completely.
>
> Cheers,
> David
>

Re: Design Doc: Mesos V1 API

[Joshua Cohen]

Overall proposal sounds reasonable to me, thanks Zameer! My main question
is whether we're setting ourselves up for possible performance regressions
by switching to the HTTP API? We'll obviously need to switch eventually,
but would be good to understand the performance impact (if any) of the
switch.

On Wed, Feb 1, 2017 at 2:23 PM, Zameer Manji  wrote:

> Hey,
>
> I have written a design doc
>  0qMxAlnMW70eOKoU3myo/edit#>
> that
> outlines the work required to adopt the Mesos HTTP V1 API in Aurora. Please
> take a look and comment.
>
> --
> Zameer Manji
>

Re: [DRAFT][REPORT]: Apache Aurora - December 2016

[Joshua Cohen]

+1, thanks for doing this Jake!

On Wed, Dec 7, 2016 at 9:55 AM, Jake Farrell  wrote:

>  Below is our draft board report which is due next week. Please let me know
> if you see any additions or changes that should be made, I'll plan on
> submitting this Friday if there are no objections
>
> -Jake
>
>
>
> Apache Aurora is a stateless and fault tolerant service scheduler used to
> schedule jobs onto Apache Mesos such as long-running services, cron jobs,
> and one off tasks.
>
> Project Status
> -
> The Apache Aurora community has continued to see growth from new users and
> contributors over the last quarter. There has been a great deal of activity
> in open
> code reviews with excellent community involvement. In September we released
> Apache Aurora 0.16.0 and since have started discussions about our next
>  0.17.0
> release candidate. The next release will focus on a number of bug
> fixes, stability
> enhancements as well as some new features outlined in our release notes [1]
>
> Community
> ---
> Latest Additions:
>
> * Committer addition: Stephan Erb, 2.3.2016
> * PMC addition:  Stephan Erb, 2.3.2016
>
> Issue backlog status since last report:
>
> * Created:   86
> * Resolved: 57
>
> Mailing list activity since last report:
>
> * @dev169 messages
> * @user   37 messages
> * @reviews  1391 messages
>
> Releases
> ---
> Last release: Apache Aurora 0.16.0 released 09.28.2016
>
>
> [1]: https://github.com/apache/aurora/blob/master/RELEASE-NOTES.md
>

Re: Preparations for 0.17.0

[Joshua Cohen]

I added https://issues.apache.org/jira/browse/AURORA-1782 to 0.17.0
Hopefully I'll have time to look into that soon.

On Wed, Nov 2, 2016 at 4:53 PM, Zameer Manji  wrote:

> Thanks for stepping up!
>
> I think picking up Mesos 1.1 is ideal for the release and we should block
> our release until it is out.
>
> On Wed, Nov 2, 2016 at 9:24 AM, Erb, Stephan 
> wrote:
>
> > Hi everyone,
> >
> > I’d like to volunteer as our next release manager and set the release
> > train for 0.17 into motion.
> >
> > Since 0.16 we have fixed several important bugs and should therefore aim
> > for a release in the next 2-4 weeks, if possible. If everything goes
> > according to plan for the current Mesos release, we should be able to
> pick
> > up Mesos 1.1 as well.
> >
> > Please tag any blocker issues with `fixVersion 0.17` so that they show up
> > on this dashboard: https://issues.apache.org/
> jira/browse/AURORA-1014?jql=
> > project%20%3D%20AURORA%20AND%20fixVersion%20%3D%200.17.0%
> > 20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%
> > 20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> >
> > Best regards,
> > Stephan
> >
> > --
> > Zameer Manji
> >
>

Re: [VOTE] Release Apache Aurora 0.16.0 packages

[Joshua Cohen]

This vote has succeeded. There were three +1 (binding) votes:

Joshua Cohen
Stephan Erb
John Sirois

The official packages are now available here:
https://dl.bintray.com/apache/aurora/

Thanks all!



On Mon, Oct 24, 2016 at 3:18 PM, Joshua Cohen <jco...@apache.org> wrote:

> Yeah, I've also fixed the release candidate script to just generate the
> email for us (c.f. https://reviews.apache.org/r/53102/) to avoid these
> problems in the future!
>
> On Mon, Oct 24, 2016 at 3:05 PM, John Sirois <john.sir...@gmail.com>
> wrote:
>
>> On Oct 24, 2016 3:50 PM, "Joshua Cohen" <jco...@apache.org> wrote:
>> >
>> > FWIW if you pull down the 0.16.x branch, it already contains fixes for
>> the
>> > missing deps. I was (perhaps unnecessarily) waiting until after the vote
>> to
>> > merge that to master though.
>>
>> Ah yes. Got thrown by the 0.15 link in the original post and failed to
>> read
>> your follow-up.
>>
>> >
>> > On Mon, Oct 24, 2016 at 2:41 PM, John Sirois <john.sir...@gmail.com>
>> wrote:
>> >
>> > > +1 (binding)
>> > >
>> > > Tested using the 3 vagrant test setups under aurora-packaging/test.
>> > > There were the `libcurl4-nss-dev` missing dep issues in each of the
>> deb
>> vms
>> > > that I worked around with `sudo apt-get -f install`, but these issues
>> were
>> > > merely test setup issues and not issues with the aurora packages which
>> > > installed and worked flawlessly.
>> > >
>> > > On Thu, Oct 20, 2016 at 5:39 PM, Joshua Cohen <jco...@apache.org>
>> wrote:
>> > >
>> > > > The git url does? This one works for me:
>> > > > https://git1-us-west.apache.org/repos/asf?p=aurora-
>> > > > packaging.git;a=tree;hb=refs/heads/0.16.x
>> > > >
>> > > > $ curl -I
>> > > > https://git1-us-west.apache.org/repos/asf?p=aurora-
>> > > > packaging.git;a=tree;hb=refs/heads/0.16.x
>> > > > HTTP/1.1 200 OK
>> > > > Date: Thu, 20 Oct 2016 21:38:26 GMT
>> > > > Server: Apache/2.4.7 (Ubuntu)
>> > > > Vary: Accept-Encoding
>> > > > Content-Type: text/html; charset=utf-8
>> > > >
>> > > >
>> > > > On Thu, Oct 20, 2016 at 4:29 PM, Henry Saputra <
>> henry.sapu...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Seems like it returns 404?
>> > > > >
>> > > > > On Thu, Oct 20, 2016 at 10:38 AM, Joshua Cohen <jco...@apache.org
>> >
>> > > > wrote:
>> > > > >
>> > > > > > Also, apparently gmail didn't update the underlying urls for the
>> git
>> > > > > links
>> > > > > > when I c/p'd the email from the 0.15.0 vote. This is the actual
>> > > branch
>> > > > is
>> > > > > > here: https://git1-us-west.apache.or
>> g/repos/asf?p=aurora-packaging
>> > > > > > .git;a=tree;hb=refs/heads/0.16.x
>> > > > > > <https://git1-us-west.apache.org/repos/asf?p=aurora-
>> > > > > > packaging.git;a=tree;hb=refs/heads/0.15.x>
>> > > > > >
>> > > > > > On Thu, Oct 20, 2016 at 10:34 AM, Joshua Cohen <
>> jco...@apache.org>
>> > > > > wrote:
>> > > > > >
>> > > > > > > I'll start the voting off with my own +1. I verified packages
>> for
>> > > all
>> > > > > > > three platforms against the test vagrant images from the
>> packaging
>> > > > > repo.
>> > > > > > >
>> > > > > > > On Thu, Oct 20, 2016 at 10:33 AM, Joshua Cohen <
>> jco...@apache.org>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > >> All,
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> I propose that we accept the following artifacts as the
>> official
>> > > deb
>> > > > > > >> and rpm packaging for Apache Aurora 0.16.0:
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> *https://dl.bintray.com/jcohen/aurora/
>> > > > > > >> <https://dl.bintray.com/

Re: [VOTE] Release Apache Aurora 0.16.0 packages

[Joshua Cohen]

Yeah, I've also fixed the release candidate script to just generate the
email for us (c.f. https://reviews.apache.org/r/53102/) to avoid these
problems in the future!

On Mon, Oct 24, 2016 at 3:05 PM, John Sirois <john.sir...@gmail.com> wrote:

> On Oct 24, 2016 3:50 PM, "Joshua Cohen" <jco...@apache.org> wrote:
> >
> > FWIW if you pull down the 0.16.x branch, it already contains fixes for
> the
> > missing deps. I was (perhaps unnecessarily) waiting until after the vote
> to
> > merge that to master though.
>
> Ah yes. Got thrown by the 0.15 link in the original post and failed to read
> your follow-up.
>
> >
> > On Mon, Oct 24, 2016 at 2:41 PM, John Sirois <john.sir...@gmail.com>
> wrote:
> >
> > > +1 (binding)
> > >
> > > Tested using the 3 vagrant test setups under aurora-packaging/test.
> > > There were the `libcurl4-nss-dev` missing dep issues in each of the deb
> vms
> > > that I worked around with `sudo apt-get -f install`, but these issues
> were
> > > merely test setup issues and not issues with the aurora packages which
> > > installed and worked flawlessly.
> > >
> > > On Thu, Oct 20, 2016 at 5:39 PM, Joshua Cohen <jco...@apache.org>
> wrote:
> > >
> > > > The git url does? This one works for me:
> > > > https://git1-us-west.apache.org/repos/asf?p=aurora-
> > > > packaging.git;a=tree;hb=refs/heads/0.16.x
> > > >
> > > > $ curl -I
> > > > https://git1-us-west.apache.org/repos/asf?p=aurora-
> > > > packaging.git;a=tree;hb=refs/heads/0.16.x
> > > > HTTP/1.1 200 OK
> > > > Date: Thu, 20 Oct 2016 21:38:26 GMT
> > > > Server: Apache/2.4.7 (Ubuntu)
> > > > Vary: Accept-Encoding
> > > > Content-Type: text/html; charset=utf-8
> > > >
> > > >
> > > > On Thu, Oct 20, 2016 at 4:29 PM, Henry Saputra <
> henry.sapu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Seems like it returns 404?
> > > > >
> > > > > On Thu, Oct 20, 2016 at 10:38 AM, Joshua Cohen <jco...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Also, apparently gmail didn't update the underlying urls for the
> git
> > > > > links
> > > > > > when I c/p'd the email from the 0.15.0 vote. This is the actual
> > > branch
> > > > is
> > > > > > here: https://git1-us-west.apache.org/repos/asf?p=aurora-
> packaging
> > > > > > .git;a=tree;hb=refs/heads/0.16.x
> > > > > > <https://git1-us-west.apache.org/repos/asf?p=aurora-
> > > > > > packaging.git;a=tree;hb=refs/heads/0.15.x>
> > > > > >
> > > > > > On Thu, Oct 20, 2016 at 10:34 AM, Joshua Cohen <
> jco...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > I'll start the voting off with my own +1. I verified packages
> for
> > > all
> > > > > > > three platforms against the test vagrant images from the
> packaging
> > > > > repo.
> > > > > > >
> > > > > > > On Thu, Oct 20, 2016 at 10:33 AM, Joshua Cohen <
> jco...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > >> All,
> > > > > > >>
> > > > > > >>
> > > > > > >> I propose that we accept the following artifacts as the
> official
> > > deb
> > > > > > >> and rpm packaging for Apache Aurora 0.16.0:
> > > > > > >>
> > > > > > >>
> > > > > > >> *https://dl.bintray.com/jcohen/aurora/
> > > > > > >> <https://dl.bintray.com/jcohen/aurora/>*
> > > > > > >>
> > > > > > >>
> > > > > > >> The Aurora deb and rpm packaging includes the following:
> > > > > > >>
> > > > > > >> ---
> > > > > > >>
> > > > > > >> The branch used to create the packaging is:
> > > > > > >>
> > > > > > >> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> > > > > > >> .git;a=tree;hb=refs/heads/0.16.x
> > > > > > >> <https://git1-us-west.apache.org/repos/asf?p=aurora-
> > > > > > packaging.git;a=tree;hb=refs/heads/0.15.x>
> > > > > > >>
> > > > > > >>
> > > > > > >> The packages are available at:
> > > > > > >>
> > > > > > >> *https://dl.bintray.com/jcohen/aurora/
> > > > > > >> <https://dl.bintray.com/jcohen/aurora/>*
> > > > > > >>
> > > > > > >>
> > > > > > >> The GPG keys used to sign the packages are available at:
> > > > > > >>
> > > > > > >> https://dist.apache.org/repos/dist/release/aurora/KEYS
> > > > > > >>
> > > > > > >>
> > > > > > >> Please download, verify, and test. Detailed test instructions
> are
> > > > > > >> available here
> > > > > > >>
> > > > > > >> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> > > > > > >> .git;a=tree;f=test;hb=refs/heads/0.16.x
> > > > > > >> <https://git1-us-west.apache.org/repos/asf?p=aurora-
> > > > > > packaging.git;a=tree;f=test;hb=refs/heads/0.15.x>
> > > > > > >>
> > > > > > >>
> > > > > > >> The vote will close on Tue Oct  25 10:30:00 CDT 2016
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> [ ] +1 Release these as the deb and rpm packages for Apache
> Aurora
> > > > > > 0.15.0
> > > > > > >>
> > > > > > >> [ ] +0
> > > > > > >>
> > > > > > >> [ ] -1 Do not release these artifacts because...
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [VOTE] Release Apache Aurora 0.16.0 packages

[Joshua Cohen]

FWIW if you pull down the 0.16.x branch, it already contains fixes for the
missing deps. I was (perhaps unnecessarily) waiting until after the vote to
merge that to master though.

On Mon, Oct 24, 2016 at 2:41 PM, John Sirois <john.sir...@gmail.com> wrote:

> +1 (binding)
>
> Tested using the 3 vagrant test setups under aurora-packaging/test.
> There were the `libcurl4-nss-dev` missing dep issues in each of the deb vms
> that I worked around with `sudo apt-get -f install`, but these issues were
> merely test setup issues and not issues with the aurora packages which
> installed and worked flawlessly.
>
> On Thu, Oct 20, 2016 at 5:39 PM, Joshua Cohen <jco...@apache.org> wrote:
>
> > The git url does? This one works for me:
> > https://git1-us-west.apache.org/repos/asf?p=aurora-
> > packaging.git;a=tree;hb=refs/heads/0.16.x
> >
> > $ curl -I
> > https://git1-us-west.apache.org/repos/asf?p=aurora-
> > packaging.git;a=tree;hb=refs/heads/0.16.x
> > HTTP/1.1 200 OK
> > Date: Thu, 20 Oct 2016 21:38:26 GMT
> > Server: Apache/2.4.7 (Ubuntu)
> > Vary: Accept-Encoding
> > Content-Type: text/html; charset=utf-8
> >
> >
> > On Thu, Oct 20, 2016 at 4:29 PM, Henry Saputra <henry.sapu...@gmail.com>
> > wrote:
> >
> > > Seems like it returns 404?
> > >
> > > On Thu, Oct 20, 2016 at 10:38 AM, Joshua Cohen <jco...@apache.org>
> > wrote:
> > >
> > > > Also, apparently gmail didn't update the underlying urls for the git
> > > links
> > > > when I c/p'd the email from the 0.15.0 vote. This is the actual
> branch
> > is
> > > > here: https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> > > > .git;a=tree;hb=refs/heads/0.16.x
> > > > <https://git1-us-west.apache.org/repos/asf?p=aurora-
> > > > packaging.git;a=tree;hb=refs/heads/0.15.x>
> > > >
> > > > On Thu, Oct 20, 2016 at 10:34 AM, Joshua Cohen <jco...@apache.org>
> > > wrote:
> > > >
> > > > > I'll start the voting off with my own +1. I verified packages for
> all
> > > > > three platforms against the test vagrant images from the packaging
> > > repo.
> > > > >
> > > > > On Thu, Oct 20, 2016 at 10:33 AM, Joshua Cohen <jco...@apache.org>
> > > > wrote:
> > > > >
> > > > >> All,
> > > > >>
> > > > >>
> > > > >> I propose that we accept the following artifacts as the official
> deb
> > > > >> and rpm packaging for Apache Aurora 0.16.0:
> > > > >>
> > > > >>
> > > > >> *https://dl.bintray.com/jcohen/aurora/
> > > > >> <https://dl.bintray.com/jcohen/aurora/>*
> > > > >>
> > > > >>
> > > > >> The Aurora deb and rpm packaging includes the following:
> > > > >>
> > > > >> ---
> > > > >>
> > > > >> The branch used to create the packaging is:
> > > > >>
> > > > >> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> > > > >> .git;a=tree;hb=refs/heads/0.16.x
> > > > >> <https://git1-us-west.apache.org/repos/asf?p=aurora-
> > > > packaging.git;a=tree;hb=refs/heads/0.15.x>
> > > > >>
> > > > >>
> > > > >> The packages are available at:
> > > > >>
> > > > >> *https://dl.bintray.com/jcohen/aurora/
> > > > >> <https://dl.bintray.com/jcohen/aurora/>*
> > > > >>
> > > > >>
> > > > >> The GPG keys used to sign the packages are available at:
> > > > >>
> > > > >> https://dist.apache.org/repos/dist/release/aurora/KEYS
> > > > >>
> > > > >>
> > > > >> Please download, verify, and test. Detailed test instructions are
> > > > >> available here
> > > > >>
> > > > >> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> > > > >> .git;a=tree;f=test;hb=refs/heads/0.16.x
> > > > >> <https://git1-us-west.apache.org/repos/asf?p=aurora-
> > > > packaging.git;a=tree;f=test;hb=refs/heads/0.15.x>
> > > > >>
> > > > >>
> > > > >> The vote will close on Tue Oct  25 10:30:00 CDT 2016
> > > > >>
> > > > >>
> > > > >>
> > > > >> [ ] +1 Release these as the deb and rpm packages for Apache Aurora
> > > > 0.15.0
> > > > >>
> > > > >> [ ] +0
> > > > >>
> > > > >> [ ] -1 Do not release these artifacts because...
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Aurora 0.16.0 packages

[Joshua Cohen]

The git url does? This one works for me:
https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;hb=refs/heads/0.16.x

$ curl -I
https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;hb=refs/heads/0.16.x
HTTP/1.1 200 OK
Date: Thu, 20 Oct 2016 21:38:26 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8


On Thu, Oct 20, 2016 at 4:29 PM, Henry Saputra <henry.sapu...@gmail.com>
wrote:

> Seems like it returns 404?
>
> On Thu, Oct 20, 2016 at 10:38 AM, Joshua Cohen <jco...@apache.org> wrote:
>
> > Also, apparently gmail didn't update the underlying urls for the git
> links
> > when I c/p'd the email from the 0.15.0 vote. This is the actual branch is
> > here: https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> > .git;a=tree;hb=refs/heads/0.16.x
> > <https://git1-us-west.apache.org/repos/asf?p=aurora-
> > packaging.git;a=tree;hb=refs/heads/0.15.x>
> >
> > On Thu, Oct 20, 2016 at 10:34 AM, Joshua Cohen <jco...@apache.org>
> wrote:
> >
> > > I'll start the voting off with my own +1. I verified packages for all
> > > three platforms against the test vagrant images from the packaging
> repo.
> > >
> > > On Thu, Oct 20, 2016 at 10:33 AM, Joshua Cohen <jco...@apache.org>
> > wrote:
> > >
> > >> All,
> > >>
> > >>
> > >> I propose that we accept the following artifacts as the official deb
> > >> and rpm packaging for Apache Aurora 0.16.0:
> > >>
> > >>
> > >> *https://dl.bintray.com/jcohen/aurora/
> > >> <https://dl.bintray.com/jcohen/aurora/>*
> > >>
> > >>
> > >> The Aurora deb and rpm packaging includes the following:
> > >>
> > >> ---
> > >>
> > >> The branch used to create the packaging is:
> > >>
> > >> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> > >> .git;a=tree;hb=refs/heads/0.16.x
> > >> <https://git1-us-west.apache.org/repos/asf?p=aurora-
> > packaging.git;a=tree;hb=refs/heads/0.15.x>
> > >>
> > >>
> > >> The packages are available at:
> > >>
> > >> *https://dl.bintray.com/jcohen/aurora/
> > >> <https://dl.bintray.com/jcohen/aurora/>*
> > >>
> > >>
> > >> The GPG keys used to sign the packages are available at:
> > >>
> > >> https://dist.apache.org/repos/dist/release/aurora/KEYS
> > >>
> > >>
> > >> Please download, verify, and test. Detailed test instructions are
> > >> available here
> > >>
> > >> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> > >> .git;a=tree;f=test;hb=refs/heads/0.16.x
> > >> <https://git1-us-west.apache.org/repos/asf?p=aurora-
> > packaging.git;a=tree;f=test;hb=refs/heads/0.15.x>
> > >>
> > >>
> > >> The vote will close on Tue Oct  25 10:30:00 CDT 2016
> > >>
> > >>
> > >>
> > >> [ ] +1 Release these as the deb and rpm packages for Apache Aurora
> > 0.15.0
> > >>
> > >> [ ] +0
> > >>
> > >> [ ] -1 Do not release these artifacts because...
> > >>
> > >
> > >
> >
>

Re: [VOTE] Release Apache Aurora 0.16.0 packages

[Joshua Cohen]

Also, apparently gmail didn't update the underlying urls for the git links
when I c/p'd the email from the 0.15.0 vote. This is the actual branch is
here: https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
.git;a=tree;hb=refs/heads/0.16.x
<https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;hb=refs/heads/0.15.x>

On Thu, Oct 20, 2016 at 10:34 AM, Joshua Cohen <jco...@apache.org> wrote:

> I'll start the voting off with my own +1. I verified packages for all
> three platforms against the test vagrant images from the packaging repo.
>
> On Thu, Oct 20, 2016 at 10:33 AM, Joshua Cohen <jco...@apache.org> wrote:
>
>> All,
>>
>>
>> I propose that we accept the following artifacts as the official deb
>> and rpm packaging for Apache Aurora 0.16.0:
>>
>>
>> *https://dl.bintray.com/jcohen/aurora/
>> <https://dl.bintray.com/jcohen/aurora/>*
>>
>>
>> The Aurora deb and rpm packaging includes the following:
>>
>> ---
>>
>> The branch used to create the packaging is:
>>
>> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
>> .git;a=tree;hb=refs/heads/0.16.x
>> <https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;hb=refs/heads/0.15.x>
>>
>>
>> The packages are available at:
>>
>> *https://dl.bintray.com/jcohen/aurora/
>> <https://dl.bintray.com/jcohen/aurora/>*
>>
>>
>> The GPG keys used to sign the packages are available at:
>>
>> https://dist.apache.org/repos/dist/release/aurora/KEYS
>>
>>
>> Please download, verify, and test. Detailed test instructions are
>> available here
>>
>> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
>> .git;a=tree;f=test;hb=refs/heads/0.16.x
>> <https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;f=test;hb=refs/heads/0.15.x>
>>
>>
>> The vote will close on Tue Oct  25 10:30:00 CDT 2016
>>
>>
>>
>> [ ] +1 Release these as the deb and rpm packages for Apache Aurora 0.15.0
>>
>> [ ] +0
>>
>> [ ] -1 Do not release these artifacts because...
>>
>
>

Re: [VOTE] Release Apache Aurora 0.16.0 packages

[Joshua Cohen]

I'll start the voting off with my own +1. I verified packages for all three
platforms against the test vagrant images from the packaging repo.

On Thu, Oct 20, 2016 at 10:33 AM, Joshua Cohen <jco...@apache.org> wrote:

> All,
>
>
> I propose that we accept the following artifacts as the official deb
> and rpm packaging for Apache Aurora 0.16.0:
>
>
> *https://dl.bintray.com/jcohen/aurora/
> <https://dl.bintray.com/jcohen/aurora/>*
>
>
> The Aurora deb and rpm packaging includes the following:
>
> ---
>
> The branch used to create the packaging is:
>
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> .git;a=tree;hb=refs/heads/0.16.x
> <https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;hb=refs/heads/0.15.x>
>
>
> The packages are available at:
>
> *https://dl.bintray.com/jcohen/aurora/
> <https://dl.bintray.com/jcohen/aurora/>*
>
>
> The GPG keys used to sign the packages are available at:
>
> https://dist.apache.org/repos/dist/release/aurora/KEYS
>
>
> Please download, verify, and test. Detailed test instructions are
> available here
>
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
> .git;a=tree;f=test;hb=refs/heads/0.16.x
> <https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;f=test;hb=refs/heads/0.15.x>
>
>
> The vote will close on Tue Oct  25 10:30:00 CDT 2016
>
>
>
> [ ] +1 Release these as the deb and rpm packages for Apache Aurora 0.15.0
>
> [ ] +0
>
> [ ] -1 Do not release these artifacts because...
>

[VOTE] Release Apache Aurora 0.16.0 packages

[Joshua Cohen]

All,


I propose that we accept the following artifacts as the official deb
and rpm packaging for Apache Aurora 0.16.0:


*https://dl.bintray.com/jcohen/aurora/
*


The Aurora deb and rpm packaging includes the following:

---

The branch used to create the packaging is:

https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
.git;a=tree;hb=refs/heads/0.16.x



The packages are available at:

*https://dl.bintray.com/jcohen/aurora/
*


The GPG keys used to sign the packages are available at:

https://dist.apache.org/repos/dist/release/aurora/KEYS


Please download, verify, and test. Detailed test instructions are available
here

https://git1-us-west.apache.org/repos/asf?p=aurora-packaging
.git;a=tree;f=test;hb=refs/heads/0.16.x



The vote will close on Tue Oct  25 10:30:00 CDT 2016



[ ] +1 Release these as the deb and rpm packages for Apache Aurora 0.15.0

[ ] +0

[ ] -1 Do not release these artifacts because...

Re: Trouble publishing Aurora 0.16.0 binaries to bintray

[Joshua Cohen]

I was just going based off of what's documented in
https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=blob;f=README.md;h=440aff22cba9da67f4d466e6bc147aee9ae065d2;hb=HEAD
(and previous votes on packages seem to match the description found
therein). I can just upload the contents of the tarball generated by the
release-candidate script manually in the meantime, but I was hoping someone
who had done this recently could clarify the process (or maybe I'm just
running into a bintray bug).

On Thu, Oct 20, 2016 at 7:57 AM, Jake Farrell <jfarr...@apache.org> wrote:

> There should be no source tar ball for the binary artifacts, just the deb's
> and rpm's which bintray requires that you upload individually. You can use
> dist.apache.org to stage the files for the vote and after it is successful
> upload them to bintray, we should also look at adding in some scripting for
> this to help automate the process
>
> -Jake
>
>
>
> On Wed, Oct 19, 2016 at 5:25 PM, Joshua Cohen <jco...@apache.org> wrote:
>
> > Hi all,
> >
> > I'm trying to get a vote out for the 0.16.0 binaries, but I'm running
> into
> > some issues publishing the binaries. I'm wondering if those who have done
> > this in the past have any advice.
> >
> > The problem seems to boil down to this: when I upload the tarball
> generated
> > by the release-candidate script, bintray does not give me the option to
> > explode it. Uploading it directly via curl, rather than the UI, I get an
> > error back telling me that it's not a valid type of file to explode, for
> > some reason it reads it as an octet-stream rather than a tarball (I even
> > tried forcing it by specifying a Content-Type header on the curl request
> to
> > no avail).
> >
> > If I change the release-candidate script to generate a tar.gz instead of
> a
> > .tar, then I am able to get bintray to give me the option to explode the
> > archive, however it still fails because for some reason it's trying to
> sign
> > the files in the archive, but the signature is already present. I've
> > confirmed that the option to GPG sign files is not selected for my
> > repository.
> >
> > At this point I'm at a loss for how to continue, as there does not seem
> to
> > be any way to upload the binaries and have bintray explode them, which is
> > required for the binary to actually be usable.
> >
> > You can see what I've uploaded in its current tar form here:
> > https://dl.bintray.com/jcohen/aurora/
> >
> > Any advice is appreciated!
> >
> > Cheers,
> >
> > Joshua
> >
>

Trouble publishing Aurora 0.16.0 binaries to bintray

[Joshua Cohen]

Hi all,

I'm trying to get a vote out for the 0.16.0 binaries, but I'm running into
some issues publishing the binaries. I'm wondering if those who have done
this in the past have any advice.

The problem seems to boil down to this: when I upload the tarball generated
by the release-candidate script, bintray does not give me the option to
explode it. Uploading it directly via curl, rather than the UI, I get an
error back telling me that it's not a valid type of file to explode, for
some reason it reads it as an octet-stream rather than a tarball (I even
tried forcing it by specifying a Content-Type header on the curl request to
no avail).

If I change the release-candidate script to generate a tar.gz instead of a
.tar, then I am able to get bintray to give me the option to explode the
archive, however it still fails because for some reason it's trying to sign
the files in the archive, but the signature is already present. I've
confirmed that the option to GPG sign files is not selected for my
repository.

At this point I'm at a loss for how to continue, as there does not seem to
be any way to upload the binaries and have bintray explode them, which is
required for the binary to actually be usable.

You can see what I've uploaded in its current tar form here:
https://dl.bintray.com/jcohen/aurora/

Any advice is appreciated!

Cheers,

Joshua

Re: A mini postmortem on snapshot failures

[Joshua Cohen]

Hi Zameer,

Thanks for this writeup!

I think one other option to consider would be using a connection for
writing the snapshots that's not bound by the pool's maximum checkout time.
I'm not sure if this is feasible or not, but I worry that there's
potentially no upper bound on raising the maximum checkout time as the size
of a cluster grows or its read traffic grows. It feels a bit heavy weight
to up the max checkout time when potentially the only connection exceeding
the limits is the one writing the snapshot.

I'd definitely be in favor of adding a flag to tune the maximum connection
checkout.

I'm neutral to negative on having Snapshot creation failures be fatal, I
don't necessarily think that one failed snapshot should take the scheduler
down, but I agree that some number of consecutive failures is a Bad Thing
that is worthy of investigation. My concern with having failures be fatal
is the pathological case where snapshots always fail causing your scheduler
to failover once every SNAPSHOT_INTERVAL. Do you think it would be
sufficient to add `scheduler_log_snapshots` to the list of important
stats[1]?

I'm also neutral on changing the defaults. I'm not sure if it's warranted,
as the behavior will vary based on cluster. It seems like you guys got bit
by this due to a comparatively heavy read load? Our cluster, on the other
hand, is probably significantly larger, but is not queried as much, and we
haven't run into issues with the defaults. However, as long as there are is
no adverse impact to bumping the default values I've got no objections.

Cheers,

Joshua

[1]
https://github.com/apache/aurora/blob/master/docs/operations/monitoring.md#important-stats

On Fri, Sep 30, 2016 at 7:34 PM, Zameer Manji  wrote:

> Aurora Developers and Users,
>
> I would like to share failure case I experienced recently. In a modestly
> sized production cluster with high read load, snapshot creation began to
> fail. The logs showed the following:
>
> 
> W0923 00:23:55.528 [LogStorage-0, LogStorage:473]
> ### Error rolling back transaction.  Cause: java.sql.SQLException: Error
> accessing PooledConnection. Connection is invalid.
> ### Cause: java.sql.SQLException: Error accessing PooledConnection.
> Connection is invalid. org.apache.ibatis.exceptions.PersistenceException:
> ### Error rolling back transaction.  Cause: java.sql.SQLException: Error
> accessing PooledConnection. Connection is invalid.
> ### Cause: java.sql.SQLException: Error accessing PooledConnection.
> Connection is invalid.
> at
> org.apache.ibatis.exceptions.ExceptionFactory.wrapException(
> ExceptionFactory.java:30)
> at
> org.apache.ibatis.session.defaults.DefaultSqlSession.
> rollback(DefaultSqlSession.java:216)
> at
> org.apache.ibatis.session.SqlSessionManager.rollback(
> SqlSessionManager.java:299)
> at
> org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(
> TransactionalMethodInterceptor.java:116)
> at
> org.apache.aurora.scheduler.storage.db.DbStorage.lambda$
> write$0(DbStorage.java:175)
> at
> org.apache.aurora.scheduler.async.GatingDelayExecutor.closeDuring(
> GatingDelayExecutor.java:62)
> at
> org.apache.aurora.scheduler.storage.db.DbStorage.write(DbStorage.java:173)
> at
> org.apache.aurora.common.inject.TimedInterceptor.
> invoke(TimedInterceptor.java:83)
> at
> org.apache.aurora.scheduler.storage.log.LogStorage.
> doInTransaction(LogStorage.java:521)
> at
> org.apache.aurora.scheduler.storage.log.LogStorage.write(
> LogStorage.java:551)
> at
> org.apache.aurora.scheduler.storage.log.LogStorage.
> doSnapshot(LogStorage.java:489)
> at
> org.apache.aurora.common.inject.TimedInterceptor.
> invoke(TimedInterceptor.java:83)
> at
> org.apache.aurora.scheduler.storage.log.LogStorage.
> snapshot(LogStorage.java:565)
> at
> org.apache.aurora.scheduler.storage.log.LogStorage.lambda$
> scheduleSnapshots$20(LogStorage.java:468)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
> ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.sql.SQLException: Error accessing PooledConnection.
> Connection is invalid.
> at
> org.apache.ibatis.datasource.pooled.PooledConnection.checkConnection(
> PooledConnection.java:254)
> at
> org.apache.ibatis.datasource.pooled.PooledConnection.
> invoke(PooledConnection.java:243)
> at com.sun.proxy.$Proxy135.getAutoCommit(Unknown Source)
> at
> org.apache.ibatis.transaction.jdbc.JdbcTransaction.rollback(
> JdbcTransaction.java:79)
> at 

[RESULT][VOTE] Release Apache Aurora 0.16.0 RC2

[Joshua Cohen]

All,
The vote to accept Apache Aurora 0.16.0 RC2
as the official Apache Aurora 0.16.0 release has passed.


+1 (Binding)
--
Joshua Cohen
Stephan Erb
Maxim Khutornenko
John Sirois
Zameer Manji
Jake Farrell


+1 (Non-binding)
--
Mehrdad Nurolahzade
Martin Hrabovčin
Kai Huang
Santhosh Shanmugham


There were no 0 or -1 votes. Thank you to all who helped make this release.


Aurora 0.16.0 includes the following:
---
The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=aurora.git=CHANGELOG=rel/0.16.0

The tag used to create the release with is rel/0.16.0:
https://git-wip-us.apache.org/repos/asf?p=aurora.git=shortlog=refs/tags/rel/0.16.0

The release is available at:
https://dist.apache.org/repos/dist/release/aurora/0.16.0/apache-aurora-0.16.0.tar.gz

The MD5 checksum of the release can be found at:
https://dist.apache.org/repos/dist/release/aurora/0.16.0/apache-aurora-0.16.0.tar.gz.md5

The signature of the release can be found at:
https://dist.apache.org/repos/dist/release/aurora/0.16.0/apache-aurora-0.16.0.tar.gz.asc

The GPG key used to sign the release are available at:
https://dist.apache.org/repos/dist/release/aurora/KEYS

Re: [VOTE] Release Apache Aurora 0.16.0 RC2

[Joshua Cohen]

+0 Missing AURORA-1779 in CHANGELOG
+1 Everything checks out via verify-release-candidate

Overall +1 (binding).

On Thu, Sep 22, 2016 at 2:18 PM, Joshua Cohen <jco...@apache.org> wrote:

> Note: I forgot to mark AURORA-1779 as fixed in 0.16.0 before cutting this
> RC, so that isn't reflected in the changelog. I don't consider that a
> blocker to release, but if others disagree I can cut rc3.
>
> On Thu, Sep 22, 2016 at 2:12 PM, Joshua Cohen <jco...@apache.org> wrote:
>
>> All,
>>
>> I propose that we accept the following release candidate as the official
>> Apache Aurora 0.16.0 release.
>>
>> Aurora 0.16.0-rc2 includes the following:
>> ---
>> The RELEASE NOTES for the release are available at:
>> https://git-wip-us.apache.org/repos/asf?p=aurora.git=RELEA
>> SE-NOTES.md=rel/0.16.0-rc2
>>
>> The CHANGELOG for the release is available at:
>> https://git-wip-us.apache.org/repos/asf?p=aurora.git=CHANG
>> ELOG=rel/0.16.0-rc2
>>
>> The tag used to create the release candidate is:
>> https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=short
>> log;h=refs/tags/rel/0.16.0-rc2
>>
>> The release candidate is available at:
>> https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc2/apa
>> che-aurora-0.16.0-rc2.tar.gz
>>
>> The MD5 checksum of the release candidate can be found at:
>> https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc2/apa
>> che-aurora-0.16.0-rc2.tar.gz.md5
>>
>> The signature of the release candidate can be found at:
>> https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc2/apa
>> che-aurora-0.16.0-rc2.tar.gz.asc
>>
>> The GPG key used to sign the release are available at:
>> https://dist.apache.org/repos/dist/dev/aurora/KEYS
>>
>> Please download, verify, and test.
>>
>> The vote will close on Sun Sep 25 14:11:09 CDT 2016
>>
>> [ ] +1 Release this as Apache Aurora 0.16.0
>> [ ] +0
>> [ ] -1 Do not release this as Apache Aurora 0.16.0 because...
>>
>> 
>> 
>>
>
>

Re: [VOTE] Release Apache Aurora 0.16.0 RC2

[Joshua Cohen]

Note: I forgot to mark AURORA-1779 as fixed in 0.16.0 before cutting this
RC, so that isn't reflected in the changelog. I don't consider that a
blocker to release, but if others disagree I can cut rc3.

On Thu, Sep 22, 2016 at 2:12 PM, Joshua Cohen <jco...@apache.org> wrote:

> All,
>
> I propose that we accept the following release candidate as the official
> Apache Aurora 0.16.0 release.
>
> Aurora 0.16.0-rc2 includes the following:
> ---
> The RELEASE NOTES for the release are available at:
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> RELEASE-NOTES.md=rel/0.16.0-rc2
>
> The CHANGELOG for the release is available at:
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> CHANGELOG=rel/0.16.0-rc2
>
> The tag used to create the release candidate is:
> https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=
> shortlog;h=refs/tags/rel/0.16.0-rc2
>
> The release candidate is available at:
> https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc2/
> apache-aurora-0.16.0-rc2.tar.gz
>
> The MD5 checksum of the release candidate can be found at:
> https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc2/
> apache-aurora-0.16.0-rc2.tar.gz.md5
>
> The signature of the release candidate can be found at:
> https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc2/
> apache-aurora-0.16.0-rc2.tar.gz.asc
>
> The GPG key used to sign the release are available at:
> https://dist.apache.org/repos/dist/dev/aurora/KEYS
>
> Please download, verify, and test.
>
> The vote will close on Sun Sep 25 14:11:09 CDT 2016
>
> [ ] +1 Release this as Apache Aurora 0.16.0
> [ ] +0
> [ ] -1 Do not release this as Apache Aurora 0.16.0 because...
>
> 
> 
>

[VOTE] Release Apache Aurora 0.16.0 RC2

[Joshua Cohen]

All,

I propose that we accept the following release candidate as the official
Apache Aurora 0.16.0 release.

Aurora 0.16.0-rc2 includes the following:
---
The RELEASE NOTES for the release are available at:
https://git-wip-us.apache.org/repos/asf?p=aurora.git=RELEASE-NOTES.md=rel/0.16.0-rc2

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=aurora.git=CHANGELOG=rel/0.16.0-rc2

The tag used to create the release candidate is:
https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=shortlog;h=refs/tags/rel/0.16.0-rc2

The release candidate is available at:
https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc2/apache-aurora-0.16.0-rc2.tar.gz

The MD5 checksum of the release candidate can be found at:
https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc2/apache-aurora-0.16.0-rc2.tar.gz.md5

The signature of the release candidate can be found at:
https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc2/apache-aurora-0.16.0-rc2.tar.gz.asc

The GPG key used to sign the release are available at:
https://dist.apache.org/repos/dist/dev/aurora/KEYS

Please download, verify, and test.

The vote will close on Sun Sep 25 14:11:09 CDT 2016

[ ] +1 Release this as Apache Aurora 0.16.0
[ ] +0
[ ] -1 Do not release this as Apache Aurora 0.16.0 because...

[RESULT] [VOTE] Release Apache Aurora 0.16.0 RC1

[Joshua Cohen]

I'm declaring this vote a failure due to AURORA-1779. RC2 is incoming
momentarily.

On Wed, Sep 21, 2016 at 4:40 PM, Erb, Stephan <stephan@blue-yonder.com>
wrote:

> Unfortunate -1 from me. I bumped into this: https://issues.apache.org/
> jira/browse/AURORA-1779
>
>
> On 20/09/16 22:37, "Joshua Cohen" <jco...@apache.org> wrote:
>
> I'll start with my own +1 vote.
>
> Verified with the verify-release-candidate script.
>
> On Tue, Sep 20, 2016 at 3:01 PM, Joshua Cohen <jco...@apache.org>
> wrote:
>
> > All,
> >
> > I propose that we accept the following release candidate as the
> official
> > Apache Aurora 0.16.0 release.
> >
> > Aurora 0.16.0-rc1 includes the following:
> > ---
> > The RELEASE NOTES for the release are available at:
> > https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> > RELEASE-NOTES.md=rel/0.16.0-rc1
> >
> > The CHANGELOG for the release is available at:
> > https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> > CHANGELOG=rel/0.16.0-rc1
> >
> > The tag used to create the release candidate is:
> > https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=
> > shortlog;h=refs/tags/rel/0.16.0-rc1
> >
> > The release candidate is available at:
> > https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc1/
> > apache-aurora-0.16.0-rc1.tar.gz
> >
> > The MD5 checksum of the release candidate can be found at:
> > https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc1/
> > apache-aurora-0.16.0-rc1.tar.gz.md5
> >
> > The signature of the release candidate can be found at:
> > https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc1/
> > apache-aurora-0.16.0-rc1.tar.gz.asc
> >
> > The GPG key used to sign the release are available at:
> > https://dist.apache.org/repos/dist/dev/aurora/KEYS
> >
> > Please download, verify, and test.
> >
> > The vote will close on Fri Sep 23 15:00:57 CDT 2016
> >
> > [ ] +1 Release this as Apache Aurora 0.16.0
> > [ ] +0
> > [ ] -1 Do not release this as Apache Aurora 0.16.0 because...
> >
> > 
> > 
> >
>
>
>
>

Re: [VOTE] Release Apache Aurora 0.16.0 RC1

[Joshua Cohen]

I'll start with my own +1 vote.

Verified with the verify-release-candidate script.

On Tue, Sep 20, 2016 at 3:01 PM, Joshua Cohen <jco...@apache.org> wrote:

> All,
>
> I propose that we accept the following release candidate as the official
> Apache Aurora 0.16.0 release.
>
> Aurora 0.16.0-rc1 includes the following:
> ---
> The RELEASE NOTES for the release are available at:
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> RELEASE-NOTES.md=rel/0.16.0-rc1
>
> The CHANGELOG for the release is available at:
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> CHANGELOG=rel/0.16.0-rc1
>
> The tag used to create the release candidate is:
> https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=
> shortlog;h=refs/tags/rel/0.16.0-rc1
>
> The release candidate is available at:
> https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc1/
> apache-aurora-0.16.0-rc1.tar.gz
>
> The MD5 checksum of the release candidate can be found at:
> https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc1/
> apache-aurora-0.16.0-rc1.tar.gz.md5
>
> The signature of the release candidate can be found at:
> https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc1/
> apache-aurora-0.16.0-rc1.tar.gz.asc
>
> The GPG key used to sign the release are available at:
> https://dist.apache.org/repos/dist/dev/aurora/KEYS
>
> Please download, verify, and test.
>
> The vote will close on Fri Sep 23 15:00:57 CDT 2016
>
> [ ] +1 Release this as Apache Aurora 0.16.0
> [ ] +0
> [ ] -1 Do not release this as Apache Aurora 0.16.0 because...
>
> 
> 
>

[VOTE] Release Apache Aurora 0.16.0 RC1

[Joshua Cohen]

All,

I propose that we accept the following release candidate as the official
Apache Aurora 0.16.0 release.

Aurora 0.16.0-rc1 includes the following:
---
The RELEASE NOTES for the release are available at:
https://git-wip-us.apache.org/repos/asf?p=aurora.git=RELEASE-NOTES.md=rel/0.16.0-rc1

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=aurora.git=CHANGELOG=rel/0.16.0-rc1

The tag used to create the release candidate is:
https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=shortlog;h=refs/tags/rel/0.16.0-rc1

The release candidate is available at:
https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc1/apache-aurora-0.16.0-rc1.tar.gz

The MD5 checksum of the release candidate can be found at:
https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc1/apache-aurora-0.16.0-rc1.tar.gz.md5

The signature of the release candidate can be found at:
https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc1/apache-aurora-0.16.0-rc1.tar.gz.asc

The GPG key used to sign the release are available at:
https://dist.apache.org/repos/dist/dev/aurora/KEYS

Please download, verify, and test.

The vote will close on Fri Sep 23 15:00:57 CDT 2016

[ ] +1 Release this as Apache Aurora 0.16.0
[ ] +0
[ ] -1 Do not release this as Apache Aurora 0.16.0 because...

[RESULT] [VOTE] Release Apache Aurora 0.16.0 RC0

[Joshua Cohen]

Due to the issue uncovered by Zameer I'm closing this vote. I'll fix this
and send out rc1 asap.

On Mon, Sep 19, 2016 at 5:26 PM, Zameer Manji <zma...@apache.org> wrote:

> -1
>
> I discovered https://issues.apache.org/jira/browse/AURORA-1777 in this
> release.
>
> On Mon, Sep 19, 2016 at 12:29 PM, Joshua Cohen <jco...@apache.org> wrote:
>
> > All,
> >
> > I propose that we accept the following release candidate as the official
> > Apache Aurora 0.16.0 release.
> >
> > Aurora 0.16.0-rc0 includes the following:
> > ---
> > The RELEASE NOTES for the release are available at:
> > https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> > RELEASE-NOTES.md=rel/0.16.0-rc0
> >
> > The CHANGELOG for the release is available at:
> > https://git-wip-us.apache.org/repos/asf?p=aurora.git=
> > CHANGELOG=rel/0.16.0-rc0
> >
> > The tag used to create the release candidate is:
> > https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=
> > shortlog;h=refs/tags/rel/0.16.0-rc0
> >
> > The release candidate is available at:
> > https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc0/
> > apache-aurora-0.16.0-rc0.tar.gz
> >
> > The MD5 checksum of the release candidate can be found at:
> > https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc0/
> > apache-aurora-0.16.0-rc0.tar.gz.md5
> >
> > The signature of the release candidate can be found at:
> > https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc0/
> > apache-aurora-0.16.0-rc0.tar.gz.asc
> >
> > The GPG key used to sign the release are available at:
> > https://dist.apache.org/repos/dist/dev/aurora/KEYS
> >
> > Please download, verify, and test.
> >
> > The vote will close on Thu Sep 22 14:28:18 CDT 2016
> >
> > [ ] +1 Release this as Apache Aurora 0.16.0
> > [ ] +0
> > [ ] -1 Do not release this as Apache Aurora 0.16.0 because...
> >
> > 
> > 
> >
>

[VOTE] Release Apache Aurora 0.16.0 RC0

[Joshua Cohen]

All,

I propose that we accept the following release candidate as the official
Apache Aurora 0.16.0 release.

Aurora 0.16.0-rc0 includes the following:
---
The RELEASE NOTES for the release are available at:
https://git-wip-us.apache.org/repos/asf?p=aurora.git=RELEASE-NOTES.md=rel/0.16.0-rc0

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=aurora.git=CHANGELOG=rel/0.16.0-rc0

The tag used to create the release candidate is:
https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=shortlog;h=refs/tags/rel/0.16.0-rc0

The release candidate is available at:
https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc0/apache-aurora-0.16.0-rc0.tar.gz

The MD5 checksum of the release candidate can be found at:
https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc0/apache-aurora-0.16.0-rc0.tar.gz.md5

The signature of the release candidate can be found at:
https://dist.apache.org/repos/dist/dev/aurora/0.16.0-rc0/apache-aurora-0.16.0-rc0.tar.gz.asc

The GPG key used to sign the release are available at:
https://dist.apache.org/repos/dist/dev/aurora/KEYS

Please download, verify, and test.

The vote will close on Thu Sep 22 14:28:18 CDT 2016

[ ] +1 Release this as Apache Aurora 0.16.0
[ ] +0
[ ] -1 Do not release this as Apache Aurora 0.16.0 because...

Re: Aurora 0.16.0 release

[Joshua Cohen]

I want to ship https://issues.apache.org/jira/browse/AURORA-1767 before I
cut 0.16.0. Once I get that in and you get your change in I'll cut the
release.

On Mon, Sep 12, 2016 at 1:40 PM, Zameer Manji <zma...@apache.org> wrote:

> Not having the webhook fix block the release SGTM.
>
> I would like to point out that I would like to add a deprecation in this
> release and there is a review out already.
> https://reviews.apache.org/r/51712/
>
> Otherwise it seems we are good to go for a release from my perspective.
>
> On Mon, Sep 12, 2016 at 11:38 AM, Joshua Cohen <jco...@apache.org> wrote:
>
> > I don't consider it a blocker for 0.16.0, so... if it's ready before I
> cut
> > the release, great. If not, we'll get it into 0.17.0?
> >
> > On Fri, Sep 9, 2016 at 8:45 PM, Zameer Manji <zma...@apache.org> wrote:
> >
> > > Resending this message so it's not stuck in people's spam.
> > >
> > > This issue is serious but can be fixed quickly I think.
> > >
> > > On Fri, Sep 9, 2016 at 6:19 PM, Dmitriy Shirchenko <shirc...@uber.com>
> > > wrote:
> > >
> > > > We discovered an issue with webhook today, so we may want to consider
> > not
> > > > releasing 0.16.0 until we fix this issue next week:
> > > > https://issues.apache.org/jira/browse/AURORA-1769
> > > >
> > > > On Tue, Sep 6, 2016 at 1:34 PM Maxim Khutornenko <ma...@apache.org>
> > > wrote:
> > > >
> > > > > I'd give it one more release as it may break any internal consumers
> > of
> > > > > that data. Same about AURORA-1708.
> > > > >
> > > > > As for AURORA-1680, it should be very safe to fix now.
> > > > >
> > > > > On Tue, Sep 6, 2016 at 1:18 PM, Zameer Manji <zma...@apache.org>
> > > wrote:
> > > > > > Should we fix AURORA-1707 in this release or bump it to the next
> > > > > release? I
> > > > > > noticed that it has been unloved for some time.
> > > > > >
> > > > > > On Tue, Sep 6, 2016 at 10:54 AM, Zameer Manji <zma...@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > >> + 1
> > > > > >>
> > > > > >> I think we are due for a release so folks can get Mesos 1.0 and
> > GPU
> > > > > >> support.
> > > > > >>
> > > > > >> On Tue, Sep 6, 2016 at 8:36 AM, Joshua Cohen <jco...@apache.org
> >
> > > > wrote:
> > > > > >>
> > > > > >>> Hi Aurorans,
> > > > > >>>
> > > > > >>> I plan to kick off the 0.16.0 release some time later this
> week.
> > > > Please
> > > > > >>> let
> > > > > >>> me know if there are any outstanding patches you'd like to ship
> > > > before
> > > > > >>> this
> > > > > >>> release.
> > > > > >>>
> > > > > >>> Thanks!
> > > > > >>>
> > > > > >>> Joshua
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Aurora 0.16.0 release

[Joshua Cohen]

I don't consider it a blocker for 0.16.0, so... if it's ready before I cut
the release, great. If not, we'll get it into 0.17.0?

On Fri, Sep 9, 2016 at 8:45 PM, Zameer Manji <zma...@apache.org> wrote:

> Resending this message so it's not stuck in people's spam.
>
> This issue is serious but can be fixed quickly I think.
>
> On Fri, Sep 9, 2016 at 6:19 PM, Dmitriy Shirchenko <shirc...@uber.com>
> wrote:
>
> > We discovered an issue with webhook today, so we may want to consider not
> > releasing 0.16.0 until we fix this issue next week:
> > https://issues.apache.org/jira/browse/AURORA-1769
> >
> > On Tue, Sep 6, 2016 at 1:34 PM Maxim Khutornenko <ma...@apache.org>
> wrote:
> >
> > > I'd give it one more release as it may break any internal consumers of
> > > that data. Same about AURORA-1708.
> > >
> > > As for AURORA-1680, it should be very safe to fix now.
> > >
> > > On Tue, Sep 6, 2016 at 1:18 PM, Zameer Manji <zma...@apache.org>
> wrote:
> > > > Should we fix AURORA-1707 in this release or bump it to the next
> > > release? I
> > > > noticed that it has been unloved for some time.
> > > >
> > > > On Tue, Sep 6, 2016 at 10:54 AM, Zameer Manji <zma...@apache.org>
> > wrote:
> > > >
> > > >> + 1
> > > >>
> > > >> I think we are due for a release so folks can get Mesos 1.0 and GPU
> > > >> support.
> > > >>
> > > >> On Tue, Sep 6, 2016 at 8:36 AM, Joshua Cohen <jco...@apache.org>
> > wrote:
> > > >>
> > > >>> Hi Aurorans,
> > > >>>
> > > >>> I plan to kick off the 0.16.0 release some time later this week.
> > Please
> > > >>> let
> > > >>> me know if there are any outstanding patches you'd like to ship
> > before
> > > >>> this
> > > >>> release.
> > > >>>
> > > >>> Thanks!
> > > >>>
> > > >>> Joshua
> > > >>>
> > > >>
> > > >>
> > >
> >
>

Re: [DRAFT][REPORT] Apache Aurora - September 2016

[Joshua Cohen]

I think the big thing in 0.16.0, if we're calling out features it will
support, is task filesystem isolation. That said, I doubt the board really
cares that much about the details, so +1 from me ;).

On Tue, Sep 6, 2016 at 2:28 PM, Maxim Khutornenko  wrote:

> +1
>
> On Tue, Sep 6, 2016 at 12:15 PM, Renan DelValle 
> wrote:
> > LGTM, +1
> >
> > Thanks for drafting this up Jake!
> >
> > On Tue, Sep 6, 2016 at 3:13 PM, Dave Lester  wrote:
> >
> >> +1
> >>
> >> On Tue, Sep 6, 2016, at 12:04 PM, Zameer Manji wrote:
> >> > +1
> >> >
> >> > Thanks for drafting this up so quickly.
> >> >
> >> > On Tue, Sep 6, 2016 at 11:52 AM, Jake Farrell 
> >> > wrote:
> >> >
> >> > >  Please take a second to review the draft board report below and
> let me
> >> > > know if there are any modifications that should be made. I will
> submit
> >> this
> >> > > in the next couple days if there are no objections
> >> > >
> >> > > -Jake
> >> > >
> >> > >
> >> > >
> >> > > Apache Aurora is a stateless and fault tolerant service scheduler
> used
> >> to
> >> > > schedule jobs onto Apache Mesos such as long-running services, cron
> >> jobs,
> >> > > and one off tasks.
> >> > >
> >> > > Project Status
> >> > > -
> >> > > The Apache Aurora community has continued to see growth from new
> users
> >> and
> >> > > contributors while releasing Apache Aurora 0.15.0 and making
> progress
> >> on
> >> > > our
> >> > > upcoming 0.16.0 release candidate. The upcoming  release will
> contain a
> >> > > number
> >> > > of bug fixes, stability enhancements as well as updating support for
> >> Apache
> >> > > Mesos
> >> > > 1.0.0, multiple executor support, and defaulting to using Apache
> >> Curator
> >> > > for
> >> > > scheduler leader election. Community design discussion have started
> >> around
> >> > > dynamic reservations and job update configuration.
> >> > >
> >> > > Community
> >> > > ---
> >> > > Latest Additions:
> >> > >
> >> > > * PMC addition: Stephan Erb, 2.3.2016
> >> > >
> >> > > Issue backlog status since last report:
> >> > >
> >> > > * Created:   53
> >> > > * Resolved: 35
> >> > >
> >> > > Mailing list activity since last report:
> >> > >
> >> > > * @dev270 messages
> >> > > * @user   73 messages
> >> > > * @reviews  863 messages
> >> > >
> >> > > Releases
> >> > > ---
> >> > > Last release: Apache Aurora 0.15.0 released 07.06.2016
> >> > > Release candidate: Apache Aurora 0.16.0 release candidate in
> progress
> >> > >
> >>
>

Aurora 0.16.0 release

[Joshua Cohen]

Hi Aurorans,

I plan to kick off the 0.16.0 release some time later this week. Please let
me know if there are any outstanding patches you'd like to ship before this
release.

Thanks!

Joshua

Re: [FEEDBACK] Transitioning Aurora leader election to Apache Curator (`-zk_use_curator`)

[Joshua Cohen]

I have this enabled in a test cluster and have not noticed any issues with
it yet. I'd like to roll it out to production before we drop the old code
though.

On Wed, Aug 24, 2016 at 1:10 PM, Zameer Manji  wrote:

> Could we change the default and drop the old code at the same time? I don't
> see any benefit of letting that hang around.
>
> I have not tested this code yet, but I hope to do it soon.
>
> On Wed, Aug 24, 2016 at 5:19 AM, Erb, Stephan  >
> wrote:
>
> > The curator backend has been working well for us so far. I believe it is
> > safe to make it the default for the next release, and to drop the old
> code
> > in the release after that.
> >
> >
> >
> > *From: *John Sirois 
> > *Reply-To: *"u...@aurora.apache.org" , "
> > jsir...@apache.org" 
> > *Date: *Thursday 7 July 2016 at 01:13
> > *To: *Martin Hrabovčin 
> > *Cc: *"dev@aurora.apache.org" , Jake Farrell <
> > jfarr...@apache.org>, "u...@aurora.apache.org" 
> > *Subject: *Re: [FEEDBACK] Transitioning Aurora leader election to Apache
> > Curator (`-zk_use_curator`)
> >
> >
> >
> > Now that 0.15.0 has been released, I thought I'd check in on any progress
> > folks have made with testing/deploying the 0.14.0+ with the Aurora
> > Scheduler `-zk_use_curator` flag in-place.
> >
> > There has been 1 fix that will go out in the 0.16.0 release to reduce
> > logger noise on shutdown [1][2] but I have heard no negative (or
> positive)
> > feedback otherwise.
> >
> >
> >
> > [1] https://issues.apache.org/jira/browse/AURORA-1729
> >
> > [2] https://reviews.apache.org/r/49578/
> >
> >
> >
> > On Thu, Jun 16, 2016 at 1:18 PM, John Sirois  wrote:
> >
> >
> >
> >
> >
> > On Thu, Jun 16, 2016 at 12:03 AM, Martin Hrabovčin <
> > martin.hrabov...@gmail.com> wrote:
> >
> > How should be this flag rolled to existing running cluster? Can it be
> done
> > using rolling update instance by instance or we need to stop the whole
> > cluster and then bring all nodes with new flag?
> >
> >
> >
> > I recommend a whole cluster down, upgrade +  new flag, up.
> >
> >
> >
> > A rolling update should work, but will likely be rocky.  My analysis:
> >
> >
> >
> > The Aurora leader election consists of 2 components, the actual leader
> > election and the resulting advertisement by the leader of itself as the
> > Aurora service endpoint.  These 2 components each use zookeeper and of
> the
> > 2 I only ensured that the advertisement was compatible with old releases
> > (old clients). The leader election portion is completely internal to the
> > Aurora scheduler instances vying for leadership and, under Curator, uses
> a
> > different (enhanced), zookeeper node scheme.  As a result, this is what
> > could happen in a slow roll:
> >
> >
> >
> > before upgrade: 0: old-lead, 1: old-follow, 2: old-follow
> >
> > upgrade 0: new-lead, 1: old-lead, 2: old-follow
> >
> >
> >
> > Here, node 0 will see itself as leader and nodes 1 and 2 will see node 1
> > as leader. The result will be both node 0 and node 1 attempting to read
> the
> > mesos distributed log.  Now the log uses its own leader election and the
> > reader must be the leader as things stand, so the Aurora-level leadership
> > "tie" will be broken by one of the 2 Aurora-level leaders failing to
> become
> > the mesos distributed log leader, and that node will restart its
> lifecycle
> > - ie flap.  This will continue to be the case with second node upgrade
> and
> > will not stabilize until the 3rd node is upgraded.
> >
> >
> >
> >
> >
> > 2016-06-16 5:03 GMT+02:00 Jake Farrell :
> >
> > +1, will enable on our test clusters to help verify
> >
> > -Jake
> >
> >
> > On Tue, Jun 14, 2016 at 7:43 PM, John Sirois  wrote:
> >
> > > I'd like to move forward with
> > > https://issues.apache.org/jira/browse/AURORA-1669 asap; ie: removing
> > > legacy
> > > (Twitter) commons zookeeper libraries used for Aurora leader election
> in
> > > favor of Apache Curator libraries. The change submitted in
> > > https://reviews.apache.org/r/46286/ is now live in Aurora 0.14.0 and
> > > Apache
> > > Curator based service discovery can be enabled with the Aurora
> scheduler
> > > flag `-zk_use_curator`.  I'd like feedback from users who enable this
> > > option.  If you have a test cluster where you can enable
> > `-zk_use_curator`
> > > and exercise leader failure and failover, I'd be grateful. If you have
> > > moved to using this option in production with demonstrable improvements
> > or
> > > even maintenance of status quo, I'd also be grateful for this news. If
> > > you've found regressions or new bugs, I'd love to know about those as
> > well.
> > >
> > > Thanks in advance to all those who find time to test this out on real
> > > systems!
> > >
> >
> >
> >
> >
> >
> >
> >
> >
>

Re: [PROPOSAL] Move Aurora discussions to Slack?

[Joshua Cohen]

I'm +0 on this. I agree that Slack is in many ways superior to IRC, but it
also feels like the barrier to entry for Slack is much higher than it is
for IRC which is potentially problematic for an open source community.

On Tue, Aug 2, 2016 at 4:08 PM, Jake Farrell  wrote:

> There is an irc bridge which relays all messages back as well as an archive
> bot. Not a huge fan of slack due to it being yet another chat client I have
> in the background, but if it helps the community stay connected and grow
> and makes things easier then +1 for whatever it is.
>
> Might make sense for us to use the mesos.slack.com #aurora channel that
> way
> the two communities stay close and its easier for users to find and ask
> questions rather than having multiple slacks they have to join and keep
> track of
>
> -Jake
>
>
>
>
> On Tue, Aug 2, 2016 at 4:48 PM, Steve Niemitz  wrote:
>
> > +1, I pretty much never remember to open my IRC client anymore.  I've
> been
> > using the Mesos Slack for a few weeks now and its way better than IRC.  I
> > believe they have chat logging still via a bot of some type too?
> >
> > On Tue, Aug 2, 2016 at 4:45 PM, Maxim Khutornenko 
> > wrote:
> >
> > > Mesos community has recently moved to Slack as their canonical chat
> > channel
> > > [1]. Thanks to Stephan, we already have some presence there via #aurora
> > > channel in Apache Mesos team.
> > >
> > > Should we move our IRC discussions to Slack too?
> > >
> > > [1] - http://markmail.org/message/azd37j64wsozmuhe
> > >
> >
>

Re: [VOTE] Release Apache Aurora 0.15.0 RC1

[Joshua Cohen]

+1 everything checks out w/ verify-release-candidate.

On Thu, Jun 30, 2016 at 9:32 PM, Maxim Khutornenko  wrote:

> +1
>
> All verification tests passed for me and scheduler has been deployed
> to our test cluster.
>
> On Thu, Jun 30, 2016 at 6:56 PM, Maxim Khutornenko 
> wrote:
> > All,
> >
> >
> > I propose that we accept the following release candidate as the official
> >
> > Apache Aurora 0.15.0 release.
> >
> >
> > Aurora 0.15.0-rc1 includes the following:
> >
> > ---
> >
> > The RELEASE NOTES for the release are available at:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=RELEASE-NOTES.md=rel/0.15.0-rc1
> >
> >
> > The CHANGELOG for the release is available at:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=CHANGELOG=rel/0.15.0-rc1
> >
> >
> > The tag used to create the release candidate is:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=shortlog;h=refs/tags/rel/0.15.0-rc1
> >
> >
> > The release candidate is available at:
> >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.15.0-rc1/apache-aurora-0.15.0-rc1.tar.gz
> >
> >
> > The MD5 checksum of the release candidate can be found at:
> >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.15.0-rc1/apache-aurora-0.15.0-rc1.tar.gz.md5
> >
> >
> > The signature of the release candidate can be found at:
> >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.15.0-rc1/apache-aurora-0.15.0-rc1.tar.gz.asc
> >
> >
> > The GPG key used to sign the release are available at:
> >
> > https://dist.apache.org/repos/dist/dev/aurora/KEYS
> >
> >
> > Please download, verify, and test.
> >
> >
> > The vote will close on Sun Jul  3 18:55:16 PDT 2016
> >
> >
> > [ ] +1 Release this as Apache Aurora 0.15.0
> >
> > [ ] +0
> >
> > [ ] -1 Do not release this as Apache Aurora 0.15.0 because...
>

Re: Aurora 0.14.0 release

[Joshua Cohen]

+1 to releasing 0.14.0 and thanks for volunteering to act as RM, Stephan!

On Tue, Jun 7, 2016 at 8:52 AM, Jake Farrell  wrote:

> Take a look at the "Creating a release" section in docs/development/
> committers-guide.md, you will also need to add you GPG key to the
> necessary
> files, feel free to ping me with any questions or bring them up in #aurora
> as a number of us have now been RM's before
>
> -Jake
>
> On Mon, Jun 6, 2016 at 6:54 PM, Erb, Stephan 
> wrote:
>
> > +1 for a RC this week.
> >
> > I'd volunteer to serve as a release manager, but will probably need
> > someone to walk me through the necessary steps.
> > 
> > From: Maxim Khutornenko 
> > Sent: Tuesday, June 7, 2016 00:15
> > To: dev@aurora.apache.org
> > Subject: Aurora 0.14.0 release
> >
> > With the resource refactoring and GPU support landed plus appc
> > container ongoing work presumably not requiring any further schema
> > changes (Joshua, please correct if that's not the case), how does
> > everyone feel about cutting a release candidate this week?
> >
> > Assuming no objections: anyone wants to be a release manager for 0.14.0?
> >
>

Re: Agent / Slave renaming

[Joshua Cohen]

+1 for me.

On Wed, May 18, 2016 at 7:15 AM, Erb, Stephan 
wrote:

> I got some spare time yesterday and used it to rebase the first of a
> series of stale review requests by Kevin that implement the Mesos agent
> renaming in Aurora (https://reviews.apache.org/r/47495/).
>
> The renaming is ranked high on the Mesos roadmap. The specific epic is
> work-in-progress but proceeding as we speak [2]. For example, the public
> documentation has been updated just yesterday [3].
>
> I believe the renaming has gained enough momentum in Mesos so that is
> worthwhile for us to tag along. We gain little in waiting this out. What do
> you think?
>
> Best Regards,
> Stephan
>
> [1] https://cwiki.apache.org/confluence/display/MESOS/Roadmap
> [2] https://issues.apache.org/jira/browse/MESOS-1478
> [3]
> https://github.com/apache/mesos/commit/24e1e098035ce918e0b73c9b3c751418d5c06064

Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.

[Joshua Cohen]

And apparently this is not part of our fork at all, the client already
supports this today! The only potential change that would be required would
be ensuring the client properly follows redirects.

On Mon, Apr 18, 2016 at 5:35 PM, Joshua Cohen <jco...@apache.org> wrote:

> Er, it's not `proxy_url`, it's  `scheduler_uri` (which makes much more
> sense ;)).
>
> On Mon, Apr 18, 2016 at 5:34 PM, Joshua Cohen <jco...@apache.org> wrote:
>
>> I'm not opposed to this, in fact we already do something similar
>> internally. We've forked clusters.py to allow configuring a `proxy_url` for
>> each cluster. If that's present, then the client will use it rather than
>> performing service discovery to communicate with the scheduler.
>>
>> On Mon, Apr 18, 2016 at 2:03 PM, John Sirois <jsir...@apache.org> wrote:
>>
>>> As part of work on switching the Aurora scheduler from using a copy of
>>> the
>>> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to
>>> enable a
>>> Curator feature (GUID protection [2]) that will make the scheduler more
>>> robust to ZooKeeper outages but has the side-effect of breaking the
>>> effectively public API formed by the ZooKeeper path names it uses to
>>> perform leader election and service advertisement.  From Aurora's
>>> standpoint, the API consumer is the Aurora command-line client.  It uses
>>> service-discovery today to find the leading scheduler to communicate
>>> with.
>>> I propose switching the client to use HTTP re-directs instead since the
>>> scheduler already has a redirect service and since this would reduce the
>>> scheduler API surface area and generally move further down the road of a
>>> pure HTTP api.
>>>
>>> The most obvious problem I see here is just the mechanics of a proper
>>> deprecation cycle for all those clients Aurora is not directly aware of
>>> that rely on its service advertisment API in ZooKeeper today.
>>>
>>> Are there other problems folks see with this?
>>>
>>> [1] https://issues.apache.org/jira/browse/AURORA-1468
>>> [2] https://issues.apache.org/jira/browse/AURORA-1670
>>>
>>
>>
>

Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.

[Joshua Cohen]

Er, it's not `proxy_url`, it's  `scheduler_uri` (which makes much more
sense ;)).

On Mon, Apr 18, 2016 at 5:34 PM, Joshua Cohen <jco...@apache.org> wrote:

> I'm not opposed to this, in fact we already do something similar
> internally. We've forked clusters.py to allow configuring a `proxy_url` for
> each cluster. If that's present, then the client will use it rather than
> performing service discovery to communicate with the scheduler.
>
> On Mon, Apr 18, 2016 at 2:03 PM, John Sirois <jsir...@apache.org> wrote:
>
>> As part of work on switching the Aurora scheduler from using a copy of the
>> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to enable
>> a
>> Curator feature (GUID protection [2]) that will make the scheduler more
>> robust to ZooKeeper outages but has the side-effect of breaking the
>> effectively public API formed by the ZooKeeper path names it uses to
>> perform leader election and service advertisement.  From Aurora's
>> standpoint, the API consumer is the Aurora command-line client.  It uses
>> service-discovery today to find the leading scheduler to communicate with.
>> I propose switching the client to use HTTP re-directs instead since the
>> scheduler already has a redirect service and since this would reduce the
>> scheduler API surface area and generally move further down the road of a
>> pure HTTP api.
>>
>> The most obvious problem I see here is just the mechanics of a proper
>> deprecation cycle for all those clients Aurora is not directly aware of
>> that rely on its service advertisment API in ZooKeeper today.
>>
>> Are there other problems folks see with this?
>>
>> [1] https://issues.apache.org/jira/browse/AURORA-1468
>> [2] https://issues.apache.org/jira/browse/AURORA-1670
>>
>
>

Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.

[Joshua Cohen]

I'm not opposed to this, in fact we already do something similar
internally. We've forked clusters.py to allow configuring a `proxy_url` for
each cluster. If that's present, then the client will use it rather than
performing service discovery to communicate with the scheduler.

On Mon, Apr 18, 2016 at 2:03 PM, John Sirois  wrote:

> As part of work on switching the Aurora scheduler from using a copy of the
> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to enable a
> Curator feature (GUID protection [2]) that will make the scheduler more
> robust to ZooKeeper outages but has the side-effect of breaking the
> effectively public API formed by the ZooKeeper path names it uses to
> perform leader election and service advertisement.  From Aurora's
> standpoint, the API consumer is the Aurora command-line client.  It uses
> service-discovery today to find the leading scheduler to communicate with.
> I propose switching the client to use HTTP re-directs instead since the
> scheduler already has a redirect service and since this would reduce the
> scheduler API surface area and generally move further down the road of a
> pure HTTP api.
>
> The most obvious problem I see here is just the mechanics of a proper
> deprecation cycle for all those clients Aurora is not directly aware of
> that rely on its service advertisment API in ZooKeeper today.
>
> Are there other problems folks see with this?
>
> [1] https://issues.apache.org/jira/browse/AURORA-1468
> [2] https://issues.apache.org/jira/browse/AURORA-1670
>

Heads up: Mesos 0.27 and Vagrant

[Joshua Cohen]

Hi Aurorans,

Just a heads up, I've landed the upgrade to Mesos 0.27.2. This is our first
Mesos version bump since switching to a custom pre-built Vagrant box. If
you try and build the Scheduler or Executor in an existing Vagrant image
you'll likely run into errors. Please destroy and re-create your vagrant
image to pick up the most recent Aurora dev environment box to resolve
these incompatibilities.

Cheers,

Joshua

Re: [VOTE] Release Apache Aurora 0.13.0 RC0

[Joshua Cohen]

Ahh, thanks for being on top of that John!

On Tue, Apr 12, 2016 at 3:05 PM, John Sirois <john.sir...@gmail.com> wrote:

> On Tue, Apr 12, 2016 at 1:48 PM, Joshua Cohen <jco...@apache.org> wrote:
>
> > +1, ran the verification script, everything looks good to me.
> >
> > Nitpick: the NEWS link in the email 404s because we renamed it to
> > RELEASE-NOTES.md. We should probably update the script that generates the
> > email?
> >
>
> See my response above - fixed here earlier this am:
> https://reviews.apache.org/r/46070/
>
>
> >
> > On Tue, Apr 12, 2016 at 10:02 AM, Erb, Stephan <
> > stephan@blue-yonder.com>
> > wrote:
> >
> > > +1 for releasing 0.13.0-rc0 as Aurora 0.13.0
> > >
> > > * tested with the verification script
> > > * deployed the RC to an inhouse test cluster
> > >
> > > I am also OK with fixing the changelog afterwards.
> > >
> > > 
> > > From: Bill Farner <wfar...@apache.org>
> > > Sent: Tuesday, April 12, 2016 16:18
> > > To: jfarr...@apache.org
> > > Cc: dev@aurora.apache.org
> > > Subject: Re: [VOTE] Release Apache Aurora 0.13.0 RC0
> > >
> > > I'm okay with not blocking the release on it.
> > >
> > > On Tuesday, April 12, 2016, Jake Farrell <jfarr...@apache.org> wrote:
> > >
> > > > My fault, I thought we had changed this awhile back to automatically
> > pull
> > > > in any tickets not having a set fix version that had been marked as
> > > > resolved and that this had become a post step to clean up jira after
> > the
> > > > release when marking the version as released.
> > > >
> > > > This should not be a blocker for the 0.13.0-rc0 release candidate as
> > the
> > > > CHANGELOG is not a requirement for a release, just a nice best
> > practice.
> > > > Will follow up with a patch to trunk to update for the missing
> section.
> > > If
> > > > this is a must have in anyones opinion then we can cancel this vote
> and
> > > > start a new release candidate, thoughts?
> > > >
> > > > -Jake
> > > >
> > > > On Tue, Apr 12, 2016 at 1:59 AM, Bill Farner <wfar...@apache.org
> > > > <javascript:_e(%7B%7D,'cvml','wfar...@apache.org');>> wrote:
> > > >
> > > >> Changelog looks sparse.  In the past we have tagged all resolved
> > tickets
> > > >> with fixVersion that don't already have fixVersion set, which I
> > believe
> > > >> lands them in the changelog.  Presumably that step is undocumented
> and
> > > not
> > > >> automated, and as a result didn't happen?
> > > >>
> > > >> On Monday, April 11, 2016, John Sirois <john.sir...@gmail.com
> > > >> <javascript:_e(%7B%7D,'cvml','john.sir...@gmail.com');>> wrote:
> > > >>
> > > >>> +1 - Tested with `./build-support/release/verify-release-candidate
> > > >>> 0.13.0-rc0`.
> > > >>>
> > > >>> On Mon, Apr 11, 2016 at 9:26 PM, Jake Farrell <jfarr...@apache.org
> >
> > > >>> wrote:
> > > >>>
> > > >>> > All,
> > > >>> >
> > > >>> > I propose that we accept the following release candidate as the
> > > >>> official
> > > >>> > Apache Aurora 0.13.0 release.
> > > >>> >
> > > >>> > Aurora 0.13.0-rc0 includes the following:
> > > >>> > ---
> > > >>> > The NEWS for the release is available at:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=NEWS=rel/0.13.0-rc0
> > > >>>
> > > >>>
> > > >>> The NEWS link above is broken, but was fixed here:
> > > >>> https://reviews.apache.org/r/46070/
> > > >>>
> > > >>>
> > > >>> >
> > > >>> >
> > > >>> > The CHANGELOG for the release is available at:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=CHANGELOG=rel/0.13.0-rc0
> > > >>>
> > > >>>
> > > >>&

Re: Looking for feedback - Setting CommandInfo.user by default when launching tasks.

[Joshua Cohen]

As things stand today, once a task is scheduled, the scheduler can die, or
be shut down for maintenance, etc. with no impact to running tasks. If the
scheduler were responsible for announcing and it maintained the current
practice of creating znodes as ephemeral, it would need to maintain a
persistent ZK connection to ensure all announceable tasks are discoverable.
This means if the scheduler went down for any reason (or just whenever it
failed over normally) all serversets would be lost. That's obviously not
acceptable, so the alternative, if we wanted the scheduler to manage
announcing tasks, would be to stop using ephemeral nodes and instead have
the scheduler create persistent znodes and manually tear them down when
tasks finish. While not as problematic as the above, this is still
potentially a degradation from the current behavior in that there can be a
long delay between a task exiting and the scheduler becoming aware of this
(e.g. task finishes during a netsplit). This might be equivalent in scope
to the issue of a zombie instance that's possible today (i.e. scheduler
thinks task has been killed and restarts it elsewhere, but Mesos fails to
clean up the cgroup for whatever reason leading to double registration for
some time). However, today, due to their ephemeral nature, cleaning up
duplicate znodes is guaranteed as soon as the executor exits. If the znodes
were persistent, we'd have to manually seek out and clean up duplicate
znodes (or build tooling to support this).

It's certainly possible to transition responsibility for announcing from
the executor to the scheduler, but I think it would be a fairly significant
amount of work and add a fair amount of complexity.

On Tue, Apr 12, 2016 at 7:35 AM, Erb, Stephan 
wrote:

> I have recently run into another issue due to Thermos running as root (
> https://issues.apache.org/jira/browse/MESOS-5187). I would therefore like
> to re-open the discussion.
>
> IIRC there was a JIRA ticket proposing that the scheduler should be
> responsible for announcing services in Zookeeper.
>
> Would that work? It looks like this could solve a couple of issues at once:
>
> * no need for credentials on slaves, we could therefore run the executor
> as a normal user
> * the announcement would also work for tasks using alternative executors
>
>
> 
> From: John Sirois 
> Sent: Tuesday, March 29, 2016 23:55
> To: Bill Farner
> Cc: dev@aurora.apache.org; John Sirois
> Subject: Re: Looking for feedback - Setting CommandInfo.user by default
> when launching tasks.
>
> On Tue, Mar 29, 2016 at 3:50 PM, Bill Farner  wrote:
>
> > Aha, i think we have different notions of the proposal.  I was under the
> > impression that the executor itself would run as the target user (e.g.
> > steve), not as a system user (e.g. aurora).  I find the former more
> > appealing, with the exception that it leaves us without a solution for
> > concealing the credentials file.
> >
>
> My sketch was nonsense anyhow, since thermos running as aurora couldn't
> launch a task as www-data.  Fundamentally we need a priviledged executor
> (thermos) that can 1: read the creds and announce 2: run the task and
> health checks nnoth as the targeted un-privileged user.
>
>
> >
> > On Tue, Mar 29, 2016 at 2:39 PM, John Sirois 
> wrote:
> >
> >> On Tue, Mar 29, 2016 at 3:26 PM, Bill Farner 
> wrote:
> >>
> >> > If i'm understanding you correctly, that doesn't address preventing
> >> users
> >> > from reading the credentials though.
> >> >
> >>
> >> It does:
> >>
> >> Say:
> >> /var/lib/aurora/creds 400 root
> >>
> >> And then if the CommandInfo has user: aurora (executor user as Steve
> >> suggested), it will get a copy of /var/lib/aurora/creds  in its sandbox
> >> chowned to 400 aurora
> >>
> >> Now aurora's executor (thermos), launches a task in role www-data
> >> announcing for it using the cred.  The www-data user will not be able to
> >> read the local sandbox 400 aurora creds.
> >>
> >>
> >> > On Tue, Mar 29, 2016 at 1:52 PM, John Sirois 
> >> wrote:
> >> >
> >> > > On Tue, Mar 29, 2016 at 2:31 PM, Steve Niemitz  >
> >> > > wrote:
> >> > >
> >> > > > So maybe we add it, but don't change the current default behavior?
> >> > > >
> >> > >
> >> > > Could we use the CommandInfo.uris [1] to solve this?  IE: the
> >> scheduler
> >> > > would need to learn the credential file path, and with that
> knowledge,
> >> > the
> >> > > local mesos (root) readable credential file could be specified as a
> >> uir
> >> > > dependency for all launched executors (or bare commands).  IIUC, if
> >> the
> >> > URI
> >> > > if a file:// the local secured credentails file will be copied into
> >> the
> >> > > sandbox where it can be read by the executor (as aurora).
> >> > >
> >> > > [1]
> >> > >
> >> >
> >>
> 

Re: [PROPOSAL] Support GPU resources in Aurora

[Joshua Cohen]

+1

On Wed, Apr 6, 2016 at 12:41 PM, Maxim Khutornenko  wrote:

> Mesos community is finalizing their MVP for supporting GPUs:
> https://issues.apache.org/jira/browse/MESOS-4424
> Design doc:
> https://docs.google.com/document/d/10GJ1A80x4nIEo8kfdeo9B11PIbS1xJrrB4Z373Ifkpo/edit
>
> Would anyone have any reservations about supporting GPUs in Aurora?
> Initial walk through our resource management codebase shows we can do
> it without too many changes or excessive refactoring. If there are no
> objections I am willing to put up a design doc for review next.
>
> Thanks,
> Maxim
>

Re: Are we ready to remove the observer?

[Joshua Cohen]

If you're suggesting just going to the task directory and pulling them out
of the executor logs. Yes, I could ssh into the host the task is running on
and grep the task directory out of the mesos agent logs and then trawl the
logs (or cat task.json), but that's much more effort than going to the
observer's task UI (i.e. it'd take a minute, rather than a few seconds).
I'd also posit that it's much easier for new Aurora operators to come to
grips with the process tree via the UI rather than a JSON blob.

If you're suggesting something else (i.e. new UI to expose these separate
from the Observer), I'm fine with that, that's what I was implying above
would be necessary before I think we could retire the Observer.

A counter question: do people feel that updating/deploying the Observer is
a major obstacle? I know we've got the process well automated, so it's
relatively painless. I'd love to replace the Observer with something
better, but I don't feel like it's a major drag on our productivity as it
exists today to warrant killing it off entirely. My opinion may be colored
by the deploy automation we have in place though!

On Mon, Apr 4, 2016 at 9:32 AM, Bill Farner <wfar...@apache.org> wrote:

> >
> > 2) Providing an easy view of a process's command-line
> > 3) Providing a holistic view of the task config
>
>
> Just to check my understanding - these could be trivially handled in
> text/log format, right?
>
> On Mon, Apr 4, 2016 at 9:30 AM, Joshua Cohen <jco...@apache.org> wrote:
>
> > I'm -1 on this until we have an actual replacement for the Observer. I
> > think that the observer provides significant value outside of just
> sandbox
> > browsing:
> >
> > 1) Exporting task-level statistics.
> > 2) Providing an easy view of a process's command-line
> > 3) Providing a holistic view of the task config
> > 4) Real time utilization stats
> >
> > As a cluster operator, I use all of these features on a daily basis
> > (especially when I'm on call) in addition to sandbox browsing, so I don't
> > think that these uses cases are that rare.
> >
> > On Fri, Apr 1, 2016 at 6:55 AM, Steve Niemitz <sniem...@apache.org>
> wrote:
> >
> > > The per-process stats have never been very useful to us (since they
> don't
> > > work for docker), however, even being able to see the processes that
> are
> > > running, how many times they've restarted, when they launched, etc is
> > > invaluable.
> > >
> > > I think there would be big pushback from users if they were to lose the
> > > functionality it provided currently (beyond log viewing).
> > >
> > > On Fri, Apr 1, 2016 at 6:58 AM, Erb, Stephan <
> > stephan@blue-yonder.com>
> > > wrote:
> > >
> > > > From an operator and Aurora developer perspective, it would be really
> > > > great to get rid of the thermos observer quickly.
> > > >
> > > > However, from a user perspective the usability gap between observer
> and
> > > > plain Mesos sandbox browsing is quite large right now. I agree with
> > > > Benjamin here that it would probably work if we generate html pages
> > ready
> > > > for user consumption.
> > > >
> > > > These are the relevant tickets in our tracker:
> > > > * https://issues.apache.org/jira/browse/AURORA-725
> > > > * https://issues.apache.org/jira/browse/AURORA-777
> > > >
> > > > 
> > > > From: ben...@gmail.com <ben...@gmail.com>
> > > > Sent: Friday, April 1, 2016 02:35
> > > > To: dev@aurora.apache.org
> > > > Subject: Re: Are we ready to remove the observer?
> > > >
> > > > Is there any chance we can keep the per-process cpu and ram
> utilization
> > > > stats?  That's one of the coolest things about aurora, imo.  The
> > executor
> > > > is already writing those checkpoints inside the mesos sandbox (I
> > think?),
> > > > so perhaps it could also produce the html pages that the observer
> > > currently
> > > > renders?
> > > >
> > > > On Thu, Mar 31, 2016 at 4:33 PM Zhitao Li <zhitaoli...@gmail.com>
> > wrote:
> > > >
> > > > > +1.
> > > > >
> > > > > On Thu, Mar 31, 2016 at 4:11 PM, Bill Farner <wfar...@apache.org>
> > > wrote:
> > > > >
> > > > > > Assuming that the vast majority of utility provided by the
> observer
> > > is
> > > > > > sandbox/log browsing - can we remove it and link to sandbox
> > browsing
> > > > that
> > > > > > mesos provides?
> > > > > >
> > > > > > The rest of the information could be (or already is) logged in
> the
> > > > > sandbox
> > > > > > for the rare debugging scenarios that call for it.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cheers,
> > > > >
> > > > > Zhitao Li
> > > > >
> > > >
> > >
> >
>

Re: Populate DiscoveryInfo in Mesos

[Joshua Cohen]

Job names are not unique though, what would happen if multiple jobs had the
same name (either across roles or across environments in the same role)?

On Wed, Mar 30, 2016 at 5:33 PM, Zhitao Li  wrote:

> Stephan,
>
> So I've managed to run the official Mesos DNS docker container
>  under the Aurora vagrant
> environment and get some SRV/A recorded pulled from Mesos master from
> Aurora.
>
> Because Mesos DNS uses 'name' field if set with some string manipulation,
> for the job 'vagrant/test/http_example_docker', my prototype generates
> these DNS records:
>
> A record: vagranttesthttp-exampled.twitterscheduler.mesos
> SRV record: _vagranttesthttp-exampled._tcp.twitterscheduler.mesos.
>
> If we want to make current prototype useful for Mesos DNS, I suggest we
> change the name field to job name, which would generate record like:
> A: http_example_docker.twitterscheduler.mesos
> SRV: _http_example_docker._tcp.twitterscheduler.slave.mesos
>
> I'll update my patch after getting some signal from you. Thanks.
>
> On Fri, Mar 25, 2016 at 1:49 PM, Zhitao Li  wrote:
>
> > Hi Stephan,
> >
> > Thanks for looking at that prototype patch.
> >
> > I'll update the patch with the review comments, and probably add a global
> > flag of "populate_discovery_info" to toggle this behavior.
> >
> > About the optional fields: I think it'll be hard to come up a good set of
> > rules applicable to all orgs using Aurora + Mesos, because cluster
> > management and service discovery stack could differ from org to org.
> >
> > In a recent Mesos work group, some experience folks (Jie Yu and Ben
> > Mahler) mentioned some ideas of *TaskInfoDecorator, *which is some
> > optional and configurable plugin on Aurora scheduler side to allow
> operator
> > to set additional fields before sending the message to Mesos. I like such
> > idea because it would enable Aurora users to experiment faster. Do you
> > think this is an interesting idea worth pursuing?
> >
> >
> > On Fri, Mar 25, 2016 at 1:42 PM, Erb, Stephan <
> stephan@blue-yonder.com
> > > wrote:
> >
> >> I had a closer look at the Mesos documentation, and a design document
> >> might be unnecessary. Most of the values are optional. We can therefore
> >> leave them out until we have a proper usecase for them.
> >>
> >> I left a couple of comments in the review request.
> >> 
> >> From: Zhitao Li 
> >> Sent: Tuesday, March 22, 2016 21:15
> >> To: dev@aurora.apache.org
> >> Subject: Re: Populate DiscoveryInfo in Mesos
> >>
> >> Hi Stephan,
> >>
> >> Sorry for the delay on follow up on this. I took a quick look at Aurora
> >> code, and it's actually quite easy to pipe this information to Mesos
> (see
> >> https://reviews.apache.org/r/45177/ for quick prototype).
> >>
> >> I'll take a stab to see how I can get Mesos-DNS to work with this
> >> prototype.
> >>
> >> IMO, if this is something the community is interested, the main
> questions
> >> would be 1) how various fields would be mapped in different Aurora
> usages,
> >> and 2) to which level should opt-in/opt-out configured for populating
> such
> >> information.
> >>
> >> I actually don't have too much insights on how these usage conventions
> >> would be set (through command line of scheduler or job configuration?)
> >>
> >> Do you think a design doc is the best action here, or a more involved
> >> questionnaire about which fields would be useful for community, or what
> >> value they should take?
> >>
> >> On Mon, Mar 7, 2016 at 1:00 AM, Erb, Stephan <
> stephan@blue-yonder.com
> >> >
> >> wrote:
> >>
> >> > That sounds like a good idea! Great.
> >> >
> >> > If you go ahead with this, please be so kind and start by posting a
> >> short
> >> > design document here on mailinglist (similar to those here
> >> > https://github.com/apache/aurora/blob/master/docs/design-documents.md
> ,
> >> > but probably shorter).
> >> >
> >> > This will allow us to split the discussion of the design from
> discussing
> >> > the actual implementation. I believe this is necessary, as the
> >> > DiscoveryInfo protocol is quite flexible (
> >> >
> >>
> http://mesos.apache.org/documentation/latest/app-framework-development-guide/
> >> > ).
> >> >
> >> > Thanks,
> >> > Stephan
> >> >
> >> >
> >> > 
> >> > From: Zhitao Li 
> >> > Sent: Monday, March 7, 2016 00:05
> >> > To: dev@aurora.apache.org
> >> > Subject: Populate DiscoveryInfo in Mesos
> >> >
> >> > Hi,
> >> >
> >> > It seems like Aurora does not populate the "discovery" field in either
> >> > TaskInfo or ExecutorInfo in mesos.proto
> >> > <
> >> >
> >>
> https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L438
> >> > >
> >> > .
> >> >
> >> > I'm considering adding this to support retrieving port map in Mesos
> >> > directly. This would enable us to discovery this 

Re: Compute event at Twitter HQ

[Joshua Cohen]

Just a heads up for those who will not be able to attend: these talks will
be recorded and made available on Twitter's YouTube channel. I'll be sure
to send out a link after the event.

For those who are attending, we look forward to seeing you tomorrow night!

Cheers,

Joshua

On Tue, Mar 15, 2016 at 4:04 PM, Joshua Cohen <jco...@apache.org> wrote:

> Hi Aurorans[1],
>
> I'd like to call attention to an event the Compute group at Twitter is
> holding at the end of the month where there will be a few of
> Aurora/Mesos-related talks:
>
>
>1. David Robinson, one of our SREs, will talk about how our small team
>of SREs manages what is possibly the largest Mesos cluster in existence (I
>have no evidence to back that up, but it sounds good ;)).
>2. David McLaughlin, Aurora committer/PMC member, will talk Workflows,
>an internal tool we've built to orchestrate deployments across Aurora
>clusters.
>3. David Hagar[2], Engineering Manager at TellApart, will talk about
>running Aurora/Mesos in AWS.
>
>
> On top of that there will be lots of other great talks about how we run
> the entirety of our compute infrastructure.
>
> The event is on the evening of March 31st at Twitter HQ in San Francisco.
> I hope to see you guys there!
>
> https://www.eventbrite.com/e/compute-tickets-22811196904
>
> Cheers,
>
> Joshua
>
> [1] I just coined that, Aurorans, what do you think?
> [2] Contrary to the evidence above, you don't need to be named David to
> work on Aurora/Mesos at Twitter.
>

Re: Handling migrations if dbScript exists in snapshot?

[Joshua Cohen]

I sent out a review for a prototype of my proposed migration system here:
https://reviews.apache.org/r/45467/

Please take a look at the review and let me know if it seems reasonable at
a high level (noting the caveats called out in the review description, this
is not production quality code, etc.).

The biggest thing I'd like to call out that makes me a bit wary is that it
uses MyBatis Migrations and, as I mentioned right now that means we have to
consume a -SNAPSHOT jar, as for some reason they have not published a new
version to Maven Central since 2014 (even though the project is active on
github.) I opened https://github.com/mybatis/migrations/issues/38 to see
about addressing that.

On Thu, Mar 24, 2016 at 11:00 AM, Joshua Cohen <jco...@apache.org> wrote:

> Sorry, I should've been more clear above. What I meant to say is that the
> lack of explicit rollback support was not a mark against Flyway since we'd
> have to do similar work to accomplish whether we did it w/ Flyway or
> whether we rolled our own.
>
> I'm currently experimenting w/ Mybatis migrations which has native support
> for downgrades as well as upgrades. The main caveat there is that we'd have
> to use a snapshot version of the jar since the one published to Maven
> Central relies on an incompatible version of the core mybatis jar.
>
> On Thu, Mar 24, 2016 at 10:53 AM, Maxim Khutornenko <ma...@apache.org>
> wrote:
>
>> >
>> > I think in the event that we have to rollback after applying a change
>> like
>> > you describe we'd either have to resort to a manual fix or restore from
>> > backup.
>>
>> I am afraid restoring from backup will not be an acceptable solution
>> if/when a new version has to be rolled back after running for awhile.
>>
>> The other thing to keep in mind is that this problem is not solved
>> > by rolling our own migration solution either.
>>
>> I don't see why we can't move data back the same way we moved it forward.
>> It takes another script to drop new objects, re-create old and migrate
>> data. What this may not help with is restoring data that has been dropped
>> completely from the DB but I'd argue that should be a non-issue as nothing
>> in the scheduler should rely on that data by the time it's dropped.
>>
>> How about having two flyway instances to support forward and reverse
>> migrations? The flyway seems to support it:
>> https://flywaydb.org/documentation/faq#multiple-schemas. We could have a
>> SQL or a Java 'afterMigrate' callback (both supported by flyway) to
>> establish a new baseline upon forward migration that may be used by a
>> reverse flyway instance to apply rollback scripts in an event of a version
>> downgrade.
>>
>> I feel this is an important problem that we need to have full clarity
>> around before moving forward with any schema mods. Happy to partake in any
>> related changes.
>>
>> On Thu, Mar 24, 2016 at 7:25 AM, Joshua Cohen <jco...@apache.org> wrote:
>>
>> > Flyway has a fairly well reasoned response to the downgrades question:
>> > https://flywaydb.org/documentation/faq#downgrade
>> >
>> > I think in the event that we have to rollback after applying a change
>> like
>> > you describe we'd either have to resort to a manual fix or restore from
>> > backup. The other thing to keep in mind is that this problem is not
>> solved
>> > by rolling our own migration solution either.
>> >
>> > Another option that may be worth considering is mybatis migrations:
>> > http://www.mybatis.org/migrations/runtime-migration.html.
>> >
>> > On Wed, Mar 23, 2016 at 11:34 PM, Maxim Khutornenko <ma...@apache.org>
>> > wrote:
>> >
>> > > +1 to giving flyway a try. I have used it before and it *mostly*
>> > > worked. There were a few hiccups occasionally when it couldn’t
>> > > properly recognize the DB drift and failed to apply the update but
>> > > that was entirely incremental build-related problem (flyway was
>> > > integrated into the build process).
>> > >
>> > > What flyway does not support is version rollbacks. It's not applicable
>> > > to this particular change but I am sure there will be changes where
>> > > we'd need to move data as part of DB upgrade. If we then decide to
>> > > rollback due to an unforeseen problem we would have to have reverse
>> > > migration script(s). Not impossible but something to keep in mind.
>> > >
>> > >
>> > >
>> > > On Wed, Mar 23, 2016 at 4:41 PM, Florian Pfeiffer

Re: Handling migrations if dbScript exists in snapshot?

[Joshua Cohen]

Flyway has a fairly well reasoned response to the downgrades question:
https://flywaydb.org/documentation/faq#downgrade

I think in the event that we have to rollback after applying a change like
you describe we'd either have to resort to a manual fix or restore from
backup. The other thing to keep in mind is that this problem is not solved
by rolling our own migration solution either.

Another option that may be worth considering is mybatis migrations:
http://www.mybatis.org/migrations/runtime-migration.html.

On Wed, Mar 23, 2016 at 11:34 PM, Maxim Khutornenko <ma...@apache.org>
wrote:

> +1 to giving flyway a try. I have used it before and it *mostly*
> worked. There were a few hiccups occasionally when it couldn’t
> properly recognize the DB drift and failed to apply the update but
> that was entirely incremental build-related problem (flyway was
> integrated into the build process).
>
> What flyway does not support is version rollbacks. It's not applicable
> to this particular change but I am sure there will be changes where
> we'd need to move data as part of DB upgrade. If we then decide to
> rollback due to an unforeseen problem we would have to have reverse
> migration script(s). Not impossible but something to keep in mind.
>
>
>
> On Wed, Mar 23, 2016 at 4:41 PM, Florian Pfeiffer
> <florian.pfeif...@gutefrage.net> wrote:
> > We're using flyway for our services, and from an ops perspective I'm
> quite
> > happy with it. On test it's allowed to apply the migrations
> > automatically... on prod we're doing it manually after reviewing. No
> > problems so far.
> >  I can't really tell anything from the dev side, beside that I haven't
> > heard them complaining about it (and normally they are complaining quite
> > fast ;) )...
> >
> > Another thing that's similar to flyway is http://www.liquibase.org/ but
> for
> > that I have zero experience.
> >
> > Flo
> >
> >
> > On 23 March 2016 at 22:41, Joshua Cohen <jco...@apache.org> wrote:
> >
> >> If we go with option 2 (which I'm leaning towards as well), does anyone
> >> have thoughts on using (or experience with) something like Flyway:
> >> https://flywaydb.org/ rather than implementing from scratch?
> >>
> >> On Wed, Mar 23, 2016 at 3:29 PM, John Sirois <jsir...@apache.org>
> wrote:
> >>
> >> > On Wed, Mar 23, 2016 at 2:21 PM, Joshua Cohen <jco...@apache.org>
> wrote:
> >> >
> >> > > Hi Aurorans,
> >> > >
> >> > > As you may have seen (
> >> https://issues.apache.org/jira/browse/AURORA-1648
> >> > ),
> >> > > we ran into an issue when upgrading a cluster that uses dbScripts in
> >> > > snapshots. To restate the problem from the ticket, when the
> scheduler
> >> > > starts up it creates the H2 database from schema.sql which contains
> >> newly
> >> > > added (or changed) tables. If the dbScript exists in the snapshot,
> it
> >> > then
> >> > > drops all tables and runs the script which re-creates the database
> in
> >> the
> >> > > form it was previously in and repopulates all the data. At this
> point,
> >> > if a
> >> > > table was added that was not in the snapshot, that table is lost.
> >> > >
> >> > > The solution here is to be run a migration after restoring the
> >> database,
> >> > my
> >> > > question is what level of complexity do we want for migrations. I
> see
> >> two
> >> > > possible options:
> >> > >
> >> > >
> >> > >1. The relatively simple approach where we have a single
> >> migration.sql
> >> > >alongside schema.sql which contains any changes to run against
> the
> >> > >database. After restoring the db from a dbScript in the
> snapshot, we
> >> > > would
> >> > >run this script. On release boundaries(?), we clear the script
> out.
> >> > The
> >> > >main caveat here is that the ddl statements in this script must
> be
> >> > >idempotent since they would potentially be run multiple times.
> >> Another
> >> > >issue is this would make it more complicated to jump over
> versions
> >> > >(upgrading from v0.12.0 to v0.14.0 means you might miss a
> migration
> >> > that
> >> > >only existed in the 0.13.0 release).
> >> > >2. A slightly more involved approac

Re: aurora job scalability

[Joshua Cohen]

Hi Christopher,

I think you already got an answer from Stephan in IRC, but just wanted to
follow up for the sake of posterity (in case anyone in the future has a
similar question and finds this thread). The only limit on the number of
jobs that Aurora can run would currently be the amount of memory available
to the Scheduler. Suffice it to say that at Twitter we're running thousands
of jobs with no issues.

Let us know if you have any follow up questions.

Cheers,

Joshua

On Fri, Mar 18, 2016 at 9:12 AM, Christopher M Luciano  wrote:

>  Hi all. It seems that we may be outgrowing Marathon. We have a problem
> with the amount of application that we are using, causing us to not exactly
> be "compliant" with Marathon goals. It seems that the unit for Aurora is a
> job+instance of that job. Does job map to a Marathon application? If
> similar is there a known limitation to how many jobs one can have?
>
>  What we discovered for Marathon more application+bigger env_vars = bigger
> zknode size and we come dangerously close to hitting the 1 MB default of
> the zk zknode size. I'm wondering if this type of a thing has potentially
> been fixed already in Aurora.
>
>
>
>
>
>
> Christopher M Luciano
>
> Staff Software Engineer, Platform Services
>
> IBM Watson Core Technology
>
>
>
>
>

Compute event at Twitter HQ

[Joshua Cohen]

Hi Aurorans[1],

I'd like to call attention to an event the Compute group at Twitter is
holding at the end of the month where there will be a few of
Aurora/Mesos-related talks:


   1. David Robinson, one of our SREs, will talk about how our small team
   of SREs manages what is possibly the largest Mesos cluster in existence (I
   have no evidence to back that up, but it sounds good ;)).
   2. David McLaughlin, Aurora committer/PMC member, will talk Workflows,
   an internal tool we've built to orchestrate deployments across Aurora
   clusters.
   3. David Hagar[2], Engineering Manager at TellApart, will talk about
   running Aurora/Mesos in AWS.


On top of that there will be lots of other great talks about how we run the
entirety of our compute infrastructure.

The event is on the evening of March 31st at Twitter HQ in San Francisco. I
hope to see you guys there!

https://www.eventbrite.com/e/compute-tickets-22811196904

Cheers,

Joshua

[1] I just coined that, Aurorans, what do you think?
[2] Contrary to the evidence above, you don't need to be named David to
work on Aurora/Mesos at Twitter.

Re: [PROPOSAL] Supporting Mesos Universal Containers

[Joshua Cohen]

I've gone ahead and filed some tickets breaking down the work involved in
this effort. They're all contained within this epic:
https://issues.apache.org/jira/browse/AURORA-1634.

I agree that we should assess our plans re: containers as a whole. My
understanding of the current state of the world is as follows:


   1. There are definite benefits to adopting the unified containerizer to
   launch image-based tasks (outlined in the design doc, but I'll reiterate
   here: better interaction w/ Mesos isolators, no need to rely on external
   daemons to coordinate containers, etc.).
   2. There are also considerations to keep in mind, specifically, we now
   rely on Mesos maintaining this image support, which is secondary to its
   primary goals, rather than relying on organizations that are invested in
   the formats. Additionally, we'll have to wait on Mesos to implement new
   features of image formats before they can be adopted by Aurora users.

I don't believe that number 2 above should be a blocker to number 1, but
it's a caveat that we must always keep in mind. I'll point out that the
design doc takes a very cautious approach towards deprecating the current
Docker support. It may be that we maintain both in perpetuity. It's also
possible, and I'm hopeful that this is the case, that the Mesos community
will show responsible custodianship of the unified container support,
allaying the concerns outlined above and allowing us to deprecate the
current native-Docker containerizer support. In either case described here,
I think Aurora users will benefit.

Does anyone think we need a stop-the-world moment to come up with a long
term, holistic plan, or is it reasonable to assess the situation as we go?


On Sun, Mar 13, 2016 at 7:23 AM, Erb, Stephan <stephan@blue-yonder.com>
wrote:

> As mentioned in IRC, I like the proposal.
>
> Still, we need a discussion regarding the future of current Docker
> support. Especially since Bill and John have now started improving it. What
> are our plans here? What  are the plans of the Mesos community (i.e.,
> deprecation of the docker containerizer)?
>
> In addition, the switch implemented for disabling Thermos when running
> Docker kind of reminded me of
> https://issues.apache.org/jira/browse/AURORA-1288. It is probably
> worthwhile to at least assess this as a whole.
>
> Kind Regards,
> Stephan
>
> ____
> From: Joshua Cohen <jco...@apache.org>
> Sent: Monday, March 7, 2016 20:58
> To: dev@aurora.apache.org
> Subject: [PROPOSAL] Supporting Mesos Universal Containers
>
> Hi all,
>
> I'd like to propose we adopt the Mesos universal container support for
> provisioning tasks from both Docker and AppC images. Please review the doc
> below and let me know what you think.
>
>
> https://docs.google.com/document/d/111T09NBF2zjjl7HE95xglsDpRdKoZqhCRM5hHmOfTLA/edit?usp=sharing
>
> Thanks!
>
> Joshua
>

Re: [VOTE] Release Apache Aurora 0.12.0 debs

[Joshua Cohen]

+1, verified under both test/deb environments.

On Tue, Mar 8, 2016 at 12:18 PM, John Sirois  wrote:

> On Mon, Mar 7, 2016 at 12:57 PM, Bill Farner  wrote:
>
> > +1
> >
> > Successfully installed and lightly exercised ubuntu and debian packages
> in
> > their respective systems (using test/ instructions in the
> aurora-packaging
> > repo).
> >
>
> I'll add my own +1 after installing and testing in the 2 respective
> aurora-packaing test/deb/ environments.
>
>
> >
> > On Mon, Mar 7, 2016 at 8:00 AM, John Sirois  wrote:
> >
> >> I propose that we accept the following artifacts as the official deb
> >> packaging for
> >> Apache Aurora 0.12.0.
> >>
> >> https://dl.bintray.com/john-sirois/aurora/ubuntu-trusty/
> >> https://dl.bintray.com/john-sirois/aurora/debian-jessie/
> >>
> >> The Aurora deb packaging includes the following:
> >> ---
> >> The CHANGELOG is available at:
> >>
> >>
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=blob_plain;f=specs/debian/changelog;hb=refs/heads/0.12.x
> >>
> >> The branch used to create the packaging is:
> >>
> >>
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;h=refs/heads/0.12.x
> >>
> >> The packages are available at:
> >> https://dl.bintray.com/john-sirois/aurora/ubuntu-trusty/
> >> https://dl.bintray.com/john-sirois/aurora/debian-jessie/
> >>
> >> The GPG keys used to sign the packages are available at:
> >> https://dist.apache.org/repos/dist/release/aurora/KEYS
> >>
> >> Please download, verify, and test.
> >>
> >> The vote will close on Thu, 07 Mar 2016 09:00:00 -0700
> >>
> >> [ ] +1 Release these as the deb packages for Apache Aurora 0.12.0
> >> [ ] +0
> >> [ ] -1 Do not release these artifacts because...
> >> ---
> >>
> >> Please consider verifying these debs using the install guide:
> >>   https://github.com/apache/aurora/blob/master/docs/installing.md
> >>
> >
> >
>

[PROPOSAL] Supporting Mesos Universal Containers

[Joshua Cohen]

Hi all,

I'd like to propose we adopt the Mesos universal container support for
provisioning tasks from both Docker and AppC images. Please review the doc
below and let me know what you think.

https://docs.google.com/document/d/111T09NBF2zjjl7HE95xglsDpRdKoZqhCRM5hHmOfTLA/edit?usp=sharing

Thanks!

Joshua

Re: Further thoughts on config deprecations

[Joshua Cohen]

I'm suggesting the latter. It wouldn't necessarily require a monorepo
(though that does make it easier for a cluster operator to run tool). If we
went with something relatively streamlined, it would be easy for job owners
to also run the tool on their individual configs.

I'm taking inspiration from the JS community which is dealing with
relatively high rate of change by providing automated tooling to make
trivial code changes in bulk. I think we could benefit from similar
tooling, but if others don't agree we can continue our less formal process
of grep'ing and sed'ing ;).

On Thu, Feb 4, 2016 at 1:55 AM, Erb, Stephan <stephan@blue-yonder.com>
wrote:

> Are you suggesting a tool that operates against a running Aurora cluster
> and performs a serverside inspection? Or are you implying a tool that works
> on .aurora files?
>
> I'd find the first one way more useful, as the latter one would suggest
> that you had to have a monorepo with access to the all aurora
> configurations.
>
> Cheers,
> Stephan
> 
> From: John Sirois <j...@conductant.com>
> Sent: Thursday, February 4, 2016 2:42 AM
> To: dev@aurora.apache.org
> Subject: Re: Further thoughts on config deprecations
>
> On Wed, Feb 3, 2016 at 6:34 PM, Joshua Cohen <jco...@apache.org> wrote:
>
> > How would folks feel about requiring any changes that deprecate job
> config
> > to include some sort of codemod[1]-like patch that would allow cluster
> > operators to automatically fix deprecated fields across their company's
> > Aurora configs?
> >
> > We could either leverage codemod directly, or create nicer tooling around
> > it that supports applying arbitrary patches (along the lines of
> > jscodeshift[2]).
> >
>
> I'd be really happy with a less ambitious step - a tool folks could run
> that gave them warnings about things that were broken between their current
> release and the targeted release - leaving fixes up to them, though
> suggesting those fixes in console text - possibly with links out to longer
> articles on the web for tougher fixes.
>
>
> >
> > Cheers,
> >
> > Joshua
> >
> > [1] https://github.com/facebook/codemod
> > [2] https://github.com/facebook/jscodeshift
> >
>
>
>
> --
> John Sirois
> 303-512-3301
>

Further thoughts on config deprecations

[Joshua Cohen]

How would folks feel about requiring any changes that deprecate job config
to include some sort of codemod[1]-like patch that would allow cluster
operators to automatically fix deprecated fields across their company's
Aurora configs?

We could either leverage codemod directly, or create nicer tooling around
it that supports applying arbitrary patches (along the lines of
jscodeshift[2]).

Cheers,

Joshua

[1] https://github.com/facebook/codemod
[2] https://github.com/facebook/jscodeshift

Re: New committer and PMC member: Stephan Erb

[Joshua Cohen]

Welcome Stephan!

On Wed, Feb 3, 2016 at 1:10 PM, Zhitao Li  wrote:

> Congrats Stephan, well deserved!
>
> On Wed, Feb 3, 2016 at 11:03 AM, Erb, Stephan  >
> wrote:
>
> > Awesome, thanks! Great to be on board! :-)
> >
> > 
> > From: Bill Farner 
> > Sent: Wednesday, February 3, 2016 7:01 PM
> > To: dev@aurora.apache.org; Erb, Stephan
> > Subject: New committer and PMC member: Stephan Erb
> >
> > Folks,
> >
> > Please join me in welcoming Stephan Erb, who is now an Aurora committer
> > and PMC member!  I'm sure anyone paying attention has noticed Stephan's
> > involvement in the community and commitment to improving Aurora!
> >
> > Welcome aboard, Stephan!
> >
>
>
>
> --
> Cheers,
>
> Zhitao Li
>

Re: NEWS Layout

[Joshua Cohen]

+1

On Tue, Feb 2, 2016 at 1:09 PM, Bill Farner  wrote:

> +1
>
> On Tuesday, February 2, 2016, Erb, Stephan 
> wrote:
>
> > Hi everyone,
> >
> > I'd like to propose that we give our NEWS file a little bit more
> > structure. Currently, it is quite cluttered [1].
> >
> > To keep it simple, I'd suggest that we adopt the style from the 0.11
> > Aurora blog post:
> >
> > * New/updated
> > * Deprecations and removals
> >
> > [1] https://github.com/apache/aurora/blob/master/NEWS
> > [2] https://aurora.apache.org/blog/aurora-0-11-0-released/
> >
> >
> > Thoughts?
> >
>

Re: Rollback Testing

[Joshua Cohen]

https://issues.apache.org/jira/browse/AURORA-1608
https://issues.apache.org/jira/browse/AURORA-1609

On Tue, Feb 2, 2016 at 7:42 PM, Joshua Cohen <jco...@apache.org> wrote:

> Ok, I'll file a ticket to at least track the intent. We'll see if I have
> time to work on it or if it just languishes (anyone else interested, feel
> free to pick it up as well of course ;)).
>
> On Mon, Feb 1, 2016 at 4:28 PM, Bill Farner <wfar...@apache.org> wrote:
>
>> Yes.  Our nightly packaging builds take advantage of this.
>>
>> On Mon, Feb 1, 2016 at 2:27 PM, Joshua Cohen <jco...@apache.org> wrote:
>>
>> > I know the blocker for e2e tests in CI previously was the ability to run
>> > vagrant. Is running docker from CI doable today?
>> >
>> > On Mon, Feb 1, 2016 at 2:40 PM, Bill Farner <wfar...@apache.org> wrote:
>> >
>> > > Definitely a nice thing to have, the big uncertainty will be whether
>> > anyone
>> > > cares enough to see the effort through.
>> > >
>> > > e2e tests in jenkins can be done, but likely only if e2e tests start
>> > using
>> > > docker instead of vagrant.  I am supportive of both, but cannot
>> > personally
>> > > invest the time at the moment.
>> > >
>> > > On Mon, Feb 1, 2016 at 12:14 PM, Joshua Cohen <jco...@apache.org>
>> wrote:
>> > >
>> > > > Another topic that came up today's IRC meeting was possibly adding
>> some
>> > > > sort of automated rollback testing between builds. This is related
>> to
>> > > > https://issues.apache.org/jira/browse/AURORA-1603 which was caused
>> by
>> > an
>> > > > inability to rollback to an earlier commit after discovering
>> > > > https://issues.apache.org/jira/browse/AURORA-1605.
>> > > >
>> > > > I'm not sure what's feasible in the immediate future. To me the most
>> > > > obvious solution would be to run the e2e tests as part of our CI
>> job,
>> > and
>> > > > have the last step of the job be to checkout HEAD^ and
>> rebuild/restart
>> > > the
>> > > > scheduler. This is predicated on actually running the e2e tests as
>> part
>> > > of
>> > > > CI though, which historically has been difficult for us to achieve
>> > (see:
>> > > > https://issues.apache.org/jira/browse/AURORA-127).
>> > > >
>> > > > Curious to hear thoughts on this. It's clearly more beneficial to
>> those
>> > > > deploying from master rather than from official releases (though it
>> > would
>> > > > be great, at least, as part of the release verification process to
>> > ensure
>> > > > we can rollback to the previous release). Do folks think this would
>> be
>> > > > beneficial, or is it overkill?
>> > > >
>> > > > Cheers,
>> > > >
>> > > > Joshua
>> > > >
>> > >
>> >
>>
>
>

Re: Rollback Testing

[Joshua Cohen]

Ok, I'll file a ticket to at least track the intent. We'll see if I have
time to work on it or if it just languishes (anyone else interested, feel
free to pick it up as well of course ;)).

On Mon, Feb 1, 2016 at 4:28 PM, Bill Farner <wfar...@apache.org> wrote:

> Yes.  Our nightly packaging builds take advantage of this.
>
> On Mon, Feb 1, 2016 at 2:27 PM, Joshua Cohen <jco...@apache.org> wrote:
>
> > I know the blocker for e2e tests in CI previously was the ability to run
> > vagrant. Is running docker from CI doable today?
> >
> > On Mon, Feb 1, 2016 at 2:40 PM, Bill Farner <wfar...@apache.org> wrote:
> >
> > > Definitely a nice thing to have, the big uncertainty will be whether
> > anyone
> > > cares enough to see the effort through.
> > >
> > > e2e tests in jenkins can be done, but likely only if e2e tests start
> > using
> > > docker instead of vagrant.  I am supportive of both, but cannot
> > personally
> > > invest the time at the moment.
> > >
> > > On Mon, Feb 1, 2016 at 12:14 PM, Joshua Cohen <jco...@apache.org>
> wrote:
> > >
> > > > Another topic that came up today's IRC meeting was possibly adding
> some
> > > > sort of automated rollback testing between builds. This is related to
> > > > https://issues.apache.org/jira/browse/AURORA-1603 which was caused
> by
> > an
> > > > inability to rollback to an earlier commit after discovering
> > > > https://issues.apache.org/jira/browse/AURORA-1605.
> > > >
> > > > I'm not sure what's feasible in the immediate future. To me the most
> > > > obvious solution would be to run the e2e tests as part of our CI job,
> > and
> > > > have the last step of the job be to checkout HEAD^ and
> rebuild/restart
> > > the
> > > > scheduler. This is predicated on actually running the e2e tests as
> part
> > > of
> > > > CI though, which historically has been difficult for us to achieve
> > (see:
> > > > https://issues.apache.org/jira/browse/AURORA-127).
> > > >
> > > > Curious to hear thoughts on this. It's clearly more beneficial to
> those
> > > > deploying from master rather than from official releases (though it
> > would
> > > > be great, at least, as part of the release verification process to
> > ensure
> > > > we can rollback to the previous release). Do folks think this would
> be
> > > > beneficial, or is it overkill?
> > > >
> > > > Cheers,
> > > >
> > > > Joshua
> > > >
> > >
> >
>

Re: Deprecation Cycles

[Joshua Cohen]

I'd propose the following: fields will be removed no less than 90 days and
1 release after they have been deprecated. This might be a little bit
conservative w.r.t to days, so feel free to push back if you think, e.g.,
60 days is more reasonable.

On Mon, Feb 1, 2016 at 2:35 PM, Bill Farner <wfar...@apache.org> wrote:

> Sounds reasonable.  Can you firm up the proposal and apply numbers?
>
> On Mon, Feb 1, 2016 at 12:09 PM, Joshua Cohen <jco...@apache.org> wrote:
>
> > This came up briefly last week, and we also discussed in the IRC meeting
> > today. Given the increased release cadence, I think we would benefit from
> > moving from release-based deprecation to time-based. That is to say,
> > instead of deprecating in release X and removing release X+1, we would
> > deprecate in release X and remove no less than N later and at least M
> > releases later.
> >
> > General consensus in the IRC meeting was in favor, however I wanted to
> > start a thread to discuss to gather additional feedback and if everyone
> is
> > in agreement, then we can solidify the actual details on what the new
> > deprecation policy should be.
> >
> > Cheers,
> >
> > Joshua
> >
>

Re: [PROPOSAL] Amend 0.12.0 release goals

[Joshua Cohen]

Sounds good to me.

On Thu, Jan 14, 2016 at 9:31 AM, Erb, Stephan 
wrote:

> +1 for catching up
> 
> From: John Sirois 
> Sent: Thursday, January 14, 2016 4:18 PM
> To: dev@aurora.apache.org
> Subject: Re: [PROPOSAL] Amend 0.12.0 release goals
>
> On Thu, Jan 14, 2016 at 8:02 AM, Bill Farner  wrote:
>
> > Given that we are still playing catch-up to mesos releases (we are on
> > 0.25.0, latest is 0.26.0, there's talk of cutting 0.27.0 soon), i would
> > like to suggest that we remove these tickets from 0.12.0:
> >
> > https://issues.apache.org/jira/browse/AURORA-987
> > https://issues.apache.org/jira/browse/AURORA-1150
> >
> > It's slightly unfortunate as these tickets represent the largest planned
> > effort for 0.12.0, we had a handful of significant contributions that
> still
> > make this a featureful release:
> >
> >
> >
> https://github.com/apache/aurora/blob/d542bd1d58bc5dcf6ead95d902c0a8cecbbffe9e/NEWS#L3-L23
> >
> > Additionally, i would like to recommend that we add the following ticket
> to
> > 0.12.0 since we have an in-fight patch that looks close to completion:
> >
> > https://issues.apache.org/jira/browse/AURORA-1109
>
>
> +1 to the proposal, catching up would be good to do.
>
> I clarified the status of AURORA-987 by stopping progress and
> un-assigning.  Although I'm working towards that ticket, its by very
> indirect means still and I'll re-assign and re-start once I'm actually
> engaged in the meat of the API proposal that is needed assuming no one else
> has dived in.
>
>
> >
> >
> >
> > Cheers,
> >
> > Bill
> >
>
>
>
> --
> John Sirois
> 303-512-3301
>

Re: [PROPOSAL] Job instance scaling APIs

[Joshua Cohen]

What happens if a job has been scaled out, but the underlying config is not
updated to take that scaling into account? Would the next update on that
job revert the number of instances (presumably, because what else could we
do)? Is there anything we can do, tooling-wise, to improve upon this?

On Thu, Jan 14, 2016 at 1:40 PM, Maxim Khutornenko  wrote:

> Our rolling update APIs can be quite inconvenient to work with when it
> comes to instance scaling [1]. It's especially frustrating when
> adding/removing instances has to be done in an automated fashion (e.g.: by
> an external autoscaling process) as it requires holding on to the original
> aurora config at all times.
>
> I propose we add simple instance scaling APIs to address the above. Since
> Aurora job may have instances at different configs at any moment, I propose
> we accept an InstanceKey as a reference point when scaling out. For
> example:
>
> /** Scales out a given job by adding more instances with the task
> config of the templateKey. */
> Response scaleOut(1: InstanceKey templateKey, 2: i32 incrementCount)
>
> /** Scales in a given job by removing existing instances. */
> Response scaleIn(1: JobKey job, 2: i32 decrementCount)
>
> A correspondent client command could then look like:
>
> aurora job scale-out devcluster/vagrant/test/hello/1 10
>
> For the above command, a scheduler would take task config of instance 1 of
> the 'hello' job and replicate it 10 more times thus adding 10 additional
> instances to the job.
>
> There are, of course, some details to work out like making sure no active
> update is in flight, scale out does not violate quota and etc. I intend to
> address those during the implementation as things progress.
>
> Does the above make sense? Any concerns/suggestions?
>
> Thanks,
> Maxim
>
> [1] - https://issues.apache.org/jira/browse/AURORA-1258
>

Re: [PROPOSAL] Replace commons-args

[Joshua Cohen]

I'm in favor of this proposal and will be very happy to see commons-args go
away :).

On Tue, Jan 12, 2016 at 6:53 AM, John Sirois  wrote:

> On Mon, Jan 11, 2016 at 9:49 PM, Bill Farner  wrote:
>
> > Any other thoughts on this?  I will proceed with initial encapsulation
> work
> > in the meantime, but am interested in any other pro/con opinions.
> >
>
> I've stayed quiet on this since Bill and I spoke about the idea offline in
> its pre-seed phase.  I think the advantages have already been well-covered
> so I'll point out the disadvantages I've seen (I implemented a system like
> this in the past):
>
> In the absence of the args system injecting args where they're needed (the
> current Args does this and so is a 2nd form of data injection in addition
> to guice), developers find it hard or inconvenient to plumb options from
> the place where args are parsed (near main) to where they belong.  If DI is
> not widely deployed - this can be a real problem.
>
> That said, In Aurora we don't have this problem.  Guice is used extensively
> for DI in aurora and it can get the options plumbed where we need them in
> the standard way.
>
> I'm very much in favor of this proposal.
>
>
> > On Fri, Jan 8, 2016 at 10:25 AM, Maxim Khutornenko 
> > wrote:
> >
> > > "Can you turn that concern into a more firm stance?" - you mastered
> > > the original library and I fully trust you with the new one :)
> > >
> > > +1.
> > >
> > > On Thu, Jan 7, 2016 at 6:58 PM, Bill Farner 
> wrote:
> > > > Can you turn that concern into a more firm stance?
> > > >
> > > > FWIW if you look closely, there's really only about 100 lines of
> > > greenfield
> > > > code in /r/42042 (CommandLineParameters.java, sure to grow of
> course).
> > > > There's a lot of bulk due to splitting out parser classes;
> commons-args
> > > has
> > > > the same.
> > > >
> > > > I should have been clear - i'm not married to this.  Mostly
> scratching
> > an
> > > > old itch, i won't be upset if it's more churn than it's worth at this
> > > time.
> > > >  (I _am_ tired of the gradle and intellij issues though.)
> > > >
> > > > On Thu, Jan 7, 2016 at 6:31 PM, Maxim Khutornenko 
> > > wrote:
> > > >
> > > >> I am a big +1 to the refactoring if only for testability reasons.
> > > >>
> > > >> My only concern is the amount of supporting code that your POC
> > > >> resulted in https://reviews.apache.org/r/42042. It may feel like
> > > >> trading code with Commons parser at the end, which is battle proven
> > > >> and well tested. That said, arg parsing is generally a well scoped
> > > >> problem, so it should be relatively easy to cover major testing
> > > >> scenarios.
> > > >>
> > > >> On Thu, Jan 7, 2016 at 5:18 PM, Bill Farner 
> > wrote:
> > > >> > All,
> > > >> >
> > > >> > I propose that we replace our current command line args system.
> For
> > > >> those
> > > >> > unaware, the args system [1] we currently use was forked out of
> > > Twitter
> > > >> > Commons several months ago.  The goal when forking was to
> > > adapt/maintain
> > > >> or
> > > >> > eventually replace these libraries.  Given the code volume and
> > > complexity
> > > >> > of commons-args, i propose we replace it and address some issues
> > along
> > > >> the
> > > >> > way.
> > > >> >
> > > >> > I suggest several goals:
> > > >> > a. sidestep current development hurdles we have encountered with
> the
> > > args
> > > >> > system (intellij/gradle not working nicely with apt)
> > > >> > b. encourage better testability of Module classes by always
> > injecting
> > > all
> > > >> > args
> > > >> > c. leverage a well-maintained third-party argument parsing library
> > > >> > d. stretch: enable user-friendly features like logical option
> groups
> > > for
> > > >> > better help/usage output
> > > >> > e. stretch: enable alternative configuration inputs like a
> > > configuration
> > > >> > file or environment variables
> > > >> >
> > > >> > (b) is currently an issue because command line arguments are
> driven
> > > from
> > > >> > pseudo-constants within the code, for example:
> > > >> >
> > > >> > @NotNull
> > > >> > @CmdLine(name = "cluster_name", help = "Name to identify the
> > > cluster
> > > >> > being served.")
> > > >> > private static final Arg CLUSTER_NAME = Arg.create();
> > > >> >
> > > >> > @NotNull
> > > >> > @NotEmpty
> > > >> > @CmdLine(name = "serverset_path", help = "ZooKeeper ServerSet
> > > path to
> > > >> > register at.")
> > > >> > private static final Arg SERVERSET_PATH =
> Arg.create();
> > > >> >
> > > >> > This makes it simple to add command line arguments.  However, it
> > means
> > > >> that
> > > >> > a level of indirection is needed to parameterize and test the code
> > > >> > consuming arg values.  We have various examples of this throughout
> > the
> > > >> > project.
> > > >> >
> > > >> > I propose that we change the way 

Re: Ticket cleanup

[Joshua Cohen]

I'm not sure I agree with this sentiment. I don't have any problems with an
aspirational backlog of tickets. At the very least it's a place to refer
people who request commonly asked for features. Perhaps the solution isn't
to close tickets that we don't imagine will see attention in the immediate
future, but rather to label them so they're easy to filter out from the
ones what will?

Also, in the future, perhaps have this discussion before closing tickets
en-masse? Especially over a holiday weekend when people aren't around to
discuss? It's not likely that first thing back from a long weekend many
folks are going to take time to scan read through a list of a hundred jira
emails to see if any of their pet issues were resolved ;).

On Mon, Dec 28, 2015 at 10:01 AM, Erb, Stephan 
wrote:

> +1. Having a well-groomed bug tracker is very helpful for everyone
> involved.
>
> In particular, it would be great if we could get the bug count to 0 over
> the course of the next months. Either bugs are important and we get them
> fixed, or we have to guts to close them as won't fix and update the
> documentation accordingly.
>
> Best Regards,
> Stephan
>
>
> 
> From: Bill Farner 
> Sent: Monday, December 28, 2015 4:44 PM
> To: dev@aurora.apache.org
> Subject: Ticket cleanup
>
> I'd like to share some rationale on my JIRA activity last night.  I'm happy
> to undo any of the changes if folks disagree.
>
> We had approximately 450 open tickets prior to last night.  Personally, I
> found it daunting to find things we should actually work on.  To remedy
> this, I started by skimming through tickets that have not been touched in
> the last 6 months.  This quickly identified a swath of tickets that may be
> valid, but I did not imagine them being valuable enough to address in the
> foreseeable future.
>
> If there is interest in doing more of this, I welcome help from others to
> continue reducing the queue to something more manageable.  My culling
> reduced the queue but it is still very large.
>
> If you do participate in this, please try to avoid using "Resolved, Fixed"
> so as to not add noise to the changelog in the next release.
>

Re: Commits without reviews

[Joshua Cohen]

It could be, I just think it's easier to comment on a reviewboard than it
is a commits@ email.

On Thu, Dec 24, 2015 at 10:29 AM, Bill Farner <wfar...@apache.org> wrote:

> Can that be handled by subscribing to commits@?
>
> On Thursday, December 24, 2015, Joshua Cohen <jco...@apache.org> wrote:
>
> > I'm generally ok with this. Just curious: what do you think about maybe
> > posting a review and then committing it right away in these cases
> though? A
> > bit noisy on the reviews@ list, but at least it'd give people a chance
> to
> > peruse/comment as they see fit (with the assumption that any comments
> would
> > be followed up in a subsequent commit if needed).
> >
> > On Wed, Dec 23, 2015 at 3:48 PM, Bill Farner <wfar...@apache.org
> > <javascript:;>> wrote:
> >
> > > All,
> > >
> > > Over the past few days, i have made several commits to the repository
> > > without code review.  Our convention has historically been to perform a
> > > code review for any change, however small.  Please see below for some
> > > rationale, but i would like to propose that we allow committers to
> > exercise
> > > judgement on skipping code reviews for changes unrelated to build or
> test
> > > of the main project (e.g. scheduler, executor, client, packaging).
> What
> > do
> > > you all think?
> > >
> > > As an example, i think the code review process is too much overhead for
> > > commits like the ones below.  With these commits i was playing
> > whack-a-mole
> > > to get alignment between markdown rendering on
> github.com/apache/aurora
> > > and
> > > aurora.apache.org.  Skipping code review allowed me to fix things in a
> > > much
> > > shorter timeframe.
> > >
> > > commit 0d9fe18
> > > Author: Bill Farner <wfar...@apache.org <javascript:;>>
> > > Date:   Wed Dec 23 08:31:27 2015 -0800
> > >
> > > Fix string interpolation for release email.
> > >
> > > commit df5200b
> > > Author: Bill Farner <wfar...@apache.org <javascript:;>>
> > > Date:   Mon Dec 21 14:19:48 2015 -0800
> > >
> > > Fix formatting and work around anchor link issues in installing.md
> > >
> > > commit 21c605e
> > > Author: Bill Farner <wfar...@apache.org <javascript:;>>
> > > Date:   Mon Dec 21 14:11:10 2015 -0800
> > >
> > > Fix anchor links in installing.md.
> > >
> > > commit 9326fa6
> > > Author: Bill Farner <wfar...@apache.org <javascript:;>>
> > > Date:   Mon Dec 21 12:21:37 2015 -0800
> > >
> > > Link to install guide from docs/README.md
> > >
> > > commit f8e59a4
> > > Author: Bill Farner <wfar...@apache.org <javascript:;>>
> > > Date:   Mon Dec 21 12:12:56 2015 -0800
> > >
> > > Fix formatting issues in installing doc.
> > >
> >
>

Re: Commits without reviews

[Joshua Cohen]

I'm generally ok with this. Just curious: what do you think about maybe
posting a review and then committing it right away in these cases though? A
bit noisy on the reviews@ list, but at least it'd give people a chance to
peruse/comment as they see fit (with the assumption that any comments would
be followed up in a subsequent commit if needed).

On Wed, Dec 23, 2015 at 3:48 PM, Bill Farner  wrote:

> All,
>
> Over the past few days, i have made several commits to the repository
> without code review.  Our convention has historically been to perform a
> code review for any change, however small.  Please see below for some
> rationale, but i would like to propose that we allow committers to exercise
> judgement on skipping code reviews for changes unrelated to build or test
> of the main project (e.g. scheduler, executor, client, packaging).  What do
> you all think?
>
> As an example, i think the code review process is too much overhead for
> commits like the ones below.  With these commits i was playing whack-a-mole
> to get alignment between markdown rendering on github.com/apache/aurora
> and
> aurora.apache.org.  Skipping code review allowed me to fix things in a
> much
> shorter timeframe.
>
> commit 0d9fe18
> Author: Bill Farner 
> Date:   Wed Dec 23 08:31:27 2015 -0800
>
> Fix string interpolation for release email.
>
> commit df5200b
> Author: Bill Farner 
> Date:   Mon Dec 21 14:19:48 2015 -0800
>
> Fix formatting and work around anchor link issues in installing.md
>
> commit 21c605e
> Author: Bill Farner 
> Date:   Mon Dec 21 14:11:10 2015 -0800
>
> Fix anchor links in installing.md.
>
> commit 9326fa6
> Author: Bill Farner 
> Date:   Mon Dec 21 12:21:37 2015 -0800
>
> Link to install guide from docs/README.md
>
> commit f8e59a4
> Author: Bill Farner 
> Date:   Mon Dec 21 12:12:56 2015 -0800
>
> Fix formatting issues in installing doc.
>

Re: [VOTE] Release Apache Aurora 0.11.0 RC1

[Joshua Cohen]

+1 non-binding

On Sun, Dec 20, 2015 at 10:21 AM, Bill Farner  wrote:

> Looks like it was orphaned - not linked against the RC ticket and not
> listed as a blocker of https://issues.apache.org/jira/browse/AURORA-1367.
> I'll move the ticket underneath it to the 0.12.0.
>
> On Sun, Dec 20, 2015 at 7:23 AM, Erb, Stephan  >
> wrote:
>
> > What's up with this ticket here:
> > https://issues.apache.org/jira/browse/AURORA-1520
> >
> > Was this forgotten? Should we do it now?
> >
> > Regards,
> > Stephan
> > 
> > From: John Sirois 
> > Sent: Friday, December 18, 2015 3:37 AM
> > To: dev@aurora.apache.org
> > Subject: Re: [VOTE] Release Apache Aurora 0.11.0 RC1
> >
> > On Thu, Dec 17, 2015 at 5:17 PM, Bill Farner  wrote:
> >
> > > Friendly reminder that verifying a release can be as easy as
> > >
> > >   ./build-support/release/verify-release-candidate  0.11.0-rc1
> > >
> > > Of course, if you have a simulated production environment, we would
> love
> > to
> > > hear how this build behaves there!
> > >
> > > On Thu, Dec 17, 2015 at 4:08 PM, Bill Farner 
> wrote:
> > >
> > > > All,
> > > >
> > > > I propose that we accept the following release candidate as the
> > official
> > > > Apache Aurora 0.11.0 release.
> > >
> >
> > +1 non-binding
> >
> > Verified on Arch Linux - kernel 4.2.5 + OpenJDK 1.8.0_66 + Python 2.7.11
> >
> > >
> > > > Aurora 0.11.0-rc1 includes the following:
> > > > ---
> > > > The NEWS for this release is available at:
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=NEWS=0.11.0-rc1
> > > >
> > > > The CHANGELOG for the release is available at:
> > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=CHANGELOG=0.11.0-rc1
> > > >
> > > > The branch used to create the release candidate is:
> > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=shortlog;h=refs/heads/0.11.0-rc1
> > > >
> > > > The release candidate is available at:
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.11.0-rc1/apache-aurora-0.11.0-rc1.tar.gz
> > > >
> > > > The MD5 checksum of the release candidate can be found at:
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.11.0-rc1/apache-aurora-0.11.0-rc1.tar.gz.md5
> > > >
> > > > The signature of the release candidate can be found at:
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.11.0-rc1/apache-aurora-0.11.0-rc1.tar.gz.asc
> > > >
> > > > The GPG key used to sign the release are available at:
> > > > https://dist.apache.org/repos/dist/dev/aurora/KEYS
> > > >
> > > > Please download, verify, and test.
> > > >
> > > > The vote will close on Tue Dec 22 15:18:55 PST 2015
> > > >
> > > > [ ] +1 Release this as Apache Aurora 0.11.0
> > > > [ ] +0
> > > > [ ] -1 Do not release this as Apache Aurora 0.11.0 because...
> > > >
> > >
> >
> >
> >
> > --
> > John Sirois
> > 303-512-3301
> >
>

Re: [RFC] REST / thrift & AURORA-987

[Joshua Cohen]

John: I share David's concerns, but it's not clear to me that they're
incompatible with your proposal. I.e. we could design the new REST API
according to any interface we like and still implement that API on top of
the thrift API (though the desired interface should always be our guide, we
should not be constrained by what's currently possible to build on top of
the thrift API). That said, I think it would be beneficial to start with
the API design and work from there back to the implementation.

On Tue, Dec 1, 2015 at 12:31 PM, Igor Morozov  wrote:

> We'll be sharing all the questions in single concise manner after we
> get the result of out POC(we're currently integrating aurora scheduler
> with internal deployment system so more questions are coming ;) )
>
> Thanks for asking!
>
> -Igor
>
> On Tue, Dec 1, 2015 at 10:21 AM, John Sirois  wrote:
> > On Tue, Dec 1, 2015 at 11:17 AM, Igor Morozov  wrote:
> >
> >> We had very similar concerns here at Uber in regards of Thrift and
> >> newer REST API that is coming to Aurora scheduler. It feels like
> >> having a general API model in a scheduler and providing whatever
> >> interface is necessary/convenient for integration would generally be a
> >> better option then building REST API layer on top of Thrift API.
> >>
> >
> > To be clear, my intent was to build on top in phase 1, then back out the
> > thrift API and gut it as phase 2, then evolve in phase 3.
> >
> > That said, its clear from both your comment and David's that there is a
> > desire to go straight to a new API side-by side with the existing API
> 1st,
> > then transition clients, then gut thrift.
> >
> > Igor, if you also have any specifics on problematic bits of the current
> API
> > - I'd love to have those.
> >
> >
> >>
> >> On Tue, Dec 1, 2015 at 9:37 AM, David McLaughlin <
> dmclaugh...@apache.org>
> >> wrote:
> >> > Shouldn't we start with the design of the API itself? Won't that
> >> influence
> >> > many of the answers to these questions?
> >> >
> >> > E.g. if you're just looking to port the Thrift API 1:1 to a JSON +
> HTTP
> >> > interface then that's a very different set of requirements to starting
> >> > fresh and doing a better job with our API.
> >> >
> >> > Personally I don't think the existing Thrift API is a very good base
> to
> >> > build an API on top off. A lot of the endpoints are fit for one
> purpose
> >> > (e.g. a specific UI view or client function) rather than being
> flexible.
> >> I
> >> > can't tell you how many times we wanted to go in and improve the UI in
> >> some
> >> > way only to find the existing API does not give us access to the data
> we
> >> > want.
> >> >
> >> > So yeah, I feel like the API should be more generic with regards to
> data
> >> > access. So fewer, more-powerful endpoints that support complex
> queries.
> >> >
> >> > On Mon, Nov 30, 2015 at 12:16 PM, John Sirois 
> >> wrote:
> >> >
> >> >> I’ve experimenting on
> https://issues.apache.org/jira/browse/AURORA-987
> >> for
> >> >> the past few weeks and I’d like to ask for feedback on the direction
> I’d
> >> >> like to head. If you’re interested in the evolution of the Aurora
> REST
> >> api,
> >> >> read on.
> >> >> --
> >> >>
> >> >> AURORA-987 aims to create a first-class REST-like scheduler
> interface.
> >> I’ve
> >> >> re-familiarized myself with the codebase and come to the conclusion
> that
> >> >> transitioning to a 1st class REST api requires maintaining the core
> >> thrift
> >> >> API as the 1st class API until the point the REST API is fully
> >> established
> >> >> and clients can all be transitioned.
> >> >>
> >> >> I think this conclusion is probably uncontroversial, but the key
> factors
> >> >> pushing this way are:
> >> >>
> >> >>1.
> >> >>
> >> >>The thrift API has both wide and deep dependencies inside the
> Aurora
> >> >>codebase - 276 imports across 97 files:
> >> >>
> >> >>$ git grep "import org.apache.aurora.gen" -- src/main/java/ | grep
> >> >> -v "import org.apache.aurora.gen.storage" | wc -l
> >> >>276
> >> >>$ git grep "import org.apache.aurora.gen" -- src/main/java/ | grep
> >> >> -v "import org.apache.aurora.gen.storage" | cut -d: -f1 | sort -u |
> wc
> >> >> -l
> >> >>97
> >> >>
> >> >>2.
> >> >>
> >> >>The thrift API is stored long-term in the log in serialized form.
> >> >>
> >> >> Both 1 & 2 dictate that the thrift API, at least its enums, structs
> and
> >> >> unions, must be maintained for the forseeable future.
> >> >> We also have the RPC API (thrift services) - which is currently a
> ~thin,
> >> >> but not insignificant, container of API processing logic. For
> example,
> >> see
> >> >> here
> >> >> <
> >> >>
> >>
> https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java#L220-L267
> >> >> >
> >> >> .
> >> >>
> >> >> As such it seems to me the 

Re: [VOTE] Release Apache Aurora 0.10.0-1 debs

[Joshua Cohen]

Yeah, I agree, that would be preferable.

On Mon, Nov 30, 2015 at 12:27 PM, John Sirois <j...@conductant.com> wrote:

> On Mon, Nov 30, 2015 at 11:18 AM, Joshua Cohen <jco...@apache.org> wrote:
>
> > +1
> >
> > One minor nit, not sure if it's worth fixing:
> > test/deb/ubuntu-trusty/README.md still references 0.9.0, should be
> updated
> > to 0.10.0?
> >
>
> I'd be more in favor of a clarifying phrase like 'for example ...' than
> changing this text for every package release.
>
>
> > On Fri, Nov 27, 2015 at 12:54 PM, John Sirois <j...@conductant.com>
> wrote:
> >
> > > On Fri, Nov 27, 2015 at 11:34 AM, Bill Farner <wfar...@apache.org>
> > wrote:
> > >
> > > > I propose that we accept the following artifacts as the official deb
> > > > packaging for
> > > > Apache Aurora 0.10.0.
> > > >
> > > >
> > > >
> > >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.10.0-1/deb/ubuntu-trusty/
> > > >
> > > > The Aurora deb packaging includes the following:
> > > > ---
> > > > The CHANGELOG is available at:
> > > >
> > > >
> > >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=blob_plain;f=specs/debian/changelog;hb=refs/heads/0.10.x
> > > >
> > > > The branch used to create the packaging is:
> > > >
> > > >
> > >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;h=refs/heads/0.10.x
> > > >
> > > > The packages are available at:
> > > >
> > > >
> > >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.10.0-1/deb/ubuntu-trusty/
> > > >
> > > > The GPG keys used to sign the packages are available at:
> > > > https://dist.apache.org/repos/dist/release/aurora/KEYS
> > > >
> > > > Please download, verify, and test.
> > > >
> > > > The vote will close in 5 business days, on Thu Dec 4 8:00:00 PT 2015
> > > >
> > > > [ ] +1 Release these as the deb packages for Apache Aurora 0.10.0
> > > > [ ] +0
> > > > [ ] -1 Do not release these artifacts because...
> > > >
> > > > I would like to get the voting started off with my own +1
> > > >
> > >
> > > +1 nonbinding
> > >
> > > Tested via the test/deb/ubuntu-trusty vm.
> > >
> > >
> > >
> > > --
> > > John Sirois
> > > 303-512-3301
> > >
> >
>
>
>
> --
> John Sirois
> 303-512-3301
>

Re: [VOTE] Release Apache Aurora 0.10.0-1 debs

[Joshua Cohen]

+1

One minor nit, not sure if it's worth fixing:
test/deb/ubuntu-trusty/README.md still references 0.9.0, should be updated
to 0.10.0?

On Fri, Nov 27, 2015 at 12:54 PM, John Sirois  wrote:

> On Fri, Nov 27, 2015 at 11:34 AM, Bill Farner  wrote:
>
> > I propose that we accept the following artifacts as the official deb
> > packaging for
> > Apache Aurora 0.10.0.
> >
> >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.10.0-1/deb/ubuntu-trusty/
> >
> > The Aurora deb packaging includes the following:
> > ---
> > The CHANGELOG is available at:
> >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=blob_plain;f=specs/debian/changelog;hb=refs/heads/0.10.x
> >
> > The branch used to create the packaging is:
> >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;h=refs/heads/0.10.x
> >
> > The packages are available at:
> >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.10.0-1/deb/ubuntu-trusty/
> >
> > The GPG keys used to sign the packages are available at:
> > https://dist.apache.org/repos/dist/release/aurora/KEYS
> >
> > Please download, verify, and test.
> >
> > The vote will close in 5 business days, on Thu Dec 4 8:00:00 PT 2015
> >
> > [ ] +1 Release these as the deb packages for Apache Aurora 0.10.0
> > [ ] +0
> > [ ] -1 Do not release these artifacts because...
> >
> > I would like to get the voting started off with my own +1
> >
>
> +1 nonbinding
>
> Tested via the test/deb/ubuntu-trusty vm.
>
>
>
> --
> John Sirois
> 303-512-3301
>

IRC Meeting notes for October 12th

[Joshua Cohen]

No one with karma was around to start the meeting official, here are the
ad-hoc meeting notes:

Mon Oct 12 18:07:58 2015   jcohen: Hi folks, it seems that no one with
karma is around to officially start the meeting, so let’s have an ad-hoc
meeting and I’ll manually send out note.
Mon Oct 12 18:08:07 2015   wickman: wfm
Mon Oct 12 18:08:07 2015   zmanji: let me try to be sure
Mon Oct 12 18:08:10 2015   zmanji: ASFBot meeting start
Mon Oct 12 18:08:18 2015   zmanji: ASFBot: meeting start
Mon Oct 12 18:08:23 2015   jcohen: Let’s start with roll-call, as always,
everyone is welcome to participate.
Mon Oct 12 18:08:25 2015   wickman: ASFBot: meeting start
Mon Oct 12 18:08:35 2015   zmanji: here
Mon Oct 12 18:08:36 2015   wickman: ASFBot: j/k lol don't start the meeting
Mon Oct 12 18:08:41 2015   wickman: :-(
Mon Oct 12 18:08:41 2015   jcohen: here
Mon Oct 12 18:08:42 2015   wickman: here
Mon Oct 12 18:09:38 2015   jcohen: Looks like a quiet week. Anyone have any
topics to discuss?
Mon Oct 12 18:10:04 2015   thalin: here
Mon Oct 12 18:10:18 2015   wickman: one sec
Mon Oct 12 18:10:20 2015   wickman: AURORA-1504
Mon Oct 12 18:10:41 2015   wickman: there was one minor comment on the
review about the naming. i called the subcommand 'config read' but stephan
erb suggested 'config bind'
Mon Oct 12 18:10:44 2015   wickman: i don't care one way or the other
Mon Oct 12 18:11:00 2015   wickman: do people feel strongly about what the
command should be called? it takes an .aurora or .json file and spits out a
fully bound .json version of it
Mon Oct 12 18:11:13 2015   wickman: that can be used with other commands
that take --read-json
Mon Oct 12 18:11:50 2015   jcohen: bind seems ok to me. I don’t feel super
strongly.
Mon Oct 12 18:12:10 2015   serb: Joined the channel
Mon Oct 12 18:12:18 2015   wickman: bind seems reasonable to me as well,
since that's basically one of the main things happening.
Mon Oct 12 18:13:05 2015   jcohen: sounds good. Anything else on that topic?
Mon Oct 12 18:13:27 2015   stephanerb: Joined the channel
Mon Oct 12 18:13:50 2015   wickman: just a plug for
https://github.com/wickman/sacker which is a package manager thing i built
for use with aurora on aws
Mon Oct 12 18:14:21 2015   wickman: i've also built a couple aurora
extensions (using the client extension feature that's out for review)
Mon Oct 12 18:14:33 2015   wickman: one is a binding helper to resolve name
-> blob in s3
Mon Oct 12 18:14:45 2015   kts: Joined the channel
Mon Oct 12 18:14:46 2015   wickman: and one is a "deploy" subcommand that
can be used for staging bound configs in s3
Mon Oct 12 18:15:07 2015   wickman:
https://github.com/wickman/sacker/commit/d28da0fed6ada566d11ecae11762ac65432b6dfa
Mon Oct 12 18:15:14 2015   wickman: still needs documentation, but i'll be
playing around with this for a few more days
Mon Oct 12 18:15:22 2015   wickman: might be useful for folks using aurora
on aws
Mon Oct 12 18:15:36 2015   kts: wickman: awesome, any desire to upstream?
Mon Oct 12 18:15:39 2015   jcohen: Sounds exciting, thanks wickman!
Mon Oct 12 18:16:18 2015   wickman: the main issue with upstreaming is the
potential for dependency creep. sacker depends on boto, but it could depend
on zookeeper or * depending on new ledger and store implementations (right
now i've only implemented ledgers using dynamo and s3, and an s3 store)
Mon Oct 12 18:16:40 2015   wickman: i'd like to just work on the plugin
ecosystem in general, and make plugins generally easy to build
Mon Oct 12 18:16:51 2015   wickman: and keep the core as light as possible
Mon Oct 12 18:18:03 2015   jcohen: Ok, anyone else have a topic?
Mon Oct 12 18:19:20 2015   mkhutornenko: Joined the channel
Mon Oct 12 18:19:34 2015   jcohen: ok then. Let’s call this meeting
concluded.
Mon Oct 12 18:19:40 2015   jcohen: Thanks all, I’ll send notes shortly.

Re: Upgrading from 0.7

[Joshua Cohen]

Filed https://issues.apache.org/jira/browse/AURORA-1497 to add
documentation on how best to upgrade the various Aurora components.

On Mon, Sep 21, 2015 at 12:58 PM, Joseph Smith  wrote:

> We also shut down all of the schedulers, upgrade to the new version, then
> bring them all back up. This isn’t a hard requirement, but it has made
> things a bit simpler for us in practice to always know we’re dealing with
> the same version of the schedulers on all hosts at once.
>
> > On Sep 21, 2015, at 10:55 AM, Zameer Manji  wrote:
> >
> > I don't think the 0.7 -> 0.9 upgrade path has been tested. I think it is
> > advisable to upgrade from 0.7 to 0.8 then later upgrade from 0.8 to 0.9.
> I
> > think Jeff's upgrade steps are reasonable and shouldn't give you any
> > trouble. Note that upgrading the executors won't affect existing tasks.
> If
> > you want the current tasks on the system to adopt the latest executor you
> > will need to upgrade the executor and then roll your tasks so they can be
> > relaunched with the updated binary. This isn't necessary to do but keep
> > this in mind if you are expecting tasks to leverage the latest executor
> > functionality.
> >
> > On Sun, Sep 20, 2015 at 8:30 PM, Jeff Schroeder <
> jeffschroe...@computer.org>
> > wrote:
> >
> >> When I did this, I updated the executors, then the observers, and then
> the
> >> schedulers, one at a time, in a rolling fashion. Seemed to work fine,
> but
> >> the clusters were lightly utilized.
> >>
> >> On Sun, Sep 20, 2015 at 9:40 PM, Mauricio Garavaglia <
> >> mauriciogaravag...@gmail.com> wrote:
> >>
> >>> Hi guys,
> >>>
> >>> I'm about to upgrade a cluster from 0.7 to 0.9 and was wondering what
> are
> >>> the things to consider to make it as seamless as possible?
> >>>
> >>> For example, update the executors first and then the schedulers; just
> >>> update the schedulers one at a time, etc.
> >>> Thanks!
> >>>
> >>> Mauricio
> >>>
> >>
> >>
> >>
> >> --
> >> Jeff Schroeder
> >>
> >> Don't drink and derive, alcohol and analysis don't mix.
> >> http://www.digitalprognosis.com
> >>
> >> --
> >> Zameer Manji
> >>
> >> 
>
>

Re: [VOTE] Release Apache Aurora 0.9.0 debs

[Joshua Cohen]

TIL that all of 127/8 is bound to the loopback interface, so while
non-standard, apparently 127.0.1.1 should (and, I just confirmed that it
does) work just fine.

So... sorry for the spam, but changing my vote to +1.

On Thu, Sep 3, 2015 at 9:36 AM, Joshua Cohen <jco...@twopensource.com>
wrote:

> To modify the above, today I *was* able to get the scheduler working with
> mesos and a job scheduled, so my only objection is to the incorrect zk IP
> in clusters.json.
>
> On Thu, Sep 3, 2015 at 9:27 AM, Joshua Cohen <jco...@twopensource.com>
> wrote:
>
>> I'm -1 to these debs. I was not able to get aurora up and running in
>> vagrant using them, which is probably due to issues on my end, but
>> regardless, in /etc/aurora/clusters.json, the default value for "zk" is set
>> to "127.01.1" which at the very least should be corrected?
>>
>> On Wed, Sep 2, 2015 at 7:55 PM, Kevin Sweeney <kevi...@apache.org> wrote:
>>
>>> +1 to these debs
>>>
>>> On Sat, Aug 29, 2015 at 9:59 AM, Bill Farner <wfar...@apache.org> wrote:
>>>
>>> > I propose that we accept the following artifacts as the official deb
>>> > packaging for
>>> > Apache Aurora 0.9.0.
>>> >
>>> >
>>> >
>>> http://people.apache.org/~wfarner/aurora/distributions/0.9.0/deb/ubuntu-trusty/
>>> >
>>> > The Aurora deb packaging includes the following:
>>> > ---
>>> > The CHANGELOG is available at:
>>> >
>>> >
>>> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=blob_plain;f=specs/debian/changelog;hb=refs/heads/0.9.x
>>> >
>>> > The branch used to create the packaging is:
>>> >
>>> >
>>> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;h=refs/heads/0.9.x
>>> >
>>> > The packages are available at:
>>> >
>>> >
>>> http://people.apache.org/~wfarner/aurora/distributions/0.9.0/deb/ubuntu-trusty/
>>> >
>>> > The GPG keys used to sign the packages are available at:
>>> > https://dist.apache.org/repos/dist/release/aurora/KEYS
>>> >
>>> > Please download, verify, and test.
>>> >
>>> > The vote will close in 4 days, on Wed Sept 2 10:00:00 PT 2015
>>> >
>>> > [ ] +1 Release these as the deb packages for Apache Aurora 0.9.0
>>> > [ ] +0
>>> > [ ] -1 Do not release these artifacts because...
>>> >
>>> > I would like to get the voting started off with my own +1
>>> >
>>> >
>>> > -=Bill
>>> >
>>>
>>
>>
>

Re: [VOTE] Release Apache Aurora 0.9.0 debs

[Joshua Cohen]

I'm -1 to these debs. I was not able to get aurora up and running in
vagrant using them, which is probably due to issues on my end, but
regardless, in /etc/aurora/clusters.json, the default value for "zk" is set
to "127.01.1" which at the very least should be corrected?

On Wed, Sep 2, 2015 at 7:55 PM, Kevin Sweeney  wrote:

> +1 to these debs
>
> On Sat, Aug 29, 2015 at 9:59 AM, Bill Farner  wrote:
>
> > I propose that we accept the following artifacts as the official deb
> > packaging for
> > Apache Aurora 0.9.0.
> >
> >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.9.0/deb/ubuntu-trusty/
> >
> > The Aurora deb packaging includes the following:
> > ---
> > The CHANGELOG is available at:
> >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=blob_plain;f=specs/debian/changelog;hb=refs/heads/0.9.x
> >
> > The branch used to create the packaging is:
> >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;h=refs/heads/0.9.x
> >
> > The packages are available at:
> >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.9.0/deb/ubuntu-trusty/
> >
> > The GPG keys used to sign the packages are available at:
> > https://dist.apache.org/repos/dist/release/aurora/KEYS
> >
> > Please download, verify, and test.
> >
> > The vote will close in 4 days, on Wed Sept 2 10:00:00 PT 2015
> >
> > [ ] +1 Release these as the deb packages for Apache Aurora 0.9.0
> > [ ] +0
> > [ ] -1 Do not release these artifacts because...
> >
> > I would like to get the voting started off with my own +1
> >
> >
> > -=Bill
> >
>

Re: [VOTE] Release Apache Aurora 0.9.0 rpms

[Joshua Cohen]

Thanks for the vagrant VM and the verification instructions. Made life much
easier!

I was able to install the rpm and successfully create a job, stat and kill
a job. +1 from me!

On Tue, Sep 1, 2015 at 9:41 PM, Bill Farner  wrote:

> Thanks for chiming in!  This vote is specifically for RPMs targeting
> centos7.  We have not yet had a contribution for centos6 RPMs, but would
> welcome them!  The contribution would amount to implementing a builder
> similar to what you see here:
> https://github.com/apache/aurora-packaging/tree/master/builder/rpm/centos-7
>
> On Tue, Sep 1, 2015 at 7:37 PM, thinker0  wrote:
>
> > Centos-6 ??
> >
> > On Tue, Sep 1, 2015 at 2:53 AM Bill Farner  wrote:
> >
> > > You can see an example test environment and instructions for installing
> > > these RPMs in a branch i'm working on here:
> > >
> > >
> >
> https://github.com/wfarner/aurora-packaging/tree/wfarner/deb_testing/test/rpm/centos-7
> > >
> > > On Mon, Aug 31, 2015 at 10:51 AM, Bill Farner 
> > wrote:
> > >
> > > > I propose that we accept the following artifacts as the official rpm
> > > > packaging for
> > > > Apache Aurora 0.9.0.
> > > >
> > > >
> > >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.9.0/rpm/centos-7/
> > > > <
> > >
> >
> http://people.apache.org/%7Ewfarner/aurora/distributions/0.9.0/deb/ubuntu-trusty/
> > > >
> > > >
> > > > The Aurora rpm packaging includes the following:
> > > > ---
> > > > The changelog is available at:
> > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora-packaging.git;a=blob;f=specs/rpm/aurora.spec;h=a559b42d73a4211891d8c7569ea7ab2a3a6993ae;hb=refs/heads/0.9.x#l328
> > > > <
> > >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=blob_plain;f=specs/debian/changelog;hb=refs/heads/0.9.x
> > > >
> > > >
> > > > The branch used to create the packaging is:
> > > >
> > > >
> > >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;h=refs/heads/0.9.x
> > > >
> > > > The packages are available at:
> > > >
> > >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.9.0/rpm/centos-7/
> > > > <
> > >
> >
> http://people.apache.org/%7Ewfarner/aurora/distributions/0.9.0/deb/ubuntu-trusty/
> > > >
> > > >
> > > > The GPG keys used to sign the packages are available at:
> > > > https://dist.apache.org/repos/dist/release/aurora/KEYS
> > > >
> > > > Please download, verify, and test.
> > > >
> > > > The vote will close in 3 business days, on Thu Sep 3 11:00:00 PT 2015
> > > >
> > > > [ ] +1 Release these as the rpm packages for Apache Aurora 0.9.0
> > > > [ ] +0
> > > > [ ] -1 Do not release these artifacts because...
> > > >
> > > > I would like to get the voting started off with my own +1
> > > >
> > >
> >
>

Summary of IRC Meeting in #aurora

[Joshua Cohen]

No one with karma was around to kick off an official meeting, so we had an
ad-hoc meeting. Here's the transcript:

11:01 jcohen: ASFBot: meeting start
11:01 ASFBot: You don't have enough karma for this request
11:01 jcohen: didn’t think so!
11:01 zmanji: ASFBot: meeting start
11:01 ASFBot: You don't have enough karma for this request
11:02 zmanji: I really thought jake added me to the karma list
11:02 zmanji: ASFBot meeting start
11:02 zmanji: dlester can you karma us here?
11:02 zmanji: ASFBot: meeting start
11:02 ASFBot: You don't have enough karma for this request
11:02 jcohen: Let’s just do an ad-hoc meeting
11:02 jcohen: we can manually send notes?
11:03 jcohen: Let’s start with roll-call, why not.
11:03 jcohen: here :)
11:03 rdelvalle: here
11:03 zmanji: here
11:03 mkhutornenko: here
11:04 thalin: here
11:04 jcohen: #topic MesosCon
11:04 jcohen: Several Aurora contributors will be at MesosCon this week,
giving Aurora-related talks.
11:05 jcohen: If you’re going to be at MesosCon, definitely look them up!
11:05 jcohen: Anyone else have any topics?
11:05 zmanji: #topic Forking Twitter Commons
11:05 Yasumoto: I'm speaking on Thursday! :)
11:05 zmanji: the IP Grant was given to fork Twitter Commons into our tree
11:06 Yasumoto: [off] too late, sorry :P
11:06 zmanji: look out for a patch this week that does that and follow up
patches that upgrade guice and guava
11:06 Yasumoto: excellent
11:06 zmanji: I’m done
11:06 thalin: GSOC update? That's about finished, correct?
11:07 jcohen: Is there a GSOC update?
11:07 jcohen: I believe Willy’s making good progress
11:07 thalin: I just remember seeing emails about it recently
11:08 jcohen: Ok, if that’s it I’ll call an end to the meeting
11:08 jcohen: have a good week everyone!
11:09 mkhutornenko: wait, one more topic
11:09 jcohen: ok!
11:09 mkhutornenko: #topic Mesos 0.23.0
11:09 mkhutornenko: Mesos 0.23.0 enforces a new runtime dependency on
MesosExecutorDriver
11:09 mkhutornenko: FYI: You will need libcurl4-nss-dev installed on every
slave
11:10 mkhutornenko: More details here:
http://markmail.org/thread/g6kkiapkartyvqn4
11:10 mkhutornenko: Aurora RB: https://reviews.apache.org/r/37379/
11:10 mkhutornenko: end of topic
11:11 jcohen: Thanks Maxim
11:11 jcohen: Last chance for topics
11:12 jcohen: now if only I could get a decently formatted transcript out
of colloquy!

Re: [VOTE] Release Apache Aurora 0.9.0 RC0

[Joshua Cohen]

+0, I've got an local environment issue that's preventing me from verifying
the Python side of things, but the Java side looks good!

If I have time to track down the environment issue I'll re-run the tests
and come back here, but don't want to block the release if it's working for
others.

On Wed, Jul 22, 2015 at 3:36 PM, Zameer Manji zma...@apache.org wrote:

 +1
 Release candidate looks good!

 On Tue, Jul 21, 2015 at 2:37 PM, Joseph Smith yasumo...@gmail.com wrote:

  +1
 
  + echo 'Release candidate looks good!’
 
   On Jul 21, 2015, at 2:29 PM, Kevin Sweeney kevi...@apache.org wrote:
  
   +1
  
   RC verification script passes
  
   On Tue, Jul 21, 2015 at 2:18 PM, Bill Farner wfar...@apache.org
 wrote:
  
   +1
  
   Successfully ran ./build-support/release/verify-release-candidate
  0.9.0-rc0
  
   -=Bill
  
   On Mon, Jul 20, 2015 at 10:59 AM, Jake Farrell jfarr...@apache.org
   wrote:
  
   I propose that we accept the following release candidate as the
  official
   Apache Aurora 0.9.0 release.
  
  
   Aurora 0.9.0-rc0 includes the following:
   ---
   The CHANGELOG for the release is available at:
  
  
  
 
 https://git-wip-us.apache.org/repos/asf?p=aurora.gitf=CHANGELOGhb=0.9.0-rc0
  
   The branch used to create the release candidate is:
  
  
  
 
 https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=shortlog;h=refs/heads/0.9.0-rc0
  
   The release candidate is available at:
  
  
  
 
 https://dist.apache.org/repos/dist/dev/aurora/0.9.0-rc0/apache-aurora-0.9.0-rc0.tar.gz
  
   The MD5 checksum of the release candidate can be found at:
  
  
  
 
 https://dist.apache.org/repos/dist/dev/aurora/0.9.0-rc0/apache-aurora-0.9.0-rc0.tar.gz.md5
  
   The signature of the release candidate can be found at:
  
  
  
 
 https://dist.apache.org/repos/dist/dev/aurora/0.9.0-rc0/apache-aurora-0.9.0-rc0.tar.gz.asc
  
   The GPG key used to sign the release are available at:
   https://dist.apache.org/repos/dist/dev/aurora/KEYS
  
   Please download, verify, and test.
  
   The vote will close on Thu Jul 23 14:00:00 EDT 2015
  
   [ ] +1 Release this as Apache Aurora 0.9.0
   [ ] +0
   [ ] -1 Do not release this as Apache Aurora 0.9.0 because...
  
  
   I would like to get the voting started off with my own +1
  
   -Jake
  
  
 
 


 --
 Zameer Manji

Re: Aurora React UI Demo/Prototype

[Joshua Cohen]

I don't really disagree with any of the points you raise, David, they were
all concerns I had going into the work, the only uncertainty was exactly
how big of a change going from Angular to React would turn out to be. The
primary reason why I ventured down this path to begin with is the looming
Angular 2.0 release. Angular 2.0 is similar to React in many ways (virtual
DOM, uses TypeScript which is a superset of ES6, etc.). My concern was that
we'd be forced to make the upgrade in the future when support for Angular
1.x was dropped. Given that I'm going to put some effort into adding test
coverage for the UI, this seemed like a good time to evaluate alternatives,
and whether we could avoid that forced upgrade in the future. I definitely
don't want us to get stuck on an unsupported tech stack for the UI,
regardless of how deep (or shallow, as the case may be) our requirements
for a UI framework are.

That said, I've been doing some deeper reading on the Angular 2.0 release
and it seems like the developers have softened their stance a bit:
http://www.infoq.com/news/2015/03/angular-2-concerns-addressed. Essentially
they've committed to support Angular 1.x until the vast majority of the
community has migrated to 2.x. There's no real timeframe attached to
Angular 1.x being EOL'd. Additionally, they seem to be providing at least
something of a migration path from 1.x to 2.x via the new router. This is
definitely still something we need to pay attention to (and I still might
advocate for React over Angular 2.x when the time comes), but I don't think
it currently bubbles up to level of concern that would require a total
overhaul of the UI.

One thing that did come up in the context of this discussion, however, was
the ability for Aurora to serve up a custom UI. There was a suggestion for
allowing Aurora operators to specify a custom UI path that would be served
up, allowing users to swap in a non-default UI. I'm curious what people
think about that idea?

On Thu, Jul 16, 2015 at 1:24 PM, David McLaughlin da...@dmclaughlin.com
wrote:

 Thanks for doing this and sharing code!

 I'd be -1 to any proposal to switch our FE technology stack for several
 reasons.

 I find this stack way too advanced for what the scheduler UI currently
 does. This is also a problem with Angular, but we're already stuck with
 that now.

 I also don't think any of the problems with the current UI are due to
 Angular. The biggest hole that I see in our FE code is lack of automated
 testing, and this technology stack doesn't change the trade-offs involved
 in solving that.

 I also don't think our current technology stack hinders our ability to
 resolve any of the currently open UI tickets, nor does it prevent us from
 doing any of the 'this would be cool' features we've talked about. The
 biggest barriers are more about abstractions or lack of concrete interfaces
 between the components in Aurora.

 I think we need to acknowledge that for the most part the Aurora project
 does not have a great deal of FE resource. Switching stacks again would
 involve retraining everyone to this new thing, but based on the nature of
 the code in the demo - it also involves making sure everyone has cutting
 edge knowledge of JS, which would be a new barrier to entry for anyone
 looking to make quick UI fixes and also make code reviews much harder. I
 think for an infrastructure product like this where most of the expertise
 is in server-side development, it's probably safer to be a couple of
 iterations behind the latest fads in the JS community.


 Cheers,
 David


 On Thu, Jul 16, 2015 at 11:07 AM, Joshua Cohen jco...@twopensource.com
 wrote:

  There are numerous large and stable sites powered by this technology. Not
  least among them are Facebook and Instagram for which these technologies
  were developed (that being React/Flux). ES6 being transpiled down to ES5
 is
  widely adopted as well (off the top of my head, I know Slack does it).
 ES6
  is an officially adopted standard and direct browser support for it is
  increasing quickly: https://kangax.github.io/compat-table/es6/. As you
 can
  see from the table Firefox and Chrome have already implemented
 significant
  parts of the spec. Predictably IE and Safari are lagging behind, but I
  think we're past the point where we need to be concerned about ES6 never
  being adopted. Transpiling in general is a common practice in the JS
  community as can be seen by the popularity of things like CoffeeScript.
 Of
  all the technologies that would potentially be introduced by this change,
  using ES6 syntax and transpiling it to ES5 is the one that I have the
 least
  qualms about.
 
  On Thu, Jul 16, 2015 at 12:53 PM, Bill Farner wfar...@apache.org
 wrote:
 
   I'm pretty ignorant of this space, please bear with me.
  
   You'll notice that the app is written in ES6 and not ES5 (i.e. the
javascript you're likely familiar with). As of React 0.13.x, ES6
 with a
transpiler (i.e. a tool to compile the ES6

Re: Aurora React UI Demo/Prototype

[Joshua Cohen]

For those who are not familiar with React, here's some more details/helpful
links on the underlying technologies.

1) You'll notice that the app is written in ES6 and not ES5 (i.e. the
javascript you're likely familiar with). As of React 0.13.x, ES6 with a
transpiler (i.e. a tool to compile the ES6 syntax into vanilla ES5 syntax)
is the preferred method of writing React apps. For more details on the new
ES6 features you can read this: http://babeljs.io/docs/learn-es2015/ (babel
is the transpiler used by the demo).

2) The next thing you'll notice is that the components weirdly intermingle
javascript and HTML. This is a React feature called JSX. JSX lets you
express DOM-like components in a more natural style. You *can* create React
components using pure-JS, but JSX is the recommended way to do so. For more
details on JSX I'd recommend reading:
https://facebook.github.io/react/docs/jsx-in-depth.html.

3) With the syntax-strangeness out of the way, the next thing to understand
is React itself. React simplifies building high performance JS apps in a
couple of ways: first, it removes two-way binding, which in complex apps
can make reasoning about what causes what to change and when very
difficult, and second: it provides a virtual DOM. The slowest part of
JS-driven apps is updating the DOM. With React, when your data changes your
components are only re-rendered if they've actually changed; this is
accomplished via the virtual DOM. Components are first rendered into the
virtual DOM and then the new tree is compared against the current one, and
only the portions of the tree that are changed are actually rendered into
the page. For more details about React in general, I'd recommend reading
the React docs themselves:
https://facebook.github.io/react/docs/getting-started.html.

4) Finally, the last architectural piece of the demo is Flux. Flux is a
replacement for the traditional MVC model. It provides a unidirectional
data flow for responding to events. Essentially the user interacts with the
app which causes Actions to be sent to the Dispatcher. Stores listen for
events from the Dispatcher and update their state. This causes Views (in
our case React components) to re-render, at which point the cycle begins
anew. You can read more about Flux here:
https://facebook.github.io/flux/docs/overview.html.

This stuff is relatively new to me as well, so there's a better than zero
chance that there's room for improvement with the way the demo is
implemented. but I think it serves as a good starting point for a
discussion about whether this style of application is palatable.

Cheers,

Joshua

On Wed, Jul 15, 2015 at 6:28 PM, Joshua Cohen jco...@twopensource.com
wrote:

 As mentioned during the IRC meeting earlier this week. I spent some time
 recently putting together a React-driven UI prototype as a strawman for a
 discussion about potentially moving away from Angular for the UI. This
 prototype is now available on Github:
 https://github.com/jcohen/aurora-react.

 As mentioned in the README, the project is very simple. It doesn't
 actually talk to the Scheduler and it only contains two pages (of which
 only one attempts to render data). It also uses the current UI layout for
 convenience. The intention to use it as a way to judge whether this style
 of app is preferable to Angular or if we should renew our investment in the
 current Angular based UI and pay down the tech debt that it has accrued.

 Interested to hear everyone's thoughts!

 Cheers,

 Joshua

Re: Aurora React UI Demo/Prototype

[Joshua Cohen]

There are numerous large and stable sites powered by this technology. Not
least among them are Facebook and Instagram for which these technologies
were developed (that being React/Flux). ES6 being transpiled down to ES5 is
widely adopted as well (off the top of my head, I know Slack does it). ES6
is an officially adopted standard and direct browser support for it is
increasing quickly: https://kangax.github.io/compat-table/es6/. As you can
see from the table Firefox and Chrome have already implemented significant
parts of the spec. Predictably IE and Safari are lagging behind, but I
think we're past the point where we need to be concerned about ES6 never
being adopted. Transpiling in general is a common practice in the JS
community as can be seen by the popularity of things like CoffeeScript. Of
all the technologies that would potentially be introduced by this change,
using ES6 syntax and transpiling it to ES5 is the one that I have the least
qualms about.

On Thu, Jul 16, 2015 at 12:53 PM, Bill Farner wfar...@apache.org wrote:

 I'm pretty ignorant of this space, please bear with me.

 You'll notice that the app is written in ES6 and not ES5 (i.e. the
  javascript you're likely familiar with). As of React 0.13.x, ES6 with a
  transpiler (i.e. a tool to compile the ES6 syntax into vanilla ES5
 syntax)
  is the preferred method of writing React apps. For more details on the
 new
  ES6 features you can read this: http://babeljs.io/docs/learn-es2015/
  (babel
  is the transpiler used by the demo).
 

 This sounds very fancy and cutting-edge.  Sometimes fancy and cutting-edge
 things are not ready for prime time.  What gives you confidence that this
 is stable and will not result in trailblazing on our part?



 -=Bill

 On Thu, Jul 16, 2015 at 8:50 AM, Joshua Cohen jco...@twopensource.com
 wrote:

  For those who are not familiar with React, here's some more
 details/helpful
  links on the underlying technologies.
 
  1) You'll notice that the app is written in ES6 and not ES5 (i.e. the
  javascript you're likely familiar with). As of React 0.13.x, ES6 with a
  transpiler (i.e. a tool to compile the ES6 syntax into vanilla ES5
 syntax)
  is the preferred method of writing React apps. For more details on the
 new
  ES6 features you can read this: http://babeljs.io/docs/learn-es2015/
  (babel
  is the transpiler used by the demo).
 
  2) The next thing you'll notice is that the components weirdly
 intermingle
  javascript and HTML. This is a React feature called JSX. JSX lets you
  express DOM-like components in a more natural style. You *can* create
 React
  components using pure-JS, but JSX is the recommended way to do so. For
 more
  details on JSX I'd recommend reading:
  https://facebook.github.io/react/docs/jsx-in-depth.html.
 
  3) With the syntax-strangeness out of the way, the next thing to
 understand
  is React itself. React simplifies building high performance JS apps in a
  couple of ways: first, it removes two-way binding, which in complex apps
  can make reasoning about what causes what to change and when very
  difficult, and second: it provides a virtual DOM. The slowest part of
  JS-driven apps is updating the DOM. With React, when your data changes
 your
  components are only re-rendered if they've actually changed; this is
  accomplished via the virtual DOM. Components are first rendered into the
  virtual DOM and then the new tree is compared against the current one,
 and
  only the portions of the tree that are changed are actually rendered into
  the page. For more details about React in general, I'd recommend reading
  the React docs themselves:
  https://facebook.github.io/react/docs/getting-started.html.
 
  4) Finally, the last architectural piece of the demo is Flux. Flux is a
  replacement for the traditional MVC model. It provides a unidirectional
  data flow for responding to events. Essentially the user interacts with
 the
  app which causes Actions to be sent to the Dispatcher. Stores listen for
  events from the Dispatcher and update their state. This causes Views (in
  our case React components) to re-render, at which point the cycle begins
  anew. You can read more about Flux here:
  https://facebook.github.io/flux/docs/overview.html.
 
  This stuff is relatively new to me as well, so there's a better than zero
  chance that there's room for improvement with the way the demo is
  implemented. but I think it serves as a good starting point for a
  discussion about whether this style of application is palatable.
 
  Cheers,
 
  Joshua
 
  On Wed, Jul 15, 2015 at 6:28 PM, Joshua Cohen jco...@twopensource.com
  wrote:
 
   As mentioned during the IRC meeting earlier this week. I spent some
 time
   recently putting together a React-driven UI prototype as a strawman
 for a
   discussion about potentially moving away from Angular for the UI. This
   prototype is now available on Github:
   https://github.com/jcohen/aurora-react.
  
   As mentioned in the README, the project

Aurora React UI Demo/Prototype

[Joshua Cohen]

As mentioned during the IRC meeting earlier this week. I spent some time
recently putting together a React-driven UI prototype as a strawman for a
discussion about potentially moving away from Angular for the UI. This
prototype is now available on Github: https://github.com/jcohen/aurora-react
.

As mentioned in the README, the project is very simple. It doesn't actually
talk to the Scheduler and it only contains two pages (of which only one
attempts to render data). It also uses the current UI layout for
convenience. The intention to use it as a way to judge whether this style
of app is preferable to Angular or if we should renew our investment in the
current Angular based UI and pay down the tech debt that it has accrued.

Interested to hear everyone's thoughts!

Cheers,

Joshua

Re: Using a config file to support custom executors: potential paradigm shift

[Joshua Cohen]

+1 to #1 for the short term, but I'd like us to assess #3 in the long term.

On Thursday, July 2, 2015, Zameer Manji zma...@apache.org wrote:

 I am in favor of #1 to prevent yak shaving.

 On Thu, Jul 2, 2015 at 12:10 PM, Bill Farner wfar...@apache.org
 javascript:; wrote:

  Thanks for starting this discussion, Renan!
 
  I think it's clear that the feature you're adding calls for a
 configuration
  file.  I'm realizing now that we do have some precedent for configuration
  files with the recently-introduced security controls [1].  In that case
 the
  sane path was obvious since we pass the configuration file in an
  established format to third-party code (Apache Shiro).
 
  I see several paths ahead:
 
  1.) start with individual feature-oriented configuration files and
  re-assess down the road
 
  2.) establish a convention for a single global configuration file
 
  3.) (2) and migrate command line arguments to a configuration file
 
  My personal preference is (1), so as to not force Renan to start a yak
  shave, and because i think willingness to change things down the road is
  important.
 
  I include (3) because people have inquired about that in the past.
 
  Does anyone have a preference which path we take?  Are there other
 options
  i'm not thinking about?
 
 
  [1]
 
 
 https://github.com/apache/aurora/blob/master/docs/security.md#http-spnego-authentication-kerberos
 
  -=Bill
 
  On Wed, Jul 1, 2015 at 3:34 PM, Renan DelValle rdelv...@binghamton.edu
 javascript:;
  wrote:
 
   Hi all,
  
   I'm currently working on bringing custom executor support to Aurora
   (AURORA-1288). As development and discussions about the most adequate
   solution to this problem have moved along, I've reached a crossroad
   where I need the community's input on the implementation path this
   feature will take.
  
   Right now, after evaluating other options,  it seems that the safest
   and most flexible way to providing users the ability to configure
   their own custom executor may be to use a configuration file.
  
   However, as there is no previous use of a config file (everything has
   been done through command line up until now), a discussion is
   necessary about this possible shift in paradigm due to the fact that,
   if this route is taken, it will set a precedent for Aurora.
  
   As Bill Farner said in his comment on Jira, all in all, this is
   discussion should be about how should approach this potential paradigm
   shift.
  
   -Renan
  
 
  --
  Zameer Manji