Re: [VOTE] Release Apache Aurora 0.10.0-1 debs

[Joshua Cohen]

Yeah, I agree, that would be preferable.

On Mon, Nov 30, 2015 at 12:27 PM, John Sirois  wrote:

> On Mon, Nov 30, 2015 at 11:18 AM, Joshua Cohen  wrote:
>
> > +1
> >
> > One minor nit, not sure if it's worth fixing:
> > test/deb/ubuntu-trusty/README.md still references 0.9.0, should be
> updated
> > to 0.10.0?
> >
>
> I'd be more in favor of a clarifying phrase like 'for example ...' than
> changing this text for every package release.
>
>
> > On Fri, Nov 27, 2015 at 12:54 PM, John Sirois 
> wrote:
> >
> > > On Fri, Nov 27, 2015 at 11:34 AM, Bill Farner 
> > wrote:
> > >
> > > > I propose that we accept the following artifacts as the official deb
> > > > packaging for
> > > > Apache Aurora 0.10.0.
> > > >
> > > >
> > > >
> > >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.10.0-1/deb/ubuntu-trusty/
> > > >
> > > > The Aurora deb packaging includes the following:
> > > > ---
> > > > The CHANGELOG is available at:
> > > >
> > > >
> > >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=blob_plain;f=specs/debian/changelog;hb=refs/heads/0.10.x
> > > >
> > > > The branch used to create the packaging is:
> > > >
> > > >
> > >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;h=refs/heads/0.10.x
> > > >
> > > > The packages are available at:
> > > >
> > > >
> > >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.10.0-1/deb/ubuntu-trusty/
> > > >
> > > > The GPG keys used to sign the packages are available at:
> > > > https://dist.apache.org/repos/dist/release/aurora/KEYS
> > > >
> > > > Please download, verify, and test.
> > > >
> > > > The vote will close in 5 business days, on Thu Dec 4 8:00:00 PT 2015
> > > >
> > > > [ ] +1 Release these as the deb packages for Apache Aurora 0.10.0
> > > > [ ] +0
> > > > [ ] -1 Do not release these artifacts because...
> > > >
> > > > I would like to get the voting started off with my own +1
> > > >
> > >
> > > +1 nonbinding
> > >
> > > Tested via the test/deb/ubuntu-trusty vm.
> > >
> > >
> > >
> > > --
> > > John Sirois
> > > 303-512-3301
> > >
> >
>
>
>
> --
> John Sirois
> 303-512-3301
>

Re: [VOTE] Release Apache Aurora 0.10.0-1 debs

[Joshua Cohen]

+1

One minor nit, not sure if it's worth fixing:
test/deb/ubuntu-trusty/README.md still references 0.9.0, should be updated
to 0.10.0?

On Fri, Nov 27, 2015 at 12:54 PM, John Sirois  wrote:

> On Fri, Nov 27, 2015 at 11:34 AM, Bill Farner  wrote:
>
> > I propose that we accept the following artifacts as the official deb
> > packaging for
> > Apache Aurora 0.10.0.
> >
> >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.10.0-1/deb/ubuntu-trusty/
> >
> > The Aurora deb packaging includes the following:
> > ---
> > The CHANGELOG is available at:
> >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=blob_plain;f=specs/debian/changelog;hb=refs/heads/0.10.x
> >
> > The branch used to create the packaging is:
> >
> >
> https://git1-us-west.apache.org/repos/asf?p=aurora-packaging.git;a=tree;h=refs/heads/0.10.x
> >
> > The packages are available at:
> >
> >
> http://people.apache.org/~wfarner/aurora/distributions/0.10.0-1/deb/ubuntu-trusty/
> >
> > The GPG keys used to sign the packages are available at:
> > https://dist.apache.org/repos/dist/release/aurora/KEYS
> >
> > Please download, verify, and test.
> >
> > The vote will close in 5 business days, on Thu Dec 4 8:00:00 PT 2015
> >
> > [ ] +1 Release these as the deb packages for Apache Aurora 0.10.0
> > [ ] +0
> > [ ] -1 Do not release these artifacts because...
> >
> > I would like to get the voting started off with my own +1
> >
>
> +1 nonbinding
>
> Tested via the test/deb/ubuntu-trusty vm.
>
>
>
> --
> John Sirois
> 303-512-3301
>

Summary of IRC Meeting in #aurora

[ASF IRC Bot]

Summary of IRC Meeting in #aurora at Mon Nov 30 19:01:45 2015:

Attendees: jsirois, jfarrell, jcohen, wfarner, Yasumoto, mkhutornenko, dlester

- Preface
- 0.10.0 packages
- AURORA-987


IRC log follows:

## Preface ##
[Mon Nov 30 19:01:54 2015] : let's start with roll call
[Mon Nov 30 19:01:55 2015] : here
[Mon Nov 30 19:01:57 2015] : here
[Mon Nov 30 19:01:58 2015] : here
[Mon Nov 30 19:02:01 2015] : howdy howdy
[Mon Nov 30 19:02:19 2015] : here
[Mon Nov 30 19:02:23 2015] : Present
## 0.10.0 packages ##
[Mon Nov 30 19:03:55 2015] : FYI for those not already following, we 
have an active vote for 0.10.0 debs: 
http://mail-archives.apache.org/mod_mbox/aurora-dev/201511.mbox/%3CCAFWq12VLCoTe%3DAJjXZiq4LQeWzLy7ZgUL9F2%2BHEqp8LnL1Qw8g%40mail.gmail.com%3E
[Mon Nov 30 19:04:02 2015] : here
[Mon Nov 30 19:04:17 2015] : i have centos7 RPMs built, hopefully 
putting up a vote for those today as well
[Mon Nov 30 19:05:04 2015] : for any centos6 users, we have an 
in-flight contribution for an el6 builder that could use more input: 
https://reviews.apache.org/r/40765/
[Mon Nov 30 19:06:10 2015] : i noticed we made no user@ announcement 
for 0.10.0.  this is our first release since having this list, should we make a 
habit of announcing there?
[Mon Nov 30 19:06:26 2015] : +1
[Mon Nov 30 19:07:53 2015] : agreed, seems a good spot to ensure 
users know of a new release
[Mon Nov 30 19:08:08 2015] : sounds good, i'll add that to my TODO 
list for today
[Mon Nov 30 19:08:18 2015] : Definitely. It'd also be nice to have a 
blog post -- I imagine these would largely be the same?
[Mon Nov 30 19:09:03 2015] : indeed.  since upgrading to OS X 10.11 
i've been having trouble building the website, so i will need to revisit those 
issues to make progress there
[Mon Nov 30 19:09:58 2015] : dlester: how’s progress on automating 
the Mesos website updates going?
[Mon Nov 30 19:10:02 2015] : wfarner: perhaps its worth putting that 
in a docker container so we can ensure state and not require additional system 
dependencies
[Mon Nov 30 19:10:17 2015] : +1 - all released things via Vagrantfile 
or Dockerfile
[Mon Nov 30 19:10:20 2015] : anything we can crib from that so that 
docs and such get published automatically?
[Mon Nov 30 19:10:32 2015] : jfarrell: i'd be +1 to that.  i'd also 
like to get the site moved over to git...anything left to do there?
[Mon Nov 30 19:10:47 2015] : jcohen: not much progress -- I've been 
enjoying my time off a little too much :)
[Mon Nov 30 19:11:01 2015] : cannot blame you :)
[Mon Nov 30 19:11:07 2015] : cant have automatic publishing of 
website, would require svn creds to be on one of the ci systems
[Mon Nov 30 19:11:32 2015] : and since that is a commit it needs to 
occur by a committer and not a bot
[Mon Nov 30 19:15:09 2015] : ok i've given myself another TODO for 
docker-based website building
[Mon Nov 30 19:15:29 2015] : i think that does it for this topic, does 
anyone else have another topic to discuss?
## AURORA-987 ##
[Mon Nov 30 19:15:45 2015] : I have a good deal of work towards the 
basis of the REST push here: 
https://github.com/apache/aurora/compare/master...jsirois:jsirois/issues/AURORA-987/experiments
[Mon Nov 30 19:15:56 2015] : ny eyes on that are welcome, trying to 
get out an RB by tomorrow.
[Mon Nov 30 19:16:10 2015] : It keeps the status quo but replaces 
apache thrift with swift and handles authparams via thrift annotations.
[Mon Nov 30 19:16:48 2015] : Hard to keep this small!
[Mon Nov 30 19:17:47 2015] : jsirois: fwiw, i'm okay with a big patch 
as long as the delta in the existing code is not the big part
[Mon Nov 30 19:18:09 2015] : thrift with swift… good to always 
favor other apache projects over external ones
[Mon Nov 30 19:18:11 2015] : The I* entities make the delta in the 
existing code larger than I'd like
[Mon Nov 30 19:19:15 2015] : I may be able to leave the I* in though 
for a 1st cut - even though they become redundant
[Mon Nov 30 19:19:54 2015] : jfarrell: the proposal for using swift 
probably deserves a brief ML discussion.  from what John has told me, swift 
gives us a lot of build-time control we currently lack with thrift
[Mon Nov 30 19:20:12 2015] : I'll start a thread - definitely deserves 
discussion
[Mon Nov 30 19:20:17 2015] : thanks jsirois
[Mon Nov 30 19:22:49 2015] : jsirois: looks interesting, I’ll take a 
look asap
[Mon Nov 30 19:22:56 2015] : any other topics?
[Mon Nov 30 19:25:36 2015] : thanks for joining, everyone!
[Mon Nov 30 19:25:38 2015] : ASFBot: meeting stop


Meeting ended at Mon Nov 30 19:25:38 2015

[RFC] REST / thrift & AURORA-987

[John Sirois]

I’ve experimenting on https://issues.apache.org/jira/browse/AURORA-987 for
the past few weeks and I’d like to ask for feedback on the direction I’d
like to head. If you’re interested in the evolution of the Aurora REST api,
read on.
--

AURORA-987 aims to create a first-class REST-like scheduler interface. I’ve
re-familiarized myself with the codebase and come to the conclusion that
transitioning to a 1st class REST api requires maintaining the core thrift
API as the 1st class API until the point the REST API is fully established
and clients can all be transitioned.

I think this conclusion is probably uncontroversial, but the key factors
pushing this way are:

   1.

   The thrift API has both wide and deep dependencies inside the Aurora
   codebase - 276 imports across 97 files:

   $ git grep "import org.apache.aurora.gen" -- src/main/java/ | grep
-v "import org.apache.aurora.gen.storage" | wc -l
   276
   $ git grep "import org.apache.aurora.gen" -- src/main/java/ | grep
-v "import org.apache.aurora.gen.storage" | cut -d: -f1 | sort -u | wc
-l
   97

   2.

   The thrift API is stored long-term in the log in serialized form.

Both 1 & 2 dictate that the thrift API, at least its enums, structs and
unions, must be maintained for the forseeable future.
We also have the RPC API (thrift services) - which is currently a ~thin,
but not insignificant, container of API processing logic. For example, see
here

.

As such it seems to me the REST API should call into the existing thrift
API to provide a stable transition and confidence in core logic of API
method implementations.

This leads to the following ideas for paths forward:

   1. Hand construct a REST forwarding layer and maintain it in tandem with
   thrift API changes.
   2. Automate 1 such that thrift API changes cause REST API changes
   automatically.

The hand construction path has the obvious maintenance issues, but is
otherwise straight-forward. The maintenance issues should not be
overstated, since good tests and some extra review vigilance could be
enough to make the approach work for the period of time both APIs are
supported.

That said, an automated solution with a single source of truth for the API
definition is clearly preferrable given the automation is free.
The automation is far from free though and so I’ve started investigating
one approach to this automation to flesh out the scope.

We already do our own thrift codegen

via a custom gradle ThriftEntitiesPlugin

that works around Apache thrift’s java codegen in order to generate
immutable wrapper entities for the storage system.
I propose taking this further and generating our own thrift API and
entities in 1 pass through our thrift files. These would be immutable
thrift entities 1st class with builders for modification and the entities
and the generated service interfaces would carry extra metadata in the form
of annotations to bind REST services and their metadata with.

There are 2 paths I’ve considered towards this end:

   1. Modify Apache thrift to support immutable-style java output with
   support for thrift annotations.
   2. Write our own thrift parser and code generator to do said same.

I’ve been pursuing option 2 even though it sounds worse on its face. The
swift  project from Facebook brings
options 1 and 2 back on the same level of undertaking since the parsing and
protocol implementations can be leveraged as libraries and only the codegen
portion need be undertaken (You can see some of that work here

).

So, with that background 2 questions of the same form:

   1. Is there some other fundamental approach I’m missing to bolting on a
   1st class REST API, or is the hand construction approach favorable?
   2. Is the approach to single point of API control using swift misguided?
   Should I be focusing on Apache thrift enhancement instead? Should I be
   generating the *.thrift files instead from a new 1st class source of truth
   in the form of a json api schema?

Any and all feedback is welcome!
​

Build failed in Jenkins: aurora-packaging-nightly #110

[Apache Jenkins Server]

See 

Changes:

[wfarner] Remove SessionKey from APIs and implementations.

[wfarner] AURORA-1512: Aurora rpm missing std output switch

--
[...truncated 28126 lines...]
00:48:08 00:08 [jvm]
00:48:08 00:08 [dup]
00:48:08 00:08   [complete]
   SUCCESS
/scratch/pants binary src/main/python/apache/aurora/kerberos:kaurora_admin
fatal: Not a git repository (or any of the parent directories): .git

00:48:09 00:00 [main]
   (To run a reporting server: ./pants server)
00:48:09 00:00   [bootstrap]
00:48:09 00:00   [setup]
00:48:09 00:00 [parse]fatal: Not a git repository (or any of the parent 
directories): .git

   Executing tasks in goals: bootstrap -> imports -> unpack-jars -> 
deferred-sources -> jvm-platform-validate -> gen -> resolve -> compile -> 
resources -> binary
00:48:09 00:00   [bootstrap]
00:48:09 00:00 [bootstrap-jvm-tools]
00:48:09 00:00   [imports]
00:48:09 00:00 [ivy-imports]
00:48:09 00:00   [unpack-jars]
00:48:09 00:00 [unpack-jars]
00:48:09 00:00   [deferred-sources]
00:48:09 00:00 [deferred-sources]
00:48:09 00:00   [jvm-platform-validate]
00:48:09 00:00 [jvm-platform-validate]
00:48:09 00:00   [gen]
00:48:09 00:00 [thrift]
00:48:09 00:00 [protoc]
00:48:09 00:00 [antlr]
00:48:09 00:00 [ragel]
00:48:09 00:00 [jaxb]
00:48:09 00:00 [wire]
00:48:09 00:00   [resolve]
00:48:09 00:00 [ivy]
00:48:09 00:00   [compile]
00:48:09 00:00 [python-eval]
00:48:09 00:00 [pythonstyle]
00:48:09 00:00 [compile]
00:48:09 00:00 [jvm]
00:48:09 00:00   [jvm-compilers]
00:48:09 00:00 [zinc-pre]
00:48:09 00:00 [zinc-post]
00:48:09 00:00 [jvm-dep-check]
00:48:09 00:00   [resources]
00:48:09 00:00 [prepare]
00:48:09 00:00 [services]
00:48:09 00:00   [binary]
00:48:09 00:00 [python-binary-create]
00:48:09 00:00   [chroot]
00:48:10 00:01 [jvm]
00:48:10 00:01 [dup]
00:48:11 00:02   [complete]
   SUCCESS
/scratch/pants binary src/main/python/apache/aurora/tools:thermos
fatal: Not a git repository (or any of the parent directories): .git

00:48:11 00:00 [main]
   (To run a reporting server: ./pants server)
00:48:11 00:00   [bootstrap]
00:48:11 00:00   [setup]
00:48:11 00:00 [parse]fatal: Not a git repository (or any of the parent 
directories): .git

   Executing tasks in goals: bootstrap -> imports -> unpack-jars -> 
jvm-platform-validate -> deferred-sources -> gen -> resolve -> resources -> 
compile -> binary
00:48:11 00:00   [bootstrap]
00:48:11 00:00 [bootstrap-jvm-tools]
00:48:11 00:00   [imports]
00:48:11 00:00 [ivy-imports]
00:48:11 00:00   [unpack-jars]
00:48:11 00:00 [unpack-jars]
00:48:11 00:00   [jvm-platform-validate]
00:48:11 00:00 [jvm-platform-validate]
00:48:11 00:00   [deferred-sources]
00:48:11 00:00 [deferred-sources]
00:48:11 00:00   [gen]
00:48:11 00:00 [thrift]
00:48:11 00:00 [protoc]
00:48:11 00:00 [antlr]
00:48:11 00:00 [ragel]
00:48:11 00:00 [jaxb]
00:48:11 00:00 [wire]
00:48:11 00:00   [resolve]
00:48:11 00:00 [ivy]
00:48:11 00:00   [resources]
00:48:11 00:00 [prepare]
00:48:11 00:00 [services]
00:48:11 00:00   [compile]
00:48:11 00:00 [python-eval]
00:48:11 00:00 [pythonstyle]
   Invalidated 4 targets.
00:48:13 00:02 [compile]
00:48:13 00:02 [jvm]
00:48:13 00:02   [jvm-compilers]
00:48:13 00:02 [zinc-pre]
00:48:13 00:02 [zinc-post]
00:48:13 00:02 [jvm-dep-check]
00:48:13 00:02   [binary]
00:48:13 00:02 [python-binary-create]
00:48:13 00:02   [chroot]
00:48:19 00:08 [jvm]
00:48:19 00:08 [dup]
00:48:19 00:08   [complete]
   SUCCESS
/scratch/pants binary src/main/python/apache/aurora/tools:thermos_observer
fatal: Not a git repository (or any of the parent directories): .git

00:48:19 00:00 [main]
   (To run a reporting server: ./pants server)
00:48:19 00:00   [bootstrap]
00:48:19 00:00   [setup]
00:48:19 00:00 [parse]fatal: Not a git repository (or any of the parent 
directories): .git

   Executing tasks in goals: bootstrap -> imports -> unpack-jars -> 
jvm-platform-validate -> deferred-sources -> gen -> resolve -> resources -> 
compile -> binary
00:48:20 00:01   [bootstrap]
00:48:20 00:01 [bootstrap-jvm-tools]
00:48:20 00:01   [imports]
00:48:20 00:01 [ivy-imports]
00:48:20 00:01   [unpack-jars]
00:48:20 00:01 [unpack-jars]
00:48:20 00:01   [jvm-platform-validate]
00:48:20 00:01 [jvm-platform-validate]
00:48:20 00:01   [deferred-sources]
00:48:20 00:01 [deferred-sources]
00:48:20 00:01   [gen]
00:48:20 00:01 [thrift]
00:48:20 00:01 [protoc]
00:48:20 00:01 [antlr]
00:48:20 00:01 [ragel]
00:48:20 00:01 [jaxb]
00:48:20 00:01 [wire]

FOSDEM 2016 - take action by 4th of December 2015

[Roman Shaposhnik]

As most of you probably know FOSDEM 2016 (the biggest,
100% free open source developer conference) is right 
around the corner:
   https://fosdem.org/2016/

We hope to have an ASF booth and we would love to see as
many ASF projects as possible present at various tracks
(AKA Developer rooms):
   https://fosdem.org/2016/schedule/#devrooms

This year, for the first time, we are running a dedicated
Big Data and HPC Developer Room and given how much of that
open source development is done at ASF it would be great
to have folks submit talks to:
   https://hpc-bigdata-fosdem16.github.io

While the CFPs for different Developer Rooms follow slightly 
different schedules, but if you submit by the end of this week 
you should be fine.

Finally if you don't want to fish for CFP submission URL,
here it is:
   https://fosdem.org/submit

If you have any questions -- please email me *directly* and
hope to see as many of you as possible in two months! 

Thanks,
Roman.