Re: [VOTE] Release Apache Aurora 0.13.0 RC0

2016-04-12 Thread Bill Farner 
+1

verification passes for me

On Tue, Apr 12, 2016 at 1:34 PM, Joshua Cohen  wrote:

> Ahh, thanks for being on top of that John!
>
> On Tue, Apr 12, 2016 at 3:05 PM, John Sirois 
> wrote:
>
> > On Tue, Apr 12, 2016 at 1:48 PM, Joshua Cohen  wrote:
> >
> > > +1, ran the verification script, everything looks good to me.
> > >
> > > Nitpick: the NEWS link in the email 404s because we renamed it to
> > > RELEASE-NOTES.md. We should probably update the script that generates
> the
> > > email?
> > >
> >
> > See my response above - fixed here earlier this am:
> > https://reviews.apache.org/r/46070/
> >
> >
> > >
> > > On Tue, Apr 12, 2016 at 10:02 AM, Erb, Stephan <
> > > stephan@blue-yonder.com>
> > > wrote:
> > >
> > > > +1 for releasing 0.13.0-rc0 as Aurora 0.13.0
> > > >
> > > > * tested with the verification script
> > > > * deployed the RC to an inhouse test cluster
> > > >
> > > > I am also OK with fixing the changelog afterwards.
> > > >
> > > > 
> > > > From: Bill Farner 
> > > > Sent: Tuesday, April 12, 2016 16:18
> > > > To: jfarr...@apache.org
> > > > Cc: dev@aurora.apache.org
> > > > Subject: Re: [VOTE] Release Apache Aurora 0.13.0 RC0
> > > >
> > > > I'm okay with not blocking the release on it.
> > > >
> > > > On Tuesday, April 12, 2016, Jake Farrell 
> wrote:
> > > >
> > > > > My fault, I thought we had changed this awhile back to
> automatically
> > > pull
> > > > > in any tickets not having a set fix version that had been marked as
> > > > > resolved and that this had become a post step to clean up jira
> after
> > > the
> > > > > release when marking the version as released.
> > > > >
> > > > > This should not be a blocker for the 0.13.0-rc0 release candidate
> as
> > > the
> > > > > CHANGELOG is not a requirement for a release, just a nice best
> > > practice.
> > > > > Will follow up with a patch to trunk to update for the missing
> > section.
> > > > If
> > > > > this is a must have in anyones opinion then we can cancel this vote
> > and
> > > > > start a new release candidate, thoughts?
> > > > >
> > > > > -Jake
> > > > >
> > > > > On Tue, Apr 12, 2016 at 1:59 AM, Bill Farner  > > > > > wrote:
> > > > >
> > > > >> Changelog looks sparse.  In the past we have tagged all resolved
> > > tickets
> > > > >> with fixVersion that don't already have fixVersion set, which I
> > > believe
> > > > >> lands them in the changelog.  Presumably that step is undocumented
> > and
> > > > not
> > > > >> automated, and as a result didn't happen?
> > > > >>
> > > > >> On Monday, April 11, 2016, John Sirois  > > > >> > wrote:
> > > > >>
> > > > >>> +1 - Tested with
> `./build-support/release/verify-release-candidate
> > > > >>> 0.13.0-rc0`.
> > > > >>>
> > > > >>> On Mon, Apr 11, 2016 at 9:26 PM, Jake Farrell <
> jfarr...@apache.org
> > >
> > > > >>> wrote:
> > > > >>>
> > > > >>> > All,
> > > > >>> >
> > > > >>> > I propose that we accept the following release candidate as the
> > > > >>> official
> > > > >>> > Apache Aurora 0.13.0 release.
> > > > >>> >
> > > > >>> > Aurora 0.13.0-rc0 includes the following:
> > > > >>> > ---
> > > > >>> > The NEWS for the release is available at:
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=NEWS=rel/0.13.0-rc0
> > > > >>>
> > > > >>>
> > > > >>> The NEWS link above is broken, but was fixed here:
> > > > >>> https://reviews.apache.org/r/46070/
> > > > >>>
> > > > >>>
> > > > >>> >
> > > > >>> >
> > > > >>> > The CHANGELOG for the release is available at:
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=CHANGELOG=rel/0.13.0-rc0
> > > > >>>
> > > > >>>
> > > > >>> The CHANGELOG looks light with 2 entries, but that may be
> correct.
> > > If
> > > > >>> its
> > > > >>> not correct, I'm not sure if this is an RC blocker or not ... I
> > voted
> > > > >>> assuming it was not.
> > > > >>>
> > > > >>>
> > > > >>> >
> > > > >>> >
> > > > >>> > The tag used to create the release candidate is:
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=shortlog;h=refs/tags/rel/0.13.0-rc0
> > > > >>> >
> > > > >>> > The release candidate is available at:
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.13.0-rc0/apache-aurora-0.13.0-rc0.tar.gz
> > > > >>> >
> > > > >>> > The MD5 checksum of the release candidate can be found at:
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.13.0-rc0/apache-aurora-0.13.0-rc0.tar.gz.md5
> > > > >>> >
> > > > >>> > The signature of the release candidate can

Build failed in Jenkins: aurora-packaging-nightly #262

2016-04-12 Thread Apache Jenkins Server 
See 

--
[...truncated 9407 lines...]
+ rm -rf 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64
++ dirname 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64
+ mkdir -p /dist/rpmbuild/BUILDROOT
+ mkdir 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64
+ cd apache-aurora-0.13.1snapshot.2016.04.13
+ rm -rf 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/bin
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/share/doc/aurora-0.13.1snapshot.2016.04.13
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/lib/aurora
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/var/lib
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/var/lib/aurora
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/var/log/aurora
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/var/log/thermos
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/var/run/thermos
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/etc/aurora
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/etc/init.d
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/etc/systemd/system
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/etc/logrotate.d
+ mkdir -p 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/etc/sysconfig
+ cp -r dist/install/aurora-scheduler/bin dist/install/aurora-scheduler/etc 
dist/install/aurora-scheduler/lib 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/lib/aurora
+ for pex_binary in aurora aurora_admin thermos thermos_executor thermos_runner 
thermos_observer
+ install -m 755 dist/aurora.pex 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/bin/aurora
+ for pex_binary in aurora aurora_admin thermos thermos_executor thermos_runner 
thermos_observer
+ install -m 755 dist/aurora_admin.pex 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/bin/aurora_admin
+ for pex_binary in aurora aurora_admin thermos thermos_executor thermos_runner 
thermos_observer
+ install -m 755 dist/thermos.pex 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/bin/thermos
+ for pex_binary in aurora aurora_admin thermos thermos_executor thermos_runner 
thermos_observer
+ install -m 755 dist/thermos_executor.pex 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/bin/thermos_executor
+ for pex_binary in aurora aurora_admin thermos thermos_executor thermos_runner 
thermos_observer
+ install -m 755 dist/thermos_runner.pex 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/bin/thermos_runner
+ for pex_binary in aurora aurora_admin thermos thermos_executor thermos_runner 
thermos_observer
+ install -m 755 dist/thermos_observer.pex 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/bin/thermos_observer
+ install -m 644 /dist/rpmbuild/SOURCES/aurora.service 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/etc/systemd/system
+ install -m 644 /dist/rpmbuild/SOURCES/thermos-observer.service 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/etc/systemd/system/thermos-observer.service
+ install -m 755 /dist/rpmbuild/SOURCES/aurora.startup.sh 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/bin/aurora-scheduler-startup
+ install -m 755 /dist/rpmbuild/SOURCES/thermos-observer.startup.sh 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/usr/bin/thermos-observer-startup
+ install -m 644 /dist/rpmbuild/SOURCES/aurora.sysconfig 
/dist/rpmbuild/BUILDROOT/aurora-scheduler-0.13.1snapshot.2016.04.13-1.el7.centos.aurora.x86_64/etc/sysconfig/aurora
+ install -m 644 /dist/rpmbuild/SOURCES/thermos-observer.sysconfig

Re: [VOTE] Release Apache Aurora 0.13.0 RC0

2016-04-12 Thread Joshua Cohen 
Ahh, thanks for being on top of that John!

On Tue, Apr 12, 2016 at 3:05 PM, John Sirois  wrote:

> On Tue, Apr 12, 2016 at 1:48 PM, Joshua Cohen  wrote:
>
> > +1, ran the verification script, everything looks good to me.
> >
> > Nitpick: the NEWS link in the email 404s because we renamed it to
> > RELEASE-NOTES.md. We should probably update the script that generates the
> > email?
> >
>
> See my response above - fixed here earlier this am:
> https://reviews.apache.org/r/46070/
>
>
> >
> > On Tue, Apr 12, 2016 at 10:02 AM, Erb, Stephan <
> > stephan@blue-yonder.com>
> > wrote:
> >
> > > +1 for releasing 0.13.0-rc0 as Aurora 0.13.0
> > >
> > > * tested with the verification script
> > > * deployed the RC to an inhouse test cluster
> > >
> > > I am also OK with fixing the changelog afterwards.
> > >
> > > 
> > > From: Bill Farner 
> > > Sent: Tuesday, April 12, 2016 16:18
> > > To: jfarr...@apache.org
> > > Cc: dev@aurora.apache.org
> > > Subject: Re: [VOTE] Release Apache Aurora 0.13.0 RC0
> > >
> > > I'm okay with not blocking the release on it.
> > >
> > > On Tuesday, April 12, 2016, Jake Farrell  wrote:
> > >
> > > > My fault, I thought we had changed this awhile back to automatically
> > pull
> > > > in any tickets not having a set fix version that had been marked as
> > > > resolved and that this had become a post step to clean up jira after
> > the
> > > > release when marking the version as released.
> > > >
> > > > This should not be a blocker for the 0.13.0-rc0 release candidate as
> > the
> > > > CHANGELOG is not a requirement for a release, just a nice best
> > practice.
> > > > Will follow up with a patch to trunk to update for the missing
> section.
> > > If
> > > > this is a must have in anyones opinion then we can cancel this vote
> and
> > > > start a new release candidate, thoughts?
> > > >
> > > > -Jake
> > > >
> > > > On Tue, Apr 12, 2016 at 1:59 AM, Bill Farner  > > > > wrote:
> > > >
> > > >> Changelog looks sparse.  In the past we have tagged all resolved
> > tickets
> > > >> with fixVersion that don't already have fixVersion set, which I
> > believe
> > > >> lands them in the changelog.  Presumably that step is undocumented
> and
> > > not
> > > >> automated, and as a result didn't happen?
> > > >>
> > > >> On Monday, April 11, 2016, John Sirois  > > >> > wrote:
> > > >>
> > > >>> +1 - Tested with `./build-support/release/verify-release-candidate
> > > >>> 0.13.0-rc0`.
> > > >>>
> > > >>> On Mon, Apr 11, 2016 at 9:26 PM, Jake Farrell  >
> > > >>> wrote:
> > > >>>
> > > >>> > All,
> > > >>> >
> > > >>> > I propose that we accept the following release candidate as the
> > > >>> official
> > > >>> > Apache Aurora 0.13.0 release.
> > > >>> >
> > > >>> > Aurora 0.13.0-rc0 includes the following:
> > > >>> > ---
> > > >>> > The NEWS for the release is available at:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=NEWS=rel/0.13.0-rc0
> > > >>>
> > > >>>
> > > >>> The NEWS link above is broken, but was fixed here:
> > > >>> https://reviews.apache.org/r/46070/
> > > >>>
> > > >>>
> > > >>> >
> > > >>> >
> > > >>> > The CHANGELOG for the release is available at:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git=CHANGELOG=rel/0.13.0-rc0
> > > >>>
> > > >>>
> > > >>> The CHANGELOG looks light with 2 entries, but that may be correct.
> > If
> > > >>> its
> > > >>> not correct, I'm not sure if this is an RC blocker or not ... I
> voted
> > > >>> assuming it was not.
> > > >>>
> > > >>>
> > > >>> >
> > > >>> >
> > > >>> > The tag used to create the release candidate is:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=shortlog;h=refs/tags/rel/0.13.0-rc0
> > > >>> >
> > > >>> > The release candidate is available at:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.13.0-rc0/apache-aurora-0.13.0-rc0.tar.gz
> > > >>> >
> > > >>> > The MD5 checksum of the release candidate can be found at:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.13.0-rc0/apache-aurora-0.13.0-rc0.tar.gz.md5
> > > >>> >
> > > >>> > The signature of the release candidate can be found at:
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> https://dist.apache.org/repos/dist/dev/aurora/0.13.0-rc0/apache-aurora-0.13.0-rc0.tar.gz.asc
> > > >>> >
> > > >>> > The GPG key used to sign the release are available at:
> > > >>> > https://dist.apache.org/repos/dist/dev/aurora/KEYS
> > > >>> >
> > > >>> > Please download, verify, and test.
> > > >>> >
> > > >>> > The vote will close on Thu Apr 14 23:24:12 EDT 2016,

Re: Looking for feedback - Setting CommandInfo.user by default when launching tasks.

2016-04-12 Thread Joshua Cohen 
As things stand today, once a task is scheduled, the scheduler can die, or
be shut down for maintenance, etc. with no impact to running tasks. If the
scheduler were responsible for announcing and it maintained the current
practice of creating znodes as ephemeral, it would need to maintain a
persistent ZK connection to ensure all announceable tasks are discoverable.
This means if the scheduler went down for any reason (or just whenever it
failed over normally) all serversets would be lost. That's obviously not
acceptable, so the alternative, if we wanted the scheduler to manage
announcing tasks, would be to stop using ephemeral nodes and instead have
the scheduler create persistent znodes and manually tear them down when
tasks finish. While not as problematic as the above, this is still
potentially a degradation from the current behavior in that there can be a
long delay between a task exiting and the scheduler becoming aware of this
(e.g. task finishes during a netsplit). This might be equivalent in scope
to the issue of a zombie instance that's possible today (i.e. scheduler
thinks task has been killed and restarts it elsewhere, but Mesos fails to
clean up the cgroup for whatever reason leading to double registration for
some time). However, today, due to their ephemeral nature, cleaning up
duplicate znodes is guaranteed as soon as the executor exits. If the znodes
were persistent, we'd have to manually seek out and clean up duplicate
znodes (or build tooling to support this).

It's certainly possible to transition responsibility for announcing from
the executor to the scheduler, but I think it would be a fairly significant
amount of work and add a fair amount of complexity.

On Tue, Apr 12, 2016 at 7:35 AM, Erb, Stephan 
wrote:

> I have recently run into another issue due to Thermos running as root (
> https://issues.apache.org/jira/browse/MESOS-5187). I would therefore like
> to re-open the discussion.
>
> IIRC there was a JIRA ticket proposing that the scheduler should be
> responsible for announcing services in Zookeeper.
>
> Would that work? It looks like this could solve a couple of issues at once:
>
> * no need for credentials on slaves, we could therefore run the executor
> as a normal user
> * the announcement would also work for tasks using alternative executors
>
>
> 
> From: John Sirois 
> Sent: Tuesday, March 29, 2016 23:55
> To: Bill Farner
> Cc: dev@aurora.apache.org; John Sirois
> Subject: Re: Looking for feedback - Setting CommandInfo.user by default
> when launching tasks.
>
> On Tue, Mar 29, 2016 at 3:50 PM, Bill Farner  wrote:
>
> > Aha, i think we have different notions of the proposal.  I was under the
> > impression that the executor itself would run as the target user (e.g.
> > steve), not as a system user (e.g. aurora).  I find the former more
> > appealing, with the exception that it leaves us without a solution for
> > concealing the credentials file.
> >
>
> My sketch was nonsense anyhow, since thermos running as aurora couldn't
> launch a task as www-data.  Fundamentally we need a priviledged executor
> (thermos) that can 1: read the creds and announce 2: run the task and
> health checks nnoth as the targeted un-privileged user.
>
>
> >
> > On Tue, Mar 29, 2016 at 2:39 PM, John Sirois 
> wrote:
> >
> >> On Tue, Mar 29, 2016 at 3:26 PM, Bill Farner 
> wrote:
> >>
> >> > If i'm understanding you correctly, that doesn't address preventing
> >> users
> >> > from reading the credentials though.
> >> >
> >>
> >> It does:
> >>
> >> Say:
> >> /var/lib/aurora/creds 400 root
> >>
> >> And then if the CommandInfo has user: aurora (executor user as Steve
> >> suggested), it will get a copy of /var/lib/aurora/creds  in its sandbox
> >> chowned to 400 aurora
> >>
> >> Now aurora's executor (thermos), launches a task in role www-data
> >> announcing for it using the cred.  The www-data user will not be able to
> >> read the local sandbox 400 aurora creds.
> >>
> >>
> >> > On Tue, Mar 29, 2016 at 1:52 PM, John Sirois 
> >> wrote:
> >> >
> >> > > On Tue, Mar 29, 2016 at 2:31 PM, Steve Niemitz  >
> >> > > wrote:
> >> > >
> >> > > > So maybe we add it, but don't change the current default behavior?
> >> > > >
> >> > >
> >> > > Could we use the CommandInfo.uris [1] to solve this?  IE: the
> >> scheduler
> >> > > would need to learn the credential file path, and with that
> knowledge,
> >> > the
> >> > > local mesos (root) readable credential file could be specified as a
> >> uir
> >> > > dependency for all launched executors (or bare commands).  IIUC, if
> >> the
> >> > URI
> >> > > if a file:// the local secured credentails file will be copied into
> >> the
> >> > > sandbox where it can be read by the executor (as aurora).
> >> > >
> >> > > [1]
> >> > >
> >> >
> >>
>