Review Request 52343: Add Criteo to Aurora user list
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52343/ --- Review request for Aurora. Repository: aurora Description --- Add Criteo to Aurora user list Diffs - README.md c314f5ab4e550328caff765deb122486820a08b8 Diff: https://reviews.apache.org/r/52343/diff/ Testing --- N/A Thanks, Pierre Cheynier
Re: Review Request 45298: Remove some dependencies linked to docker usage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45298/ --- (Updated April 20, 2016, 3:38 p.m.) Review request for Aurora and John Sirois. Bugs: AURORA-1568 https://issues.apache.org/jira/browse/AURORA-1568 Repository: aurora-packaging Description --- An operator should be able to build Aurora on his platform without relying on Docker. Even if this is an interesting way of doing it, it sometimes introduce missed (build-)dependencies (in this case openssl) and context-specific behaviour (here, the way pants.ini was added to the build context) Diffs - builder/deb/debian-jessie/Dockerfile fd15dfe6fd4518cfe9915b7339c1b3e609e968d7 builder/deb/debian-jessie/pants.ini 446ea8d8ebb4fce2665dc4c7c1a2ea32d5f18707 builder/deb/ubuntu-trusty/Dockerfile a4a9e648537841583fdc02c006bfa00a3a095aca builder/deb/ubuntu-trusty/pants.ini builder/rpm/centos-7/Dockerfile a4d9dfcee5637d2c31d0c2b63a9ebcf04e8d builder/rpm/centos-7/pants.ini specs/debian/control 9dd927df63f68afc10506a7dcab8d26f21fbee33 specs/debian/rules 974fddfda6da04686daae691fa32c5b4c11a0b5d specs/rpm/Makefile 0476bcbda489bf53edc2101831475fe4a4ea3241 specs/rpm/aurora.spec fa89242f0c39ca058bc791dab18f67f75150e2d2 Diff: https://reviews.apache.org/r/45298/diff/ Testing --- Able to build on CentOS 6 using mock. Able to build on Debian. Limitation = need the patched version of Gradle. Still able to build using the current Docker build system Thanks, Pierre Cheynier
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
> On March 22, 2016, 1:02 a.m., Joshua Cohen wrote:
> > Sorry for the delay on this. After you filed the pull request, I
> > investigated a bit what will be required once Mesos 0.30.0 lands:
> > https://issues.apache.org/jira/browse/AURORA-1632. I think the problem goes
> > beyond the failure to find `sys.executable` when $PATH is not set. As even
> > after switching back to chmod+x on the runner, the task failed further down
> > the stack.
> >
> > I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which
> > should allow `sys.executable` to continue working and will allow any tasks
> > users have running which have come to rely on Thermos setting it for them
> > to behave as expected. The problem is, I haven't had time to figure out
> > what we should set $PATH to yet ;) (anyone have any thoughts?).
> >
> > I know this is probably more info than you bargained for when you opened
> > what seemed like a simple pull request. I'm not opposed to accepting this
> > patch (with a TODO to restore `sys.executable` when we figure out what to
> > do about setting $PATH) if it unblocks your use case, but can you confirm
> > that you're actually able to run the Mesos agent with
> > `--executor_environment_variables='{}'` and still launch tasks?
>
> Pierre Cheynier wrote:
> Hi,
>
> Sorry for my delay on opening this review on apache.org.
> Actually, I started to use Aurora and faced this issue cause my setup use
> `--executor_environment_variables`.
> I first investigated in the Python default setup on CentOS, the pants/pex
> build system and then the differences between the vagrant box provided in the
> repo and mine before discovering that this is the origin of the issue.
>
> I admit this is a simple fix, I just tried to understand the reason of
> using this mechanism and found that it was changed 2 years ago.
> In 0.27.2 it works well by using this patch (I'm now able to run
> something).
>
> Joshua Cohen wrote:
> So he's the behavior I see when I run the Mesos agent with
> `--executor_environment_variables='{}'`. Without this patch, as you note,
> tasks fail to start with a permission denied error (due to `sys.executable`
> being unset). After applying this patch, tasks start up, but processes fail
> to fork:
> https://www.dropbox.com/s/7zzy574xdy02ukq/Screen%20Shot%202016-03-24%20at%203.49.24%20PM.png?dl=0.
>
> Are you by chance running Docker tasks?
>
> Pierre Cheynier wrote:
> I'm running tasks on the Mesos containerizer for now.
> Your problem is not purely a side-effect of not passing any environment
> variables ? (your tasks themselves maybe need PATH or LD_LIBRARY_PATH ?)
>
> Joshua Cohen wrote:
> It's not the task that requires PATH to be set, it's the thermos runner
> itself:
> https://github.com/apache/aurora/blob/master/src/main/python/apache/thermos/core/process.py#L397-L402
>
> This is why I'm confused as to how this works for you unless PATH is
> somehow being set elsewhere (and if PATH is set elsewhere, then
> `sys.executable` should work as well).
Using the Mesos Containerizer, it probably depends on the shell used and the
need for the tasks itself to get a specific environment.
But I'm really surprised, cause even when I'm using your vagrant setup (patch
cherry-picked on top of the master branch + vagrant up), I'm able to launch
your two sample tasks (hello_world + cron_hello_world).
I can provide screenshots if needed.
The portion of code you pointed me to is supposed to enrich the environment of
the subshell. I presume that if the child absolutely need the
PATH/LD_LIBRARY_PATH/whatever, the operator will have to add it in
--executor_environment_variable OR the user will add it by itself. This is what
it's supposed to happen when Mesos 0.30 will be released, right ?
- Pierre
-------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45104/#review124697
---
On March 22, 2016, 9:50 a.m., Pierre Cheynier wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45104/
> ---
>
> (Updated March 22, 2016, 9:50 a.m.)
>
>
> Review request for Aurora, Joshua Cohen and Bill Farner.
>
>
> Repository: aurora
>
>
> Description
> ---
>
> When using `--executor_environment_variables` without explicitely
> passing LD_LIBRARY_PATH, `sys.executable` returns an empty strin
Re: Review Request 45298: Remove some dependencies linked to docker usage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45298/ --- (Updated avr. 19, 2016, 11:34 après-midi) Review request for Aurora and John Sirois. Changes --- Re-introduce `tr '-' '_'` on AURORA_VERSION to be able to build suffixed version number with rpmbuild. Repository: aurora-packaging Description --- An operator should be able to build Aurora on his platform without relying on Docker. Even if this is an interesting way of doing it, it sometimes introduce missed (build-)dependencies (in this case openssl) and context-specific behaviour (here, the way pants.ini was added to the build context) Diffs (updated) - builder/deb/debian-jessie/Dockerfile fd15dfe6fd4518cfe9915b7339c1b3e609e968d7 builder/deb/debian-jessie/pants.ini 446ea8d8ebb4fce2665dc4c7c1a2ea32d5f18707 builder/deb/ubuntu-trusty/Dockerfile a4a9e648537841583fdc02c006bfa00a3a095aca builder/deb/ubuntu-trusty/pants.ini builder/rpm/centos-7/Dockerfile a4d9dfcee5637d2c31d0c2b63a9ebcf04e8d builder/rpm/centos-7/pants.ini specs/debian/control 9dd927df63f68afc10506a7dcab8d26f21fbee33 specs/debian/rules 974fddfda6da04686daae691fa32c5b4c11a0b5d specs/rpm/Makefile 0476bcbda489bf53edc2101831475fe4a4ea3241 specs/rpm/aurora.spec fa89242f0c39ca058bc791dab18f67f75150e2d2 Diff: https://reviews.apache.org/r/45298/diff/ Testing --- Able to build on CentOS 6 using mock. Able to build on Debian. Limitation = need the patched version of Gradle. Still able to build using the current Docker build system Thanks, Pierre Cheynier
Re: Review Request 45298: Remove some dependencies linked to docker usage
> On avr. 19, 2016, 6:32 après-midi, John Sirois wrote: > > Thanks for plugging away at this. It sounds like you were trying to get > > versions like '0.13.1-SNAPSHOT' working but they still don't (see below). > > As such I'm not sure what the new AURORA_INTERNAL_VERSION is buying. > > Pierre Cheynier wrote: > AURORA_INTERNAL_VERSION is a way for rpmbuild to properly unpack the tgz > archive created with git archive (used in %setup -n). > > But the whole thing is working by passing a version number without > "-SNAPSHOT". > As stated above, I'm now able to build using `./build-artifact.sh > /tmp/snapshot.tar.gz 0.13.2`, even on the 0.13.1-SNAPSHOT sources. > This will produce 0.13.2 debs and rpms. > > This doesn't fully resolve AURORA-1568 if the purpose was to build > -SNAPSHOT rpms. The idea here would be to use the Release attribute of the > spec file. From my point of view, this is out of the scope of this review. > What do you think ? All things considered, this has no impact on my side. I'll re-introduce the `export AURORA_VERSION | tr '-' '_'` in the build.sh script, so we are able to build using the exact content of .auroraversion. A new feature here is that we are now free to put an arbitrary version number, and the rpm part is able to unpack the tgz properly, using AURORA_INTERNAL_VERSION. - Pierre --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45298/#review129587 ------- On avr. 19, 2016, 5:48 après-midi, Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45298/ > --- > > (Updated avr. 19, 2016, 5:48 après-midi) > > > Review request for Aurora and John Sirois. > > > Repository: aurora-packaging > > > Description > --- > > An operator should be able to build Aurora on his platform without > relying on Docker. > > Even if this is an interesting way of doing it, it sometimes introduce missed > (build-)dependencies (in this case openssl) and context-specific behaviour > (here, the way pants.ini was added to the build context) > > > Diffs > - > > builder/deb/debian-jessie/Dockerfile > fd15dfe6fd4518cfe9915b7339c1b3e609e968d7 > builder/deb/debian-jessie/pants.ini > 446ea8d8ebb4fce2665dc4c7c1a2ea32d5f18707 > builder/deb/ubuntu-trusty/Dockerfile > a4a9e648537841583fdc02c006bfa00a3a095aca > builder/deb/ubuntu-trusty/pants.ini > builder/rpm/centos-7/Dockerfile a4d9dfcee5637d2c31d0c2b63a9ebcf04e8d > builder/rpm/centos-7/build.sh be3c6fb970dbdd1b94630473e6f67e4a91700e43 > builder/rpm/centos-7/pants.ini > specs/debian/control 9dd927df63f68afc10506a7dcab8d26f21fbee33 > specs/debian/rules 974fddfda6da04686daae691fa32c5b4c11a0b5d > specs/rpm/Makefile 0476bcbda489bf53edc2101831475fe4a4ea3241 > specs/rpm/aurora.spec fa89242f0c39ca058bc791dab18f67f75150e2d2 > > Diff: https://reviews.apache.org/r/45298/diff/ > > > Testing > --- > > Able to build on CentOS 6 using mock. > Able to build on Debian. Limitation = need the patched version of Gradle. > Still able to build using the current Docker build system > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45298: Remove some dependencies linked to docker usage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45298/ --- (Updated April 19, 2016, 11:44 a.m.) Review request for Aurora and John Sirois. Changes --- Pierre - looks like you solved AURORA-1568 so linking that bug. Bugs: AURORA-1568 https://issues.apache.org/jira/browse/AURORA-1568 Repository: aurora-packaging Description --- An operator should be able to build Aurora on his platform without relying on Docker. Even if this is an interesting way of doing it, it sometimes introduce missed (build-)dependencies (in this case openssl) and context-specific behaviour (here, the way pants.ini was added to the build context) Diffs - builder/deb/debian-jessie/Dockerfile fd15dfe6fd4518cfe9915b7339c1b3e609e968d7 builder/deb/debian-jessie/pants.ini 446ea8d8ebb4fce2665dc4c7c1a2ea32d5f18707 builder/deb/ubuntu-trusty/Dockerfile a4a9e648537841583fdc02c006bfa00a3a095aca builder/deb/ubuntu-trusty/pants.ini builder/rpm/centos-7/Dockerfile a4d9dfcee5637d2c31d0c2b63a9ebcf04e8d builder/rpm/centos-7/build.sh be3c6fb970dbdd1b94630473e6f67e4a91700e43 builder/rpm/centos-7/pants.ini specs/debian/control 9dd927df63f68afc10506a7dcab8d26f21fbee33 specs/debian/rules 974fddfda6da04686daae691fa32c5b4c11a0b5d specs/rpm/Makefile 0476bcbda489bf53edc2101831475fe4a4ea3241 specs/rpm/aurora.spec fa89242f0c39ca058bc791dab18f67f75150e2d2 Diff: https://reviews.apache.org/r/45298/diff/ Testing --- Able to build on CentOS 6 using mock. Able to build on Debian. Limitation = need the patched version of Gradle. Still able to build using the current Docker build system Thanks, Pierre Cheynier
Re: Review Request 45298: Remove some dependencies linked to docker usage
> On April 13, 2016, 2:51 p.m., John Sirois wrote: > > Thanks for following up - the source change from github to apache's > > download redirector service was a good idea and it seems a more appropriate > > source to boot! > > > > This worked for me using `./build-artifact.sh /tmp/snapshot.tar.gz 0.13.0` > > only after: > > ```diff > > diff --git a/specs/rpm/Makefile b/specs/rpm/Makefile > > index 0476bcb..950f91e 100644 > > --- a/specs/rpm/Makefile > > +++ b/specs/rpm/Makefile > > @@ -21,7 +21,7 @@ DATETIME ?= $(shell date +%Y%m%d%H%M) > > AURORA_VERSION ?= $(shell cat $(TOPDIR)/.auroraversion | tr '-' '.') > > MESOS_VERSION ?= $(shell tr -d "'\" \t" < $(TOPDIR)/3rdparty/python/BUILD > > | sed -n 's/MESOS_REV=(.*)/\1/p' ) > > > > -SOURCE_TARGET ?= $(RPM_TOPDIR)/rpmbuild/SOURCES/$(AURORA_VERSION).tar.gz > > +SOURCE_TARGET ?= > > $(RPM_TOPDIR)/rpmbuild/SOURCES/apache-aurora-$(AURORA_VERSION).tar.gz > > > > PHONY: all nightly_version release_version get_source_target clean mkdir > > srpm nightly_srpm nightly_rpm release_srpm release_rpm tar_source > > ``` > > > > If you add that change and rebase (there were changes to the removed/moved > > pants.ini in https://reviews.apache.org/r/45662/) - this change LGTM and > > I'll patch it in. Hi, I think this is not sufficient. Therotically, we should be able to build Aurora from the spec file "as is", but also from any git branch (including a SNAPSHOT version). This is why I propose this patch : for the centos builder, we introduce the notion of AURORA_INTERNAL_VERSION, which is the content of .auroraversion. It helps locating the sources, but the final RPM will be versioned using the version arg passed to build-artifact.sh (as it is done for debian/ubuntu builders). I tested it by building the current 0.13.1-SNAPSHOT from sources, and I rebased from master. - Pierre --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45298/#review128683 --- On April 19, 2016, 10:48 a.m., Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45298/ > --- > > (Updated April 19, 2016, 10:48 a.m.) > > > Review request for Aurora and John Sirois. > > > Repository: aurora-packaging > > > Description > --- > > An operator should be able to build Aurora on his platform without > relying on Docker. > > Even if this is an interesting way of doing it, it sometimes introduce missed > (build-)dependencies (in this case openssl) and context-specific behaviour > (here, the way pants.ini was added to the build context) > > > Diffs > - > > builder/deb/debian-jessie/Dockerfile > fd15dfe6fd4518cfe9915b7339c1b3e609e968d7 > builder/deb/debian-jessie/pants.ini > 446ea8d8ebb4fce2665dc4c7c1a2ea32d5f18707 > builder/deb/ubuntu-trusty/Dockerfile > a4a9e648537841583fdc02c006bfa00a3a095aca > builder/deb/ubuntu-trusty/pants.ini > builder/rpm/centos-7/Dockerfile a4d9dfcee5637d2c31d0c2b63a9ebcf04e8d > builder/rpm/centos-7/build.sh be3c6fb970dbdd1b94630473e6f67e4a91700e43 > builder/rpm/centos-7/pants.ini > specs/debian/control 9dd927df63f68afc10506a7dcab8d26f21fbee33 > specs/debian/rules 974fddfda6da04686daae691fa32c5b4c11a0b5d > specs/rpm/Makefile 0476bcbda489bf53edc2101831475fe4a4ea3241 > specs/rpm/aurora.spec fa89242f0c39ca058bc791dab18f67f75150e2d2 > > Diff: https://reviews.apache.org/r/45298/diff/ > > > Testing > --- > > Able to build on CentOS 6 using mock. > Able to build on Debian. Limitation = need the patched version of Gradle. > Still able to build using the current Docker build system > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45298: Remove some dependencies linked to docker usage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45298/ --- (Updated April 19, 2016, 10:48 a.m.) Review request for Aurora and John Sirois. Changes --- [CentOS] Change the Makefile to consider the target version given from the command-line Repository: aurora-packaging Description --- An operator should be able to build Aurora on his platform without relying on Docker. Even if this is an interesting way of doing it, it sometimes introduce missed (build-)dependencies (in this case openssl) and context-specific behaviour (here, the way pants.ini was added to the build context) Diffs (updated) - builder/deb/debian-jessie/Dockerfile fd15dfe6fd4518cfe9915b7339c1b3e609e968d7 builder/deb/debian-jessie/pants.ini 446ea8d8ebb4fce2665dc4c7c1a2ea32d5f18707 builder/deb/ubuntu-trusty/Dockerfile a4a9e648537841583fdc02c006bfa00a3a095aca builder/deb/ubuntu-trusty/pants.ini builder/rpm/centos-7/Dockerfile a4d9dfcee5637d2c31d0c2b63a9ebcf04e8d builder/rpm/centos-7/build.sh be3c6fb970dbdd1b94630473e6f67e4a91700e43 builder/rpm/centos-7/pants.ini specs/debian/control 9dd927df63f68afc10506a7dcab8d26f21fbee33 specs/debian/rules 974fddfda6da04686daae691fa32c5b4c11a0b5d specs/rpm/Makefile 0476bcbda489bf53edc2101831475fe4a4ea3241 specs/rpm/aurora.spec fa89242f0c39ca058bc791dab18f67f75150e2d2 Diff: https://reviews.apache.org/r/45298/diff/ Testing --- Able to build on CentOS 6 using mock. Able to build on Debian. Limitation = need the patched version of Gradle. Still able to build using the current Docker build system Thanks, Pierre Cheynier
Re: Review Request 45212: Remove hard dependency on a specific mesos-version
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45212/#review128656 --- Ship it! Ship It! - Pierre Cheynier On March 23, 2016, 2:56 p.m., Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45212/ > --- > > (Updated March 23, 2016, 2:56 p.m.) > > > Review request for Aurora, Jake Farrell, John Sirois, Stephan Erb, Bill > Farner, and Zameer Manji. > > > Repository: aurora-packaging > > > Description > --- > > We should consider MESOS_VERSION as the minimal requirement to install > the current Aurora version instead of enforce a specific Mesos version. > > > Diffs > - > > specs/rpm/aurora.spec 61e7d146108ae7dd5e129d8288a05773c2659d25 > > Diff: https://reviews.apache.org/r/45212/diff/ > > > Testing > --- > > Install Aurora through the RPM built with aurora-packaging on a Mesos 0.27 > running install. > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45298: Remove some dependencies linked to docker usage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45298/ --- (Updated April 13, 2016, 10:47 a.m.) Review request for Aurora and John Sirois. Changes --- Move RPM source from github.com to apache.org Repository: aurora-packaging Description --- An operator should be able to build Aurora on his platform without relying on Docker. Even if this is an interesting way of doing it, it sometimes introduce missed (build-)dependencies (in this case openssl) and context-specific behaviour (here, the way pants.ini was added to the build context) Diffs (updated) - builder/deb/debian-jessie/Dockerfile 63e89deed9a411b1859ba28ea2572ef4a210da1f builder/deb/debian-jessie/pants.ini 2ff2d5eb6128d4be3e15c0488d8d23bed81a8d5b builder/deb/ubuntu-trusty/Dockerfile 52739100c9593292a7aea2459412c8e49f0155a4 builder/deb/ubuntu-trusty/pants.ini builder/rpm/centos-7/Dockerfile a4d9dfcee5637d2c31d0c2b63a9ebcf04e8d builder/rpm/centos-7/pants.ini specs/debian/control a30e987e4423c611c86518a8f5e3e68ee996982a specs/debian/rules 974fddfda6da04686daae691fa32c5b4c11a0b5d specs/rpm/aurora.spec 61e7d146108ae7dd5e129d8288a05773c2659d25 Diff: https://reviews.apache.org/r/45298/diff/ Testing --- Able to build on CentOS 6 using mock. Able to build on Debian. Limitation = need the patched version of Gradle. Still able to build using the current Docker build system Thanks, Pierre Cheynier
Re: Review Request 45298: Remove some dependencies linked to docker usage
> On mars 24, 2016, 6:04 après-midi, John Sirois wrote: > > specs/rpm/aurora.spec, line 146 > > <https://reviews.apache.org/r/45298/diff/1/?file=1314067#file1314067line146> > > > > This breaks the release script, you can try something like this - it > > needs to run green: `./build-artifact.sh /tmp/apache-aurora-0.12.0.tar.gz > > 0.12.0` > > Pierre Cheynier wrote: > Hi John. > This is why I edited the README (AFAIK, this is a pure naming convention > introduced by the README). > > I changed it to be able to make the aurora.spec work (see Source0 : it > fetch Aurora sources from github). Let me know, maybe it was not the good approach. I was initially wondering why I was not able to simply build the RPM with spectool + mock (= fetch sources + rpmbuild in a chrooted context). - Pierre --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45298/#review125302 --- On mars 24, 2016, 6:04 après-midi, Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45298/ > --- > > (Updated mars 24, 2016, 6:04 après-midi) > > > Review request for Aurora and John Sirois. > > > Repository: aurora-packaging > > > Description > --- > > An operator should be able to build Aurora on his platform without > relying on Docker. > > Even if this is an interesting way of doing it, it sometimes introduce missed > (build-)dependencies (in this case openssl) and context-specific behaviour > (here, the way pants.ini was added to the build context) > > > Diffs > - > > README.md 3a7cf45034b7896c23588fed83176468ca627ebc > builder/deb/debian-jessie/Dockerfile > 63e89deed9a411b1859ba28ea2572ef4a210da1f > builder/deb/debian-jessie/pants.ini > 2ff2d5eb6128d4be3e15c0488d8d23bed81a8d5b > builder/deb/ubuntu-trusty/Dockerfile > 52739100c9593292a7aea2459412c8e49f0155a4 > builder/deb/ubuntu-trusty/pants.ini > builder/rpm/centos-7/Dockerfile a4d9dfcee5637d2c31d0c2b63a9ebcf04e8d > builder/rpm/centos-7/pants.ini > specs/debian/control a30e987e4423c611c86518a8f5e3e68ee996982a > specs/debian/rules 974fddfda6da04686daae691fa32c5b4c11a0b5d > specs/rpm/aurora.spec 61e7d146108ae7dd5e129d8288a05773c2659d25 > > Diff: https://reviews.apache.org/r/45298/diff/ > > > Testing > --- > > Able to build on CentOS 6 using mock. > Able to build on Debian. Limitation = need the patched version of Gradle. > Still able to build using the current Docker build system > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45298: Remove some dependencies linked to docker usage
> On mars 24, 2016, 6:04 après-midi, John Sirois wrote: > > specs/rpm/aurora.spec, line 146 > > <https://reviews.apache.org/r/45298/diff/1/?file=1314067#file1314067line146> > > > > This breaks the release script, you can try something like this - it > > needs to run green: `./build-artifact.sh /tmp/apache-aurora-0.12.0.tar.gz > > 0.12.0` Hi John. This is why I edited the README (AFAIK, this is a pure naming convention introduced by the README). I changed it to be able to make the aurora.spec work (see Source0 : it fetch Aurora sources from github). - Pierre --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45298/#review125302 --- On mars 24, 2016, 6:04 après-midi, Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45298/ > --- > > (Updated mars 24, 2016, 6:04 après-midi) > > > Review request for Aurora and John Sirois. > > > Repository: aurora-packaging > > > Description > --- > > An operator should be able to build Aurora on his platform without > relying on Docker. > > Even if this is an interesting way of doing it, it sometimes introduce missed > (build-)dependencies (in this case openssl) and context-specific behaviour > (here, the way pants.ini was added to the build context) > > > Diffs > - > > README.md 3a7cf45034b7896c23588fed83176468ca627ebc > builder/deb/debian-jessie/Dockerfile > 63e89deed9a411b1859ba28ea2572ef4a210da1f > builder/deb/debian-jessie/pants.ini > 2ff2d5eb6128d4be3e15c0488d8d23bed81a8d5b > builder/deb/ubuntu-trusty/Dockerfile > 52739100c9593292a7aea2459412c8e49f0155a4 > builder/deb/ubuntu-trusty/pants.ini > builder/rpm/centos-7/Dockerfile a4d9dfcee5637d2c31d0c2b63a9ebcf04e8d > builder/rpm/centos-7/pants.ini > specs/debian/control a30e987e4423c611c86518a8f5e3e68ee996982a > specs/debian/rules 974fddfda6da04686daae691fa32c5b4c11a0b5d > specs/rpm/aurora.spec 61e7d146108ae7dd5e129d8288a05773c2659d25 > > Diff: https://reviews.apache.org/r/45298/diff/ > > > Testing > --- > > Able to build on CentOS 6 using mock. > Able to build on Debian. Limitation = need the patched version of Gradle. > Still able to build using the current Docker build system > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
> On mars 22, 2016, 1:02 matin, Joshua Cohen wrote:
> > Sorry for the delay on this. After you filed the pull request, I
> > investigated a bit what will be required once Mesos 0.30.0 lands:
> > https://issues.apache.org/jira/browse/AURORA-1632. I think the problem goes
> > beyond the failure to find `sys.executable` when $PATH is not set. As even
> > after switching back to chmod+x on the runner, the task failed further down
> > the stack.
> >
> > I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which
> > should allow `sys.executable` to continue working and will allow any tasks
> > users have running which have come to rely on Thermos setting it for them
> > to behave as expected. The problem is, I haven't had time to figure out
> > what we should set $PATH to yet ;) (anyone have any thoughts?).
> >
> > I know this is probably more info than you bargained for when you opened
> > what seemed like a simple pull request. I'm not opposed to accepting this
> > patch (with a TODO to restore `sys.executable` when we figure out what to
> > do about setting $PATH) if it unblocks your use case, but can you confirm
> > that you're actually able to run the Mesos agent with
> > `--executor_environment_variables='{}'` and still launch tasks?
>
> Pierre Cheynier wrote:
> Hi,
>
> Sorry for my delay on opening this review on apache.org.
> Actually, I started to use Aurora and faced this issue cause my setup use
> `--executor_environment_variables`.
> I first investigated in the Python default setup on CentOS, the pants/pex
> build system and then the differences between the vagrant box provided in the
> repo and mine before discovering that this is the origin of the issue.
>
> I admit this is a simple fix, I just tried to understand the reason of
> using this mechanism and found that it was changed 2 years ago.
> In 0.27.2 it works well by using this patch (I'm now able to run
> something).
>
> Joshua Cohen wrote:
> So he's the behavior I see when I run the Mesos agent with
> `--executor_environment_variables='{}'`. Without this patch, as you note,
> tasks fail to start with a permission denied error (due to `sys.executable`
> being unset). After applying this patch, tasks start up, but processes fail
> to fork:
> https://www.dropbox.com/s/7zzy574xdy02ukq/Screen%20Shot%202016-03-24%20at%203.49.24%20PM.png?dl=0.
>
> Are you by chance running Docker tasks?
I'm running tasks on the Mesos containerizer for now.
Your problem is not purely a side-effect of not passing any environment
variables ? (your tasks themselves maybe need PATH or LD_LIBRARY_PATH ?)
- Pierre
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45104/#review124697
---
On mars 22, 2016, 9:50 matin, Pierre Cheynier wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45104/
> ---
>
> (Updated mars 22, 2016, 9:50 matin)
>
>
> Review request for Aurora, Joshua Cohen and Bill Farner.
>
>
> Repository: aurora
>
>
> Description
> ---
>
> When using `--executor_environment_variables` without explicitely
> passing LD_LIBRARY_PATH, `sys.executable` returns an empty string
> resulting in a '[Errno 13] Permission denied' error for every launched
> task.
>
> Moreover, it seems that this feature is coming in 0.30: "Executors no
> longer inherit environment variables from the agent".
>
> This patch partially revert back 07ce21d where chmod_x method was
> removed in favor of using sys.executable.
>
>
> Diffs
> -
>
> src/main/python/apache/aurora/executor/thermos_task_runner.py
> 3896e3841562600379705dbf78a6f62728246348
>
> Diff: https://reviews.apache.org/r/45104/diff/
>
>
> Testing
> ---
>
> Make Aurora executor run on CentOS7, while running mesos agent with
> `--executor_environment_variables` option and no excplicit $PATH set.
>
>
> Thanks,
>
> Pierre Cheynier
>
>
Review Request 45298: Remove some dependencies linked to docker usage
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45298/ --- Review request for Aurora. Repository: aurora-packaging Description --- An operator should be able to build Aurora on his platform without relying on Docker. Even if this is an interesting way of doing it, it sometimes introduce missed (build-)dependencies (in this case openssl) and context-specific behaviour (here, the way pants.ini was added to the build context) Diffs - README.md 3a7cf45034b7896c23588fed83176468ca627ebc builder/deb/debian-jessie/Dockerfile 63e89deed9a411b1859ba28ea2572ef4a210da1f builder/deb/debian-jessie/pants.ini 2ff2d5eb6128d4be3e15c0488d8d23bed81a8d5b builder/deb/ubuntu-trusty/Dockerfile 52739100c9593292a7aea2459412c8e49f0155a4 builder/deb/ubuntu-trusty/pants.ini builder/rpm/centos-7/Dockerfile a4d9dfcee5637d2c31d0c2b63a9ebcf04e8d builder/rpm/centos-7/pants.ini specs/debian/control a30e987e4423c611c86518a8f5e3e68ee996982a specs/debian/rules 974fddfda6da04686daae691fa32c5b4c11a0b5d specs/rpm/aurora.spec 61e7d146108ae7dd5e129d8288a05773c2659d25 Diff: https://reviews.apache.org/r/45298/diff/ Testing --- Able to build on CentOS 6 using mock. Able to build on Debian. Limitation = need the patched version of Gradle. Still able to build using the current Docker build system Thanks, Pierre Cheynier
Re: Review Request 45212: Remove hard dependency on a specific mesos-version
> On March 23, 2016, 2:31 p.m., Stephan Erb wrote: > > Thinking out loud here, so please comment: > > > > We could move to a mode where we build against a specific Mesos version, > > and recommend that version for deployment, but leave it up to the cluster > > operator to select and deploy a compatible Mesos version. > > > > This would enable the following usecase: > > > > * Aurora 0.12 currently depends on Mesos 0.25 > > * By the given compatability, cluster operators can safely update to Mesos > > 0.26. > > * Instead of releasing Aurora with Mesos 0.26 as currently planned, we > > release a version build against 0.27. This one will be backwards > > compatible, and will therefore work with the deployed Mesos 0.26. > > * Cluster operators can then safely update to Mesos 0.27 and Mesos 0.28 > > > > This should make it easier for us to keep up with the Mesos release train... > > John Sirois wrote: > Leaving off the package dependency (which we already do by mistake for > the aurora-executor deb) certainly has a maintenance appeal, since we already > need to maintain the install guide in the aurora repo, which could contain or > point to a compatibility matrix we maintain. If we do go with no explicit > mesos dependency in our binary packages (or a floating one), I think its > important a compatibility matrix be prominent in the install docs since the > questions and install problems will happen. But if we do maintain a > compatibility matrix we could ~just as easily be adding the compatibility > constraints into the package dependencies too and avoiding a wider swath of > bug reports / questions. > > I've widened the reviewer scope a bit to gather more opions here. This > may need a dev@ thread. > > Stephan Erb wrote: > It's not a real bug for the executor. For the executor we boundle the > `mesos.native` wheel in the pex-file. > > Bill Farner wrote: > I'm pro min-version. Without true semantic versioning in mesos, i'm > doubtful we could be accurate with an upper bound. > > Zameer Manji wrote: > I am pro min and max version. I believe the range that John proposes > above is the best way to go. Mesos only guarantees -1/+1 and we should > reflect that in the packaging. In my experience I have been bit by > incompatabilities that can exist beyond +1/-1 and they were very difficult to > debug. > > A more sophisticated cluster operator that knows what they are doing can > use the facilities of rpm/deb to force a version of the package beyond our > constraints if neeed. > > I'm not in favor of a compatability matrix, it seems like it would be a > lot of work to maintain and test out, I suggest just rolling with what the > Mesos project recomends until a better story comes out. > > Maxim Khutornenko wrote: > -1 to this patch. I don't see any real benefits here and certainly would > not want to _guess_ the compat matrix. >From my point of view, this compatibility (or at least approval) matrix could >be simply written in the release note at each release. For ex: "0.12.0 supports only 0.25+ (enforced by constraint) and was tested on: 0.25/0.26/0.27". No guarantee on the newer versions, but it avoid everyone that want to test it to patch the aurora-packaging project. Operators on such big perimeters should be aware of the possible impacts in upgrading one of the building block of their infrastructure. - Pierre ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45212/#review125027 --- On March 23, 2016, 2:56 p.m., Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45212/ > --- > > (Updated March 23, 2016, 2:56 p.m.) > > > Review request for Aurora, Jake Farrell, John Sirois, Stephan Erb, Bill > Farner, and Zameer Manji. > > > Repository: aurora-packaging > > > Description > --- > > We should consider MESOS_VERSION as the minimal requirement to install > the current Aurora version instead of enforce a specific Mesos version. > > > Diffs > - > > specs/rpm/aurora.spec 61e7d146108ae7dd5e129d8288a05773c2659d25 > > Diff: https://reviews.apache.org/r/45212/diff/ > > > Testing > --- > > Install Aurora through the RPM built with aurora-packaging on a Mesos 0.27 > running install. > > > Thanks, > > Pierre Cheynier > >
Review Request 45212: Remove hard dependency on a specific mesos-version
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45212/ --- Review request for Aurora. Repository: aurora-packaging Description --- We should consider MESOS_VERSION as the minimal requirement to install the current Aurora version instead of enforce a specific Mesos version. Diffs - specs/rpm/aurora.spec 61e7d146108ae7dd5e129d8288a05773c2659d25 Diff: https://reviews.apache.org/r/45212/diff/ Testing --- Install Aurora through the RPM built with aurora-packaging on a Mesos 0.27 running install. Thanks, Pierre Cheynier
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/ --- (Updated March 22, 2016, 9:50 a.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Repository: aurora Description --- When using `--executor_environment_variables` without explicitely passing LD_LIBRARY_PATH, `sys.executable` returns an empty string resulting in a '[Errno 13] Permission denied' error for every launched task. Moreover, it seems that this feature is coming in 0.30: "Executors no longer inherit environment variables from the agent". This patch partially revert back 07ce21d where chmod_x method was removed in favor of using sys.executable. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 3896e3841562600379705dbf78a6f62728246348 Diff: https://reviews.apache.org/r/45104/diff/ Testing (updated) --- Make Aurora executor run on CentOS7, while running mesos agent with `--executor_environment_variables` option and no excplicit $PATH set. Thanks, Pierre Cheynier
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
> On March 22, 2016, 1:02 a.m., Joshua Cohen wrote:
> > Sorry for the delay on this. After you filed the pull request, I
> > investigated a bit what will be required once Mesos 0.30.0 lands:
> > https://issues.apache.org/jira/browse/AURORA-1632. I think the problem goes
> > beyond the failure to find `sys.executable` when $PATH is not set. As even
> > after switching back to chmod+x on the runner, the task failed further down
> > the stack.
> >
> > I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which
> > should allow `sys.executable` to continue working and will allow any tasks
> > users have running which have come to rely on Thermos setting it for them
> > to behave as expected. The problem is, I haven't had time to figure out
> > what we should set $PATH to yet ;) (anyone have any thoughts?).
> >
> > I know this is probably more info than you bargained for when you opened
> > what seemed like a simple pull request. I'm not opposed to accepting this
> > patch (with a TODO to restore `sys.executable` when we figure out what to
> > do about setting $PATH) if it unblocks your use case, but can you confirm
> > that you're actually able to run the Mesos agent with
> > `--executor_environment_variables='{}'` and still launch tasks?
Hi,
Sorry for my delay on opening this review on apache.org.
Actually, I started to use Aurora and faced this issue cause my setup use
`--executor_environment_variables`.
I first investigated in the Python default setup on CentOS, the pants/pex build
system and then the differences between the vagrant box provided in the repo
and mine before discovering that this is the origin of the issue.
I admit this is a simple fix, I just tried to understand the reason of using
this mechanism and found that it was changed 2 years ago.
In 0.27.2 it works well by using this patch (I'm now able to run something).
- Pierre
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45104/#review124697
---
On March 21, 2016, 1:21 p.m., Pierre Cheynier wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45104/
> ---
>
> (Updated March 21, 2016, 1:21 p.m.)
>
>
> Review request for Aurora, Joshua Cohen and Bill Farner.
>
>
> Repository: aurora
>
>
> Description
> ---
>
> When using `--executor_environment_variables` without explicitely
> passing LD_LIBRARY_PATH, `sys.executable` returns an empty string
> resulting in a '[Errno 13] Permission denied' error for every launched
> task.
>
> Moreover, it seems that this feature is coming in 0.30: "Executors no
> longer inherit environment variables from the agent".
>
> This patch partially revert back 07ce21d where chmod_x method was
> removed in favor of using sys.executable.
>
>
> Diffs
> -
>
> src/main/python/apache/aurora/executor/thermos_task_runner.py
> 3896e3841562600379705dbf78a6f62728246348
>
> Diff: https://reviews.apache.org/r/45104/diff/
>
>
> Testing
> ---
>
> Make Aurora run on CentOS7
>
>
> Thanks,
>
> Pierre Cheynier
>
>
Review Request 45104: Use chmod+x to make termos_runner.pex executable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/ --- Review request for Aurora. Repository: aurora Description --- When using `--executor_environment_variables` without explicitely passing LD_LIBRARY_PATH, `sys.executable` returns an empty string resulting in a '[Errno 13] Permission denied' error for every launched task. Moreover, it seems that this feature is coming in 0.30: "Executors no longer inherit environment variables from the agent". This patch partially revert back 07ce21d where chmod_x method was removed in favor of using sys.executable. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 3896e3841562600379705dbf78a6f62728246348 Diff: https://reviews.apache.org/r/45104/diff/ Testing --- Make Aurora run on CentOS7 Thanks, Pierre Cheynier
