date:20161019

Re: Review Request 52437: Adding support for Ubuntu Xenial packages

2016-10-19 Thread Renan DelValle



> On Oct. 16, 2016, 2:35 p.m., Renan DelValle wrote:
> > Attempted to deploy binaries generated by this on my cluster. Everything 
> > seemed to install fine, but I had issues deploying docker containers. 
> > Thermos crashes with the following error:
> > ```
> > ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version 
> > `GLIBCXX_3.4.20' not found
> > ```
> > I suspect that the GLIBC that thermos is compiled with is one of the newer 
> > ones, causing issues with Docker containers that contain older versions of 
> > it. I'll have to experiment a bit to see if this is true.
> 
> Renan DelValle wrote:
> Looked into his and the take away is that if Mesos is compiled with a 
> version of GLIBC, we can't compile an egg for Mesos with a lower version of 
> the GLIBCXX. Containers will just have to have a compatible version of the 
> libstdc++ as a side effect of the way we load thermos into the container
> 
> Zameer Manji wrote:
> To be clear, this is with the DockerContainerizer right?

Oops, forgot to mention that. Yep! This only applies to DockerContainerizer.


- Renan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52437/#review152813
---


On Oct. 12, 2016, 4:17 p.m., Renan DelValle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52437/
> ---
> 
> (Updated Oct. 12, 2016, 4:17 p.m.)
> 
> 
> Review request for Aurora, Dmitriy Shirchenko and Zameer Manji.
> 
> 
> Repository: aurora-packaging
> 
> 
> Description
> ---
> 
> Added builder and test environment for Xenial as well as updated instructions 
> on how to test it.
> 
> Added distributtion to release-candidate script.
> 
> 
> Diffs
> -
> 
>   build-support/release/release-candidate 
> c08c88529ec036989032869198da7a21dcf6ac35 
>   builder/deb/ubuntu-xenial/Dockerfile PRE-CREATION 
>   builder/deb/ubuntu-xenial/build.sh PRE-CREATION 
>   specs/debian/aurora-scheduler.startup.sh PRE-CREATION 
>   specs/ubuntu-xenial/aurora-doc.docs PRE-CREATION 
>   specs/ubuntu-xenial/aurora-doc.examples PRE-CREATION 
>   specs/ubuntu-xenial/aurora-pants.ini PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.default PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.install PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.links PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.postinst PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.service PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.startup.sh PRE-CREATION 
>   specs/ubuntu-xenial/aurora-tools.install PRE-CREATION 
>   specs/ubuntu-xenial/aurora-tools.links PRE-CREATION 
>   specs/ubuntu-xenial/changelog PRE-CREATION 
>   specs/ubuntu-xenial/clusters.json PRE-CREATION 
>   specs/ubuntu-xenial/compat PRE-CREATION 
>   specs/ubuntu-xenial/control PRE-CREATION 
>   specs/ubuntu-xenial/copyright PRE-CREATION 
>   specs/ubuntu-xenial/rules PRE-CREATION 
>   specs/ubuntu-xenial/source/format PRE-CREATION 
>   specs/ubuntu-xenial/thermos.default PRE-CREATION 
>   specs/ubuntu-xenial/thermos.dirs PRE-CREATION 
>   specs/ubuntu-xenial/thermos.install PRE-CREATION 
>   specs/ubuntu-xenial/thermos.links PRE-CREATION 
>   specs/ubuntu-xenial/thermos.service PRE-CREATION 
>   test/deb/ubuntu-xenial/README.md PRE-CREATION 
>   test/deb/ubuntu-xenial/Vagrantfile PRE-CREATION 
>   test/deb/ubuntu-xenial/provision.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52437/diff/
> 
> 
> Testing
> ---
> 
> Created artifacts using the build-artifacts script.
> 
> Brought a vagrant image up, installed all deb files created by the artifacts 
> script, started both aurora-scheduler and thermos services, and launched a 
> sample job.
> 
> 
> Thanks,
> 
> Renan DelValle
> 
>

Re: Review Request 52437: Adding support for Ubuntu Xenial packages

2016-10-19 Thread Zameer Manji



> On Oct. 16, 2016, 2:35 p.m., Renan DelValle wrote:
> > Attempted to deploy binaries generated by this on my cluster. Everything 
> > seemed to install fine, but I had issues deploying docker containers. 
> > Thermos crashes with the following error:
> > ```
> > ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version 
> > `GLIBCXX_3.4.20' not found
> > ```
> > I suspect that the GLIBC that thermos is compiled with is one of the newer 
> > ones, causing issues with Docker containers that contain older versions of 
> > it. I'll have to experiment a bit to see if this is true.
> 
> Renan DelValle wrote:
> Looked into his and the take away is that if Mesos is compiled with a 
> version of GLIBC, we can't compile an egg for Mesos with a lower version of 
> the GLIBCXX. Containers will just have to have a compatible version of the 
> libstdc++ as a side effect of the way we load thermos into the container

To be clear, this is with the DockerContainerizer right?


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52437/#review152813
---


On Oct. 12, 2016, 4:17 p.m., Renan DelValle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52437/
> ---
> 
> (Updated Oct. 12, 2016, 4:17 p.m.)
> 
> 
> Review request for Aurora, Dmitriy Shirchenko and Zameer Manji.
> 
> 
> Repository: aurora-packaging
> 
> 
> Description
> ---
> 
> Added builder and test environment for Xenial as well as updated instructions 
> on how to test it.
> 
> Added distributtion to release-candidate script.
> 
> 
> Diffs
> -
> 
>   build-support/release/release-candidate 
> c08c88529ec036989032869198da7a21dcf6ac35 
>   builder/deb/ubuntu-xenial/Dockerfile PRE-CREATION 
>   builder/deb/ubuntu-xenial/build.sh PRE-CREATION 
>   specs/debian/aurora-scheduler.startup.sh PRE-CREATION 
>   specs/ubuntu-xenial/aurora-doc.docs PRE-CREATION 
>   specs/ubuntu-xenial/aurora-doc.examples PRE-CREATION 
>   specs/ubuntu-xenial/aurora-pants.ini PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.default PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.install PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.links PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.postinst PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.service PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.startup.sh PRE-CREATION 
>   specs/ubuntu-xenial/aurora-tools.install PRE-CREATION 
>   specs/ubuntu-xenial/aurora-tools.links PRE-CREATION 
>   specs/ubuntu-xenial/changelog PRE-CREATION 
>   specs/ubuntu-xenial/clusters.json PRE-CREATION 
>   specs/ubuntu-xenial/compat PRE-CREATION 
>   specs/ubuntu-xenial/control PRE-CREATION 
>   specs/ubuntu-xenial/copyright PRE-CREATION 
>   specs/ubuntu-xenial/rules PRE-CREATION 
>   specs/ubuntu-xenial/source/format PRE-CREATION 
>   specs/ubuntu-xenial/thermos.default PRE-CREATION 
>   specs/ubuntu-xenial/thermos.dirs PRE-CREATION 
>   specs/ubuntu-xenial/thermos.install PRE-CREATION 
>   specs/ubuntu-xenial/thermos.links PRE-CREATION 
>   specs/ubuntu-xenial/thermos.service PRE-CREATION 
>   test/deb/ubuntu-xenial/README.md PRE-CREATION 
>   test/deb/ubuntu-xenial/Vagrantfile PRE-CREATION 
>   test/deb/ubuntu-xenial/provision.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52437/diff/
> 
> 
> Testing
> ---
> 
> Created artifacts using the build-artifacts script.
> 
> Brought a vagrant image up, installed all deb files created by the artifacts 
> script, started both aurora-scheduler and thermos services, and launched a 
> sample job.
> 
> 
> Thanks,
> 
> Renan DelValle
> 
>

Re: Review Request 52437: Adding support for Ubuntu Xenial packages

2016-10-19 Thread Renan DelValle



> On Oct. 16, 2016, 2:35 p.m., Renan DelValle wrote:
> > Attempted to deploy binaries generated by this on my cluster. Everything 
> > seemed to install fine, but I had issues deploying docker containers. 
> > Thermos crashes with the following error:
> > ```
> > ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version 
> > `GLIBCXX_3.4.20' not found
> > ```
> > I suspect that the GLIBC that thermos is compiled with is one of the newer 
> > ones, causing issues with Docker containers that contain older versions of 
> > it. I'll have to experiment a bit to see if this is true.

Looked into his and the take away is that if Mesos is compiled with a version 
of GLIBC, we can't compile an egg for Mesos with a lower version of the 
GLIBCXX. Containers will just have to have a compatible version of the 
libstdc++ as a side effect of the way we load thermos into the container


- Renan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52437/#review152813
---


On Oct. 12, 2016, 4:17 p.m., Renan DelValle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52437/
> ---
> 
> (Updated Oct. 12, 2016, 4:17 p.m.)
> 
> 
> Review request for Aurora, Dmitriy Shirchenko and Zameer Manji.
> 
> 
> Repository: aurora-packaging
> 
> 
> Description
> ---
> 
> Added builder and test environment for Xenial as well as updated instructions 
> on how to test it.
> 
> Added distributtion to release-candidate script.
> 
> 
> Diffs
> -
> 
>   build-support/release/release-candidate 
> c08c88529ec036989032869198da7a21dcf6ac35 
>   builder/deb/ubuntu-xenial/Dockerfile PRE-CREATION 
>   builder/deb/ubuntu-xenial/build.sh PRE-CREATION 
>   specs/debian/aurora-scheduler.startup.sh PRE-CREATION 
>   specs/ubuntu-xenial/aurora-doc.docs PRE-CREATION 
>   specs/ubuntu-xenial/aurora-doc.examples PRE-CREATION 
>   specs/ubuntu-xenial/aurora-pants.ini PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.default PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.install PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.links PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.postinst PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.service PRE-CREATION 
>   specs/ubuntu-xenial/aurora-scheduler.startup.sh PRE-CREATION 
>   specs/ubuntu-xenial/aurora-tools.install PRE-CREATION 
>   specs/ubuntu-xenial/aurora-tools.links PRE-CREATION 
>   specs/ubuntu-xenial/changelog PRE-CREATION 
>   specs/ubuntu-xenial/clusters.json PRE-CREATION 
>   specs/ubuntu-xenial/compat PRE-CREATION 
>   specs/ubuntu-xenial/control PRE-CREATION 
>   specs/ubuntu-xenial/copyright PRE-CREATION 
>   specs/ubuntu-xenial/rules PRE-CREATION 
>   specs/ubuntu-xenial/source/format PRE-CREATION 
>   specs/ubuntu-xenial/thermos.default PRE-CREATION 
>   specs/ubuntu-xenial/thermos.dirs PRE-CREATION 
>   specs/ubuntu-xenial/thermos.install PRE-CREATION 
>   specs/ubuntu-xenial/thermos.links PRE-CREATION 
>   specs/ubuntu-xenial/thermos.service PRE-CREATION 
>   test/deb/ubuntu-xenial/README.md PRE-CREATION 
>   test/deb/ubuntu-xenial/Vagrantfile PRE-CREATION 
>   test/deb/ubuntu-xenial/provision.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52437/diff/
> 
> 
> Testing
> ---
> 
> Created artifacts using the build-artifacts script.
> 
> Brought a vagrant image up, installed all deb files created by the artifacts 
> script, started both aurora-scheduler and thermos services, and launched a 
> sample job.
> 
> 
> Thanks,
> 
> Renan DelValle
> 
>

Re: Review Request 53003: Adding logic to copy network files when using the Mesos containierizer with a Docker image.

2016-10-19 Thread Justin Pinkul



> On Oct. 18, 2016, 11:59 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/executor/common/sandbox.py, lines 308-313
> > 
> >
> > Is this always necessary, or only necessary when filesystem isolation 
> > is used in conjunction with the network/cni isolator? If the latter, does 
> > it make more sense to just configure these as global mounts via the 
> > scheduler's `-global_container_mounts` command line flag, rather than doing 
> > this for everyone where it may not be necessary/desirable?
> > 
> > Alternately, I'm not super familiar w/ CNI, but is it possible to infer 
> > from the TaskInfo whether CNI is enabled (e.g. is NetworkInfo set 
> > somewhere)?
> 
> Justin Pinkul wrote:
> This is always nessisary when using a Docker image with the Mesos 
> containierizer. The reason I brought up the network/cni isolator is that when 
> you are running with a Docker image set as the rootfs this isolator will copy 
> these files in, even if no CNI networks are defined. Since the current 
> Thermos executor is using a volume instead of a rootfs this logic is 
> completely bypassed. It makes sense for this change to be in the executor 
> since it is required for DNS to function properly.
> 
> Pod support can be used as a longer term fix. This will allow us to set 
> the rootfs for processes and the ownership of this logic can return to Mesos.
> 
> Joshua Cohen wrote:
> Gotcha, thanks for clarifying. Given the above, does it make sense to 
> only do this when the container is being launched with a Docker image?
> 
> Justin Pinkul wrote:
> Definitely, I placed the code in `FileSystemImageSandbox` which I believe 
> is only used when using a Docker image with the Mesos containierizer.
> 
> Joshua Cohen wrote:
> `FileSystemImageSandbox` is used for a task launched with any filesystem 
> image, not just a docker image. I.e. it's also currently applicable to AppC 
> images, and will be applicable to OCI images when they land as well.

Oh I see, are these other images mounted as volumes in Mesos? If so they will 
require this copy to configure DNS properly.


- Justin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53003/#review153217
---


On Oct. 18, 2016, 11:41 p.m., Justin Pinkul wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53003/
> ---
> 
> (Updated Oct. 18, 2016, 11:41 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Bugs: AURORA-1798
> https://issues.apache.org/jira/browse/AURORA-1798
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> The networking files /etc/resolv.conf, /etc/hosts and /etc/hostname are now 
> copied into the taskfs when using the Mesos containierizer with a Docker 
> image.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/executor/common/sandbox.py 
> 4a0f3b5094940cc3dad34689a0b004fb33b348a0 
>   src/test/python/apache/aurora/executor/common/test_sandbox.py 
> 41ee884a309e8cc8fedecf19cab2fbc397fcf1dc 
> 
> Diff: https://reviews.apache.org/r/53003/diff/
> 
> 
> Testing
> ---
> 
> Ran unit tests and launched a simple ping Aurora job with and without the 
> change.
> 
> 
> Thanks,
> 
> Justin Pinkul
> 
>

Re: Review Request 53003: Adding logic to copy network files when using the Mesos containierizer with a Docker image.

2016-10-19 Thread Joshua Cohen



> On Oct. 18, 2016, 11:59 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/executor/common/sandbox.py, lines 308-313
> > 
> >
> > Is this always necessary, or only necessary when filesystem isolation 
> > is used in conjunction with the network/cni isolator? If the latter, does 
> > it make more sense to just configure these as global mounts via the 
> > scheduler's `-global_container_mounts` command line flag, rather than doing 
> > this for everyone where it may not be necessary/desirable?
> > 
> > Alternately, I'm not super familiar w/ CNI, but is it possible to infer 
> > from the TaskInfo whether CNI is enabled (e.g. is NetworkInfo set 
> > somewhere)?
> 
> Justin Pinkul wrote:
> This is always nessisary when using a Docker image with the Mesos 
> containierizer. The reason I brought up the network/cni isolator is that when 
> you are running with a Docker image set as the rootfs this isolator will copy 
> these files in, even if no CNI networks are defined. Since the current 
> Thermos executor is using a volume instead of a rootfs this logic is 
> completely bypassed. It makes sense for this change to be in the executor 
> since it is required for DNS to function properly.
> 
> Pod support can be used as a longer term fix. This will allow us to set 
> the rootfs for processes and the ownership of this logic can return to Mesos.
> 
> Joshua Cohen wrote:
> Gotcha, thanks for clarifying. Given the above, does it make sense to 
> only do this when the container is being launched with a Docker image?
> 
> Justin Pinkul wrote:
> Definitely, I placed the code in `FileSystemImageSandbox` which I believe 
> is only used when using a Docker image with the Mesos containierizer.

`FileSystemImageSandbox` is used for a task launched with any filesystem image, 
not just a docker image. I.e. it's also currently applicable to AppC images, 
and will be applicable to OCI images when they land as well.


- Joshua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53003/#review153217
---


On Oct. 18, 2016, 11:41 p.m., Justin Pinkul wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53003/
> ---
> 
> (Updated Oct. 18, 2016, 11:41 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Bugs: AURORA-1798
> https://issues.apache.org/jira/browse/AURORA-1798
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> The networking files /etc/resolv.conf, /etc/hosts and /etc/hostname are now 
> copied into the taskfs when using the Mesos containierizer with a Docker 
> image.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/executor/common/sandbox.py 
> 4a0f3b5094940cc3dad34689a0b004fb33b348a0 
>   src/test/python/apache/aurora/executor/common/test_sandbox.py 
> 41ee884a309e8cc8fedecf19cab2fbc397fcf1dc 
> 
> Diff: https://reviews.apache.org/r/53003/diff/
> 
> 
> Testing
> ---
> 
> Ran unit tests and launched a simple ping Aurora job with and without the 
> change.
> 
> 
> Thanks,
> 
> Justin Pinkul
> 
>

Re: Review Request 53003: Adding logic to copy network files when using the Mesos containierizer with a Docker image.

2016-10-19 Thread Justin Pinkul



> On Oct. 18, 2016, 11:59 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/executor/common/sandbox.py, lines 308-313
> > 
> >
> > Is this always necessary, or only necessary when filesystem isolation 
> > is used in conjunction with the network/cni isolator? If the latter, does 
> > it make more sense to just configure these as global mounts via the 
> > scheduler's `-global_container_mounts` command line flag, rather than doing 
> > this for everyone where it may not be necessary/desirable?
> > 
> > Alternately, I'm not super familiar w/ CNI, but is it possible to infer 
> > from the TaskInfo whether CNI is enabled (e.g. is NetworkInfo set 
> > somewhere)?
> 
> Justin Pinkul wrote:
> This is always nessisary when using a Docker image with the Mesos 
> containierizer. The reason I brought up the network/cni isolator is that when 
> you are running with a Docker image set as the rootfs this isolator will copy 
> these files in, even if no CNI networks are defined. Since the current 
> Thermos executor is using a volume instead of a rootfs this logic is 
> completely bypassed. It makes sense for this change to be in the executor 
> since it is required for DNS to function properly.
> 
> Pod support can be used as a longer term fix. This will allow us to set 
> the rootfs for processes and the ownership of this logic can return to Mesos.
> 
> Joshua Cohen wrote:
> Gotcha, thanks for clarifying. Given the above, does it make sense to 
> only do this when the container is being launched with a Docker image?

Definitely, I placed the code in `FileSystemImageSandbox` which I believe is 
only used when using a Docker image with the Mesos containierizer.


- Justin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53003/#review153217
---


On Oct. 18, 2016, 11:41 p.m., Justin Pinkul wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53003/
> ---
> 
> (Updated Oct. 18, 2016, 11:41 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Bugs: AURORA-1798
> https://issues.apache.org/jira/browse/AURORA-1798
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> The networking files /etc/resolv.conf, /etc/hosts and /etc/hostname are now 
> copied into the taskfs when using the Mesos containierizer with a Docker 
> image.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/executor/common/sandbox.py 
> 4a0f3b5094940cc3dad34689a0b004fb33b348a0 
>   src/test/python/apache/aurora/executor/common/test_sandbox.py 
> 41ee884a309e8cc8fedecf19cab2fbc397fcf1dc 
> 
> Diff: https://reviews.apache.org/r/53003/diff/
> 
> 
> Testing
> ---
> 
> Ran unit tests and launched a simple ping Aurora job with and without the 
> change.
> 
> 
> Thanks,
> 
> Justin Pinkul
> 
>

Re: Review Request 52437: Adding support for Ubuntu Xenial packages

Re: Review Request 52437: Adding support for Ubuntu Xenial packages

Re: Review Request 52437: Adding support for Ubuntu Xenial packages

Re: Review Request 53003: Adding logic to copy network files when using the Mesos containierizer with a Docker image.

Re: Review Request 53003: Adding logic to copy network files when using the Mesos containierizer with a Docker image.

Re: Review Request 53003: Adding logic to copy network files when using the Mesos containierizer with a Docker image.

6 matches

Site Navigation

Mail list logo

Footer information