[Distutils] Re: The base docker images for manylinux appear
Sent from Yahoo Mail on Android On Thu, Feb 27, 2020 at 7:42 AM, Sylvain Corlay wrote: -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/GPSS36SBBE46MZJ5TVALMIA6OU44LS4V/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/GPSS36SBBE46MZJ5TVALMIA6OU44LS4V/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/OO6FMCWVGQKA4UB6OUTYBPMLTKEEYH23/
[Distutils] Re: The base docker images for manylinux appear to be private
No problem. It is a bit confusing that it's tagged as though it's going to be uploaded. On Thu, Feb 27, 2020, at 12:41 PM, Sylvain Corlay wrote: > Thanks Thomas, I was mistaken by the first line of the docker image which > includes a full URL on quay.io > > > FROM quay.io/pypa/manylinux2010_centos-6-no-vsyscall > > We will build this image ourself. > > Best, > > > On Wed, Feb 26, 2020 at 10:38 AM Thomas Kluyver wrote: >> __ >> To be precise, these are not uploaded at all - there's no hidden repository >> on quay.io AFAICT, and I'm an owner of the pypa team there. >> >> The CI on the manylinux repo always builds both stages one after the other. >> They're split as an implementation detail - if I recall correctly, doing >> everything in one Dockerfile tended to hit a timeout on Travis. So I'm >> inclined to say that the intermediate image is not any kind of a public >> interface, and you should build it from the git repository if you need it. >> >> Thomas >> >> On Wed, Feb 26, 2020, at 9:12 AM, Sylvain Corlay wrote: >>> I am fine with there being a specific base image but I think that it should >>> be pullable. >>> >>> On Wed, Feb 26, 2020, 04:06 Wes Turner wrote: On Tue, Feb 25, 2020, 10:03 PM Wes Turner wrote: > > Is there a reason the new manylinux does not just extend centos:6? https://github.com/pypa/manylinux/blob/master/docker/glibc/README.rst : > Summary: Because of > https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html, > this a CentOS 6.10 Docker image that rebuilds glibc without vsyscall is > necessary to reliably run manylinux2010 on 64-bit hosts. This requires > building the image on a system with vsyscall=emulate but allows the > resulting container to run on systems with vsyscall=none or > vsyscall=emulate. > > vsyscall is an antiquated optimization for a small number of > frequently-used system calls. A vsyscall-enabled Linux kernel maps a > read-only page of data and system calls into a process' memory at a > fixed address. These system calls can then be invoked by dereferencing a > function pointers to fixed offsets in that page, saving a relatively > expensive context switch. [1] > > Unfortunately, because the code and its location in memory are fixed and > well-known, the vsyscall mechanism has become a source of gadgets for > ROP attacks (specifically, Sigreturn-Oriented Programs). [2] Linux 3.1 > introduced vsyscall emulation that prevents attackers from jumping into > the middle of the system calls' code at the expense of speed, as well as > the ability to disable it entirely. [3] [4] The vsyscall mechanism could > not be eliminated at the time because glibc versions earlier than 2.14 > contained hard-coded references to the fixed memory address, > specifically in time(2). [5] These segfault when attempting to issue a > vsyscall-optimized system call against a kernel that has disabled it. >>> -- >>> Distutils-SIG mailing list -- distutils-sig@python.org >>> To unsubscribe send an email to distutils-sig-le...@python.org >>> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ >>> Message archived at >>> https://mail.python.org/archives/list/distutils-sig@python.org/message/T6KDLHKSE4DHCVIACR7QQH3XG6RGBRRO/ >>> >> >> -- >> Distutils-SIG mailing list -- distutils-sig@python.org >> To unsubscribe send an email to distutils-sig-le...@python.org >> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ >> Message archived at >> https://mail.python.org/archives/list/distutils-sig@python.org/message/OR3MYEWUCHGHMK6YBJCDBUYIZJ3JJF7P/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/A6P3WE2OW7QR7H7C57EKRZYKHURCGIWN/
[Distutils] Re: The base docker images for manylinux appear to be private
Thanks Thomas, I was mistaken by the first line of the docker image which includes a full URL on quay.io FROM quay.io/pypa/manylinux2010_centos-6-no-vsyscall We will build this image ourself. Best, On Wed, Feb 26, 2020 at 10:38 AM Thomas Kluyver wrote: > To be precise, these are not uploaded at all - there's no hidden > repository on quay.io AFAICT, and I'm an owner of the pypa team there. > > The CI on the manylinux repo always builds both stages one after the > other. They're split as an implementation detail - if I recall correctly, > doing everything in one Dockerfile tended to hit a timeout on Travis. So > I'm inclined to say that the intermediate image is not any kind of a public > interface, and you should build it from the git repository if you need it. > > Thomas > > On Wed, Feb 26, 2020, at 9:12 AM, Sylvain Corlay wrote: > > I am fine with there being a specific base image but I think that it > should be pullable. > > On Wed, Feb 26, 2020, 04:06 Wes Turner wrote: > > > > On Tue, Feb 25, 2020, 10:03 PM Wes Turner wrote: > > > Is there a reason the new manylinux does not just extend centos:6? > > > https://github.com/pypa/manylinux/blob/master/docker/glibc/README.rst : > > > Summary: Because of > https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html, > this a CentOS 6.10 Docker image that rebuilds glibc without vsyscall is > necessary to reliably run manylinux2010 on 64-bit hosts. This requires > building the image on a system with vsyscall=emulate but allows the > resulting container to run on systems with vsyscall=none or > vsyscall=emulate. > > > > vsyscall is an antiquated optimization for a small number of > frequently-used system calls. A vsyscall-enabled Linux kernel maps a > read-only page of data and system calls into a process' memory at a fixed > address. These system calls can then be invoked by dereferencing a function > pointers to fixed offsets in that page, saving a relatively expensive > context switch. [1] > > > > Unfortunately, because the code and its location in memory are fixed and > well-known, the vsyscall mechanism has become a source of gadgets for ROP > attacks (specifically, Sigreturn-Oriented Programs). [2] Linux 3.1 > introduced vsyscall emulation that prevents attackers from jumping into the > middle of the system calls' code at the expense of speed, as well as the > ability to disable it entirely. [3] [4] The vsyscall mechanism could not be > eliminated at the time because glibc versions earlier than 2.14 contained > hard-coded references to the fixed memory address, specifically in time(2). > [5] These segfault when attempting to issue a vsyscall-optimized system > call against a kernel that has disabled it. > > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/T6KDLHKSE4DHCVIACR7QQH3XG6RGBRRO/ > > > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/OR3MYEWUCHGHMK6YBJCDBUYIZJ3JJF7P/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/GPSS36SBBE46MZJ5TVALMIA6OU44LS4V/
[Distutils] Re: The base docker images for manylinux appear to be private
To be precise, these are not uploaded at all - there's no hidden repository on quay.io AFAICT, and I'm an owner of the pypa team there. The CI on the manylinux repo always builds both stages one after the other. They're split as an implementation detail - if I recall correctly, doing everything in one Dockerfile tended to hit a timeout on Travis. So I'm inclined to say that the intermediate image is not any kind of a public interface, and you should build it from the git repository if you need it. Thomas On Wed, Feb 26, 2020, at 9:12 AM, Sylvain Corlay wrote: > I am fine with there being a specific base image but I think that it should > be pullable. > > On Wed, Feb 26, 2020, 04:06 Wes Turner wrote: >> >> >> On Tue, Feb 25, 2020, 10:03 PM Wes Turner wrote: >>> >>> Is there a reason the new manylinux does not just extend centos:6? >> >> https://github.com/pypa/manylinux/blob/master/docker/glibc/README.rst : >> >> > Summary: Because of >> > https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html, >> > this a CentOS 6.10 Docker image that rebuilds glibc without vsyscall is >> > necessary to reliably run manylinux2010 on 64-bit hosts. This requires >> > building the image on a system with vsyscall=emulate but allows the >> > resulting container to run on systems with vsyscall=none or >> > vsyscall=emulate. >> > >> > vsyscall is an antiquated optimization for a small number of >> > frequently-used system calls. A vsyscall-enabled Linux kernel maps a >> > read-only page of data and system calls into a process' memory at a fixed >> > address. These system calls can then be invoked by dereferencing a >> > function pointers to fixed offsets in that page, saving a relatively >> > expensive context switch. [1] >> > >> > Unfortunately, because the code and its location in memory are fixed and >> > well-known, the vsyscall mechanism has become a source of gadgets for ROP >> > attacks (specifically, Sigreturn-Oriented Programs). [2] Linux 3.1 >> > introduced vsyscall emulation that prevents attackers from jumping into >> > the middle of the system calls' code at the expense of speed, as well as >> > the ability to disable it entirely. [3] [4] The vsyscall mechanism could >> > not be eliminated at the time because glibc versions earlier than 2.14 >> > contained hard-coded references to the fixed memory address, specifically >> > in time(2). [5] These segfault when attempting to issue a >> > vsyscall-optimized system call against a kernel that has disabled it. > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/T6KDLHKSE4DHCVIACR7QQH3XG6RGBRRO/ > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/OR3MYEWUCHGHMK6YBJCDBUYIZJ3JJF7P/
[Distutils] Re: The base docker images for manylinux appear to be private
I am fine with there being a specific base image but I think that it should be pullable. On Wed, Feb 26, 2020, 04:06 Wes Turner wrote: > > > On Tue, Feb 25, 2020, 10:03 PM Wes Turner wrote: > >> >> Is there a reason the new manylinux does not just extend centos:6? >> > > https://github.com/pypa/manylinux/blob/master/docker/glibc/README.rst : > > > Summary: Because of > https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html, > this a CentOS 6.10 Docker image that rebuilds glibc without vsyscall is > necessary to reliably run manylinux2010 on 64-bit hosts. This requires > building the image on a system with vsyscall=emulate but allows the > resulting container to run on systems with vsyscall=none or > vsyscall=emulate. > > > > vsyscall is an antiquated optimization for a small number of > frequently-used system calls. A vsyscall-enabled Linux kernel maps a > read-only page of data and system calls into a process' memory at a fixed > address. These system calls can then be invoked by dereferencing a function > pointers to fixed offsets in that page, saving a relatively expensive > context switch. [1] > > > > Unfortunately, because the code and its location in memory are fixed and > well-known, the vsyscall mechanism has become a source of gadgets for ROP > attacks (specifically, Sigreturn-Oriented Programs). [2] Linux 3.1 > introduced vsyscall emulation that prevents attackers from jumping into the > middle of the system calls' code at the expense of speed, as well as the > ability to disable it entirely. [3] [4] The vsyscall mechanism could not be > eliminated at the time because glibc versions earlier than 2.14 contained > hard-coded references to the fixed memory address, specifically in time(2). > [5] These segfault when attempting to issue a vsyscall-optimized system > call against a kernel that has disabled it. > -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/T6KDLHKSE4DHCVIACR7QQH3XG6RGBRRO/
[Distutils] Re: The base docker images for manylinux appear to be private
On Tue, Feb 25, 2020, 10:03 PM Wes Turner wrote: > > Is there a reason the new manylinux does not just extend centos:6? > https://github.com/pypa/manylinux/blob/master/docker/glibc/README.rst : > Summary: Because of https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html, this a CentOS 6.10 Docker image that rebuilds glibc without vsyscall is necessary to reliably run manylinux2010 on 64-bit hosts. This requires building the image on a system with vsyscall=emulate but allows the resulting container to run on systems with vsyscall=none or vsyscall=emulate. > > vsyscall is an antiquated optimization for a small number of frequently-used system calls. A vsyscall-enabled Linux kernel maps a read-only page of data and system calls into a process' memory at a fixed address. These system calls can then be invoked by dereferencing a function pointers to fixed offsets in that page, saving a relatively expensive context switch. [1] > > Unfortunately, because the code and its location in memory are fixed and well-known, the vsyscall mechanism has become a source of gadgets for ROP attacks (specifically, Sigreturn-Oriented Programs). [2] Linux 3.1 introduced vsyscall emulation that prevents attackers from jumping into the middle of the system calls' code at the expense of speed, as well as the ability to disable it entirely. [3] [4] The vsyscall mechanism could not be eliminated at the time because glibc versions earlier than 2.14 contained hard-coded references to the fixed memory address, specifically in time(2). [5] These segfault when attempting to issue a vsyscall-optimized system call against a kernel that has disabled it. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/HDNTKOTF2QNNPVGFVHYWP6CXOWN4AXO3/
[Distutils] Re: The base docker images for manylinux appear to be private
I agree. The manylinux docker images should be entirely reproducible from
source.
Is there a reason the new manylinux does not just extend centos:6?
https://github.com/pypa/manylinux/blame/master/docker/Dockerfile-x86_64
https://discuss.python.org/t/manylinux2010-docker-image-now-available/1471
(There are also centos:6 ARM and ARM64 builds:
https://hub.docker.com/_/centos )
FWIW, I recently learned about vmware/tern:
> Tern is an inspection tool to find the metadata of the packages installed
in a container image.
https://github.com/vmware/tern
On Tue, Feb 25, 2020, 6:09 PM Sylvain Corlay
wrote:
> It seems that the base images used for manylinux wheels are not publicly
> available.
>
> For example, https://quay.io/pypa/manylinux2010_centos-6-no-vsyscall gives
> a 403 error ("you are not authorized to view this resource").
>
> I think that it would make sense for reproducibility purposes to make
> these images public. (I have opened an issue about this on the GitHub
> repository: https://github.com/pypa/manylinux/issues/481).
>
> Best,
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/EMU5KOCLFACJ4HIORKBYAEZBBNN5VEPH/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/PY7GCMCF2MGUFXJPLQITCWLJNJ26JG5G/
[Distutils] Re: The base docker images for manylinux appear to be private
Like to no more distutils images. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/7QUK2PUGYFXBBISRSUAVDMDNCEUZRAL3/
