Re: Propose requiring Python 3.4 or later for building glibc.

[Carlos O'Donell]

On 10/24/18 4:37 PM, Joseph Myers wrote:
> On Wed, 24 Oct 2018, Carlos O'Donell wrote:
> 
>>> Carlos, do you agree we have consensus on the Python 3.4 requirement (the 
>>> patch  to add 
>>> that requirement still needs to be reviewed)?
>>  
>> I agree we have consensus and that the distribution maintainer are
>> sufficiently aware of the change that they will have time to adjust
>> before the release on February 1st.
> 
> Thanks.  I've now committed the patch to use gen-libm-test.py to generate 
> the ulps table for the manual (which the above referenced patch builds 
> on).

I'm glad to see this move forward. Thank you very much for doing this
work.

-- 
Cheers,
Carlos.

Re: Propose requiring Python 3.4 or later for building glibc.

[Joseph Myers]

On Wed, 24 Oct 2018, Carlos O'Donell wrote:

> > Carlos, do you agree we have consensus on the Python 3.4 requirement (the 
> > patch  to add 
> > that requirement still needs to be reviewed)?
>  
> I agree we have consensus and that the distribution maintainer are
> sufficiently aware of the change that they will have time to adjust
> before the release on February 1st.

Thanks.  I've now committed the patch to use gen-libm-test.py to generate 
the ulps table for the manual (which the above referenced patch builds 
on).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Propose requiring Python 3.4 or later for building glibc.

[Carlos O'Donell]

On 10/24/18 1:43 PM, Joseph Myers wrote:
> On Wed, 24 Oct 2018, Joseph Myers wrote:
> 
>> On Mon, 22 Oct 2018, Helmut Grohne wrote:
>>
>>> Daniel Schepler is working on a native bootstrap approach. As far as I
>>> understand, he natively bootstraps Debian from non-Debian (same
>>> processor architecure and kernel). I expect that his work will be
>>> impacted by the proposed change. I've added him to Cc to let him speak
>>> up.
>>
>> Daniel, any comments?  I believe we have all the other distribution 
>> responses required for reaching consensus on this issue.
> 
> My attention has been drawn to a reply at 
>  that didn't 
> go to libc-alpha.
> 
> Carlos, do you agree we have consensus on the Python 3.4 requirement (the 
> patch  to add 
> that requirement still needs to be reviewed)?
 
I agree we have consensus and that the distribution maintainer are
sufficiently aware of the change that they will have time to adjust
before the release on February 1st.

-- 
Cheers,
Carlos.

Re: Propose requiring Python 3.4 or later for building glibc.

[Joseph Myers]

On Wed, 24 Oct 2018, Joseph Myers wrote:

> On Mon, 22 Oct 2018, Helmut Grohne wrote:
> 
> > Daniel Schepler is working on a native bootstrap approach. As far as I
> > understand, he natively bootstraps Debian from non-Debian (same
> > processor architecure and kernel). I expect that his work will be
> > impacted by the proposed change. I've added him to Cc to let him speak
> > up.
> 
> Daniel, any comments?  I believe we have all the other distribution 
> responses required for reaching consensus on this issue.

My attention has been drawn to a reply at 
 that didn't 
go to libc-alpha.

Carlos, do you agree we have consensus on the Python 3.4 requirement (the 
patch  to add 
that requirement still needs to be reviewed)?

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Propose requiring Python 3.4 or later for building glibc.

[Joseph Myers]

On Mon, 22 Oct 2018, Helmut Grohne wrote:

> Daniel Schepler is working on a native bootstrap approach. As far as I
> understand, he natively bootstraps Debian from non-Debian (same
> processor architecure and kernel). I expect that his work will be
> impacted by the proposed change. I've added him to Cc to let him speak
> up.

Daniel, any comments?  I believe we have all the other distribution 
responses required for reaching consensus on this issue.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Propose requiring Python 3.4 or later for building glibc.

[Helmut Grohne]

On Mon, Oct 22, 2018 at 11:38:15AM -0400, Carlos O'Donell wrote:
> It is possible that the build *and* host require python 3.4.
> 
> The reason being that when cross-testing glibc with the test-wrapper-env
> script the build system may execute a command on the host system to 
> run python (which is usually implemented as a ssh to a target system
> with a shared mounted filesystem).
> 
> There are at least several pretty-printing tests which use python, and
> require PExpect, and those run on the host during testing via the
> test-wrapper-env abstraction.
> 
> I've started a new thread of discussion here, but in general my expectation
> has always been that host and build environments need the same set of tools.
> 
> https://www.sourceware.org/ml/libc-alpha/2018-10/msg00395.html

I should have thought of testing indeed. In Debian, we tend to turn
testing off completely for a bootstrap and then rebuild the world (using
the bootstraped packages) with testing enabled. This is done, because it
removes a pile of dependencies and makes the problem a bit more
manageable. So as long as tests can be disabled (preferably without
changing the build result in terms of reproducible builds[1]), the cross
bootstrap won't be impacted.

In any case, requiring Python 3 for testing does not seem to be a new
thing. The dependency is already there and it is not causing problems
now. So from a cross bootstrap pov, I'm fine here.

Helmut

[1] We also use reproducible builds to validate cross buildt packages
against natively built (with tests enabled) ones.

Re: Propose requiring Python 3.4 or later for building glibc.

[Daniel Schepler]

On Mon, Oct 22, 2018 at 7:15 AM Helmut Grohne  wrote:
> Daniel Schepler is working on a native bootstrap approach. As far as I
> understand, he natively bootstraps Debian from non-Debian (same
> processor architecure and kernel). I expect that his work will be
> impacted by the proposed change. I've added him to Cc to let him speak
> up.

That shouldn't be an issue - already, Python 3 tends to come
significantly earlier than the rebuild of glibc in the process.

(On the other hand, I don't see anybody from Linux From Scratch on the
distribution list - and I'd expect they would have to add Python 3 to
the host system dependencies, and also add it to the initial bootstrap
system.)
-- 
Daniel Schepler

Re: Propose requiring Python 3.4 or later for building glibc.

[Joseph Myers]

On Mon, 22 Oct 2018, Carlos O'Donell wrote:

> There are at least several pretty-printing tests which use python, and
> require PExpect, and those run on the host during testing via the
> test-wrapper-env abstraction.

The normal case for tests written using Python is that it runs on the 
build system.  The pretty printers tests are very much an exception 
(they're also an exception in that they use PExpect, rather than the norm 
of being limited to the standad library).

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Propose requiring Python 3.4 or later for building glibc.

[Carlos O'Donell]

On 10/22/18 10:15 AM, Helmut Grohne wrote:
> On Mon, Oct 22, 2018 at 11:51:25AM +0200, Aurelien Jarno wrote:
>> On 2018-10-19 09:47, Carlos O'Donell wrote:
>>> This proposal is to being circulated to all the distribution
>>> maintainers to gain their acceptance surrounding the use of
>>> python 3.4 or greater for building glibc.
>>>
>>> There has been concern expressed that requiring python 3.4
>>> or greater for the bootstrap process will add an additional
>>> tool the the bootstrap, and specifically a tool that may not
>>> be available on older distributions.
>>>
>>> Python is already mostly available for distributions because
>>> of the integration into key OS components. Python can be 
>>> built on older distributions, and on older distributions you
>>> already have to build a lot of things to compile glibc (like
>>> a newer gcc, and binutils).
>>>
>>> The question today is:
>>>
>>> * Is it OK to require python 3.4 or later to build glibc?
>>
>> [...]
>>
>>> Aurelian,
>>>
>>> Any input from Debian?
>>
>> From the Debian side, the version 3.4 is not an issue at all. The
>> default Python 3 version in unstable (which is relevant for future
>> versions of glibc) is already 3.6.
>>
>> We also need to consider the process of bootstrapping Debian. In that
>> case I believe it should also be fine as Python 3 from the host system
>> can be used. I have added Helmut Grohne in Cc: who is much more aware
>> of the bootstrapping process than me, and can confirm or infirm that.
> 
> I fear we need more precision here. There are multiple ways to bootstrap
> Debian. There is "cross bootstrap" for jumping from one Debian
> architecture to another, which is what I work on. As long as glibc only
> needs to run the build architecture Python, I see no problems here. 

OK, good.

> I assume that you are not trying to integrate with the host architecture
> Python as that would pose interesting problems. Given that Debian's
> cross bootstraps are version locked, I don't see problems with requiring
> recent versions either. In contrast, I'd love to be able to use Python 3
> in Debian's glibc packaging. Also note that Debian's linux packaging
> already requires Python 3 (for the build architecture) for quite a while
> already.

It is possible that the build *and* host require python 3.4.

The reason being that when cross-testing glibc with the test-wrapper-env
script the build system may execute a command on the host system to 
run python (which is usually implemented as a ssh to a target system
with a shared mounted filesystem).

There are at least several pretty-printing tests which use python, and
require PExpect, and those run on the host during testing via the
test-wrapper-env abstraction.

I've started a new thread of discussion here, but in general my expectation
has always been that host and build environments need the same set of tools.

https://www.sourceware.org/ml/libc-alpha/2018-10/msg00395.html

> From a Debian cross bootstrap pov, no problems are to be expected.

OK.

> Daniel Schepler is working on a native bootstrap approach. As far as I
> understand, he natively bootstraps Debian from non-Debian (same
> processor architecure and kernel). I expect that his work will be
> impacted by the proposed change. I've added him to Cc to let him speak
> up.

Yes, in that case you must build a minimal python 3.4 for the host system.

> Requiring Python modules or extensions is another question. Some modules
> (e.g. six) are easy, but ones with dependencies of their own tend to not
> be. That's rooted in the way Debian handles cross architecture
> dependencies unfortunately.

Yes, we plan to limit the set of imports based on a small subset we need
for glibc.

> Thank you for reaching out in advance of the change and for relaying the
> message to non-subscribers.

You are welcome. This kind of change can be very disruptive, so I wanted
to get input from the distribution maintainers.

I'd like to get this resolved in a timely fashion so we can have an answer
for cleanups we'd like to do in glibc 2.29 (Feb 1, 2019 release). Having
an answer sooner makes the cleanup happen earlier and thus is safer overall
for distro testing.

-- 
Cheers,
Carlos.

Re: Propose requiring Python 3.4 or later for building glibc.

[Helmut Grohne]

On Mon, Oct 22, 2018 at 11:51:25AM +0200, Aurelien Jarno wrote:
> On 2018-10-19 09:47, Carlos O'Donell wrote:
> > This proposal is to being circulated to all the distribution
> > maintainers to gain their acceptance surrounding the use of
> > python 3.4 or greater for building glibc.
> > 
> > There has been concern expressed that requiring python 3.4
> > or greater for the bootstrap process will add an additional
> > tool the the bootstrap, and specifically a tool that may not
> > be available on older distributions.
> > 
> > Python is already mostly available for distributions because
> > of the integration into key OS components. Python can be 
> > built on older distributions, and on older distributions you
> > already have to build a lot of things to compile glibc (like
> > a newer gcc, and binutils).
> > 
> > The question today is:
> > 
> > * Is it OK to require python 3.4 or later to build glibc?
> 
> [...]
> 
> > Aurelian,
> > 
> > Any input from Debian?
> 
> From the Debian side, the version 3.4 is not an issue at all. The
> default Python 3 version in unstable (which is relevant for future
> versions of glibc) is already 3.6.
> 
> We also need to consider the process of bootstrapping Debian. In that
> case I believe it should also be fine as Python 3 from the host system
> can be used. I have added Helmut Grohne in Cc: who is much more aware
> of the bootstrapping process than me, and can confirm or infirm that.

I fear we need more precision here. There are multiple ways to bootstrap
Debian. There is "cross bootstrap" for jumping from one Debian
architecture to another, which is what I work on. As long as glibc only
needs to run the build architecture Python, I see no problems here. I
assume that you are not trying to integrate with the host architecture
Python as that would pose interesting problems. Given that Debian's
cross bootstraps are version locked, I don't see problems with requiring
recent versions either. In contrast, I'd love to be able to use Python 3
in Debian's glibc packaging. Also note that Debian's linux packaging
already requires Python 3 (for the build architecture) for quite a while
already.

>From a Debian cross bootstrap pov, no problems are to be expected.

Daniel Schepler is working on a native bootstrap approach. As far as I
understand, he natively bootstraps Debian from non-Debian (same
processor architecure and kernel). I expect that his work will be
impacted by the proposed change. I've added him to Cc to let him speak
up.

Requiring Python modules or extensions is another question. Some modules
(e.g. six) are easy, but ones with dependencies of their own tend to not
be. That's rooted in the way Debian handles cross architecture
dependencies unfortunately.

Thank you for reaching out in advance of the change and for relaying the
message to non-subscribers.

Helmut

Re: Propose requiring Python 3.4 or later for building glibc.

[Aurelien Jarno]

Hi Carlos, hi all,

On 2018-10-19 09:47, Carlos O'Donell wrote:
> This proposal is to being circulated to all the distribution
> maintainers to gain their acceptance surrounding the use of
> python 3.4 or greater for building glibc.
> 
> There has been concern expressed that requiring python 3.4
> or greater for the bootstrap process will add an additional
> tool the the bootstrap, and specifically a tool that may not
> be available on older distributions.
> 
> Python is already mostly available for distributions because
> of the integration into key OS components. Python can be 
> built on older distributions, and on older distributions you
> already have to build a lot of things to compile glibc (like
> a newer gcc, and binutils).
> 
> The question today is:
> 
> * Is it OK to require python 3.4 or later to build glibc?

[...]

> Aurelian,
> 
> Any input from Debian?

From the Debian side, the version 3.4 is not an issue at all. The
default Python 3 version in unstable (which is relevant for future
versions of glibc) is already 3.6.

We also need to consider the process of bootstrapping Debian. In that
case I believe it should also be fine as Python 3 from the host system
can be used. I have added Helmut Grohne in Cc: who is much more aware
of the bootstrapping process than me, and can confirm or infirm that.

Cheers,
Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net


signature.asc
Description: PGP signature

Re: Propose requiring Python 3.4 or later for building glibc.

[Khem Raj]

On Fri, Oct 19, 2018 at 2:47 PM Carlos O'Donell  wrote:
>
> This proposal is to being circulated to all the distribution
> maintainers to gain their acceptance surrounding the use of
> python 3.4 or greater for building glibc.
>
> There has been concern expressed that requiring python 3.4
> or greater for the bootstrap process will add an additional
> tool the the bootstrap, and specifically a tool that may not
> be available on older distributions.
>
> Python is already mostly available for distributions because
> of the integration into key OS components. Python can be
> built on older distributions, and on older distributions you
> already have to build a lot of things to compile glibc (like
> a newer gcc, and binutils).
>
> The question today is:
>
> * Is it OK to require python 3.4 or later to build glibc?
>
> Dmitry Levin has provided input for ALT Linux. Specifically that
> despite objections over the addition of Python the bootstrap, that
> technically speaking ALT Linux should be OK.
>
> Allan,
>
> Any input from Arch Linux?
>
> Aurelian,
>
> Any input from Debian?
>
> Siddhesh,
>
> Any input from Linaro for toolchains being built?
>
> Florian has provided input for RHEL and Fedora, with a statement
> that the distributions have sufficiently new versions of Python 3
> that this should not be a problem.
>
> Andreas,
>
> Any input from Gentoo?
>
> Khem,
>
> Any input from your RDK work?

We already depend on python 3.5 for other reasons
and build our own python3 during cross compile, this
change essentially is therefore fine for us, only difference
is that now we have to build python3-native before glibc
which might add a bit to build time due
to serialization.

>
> Andreas,
>
> Any further input from SUSE? You object on the basis that the addition
> of Python would add to the SUSE bootstrap cycle that is used frequently
> in package build and test. I'm recording your objection here as a
> non-sustained objection (does not block consensus).
>
> Adam,
>
> Any input from Ubuntu?
>
> Tulio,
>
> Any input from IBM for the AT toolchains being built?
>
> Joseph,
>
> Any input from Mentor Graphics?
>
> --
> Cheers,
> Carlos.

Re: Propose requiring Python 3.4 or later for building glibc.

[Andreas K. Huettel]

Am Freitag, 19. Oktober 2018, 15:47:14 CEST schrieb Carlos O'Donell:
> This proposal is to being circulated to all the distribution
> maintainers to gain their acceptance surrounding the use of
> python 3.4 or greater for building glibc.
> 
[...]

> Andreas,
> 
> Any input from Gentoo?
> 

Well, theoretically it's still possible to have a python-2 only Gentoo 
install. In practice that is probably not very useful anymore since more and 
more packages require python-3 (meson, future icu, ...). So I don't really 
mind (and I suspect that some people from our python team will be secretly 
glad about it).

As for python-3 versions, python-3.4 is our "oldest", so no problem.

Cheers, Andreas

-- 
Andreas K. Hüttel
dilfri...@gentoo.org
Gentoo Linux developer
(council, toolchain, perl, libreoffice)


signature.asc
Description: This is a digitally signed message part.

Re: Propose requiring Python 3.4 or later for building glibc.

[Allan McRae]

On 19/10/18 11:47 pm, Carlos O'Donell wrote:
> Allan,
> 
> Any input from Arch Linux?

Arch Linux has no issue with that requirement.

Allan

Re: Propose requiring Python 3.4 or later for building glibc.

[Joseph Myers]

On Fri, 19 Oct 2018, Carlos O'Donell wrote:

> Joseph,
> 
> Any input from Mentor Graphics?

All our builds of current glibc have Python 3.5 available at build time 
(locally built; the Python 3 installation from the OS is 3.4), so are 
unaffected.

-- 
Joseph S. Myers
jos...@codesourcery.com

Re: Propose requiring Python 3.4 or later for building glibc.

[Adam Conrad]

On Fri, Oct 19, 2018 at 09:47:14AM -0400, Carlos O'Donell wrote:
> 
> Adam,
> 
> Any input from Ubuntu?

I think the more tools you involve in any bootstrap process, the more
painful you're going to make it for people bringing up a new arch.

Python isn't exactly lightweight (though there are ways to build just
stdlib and not much else), and intertwining a python bootstrap with
the already tiresome gcc/glibc/etc bootstraps won't fill me with warm
fuzzies.

With the above objection registered, I'll note that for both Debian
and Ubuntu, python3.4 is sufficiently ancient that we'll have no issues
meeting the actual requirement, and for most cases, we shouldn't really
see an issue with the above "oh noes, more bootstrap bits", cause we
can reliably cross bootstrap and ignore the whole issue.

... Adam