Re: [Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

On 15 March 2016 at 04:32, Nick Coghlan  wrote:
> On 15 March 2016 at 10:49, Martin Panter  wrote:
>> One more idea I am considering is to decouple the makefile rules from
>> the main build. So to update the generated files you would have to run
>> a separate command like “make graminit” or “make frozen”. The normal
>> build would never regenerate them; although perhaps it could still
>> result in an error or warning if they appear out of date.
>
> Some of them used to work that way and it's an incredible PITA when
> you actually *are* working on one of the affected bits of the
> interpreter - working on those parts is rare, so if there are special
> cases to remember, you can pretty much guarantee we'll have forgotten
> them by the time we work on that piece again.

Perhaps if we wrapped them all up in a common “make regenerate”
target, so it is only one special case to remember? Maybe you could
include other stuff like “make clinic” in that as well. Or you could
include the special commands in the warning messages.

> However, it would be worth reviewing the explicit dependencies on
> "Makefile" and see whether they could be replaced by dependencies on
> Makefile.pre.in instead. I'm confident that will work just fine for
> the importlib bootstrapping, and I suspect it will work for the other
> pregenerated-and-checked-in files as well.

The problem is not the reference to Makefile. The graminit files do
not depend on Makefile. The bigger problem is that the checked-in
files depend on compiled programs. This is a summary of the current
rules for importlib:

_freeze_importlib.o: _freeze_importlib.c Makefile

_freeze_importlib: _freeze_importlib.o [. . .]
$(LINKCC) [. . .]

importlib_external.h: _bootstrap_external.py _freeze_importlib
_freeze_importlib _bootstrap_external.py importlib_external.h

importlib.h: _bootstrap.py _freeze_importlib
_freeze_importlib _bootstrap.py importlib.h

So importlib.h depends on the _freeze_importlib compiled program (and
only indirectly on Makefile). The makefile says we have to compile
_freeze_importlib before checking if importlib.h is up to date.

Gnu Make has order-only prerequisites
,
which it seems we could abuse to do most of what we want. But (1) I’m
not sure if we can restrict ourselves to Gnu Make, and (2) it is a
horrible hack and would always compile _freeze_importlib even if it is
never run.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Nick Coghlan]

On 15 March 2016 at 10:49, Martin Panter  wrote:
> One more idea I am considering is to decouple the makefile rules from
> the main build. So to update the generated files you would have to run
> a separate command like “make graminit” or “make frozen”. The normal
> build would never regenerate them; although perhaps it could still
> result in an error or warning if they appear out of date.

Some of them used to work that way and it's an incredible PITA when
you actually *are* working on one of the affected bits of the
interpreter - working on those parts is rare, so if there are special
cases to remember, you can pretty much guarantee we'll have forgotten
them by the time we work on that piece again.

However, it would be worth reviewing the explicit dependencies on
"Makefile" and see whether they could be replaced by dependencies on
Makefile.pre.in instead. I'm confident that will work just fine for
the importlib bootstrapping, and I suspect it will work for the other
pregenerated-and-checked-in files as well.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New OpenSSL - has anyone ever looked at (in)compatibility with LibreSSL

[Jim Baker]

I have no vested interest in this, other than the continuing work we have
done to make Jython compatible with OpenSSL's model, warts and all.

But the fact that BoringSSL cleans up the OpenSSL API (
https://boringssl.googlesource.com/boringssl/+/HEAD/PORTING.md), at the
cost of possible backwards breaking API changes looks reasonable. I suppose
there is some risk - perhaps the maintainers will decide that returning 1
should mean OK, but that's not going to happen, is it. The real issue here
is that no direct exposure of BoringSSL to other packages. I don't think
that happens with CPython. (Ironically it happens with Jython, due to how
signed jars poorly interact with shading/Java namespace remapping.)

Maintaining security means dealing with the inevitable churn. Did I mention
Jython's support of Python-compatible SSL? I think I did :p

- Jim

On Mon, Mar 14, 2016 at 6:06 PM, Gregory P. Smith  wrote:

>
> On Mon, Mar 14, 2016 at 4:56 PM Nathaniel Smith  wrote:
>
>> Should people outside google pay attention to boringssl? The first
>> thing it says on the website is:
>>
>> "Although BoringSSL is an open source project, it is not intended for
>> general use, as OpenSSL is. We don’t recommend that third parties
>> depend upon it. Doing so is likely to be frustrating because there are
>> no guarantees of API or ABI stability."
>>
>
> Heh, good point.  I guess not.  :)
>
>
>> On Mon, Mar 14, 2016 at 4:40 PM, Gregory P. Smith 
>> wrote:
>> > Don't forget BoringSSL.
>> >
>> > On Wed, Mar 9, 2016 at 9:30 AM Michael Felt  wrote:
>> >>
>> >> Can look at it. There has been a lot of discussion, iirc, between
>> OpenSSL
>> >> and LibreSSL re: version identification.
>> >> Thx for the reference.
>> >>
>> >>
>> >> On 08-Mar-16 14:55, Hasan Diwan wrote:
>> >>
>> >>
>> >> On 8 March 2016 at 00:49, Michael Felt  wrote:
>> >>>
>> >>> As a relative newcomer I may have missed a long previous discussion
>> re:
>> >>> linking with OpenSSL and/or LibreSSL.
>> >>> In an ideal world this would be rtl linking, i.e., underlying
>> >>> complexities of *SSL libraries are hidden from applications.
>> >>>
>> >>> In short, when I saw this http://bugs.python.org/issue26465 Title:
>> >>> Upgrade OpenSSL shipped with python installers, it reminded me I need
>> to
>> >>> start looking at LibreSSL again - and that, if not already done -
>> might be
>> >>> something "secure" for python as well.
>> >>
>> >>
>> >> According to the libressl website, one of the projects primary goals
>> is to
>> >> remain "backwards-compatible with OpenSSL", which is to say, to either
>> have
>> >> code work without changes or to fail gracefully when it uses the
>> deprecated
>> >> bits. It does seem it ships with OpenBSD. There is an issue open on
>> bugs to
>> >> address whatever incompatibilities remain between LibreSSL and
>> OpenSSL[1].
>> >> Perhaps you might want to take a look at that? -- H
>> >> 1. https://bugs.python.org/issue23177
>> >>>
>> >>>
>> >>> Michael
>> >>> ___
>> >>> Python-Dev mailing list
>> >>> Python-Dev@python.org
>> >>> https://mail.python.org/mailman/listinfo/python-dev
>> >>> Unsubscribe:
>> >>>
>> https://mail.python.org/mailman/options/python-dev/hasan.diwan%40gmail.com
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> OpenPGP: http://hasan.d8u.us/gpg.asc
>> >> Sent from my mobile device
>> >> Envoyé de mon portable
>> >>
>> >>
>> >> ___
>> >> Python-Dev mailing list
>> >> Python-Dev@python.org
>> >> https://mail.python.org/mailman/listinfo/python-dev
>> >> Unsubscribe:
>> >> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>> >
>> >
>> > ___
>> > Python-Dev mailing list
>> > Python-Dev@python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> > https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
>> >
>>
>>
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

On 14 March 2016 at 13:26, R. David Murray  wrote:
> On Mon, 14 Mar 2016 03:04:08 -, "Gregory P. Smith"  
> wrote:
>> On Sun, Mar 13, 2016 at 7:41 PM Martin Panter  wrote:
>> > Include/graminit.h
>> > Python/graminit.c
>> > Python/importlib.h
>> > Python/importlib_external.h
>> >
>> > A question for other Python developers: Why are these generated files
>> > stored in the repository? [. . .]
>>
>> They should not be regenerated every build, if they are, that seems like a
>> bug in the makefile to me (or else the timestamp checks that make does vs
>> how your code checkout happened).

The reason the current Python 3 build regenerates some files, is
because of the makefile prerequisites. For example, Include/graminit.h
currently depends on Parser/pgen, which needs to be compiled for the
native build host.

>> Having them checked in is convenient for
>> cross builds as it is one less thing that needs a build-host-arch build.
>
> [. . .]
> And yes, checking in these platform-independent artifacts is very
> intentional: less to build, fewer external dependencies in the build
> process...you don't need to *have* python to *build* python, which you
> would have to if they were not checked in.

Okay so it sounds like the generated files (more listed in .hgtouch)
have to stay. Reasons given:

* Some need Python to generate them (bootstrap problem)
* Relied on by Windows build system
* General convenience (less build steps, less prerequisites, less
things to go wrong)

One more idea I am considering is to decouple the makefile rules from
the main build. So to update the generated files you would have to run
a separate command like “make graminit” or “make frozen”. The normal
build would never regenerate them; although perhaps it could still
result in an error or warning if they appear out of date.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New OpenSSL - has anyone ever looked at (in)compatibility with LibreSSL

[Gregory P. Smith]

On Mon, Mar 14, 2016 at 4:56 PM Nathaniel Smith  wrote:

> Should people outside google pay attention to boringssl? The first
> thing it says on the website is:
>
> "Although BoringSSL is an open source project, it is not intended for
> general use, as OpenSSL is. We don’t recommend that third parties
> depend upon it. Doing so is likely to be frustrating because there are
> no guarantees of API or ABI stability."
>

Heh, good point.  I guess not.  :)


> On Mon, Mar 14, 2016 at 4:40 PM, Gregory P. Smith  wrote:
> > Don't forget BoringSSL.
> >
> > On Wed, Mar 9, 2016 at 9:30 AM Michael Felt  wrote:
> >>
> >> Can look at it. There has been a lot of discussion, iirc, between
> OpenSSL
> >> and LibreSSL re: version identification.
> >> Thx for the reference.
> >>
> >>
> >> On 08-Mar-16 14:55, Hasan Diwan wrote:
> >>
> >>
> >> On 8 March 2016 at 00:49, Michael Felt  wrote:
> >>>
> >>> As a relative newcomer I may have missed a long previous discussion re:
> >>> linking with OpenSSL and/or LibreSSL.
> >>> In an ideal world this would be rtl linking, i.e., underlying
> >>> complexities of *SSL libraries are hidden from applications.
> >>>
> >>> In short, when I saw this http://bugs.python.org/issue26465 Title:
> >>> Upgrade OpenSSL shipped with python installers, it reminded me I need
> to
> >>> start looking at LibreSSL again - and that, if not already done -
> might be
> >>> something "secure" for python as well.
> >>
> >>
> >> According to the libressl website, one of the projects primary goals is
> to
> >> remain "backwards-compatible with OpenSSL", which is to say, to either
> have
> >> code work without changes or to fail gracefully when it uses the
> deprecated
> >> bits. It does seem it ships with OpenBSD. There is an issue open on
> bugs to
> >> address whatever incompatibilities remain between LibreSSL and
> OpenSSL[1].
> >> Perhaps you might want to take a look at that? -- H
> >> 1. https://bugs.python.org/issue23177
> >>>
> >>>
> >>> Michael
> >>> ___
> >>> Python-Dev mailing list
> >>> Python-Dev@python.org
> >>> https://mail.python.org/mailman/listinfo/python-dev
> >>> Unsubscribe:
> >>>
> https://mail.python.org/mailman/options/python-dev/hasan.diwan%40gmail.com
> >>
> >>
> >>
> >>
> >> --
> >> OpenPGP: http://hasan.d8u.us/gpg.asc
> >> Sent from my mobile device
> >> Envoyé de mon portable
> >>
> >>
> >> ___
> >> Python-Dev mailing list
> >> Python-Dev@python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
> >
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
> >
>
>
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New OpenSSL - has anyone ever looked at (in)compatibility with LibreSSL

[Nathaniel Smith]

Should people outside google pay attention to boringssl? The first
thing it says on the website is:

"Although BoringSSL is an open source project, it is not intended for
general use, as OpenSSL is. We don’t recommend that third parties
depend upon it. Doing so is likely to be frustrating because there are
no guarantees of API or ABI stability."

On Mon, Mar 14, 2016 at 4:40 PM, Gregory P. Smith  wrote:
> Don't forget BoringSSL.
>
> On Wed, Mar 9, 2016 at 9:30 AM Michael Felt  wrote:
>>
>> Can look at it. There has been a lot of discussion, iirc, between OpenSSL
>> and LibreSSL re: version identification.
>> Thx for the reference.
>>
>>
>> On 08-Mar-16 14:55, Hasan Diwan wrote:
>>
>>
>> On 8 March 2016 at 00:49, Michael Felt  wrote:
>>>
>>> As a relative newcomer I may have missed a long previous discussion re:
>>> linking with OpenSSL and/or LibreSSL.
>>> In an ideal world this would be rtl linking, i.e., underlying
>>> complexities of *SSL libraries are hidden from applications.
>>>
>>> In short, when I saw this http://bugs.python.org/issue26465 Title:
>>> Upgrade OpenSSL shipped with python installers, it reminded me I need to
>>> start looking at LibreSSL again - and that, if not already done - might be
>>> something "secure" for python as well.
>>
>>
>> According to the libressl website, one of the projects primary goals is to
>> remain "backwards-compatible with OpenSSL", which is to say, to either have
>> code work without changes or to fail gracefully when it uses the deprecated
>> bits. It does seem it ships with OpenBSD. There is an issue open on bugs to
>> address whatever incompatibilities remain between LibreSSL and OpenSSL[1].
>> Perhaps you might want to take a look at that? -- H
>> 1. https://bugs.python.org/issue23177
>>>
>>>
>>> Michael
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/hasan.diwan%40gmail.com
>>
>>
>>
>>
>> --
>> OpenPGP: http://hasan.d8u.us/gpg.asc
>> Sent from my mobile device
>> Envoyé de mon portable
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
>



-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New OpenSSL - has anyone ever looked at (in)compatibility with LibreSSL

[Gregory P. Smith]

Don't forget BoringSSL.

On Wed, Mar 9, 2016 at 9:30 AM Michael Felt  wrote:

> Can look at it. There has been a lot of discussion, iirc, between OpenSSL
> and LibreSSL re: version identification.
> Thx for the reference.
>
>
> On 08-Mar-16 14:55, Hasan Diwan wrote:
>
>
> On 8 March 2016 at 00:49, Michael Felt  wrote:
>
>> As a relative newcomer I may have missed a long previous discussion re:
>> linking with OpenSSL and/or LibreSSL.
>> In an ideal world this would be rtl linking, i.e., underlying
>> complexities of *SSL libraries are hidden from applications.
>>
>> In short, when I saw this http://bugs.python.org/issue26465 Title:
>> Upgrade OpenSSL shipped with python installers, it reminded me I need to
>> start looking at LibreSSL again - and that, if not already done - might be
>> something "secure" for python as well.
>>
>
> According to the libressl website, one of the projects primary goals is to
> remain "backwards-compatible with OpenSSL", which is to say, to either
> have code work without changes or to fail gracefully when it uses the
> deprecated bits. It does seem it ships with OpenBSD. There is an issue open
> on bugs to address whatever incompatibilities remain between LibreSSL and
> OpenSSL[1]. Perhaps you might want to take a look at that? -- H
> 1. https://bugs.python.org/issue23177
>
>>
>> Michael
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/hasan.diwan%40gmail.com
>>
>
>
>
> --
> OpenPGP: http://hasan.d8u.us/gpg.asc
> Sent from my mobile device
> Envoyé de mon portable
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

2016-02-12 14:31 GMT+01:00 M.-A. Lemburg :
>>> If your program has bugs, you can use a debug build of Python 3.5 to
>>> detect misusage of the API.
>
> Yes, but people don't necessarily do this, e.g. I have
> for a very long time ignored debug builds completely
> and when I started to try them, I found that some of the
> things I had been doing with e.g. free list implementations
> did not work in debug builds.

I just added support for debug hooks on Python memory allocators on
Python compiled in *release* mode. Set the environment variable
PYTHONMALLOC to debug to try with Python 3.6.

I added a check on PyObject_Malloc() debug hook to ensure that the
function is called with the GIL held. I opened an issue to add a
similar check on PyMem_Malloc():
https://bugs.python.org/issue26563


> Yes, but those are part of the stdlib. You'd need to check
> a few C extensions which are not tested as part of the stdlib,
> e.g. numpy, scipy, lxml, pillow, etc. (esp. ones which implement custom
> types in C since these will often need the memory management
> APIs).
>
> It may also be a good idea to check wrapper generators such
> as cython, swig, cffi, etc.

I ran the test suite of numpy, lxml, Pillow and cryptography (used cffi).

I found a bug in numpy. numpy calls PyMem_Malloc() without holding the GIL:
https://github.com/numpy/numpy/pull/7404

Except of this bug, all other tests pass with PyMem_Malloc() using
pymalloc and all debug checks.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

2016-03-09 18:54 GMT+01:00 Brett Cannon :
>>> https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks
>>
>> The main advantage of this variable is that you don't have to
>> recompile Python in debug mode to benefit of these checks.
>
> I just wanted to say this all sounds awesome! Thanks for all the hard work
> on making our memory management story easier to work with, Victor.

You're welcome. I pushed my patch adding PYTHONMALLOC environment variable:
https://docs.python.org/dev/whatsnew/3.6.html#pythonmalloc-environment-variable

Please test PYTHONMALLOC=debug and PYTHONMALLOC=malloc with your
favorite application.

I also adjusted code (like code handling PYTHONMALLOCSTATS env var) to
be able to use debug checks in all cases. For example, debug hooks are
now also installed by default when Python is configured in debug mode
without pymalloc support.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Xavier de Gaye]

On 03/14/2016 05:34 PM, Xavier de Gaye wrote:
> Changeset c2a53aa27cad [1] was commited in issue 22359 [2] to remove incorrect
> uses of recursive make.  The changeset added executable binaries as
> prerequisites to the existing rules (Python/importlib.h and $(GRAMMAR_H)).
> This broke cross-compilation:
> * the executables do not exist and must be cross-compiled
> * then the Python/importlib.h or $(GRAMMAR_H) target recipes must be run since
>the prerequisites are newer
> * but the executables cannot run on the build system
>
> Actually the files need not be re-generated as their timestamps have been
> setup for that purpose with 'make touch'. So a solution to the problem
> introduced by this changeset when cross-compiling could be to remove the
> binaries as prerequisites of these rules and include the recipe of their
> corresponding rules, the one used to build the executable, into the recipes of
> the original rule.  Also IMHO the Programs/_freeze_importlib.c can be used
> instead of Programs/_freeze_importlib.o as a prerequisite in the
> Python/importlib.h rule.
>
> [1] https://hg.python.org/cpython/rev/c2a53aa27cad/
> [2] http://bugs.python.org/issue22359


The pgen dependencies are lost when following my previous suggestion, which is
wrong.  I have uploaded a patch at issue 22359 that uses a conditional to 
change the
rules, based on whether a cross-compilation is being run.

Xavier

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Xavier de Gaye]

On 03/14/2016 02:26 PM, R. David Murray wrote:
>
> The repo-timestamp problem is addressed by the 'make touch' target.
>
> And yes, checking in these platform-independent artifacts is very
> intentional: less to build, fewer external dependencies in the build
> process...you don't need to *have* python to *build* python, which you
> would have to if they were not checked in.


Changeset c2a53aa27cad [1] was commited in issue 22359 [2] to remove incorrect
uses of recursive make.  The changeset added executable binaries as
prerequisites to the existing rules (Python/importlib.h and $(GRAMMAR_H)).
This broke cross-compilation:
* the executables do not exist and must be cross-compiled
* then the Python/importlib.h or $(GRAMMAR_H) target recipes must be run since
  the prerequisites are newer
* but the executables cannot run on the build system

Actually the files need not be re-generated as their timestamps have been
setup for that purpose with 'make touch'. So a solution to the problem
introduced by this changeset when cross-compiling could be to remove the
binaries as prerequisites of these rules and include the recipe of their
corresponding rules, the one used to build the executable, into the recipes of
the original rule.  Also IMHO the Programs/_freeze_importlib.c can be used
instead of Programs/_freeze_importlib.o as a prerequisite in the
Python/importlib.h rule.

[1] https://hg.python.org/cpython/rev/c2a53aa27cad/
[2] http://bugs.python.org/issue22359

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[R. David Murray]

On Mon, 14 Mar 2016 03:04:08 -, "Gregory P. Smith"  wrote:
> On Sun, Mar 13, 2016 at 7:41 PM Martin Panter  wrote:
> 
> > On 13 March 2016 at 01:13, Russell Keith-Magee 
> > wrote:
> > > The patches that I've uploaded to Issue23670 [1] show a full
> > cross-platform
> > > [1] http://bugs.python.org/issue23670
> > > build process. After you apply that patch, the iOS directory contains a
> > > meta-Makefile that manages the build process.
> >
> > Thanks very much for pointing that out. This has helped me understand
> > a lot more things. Only now do I realize that the four files generated
> > by pgen and _freeze_importlib are actually already committed into the
> > Mercurial repository:
> >
> > Include/graminit.h
> > Python/graminit.c
> > Python/importlib.h
> > Python/importlib_external.h
> >
> > A question for other Python developers: Why are these generated files
> > stored in the repository? The graminit ones seem to have been there
> > since forever (1990). It seems the importlib ones were there due to a
> > bootstrapping problem, but now that is solved. Antoine
> >  said he kept them in
> > the repository on purpose, but I want to know why.
> >
> > If we ignore the cross compiling use case, would there be any other
> > consequences of removing these generated files from the repository?
> > E.g. would it affect the Windows build process?
> >
> > I have two possible solutions in mind: either remove the generated
> > files from the repository and always build them, or keep them but do
> > not automatically regenerate them every build. Since they are
> > generated files, not source files, I would prefer to remove them, but
> > I want to know the consequences first.
> >
> 
> They should not be regenerated every build, if they are, that seems like a
> bug in the makefile to me (or else the timestamp checks that make does vs
> how your code checkout happened).  Having them checked in is convenient for
> cross builds as it is one less thing that needs a build-host-arch build.

The repo-timestamp problem is addressed by the 'make touch' target.

And yes, checking in these platform-independent artifacts is very
intentional: less to build, fewer external dependencies in the build
process...you don't need to *have* python to *build* python, which you
would have to if they were not checked in.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Nick Coghlan]

On 14 March 2016 at 13:04, Gregory P. Smith  wrote:
> They should not be regenerated every build, if they are, that seems like a
> bug in the makefile to me (or else the timestamp checks that make does vs
> how your code checkout happened).  Having them checked in is convenient for
> cross builds as it is one less thing that needs a build-host-arch build.

It's also two less things to go wrong for folks just wanting to work
on the 99.9% of CPython that is neither the Grammar file nor
importlib._bootstrap.

I'm trying to remember the problem I was solving in making
freezeimportlib.o explicitly depend on the Makefile (while cursing
past me for not writing it down in the commit message), and I think
the issue was that it wasn't correctly picking up changes to the
builtin module list. If that's right, then fixing the dependency to be
on "$(srcdir)/Makefile.pre.in" instead of the generated Makefile
should keep it from getting confused in the cross-compilation
scenario, and also restore the behaviour of using the checked in copy
rather than rebuilding it just because you ran "./configure".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com