Re: [Python-Dev] Bug in build system for cross-platform builds

2016-03-13 Thread Steve Dower 
Simply removing them would break the Windows build, and it may not be easily 
fixable as the dependency system we use doesn't allow building a project twice. 

Currently we fail the build if a generated file changes and then the user has 
to trigger a second build with the new file, but typically they're fine and the 
first build succeeds.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Martin Panter" 
Sent: ‎3/‎13/‎2016 19:43
To: "Russell Keith-Magee" ; "python-dev" 

Cc: "anto...@python.org" 
Subject: Re: [Python-Dev] Bug in build system for cross-platform builds

On 13 March 2016 at 01:13, Russell Keith-Magee  wrote:
> The patches that I've uploaded to Issue23670 [1] show a full cross-platform
> [1] http://bugs.python.org/issue23670
> build process. After you apply that patch, the iOS directory contains a
> meta-Makefile that manages the build process.

Thanks very much for pointing that out. This has helped me understand
a lot more things. Only now do I realize that the four files generated
by pgen and _freeze_importlib are actually already committed into the
Mercurial repository:

Include/graminit.h
Python/graminit.c
Python/importlib.h
Python/importlib_external.h

A question for other Python developers: Why are these generated files
stored in the repository? The graminit ones seem to have been there
since forever (1990). It seems the importlib ones were there due to a
bootstrapping problem, but now that is solved. Antoine
 said he kept them in
the repository on purpose, but I want to know why.

If we ignore the cross compiling use case, would there be any other
consequences of removing these generated files from the repository?
E.g. would it affect the Windows build process?

I have two possible solutions in mind: either remove the generated
files from the repository and always build them, or keep them but do
not automatically regenerate them every build. Since they are
generated files, not source files, I would prefer to remove them, but
I want to know the consequences first.

> On Sat, Mar 12, 2016 at 8:48 AM, Martin Panter  wrote:
>> On 11 March 2016 at 23:16, Russell Keith-Magee 
>> wrote:
>> >
>> > On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter 
>> > wrote:
>> >> I don't understand. After I run Make, it looks like I get working
>> >> executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
>> >> you mean to install these programs with "make install" or something?
>> >
>> >
>> > Making them part of the installable artefacts would be one option, but
>> > they
>> > don't have to be installed, just preserved.
>>
>> What commands are you running that cause them to not be preserved at
>> the end of the build?
>
>
> I don't know - this is where I hit the end of my understanding of the build
> process. All I know for certain is that 3.4.2 doesn't have this problem;
> 3.5.1 does, and Issue22359 (fixed in [3]) is the source of the problem. A
> subsequent fix [4] introduced an additional problem with _freeze_importlib.
>
> [3] https://hg.python.org/cpython/rev/565b96093ec8
> [4] https://hg.python.org/cpython/rev/02e3bf65b2f8

After my realization about the generated files, I think I can solve
this one. Before the changes you identified, the build process
probably thought the generated files were up to date, so it didn't
need to cross-compile pgen or _freeze_importlib. If the generated file
timestamps were out of date (e.g. depending on the order they are
checked out or extracted), the first native build stage would have
fixed them up.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

2016-03-13 Thread Gregory P. Smith 
On Sun, Mar 13, 2016 at 7:41 PM Martin Panter  wrote:

> On 13 March 2016 at 01:13, Russell Keith-Magee 
> wrote:
> > The patches that I've uploaded to Issue23670 [1] show a full
> cross-platform
> > [1] http://bugs.python.org/issue23670
> > build process. After you apply that patch, the iOS directory contains a
> > meta-Makefile that manages the build process.
>
> Thanks very much for pointing that out. This has helped me understand
> a lot more things. Only now do I realize that the four files generated
> by pgen and _freeze_importlib are actually already committed into the
> Mercurial repository:
>
> Include/graminit.h
> Python/graminit.c
> Python/importlib.h
> Python/importlib_external.h
>
> A question for other Python developers: Why are these generated files
> stored in the repository? The graminit ones seem to have been there
> since forever (1990). It seems the importlib ones were there due to a
> bootstrapping problem, but now that is solved. Antoine
>  said he kept them in
> the repository on purpose, but I want to know why.
>
> If we ignore the cross compiling use case, would there be any other
> consequences of removing these generated files from the repository?
> E.g. would it affect the Windows build process?
>
> I have two possible solutions in mind: either remove the generated
> files from the repository and always build them, or keep them but do
> not automatically regenerate them every build. Since they are
> generated files, not source files, I would prefer to remove them, but
> I want to know the consequences first.
>

They should not be regenerated every build, if they are, that seems like a
bug in the makefile to me (or else the timestamp checks that make does vs
how your code checkout happened).  Having them checked in is convenient for
cross builds as it is one less thing that needs a build-host-arch build.

my 2 cents,
-gps


>
> > On Sat, Mar 12, 2016 at 8:48 AM, Martin Panter 
> wrote:
> >> On 11 March 2016 at 23:16, Russell Keith-Magee  >
> >> wrote:
> >> >
> >> > On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter 
> >> > wrote:
> >> >> I don't understand. After I run Make, it looks like I get working
> >> >> executables leftover at Programs/_freeze_importlib and Parser/pgen.
> Do
> >> >> you mean to install these programs with "make install" or something?
> >> >
> >> >
> >> > Making them part of the installable artefacts would be one option, but
> >> > they
> >> > don't have to be installed, just preserved.
> >>
> >> What commands are you running that cause them to not be preserved at
> >> the end of the build?
> >
> >
> > I don't know - this is where I hit the end of my understanding of the
> build
> > process. All I know for certain is that 3.4.2 doesn't have this problem;
> > 3.5.1 does, and Issue22359 (fixed in [3]) is the source of the problem. A
> > subsequent fix [4] introduced an additional problem with
> _freeze_importlib.
> >
> > [3] https://hg.python.org/cpython/rev/565b96093ec8
> > [4] https://hg.python.org/cpython/rev/02e3bf65b2f8
>
> After my realization about the generated files, I think I can solve
> this one. Before the changes you identified, the build process
> probably thought the generated files were up to date, so it didn't
> need to cross-compile pgen or _freeze_importlib. If the generated file
> timestamps were out of date (e.g. depending on the order they are
> checked out or extracted), the first native build stage would have
> fixed them up.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

2016-03-13 Thread Martin Panter 
On 13 March 2016 at 01:13, Russell Keith-Magee  wrote:
> The patches that I've uploaded to Issue23670 [1] show a full cross-platform
> [1] http://bugs.python.org/issue23670
> build process. After you apply that patch, the iOS directory contains a
> meta-Makefile that manages the build process.

Thanks very much for pointing that out. This has helped me understand
a lot more things. Only now do I realize that the four files generated
by pgen and _freeze_importlib are actually already committed into the
Mercurial repository:

Include/graminit.h
Python/graminit.c
Python/importlib.h
Python/importlib_external.h

A question for other Python developers: Why are these generated files
stored in the repository? The graminit ones seem to have been there
since forever (1990). It seems the importlib ones were there due to a
bootstrapping problem, but now that is solved. Antoine
 said he kept them in
the repository on purpose, but I want to know why.

If we ignore the cross compiling use case, would there be any other
consequences of removing these generated files from the repository?
E.g. would it affect the Windows build process?

I have two possible solutions in mind: either remove the generated
files from the repository and always build them, or keep them but do
not automatically regenerate them every build. Since they are
generated files, not source files, I would prefer to remove them, but
I want to know the consequences first.

> On Sat, Mar 12, 2016 at 8:48 AM, Martin Panter  wrote:
>> On 11 March 2016 at 23:16, Russell Keith-Magee 
>> wrote:
>> >
>> > On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter 
>> > wrote:
>> >> I don't understand. After I run Make, it looks like I get working
>> >> executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
>> >> you mean to install these programs with "make install" or something?
>> >
>> >
>> > Making them part of the installable artefacts would be one option, but
>> > they
>> > don't have to be installed, just preserved.
>>
>> What commands are you running that cause them to not be preserved at
>> the end of the build?
>
>
> I don't know - this is where I hit the end of my understanding of the build
> process. All I know for certain is that 3.4.2 doesn't have this problem;
> 3.5.1 does, and Issue22359 (fixed in [3]) is the source of the problem. A
> subsequent fix [4] introduced an additional problem with _freeze_importlib.
>
> [3] https://hg.python.org/cpython/rev/565b96093ec8
> [4] https://hg.python.org/cpython/rev/02e3bf65b2f8

After my realization about the generated files, I think I can solve
this one. Before the changes you identified, the build process
probably thought the generated files were up to date, so it didn't
need to cross-compile pgen or _freeze_importlib. If the generated file
timestamps were out of date (e.g. depending on the order they are
checked out or extracted), the first native build stage would have
fixed them up.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Question about sys.path and sys.argv and how packaging (may) affects default values

2016-03-13 Thread Michael Felt 

On 2016-03-02 18:45, Thomas Wouters wrote:

On Wed, Mar 2, 2016 at 3:50 AM, Michael Felt  wrote:


Hello all,

1) There are many lists to choose from - if this is the wrong one for
questions about packaging - please forgive me, and point me in the right
direction.


It's hard to say where this belongs best, but python-list would probably
have done as well.



2) Normally, I have just packaged python, and then moved on. However,
recently I have been asked to help with packaging an 'easier to install'
python by people using cloud-init, and more recently people wanting to use
salt-stack (on AIX).

FYI: I have been posting about my complete failure to build 2.7.11 (
http://bugs.python.org/issue26466) - so, what I am testing is based on
2.7.10 - which built easily for me.

Going through the 'base documentation' I saw a reference to both sys.argv
and sys.path. atm, I am looking for an easy way to get the program name
(e.g., /opt/bin/python, versus ./python).
I have my reasons (basically, looking for a compiled-in library search
path to help with http://bugs.python.org/issue26439)


I think the only way to get at the compiled-in search path is to recreate
it based on the compiled-in prefix, which you can get through distutils.
Python purposely only uses the compiled-in path as the last resort.
Instead, it searches for its home relative to the executable and adds a set
of directories relative to its home (if they exist).

It's not clear to me why you're focusing on these differences, as (as I
describe below) they are immaterial.



Looking on two platforms (AIX, my build, and debian for power) I am
surprised that sys.argv is empty in both cases, and sys.path returns
/opt/lib/python27.zip with AIX, but not with debian.


When you run python interactively, sys.argv[0] will be '', yes. Since
you're not launching a program, there's nothing else to set it to. 'python'
(or the path to the executable) wouldn't be the right thing to set it to,
because python itself isn't a Python program :)

The actual python executable is sys.executable, not sys.argv[0], but you
shouldn't usually care about that, either. If you want to know where to
install things, distutils is the thing to use. If you want to know where
Python thinks it's installed (for debugging purposes only, really),
sys.prefix will tell you.



root@x064:[/data/prj/aixtools/python/python-2.7.10]/opt/bin/python
Python 2.7.10 (default, Nov  3 2015, 14:36:51) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.

import sys
sys.argv

['']

sys.path

['', '/opt/lib/python27.zip', '/opt/lib/python2.7',
'/opt/lib/python2.7/plat-aix5', '/opt/lib/python2.7/lib-tk',
'/opt/lib/python2.7/lib-old', '/opt/lib/python2.7/lib-dynload',
'/opt/lib/python2.7/site-packages']

michael@ipv4:~$ python
Python 2.7.9 (default, Mar  1 2015, 13:01:00)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import sys
sys.argv

['']

sys.path

['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-powerpc-linux-gnu',
'/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/PILcompat',
'/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7']


In sys.path, you're seeing the difference between a vanilla Python and
Debian's patched Python. Vanilla Python adds $prefix/lib/python27.zip to
sys.path unconditionally, whereas Debian removes it when it doesn't exist.
Likewise, the dist-packages directory is a local modification by Debian; in
vanilla Python it's called 'site-packages' instead. The subdirectories in
dist-packages that you see in the Debian case are added by .pth files
installed in $prefix -- third-party packages, in other words, adding their
own directories to the module search path.



And I guess I would be interested in getting
'/opt/lib/python2.7/dist-packages' in there as well (or learn a way to
later add it for pre-compiled packages such as cloud-init AND that those
would also look 'first' in /opt/lib/python2.7/dist-packages/cloud-init for
modules added to support cloud-init - should I so choose (mainly in case of
compatibility issues between say cloud-init and salt-stack that have common
modules BUT may have conflicts) - Hopefully never needed for that reason,
but it might also simplify packaging applications that depend on python.


A vanilla Python (or non-Debian-built python, even) has no business looking
in dist-packages. It should just use site-packages. Third-party packages
shouldn't care whether they're installed in site-packages or dist-packages,
and instead should use distutils one way or another (if not by having an
actual setup.py that uses distutils or setuptools, then at least by
querying distutils for the installation directory the way python-config
does).



Many thanks for your time and pointers into the documentation,