Re: [Python-Dev] Why are generated files in the repository?
On 20 January 2015 at 10:53, Benjamin Peterson wrote: > > > On Mon, Jan 19, 2015, at 19:40, Neil Girdhar wrote: >> I was also wondering why files like Python/graminit.c are in the >> respository? They generate spurious merge conflicts. > > Convenience mostly. It also gets us a round a couple of bootstrapping problems, where generating some of those files requires a working Python interpreter, which you may not have if you just cloned the source tree or unpacked the tarball. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 468 (Ordered kwargs)
Hi I would like to point out that we implemented rhettingers idea in PyPy that makes all the dicts ordered by default and we don't have any adverse performance effects (in fact, there is quite significant memory saving coming from it). The measurments on CPython could be different, but in principle OrderedDict can be implemented as efficiently as normal dict. Writeup: http://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html Previous discussion: https://mail.python.org/pipermail/python-dev/2012-December/123028.html Cheers, fijal ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 468 (Ordered kwargs)
On Sat, Jan 24, 2015 at 12:50 PM, Maciej Fijalkowski wrote: > Hi > > I would like to point out that we implemented rhettingers idea in PyPy > that makes all the dicts ordered by default and we don't have any > adverse performance effects (in fact, there is quite significant > memory saving coming from it). The measurments on CPython could be > different, but in principle OrderedDict can be implemented as > efficiently as normal dict. > > Writeup: > http://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html > > Previous discussion: > https://mail.python.org/pipermail/python-dev/2012-December/123028.html > > Cheers, > fijal also as a sidenote: PEP should maybe mention that PyPy is already supporting it, a bit by chance ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can Python Be Built Without Distutils
On Fri, Jan 23, 2015 at 12:48 PM, Matthias Klose wrote: > On 01/23/2015 06:30 PM, Cyd Haselton wrote: >> Related to my earlier question regarding building Python on Android >> and an undefined reference to dlopen error...I have the following >> question: Is it possible to build and install Python without having >> to build and install...or use...distutils? >> >> Some background: >> I can build the python interpreter on my device, and I can build a >> bunch of modules. The problem appears when make reaches the part >> where setup.py is used to build and import modules...specifically when >> setup.py attempts to import distutils.core. > > you can do this using Setup.local. This works for me building additional > extensions as builtins. It might require some tweaking to build everything. > Otoh, I would like to get rid off the setup.py altogether (/me ducks ...). > > Matthias Regarding previous question, build completes when using Setup, and changing *shared* to *static* but 'make install' fails. If I run 'make altbininstall' or 'make altmaninstall' thst works...actually they all work except any of the libinstall targets, which is probably because in the makefile all lib install targets run setup.py. Is there a special way to install python without setup.py ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can Python Be Built Without Distutils
On Fri, Jan 23, 2015 at 12:48 PM, Matthias Klose wrote: > On 01/23/2015 06:30 PM, Cyd Haselton wrote: >> Related to my earlier question regarding building Python on Android >> and an undefined reference to dlopen error...I have the following >> question: Is it possible to build and install Python without having >> to build and install...or use...distutils? >> >> Some background: >> I can build the python interpreter on my device, and I can build a >> bunch of modules. The problem appears when make reaches the part >> where setup.py is used to build and import modules...specifically when >> setup.py attempts to import distutils.core. > > you can do this using Setup.local. This works for me building additional > extensions as builtins. It might require some tweaking to build everything. > Otoh, I would like to get rid off the setup.py altogether (/me ducks ...). > > Matthias > Was that Setup.local or Setup? Setup.local is mostly empty... ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why are generated files in the repository?
On Sat, Jan 24, 2015, at 03:00, Nick Coghlan wrote: > On 20 January 2015 at 10:53, Benjamin Peterson > wrote: > > > > > > On Mon, Jan 19, 2015, at 19:40, Neil Girdhar wrote: > >> I was also wondering why files like Python/graminit.c are in the > >> respository? They generate spurious merge conflicts. > > > > Convenience mostly. > > It also gets us a round a couple of bootstrapping problems, where > generating some of those files requires a working Python interpreter, > which you may not have if you just cloned the source tree or unpacked > the tarball. We could distribute the generated files in tarballs as part of the release process. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Overriding stdlib http package
> On Jan 24, 2015, at 2:57 AM, Nick Coghlan wrote: > > On 15 January 2015 at 07:35, Donald Stufft wrote: >> >> On Jan 14, 2015, at 4:19 PM, Brett Cannon wrote: >> >> But as Guido pointed out, we _like_ it being difficult to do because we >> don't want this kind of substitution happening as code ends up depending on >> bugs and quirks that you may fix. >> >> Not all of us, I hate the default order of sys.path. > > It's mostly an opinion that arises from debugging other people's > problems after they've managed to import the wrong thing without > realising it (cf. the "don't use 'socket.py' as the name of your > script for learning about how TCP sockets work" problem). We're aware > that annoys power users, but they're far better equipped to handle the > problem than if we inverted the situation. It’s not just power users that it’s good for, it makes it harder for even beginners to use things like backports of modules. For example unittest2 and explaining to people the difference between unittest and unittest2 and that unittest2 isn’t actually any different than unittest on newer versions of Python. Or, for example, PEP 453 could have been like 100x better if it would have been reasonable to just add pip to the stdlib but still enabling the ability to install an upgraded version of it that would take precedence. Or you have things like pdb++ which needs to replace the pdb import because a lot of tools only have a flag like —pdb and do not provide a way to switch it to a different import. The sys.path ordering means that pdb++ has to do hacks in its setup.py[1] which means it won’t be compatible with Wheel files or with a world where sdists don’t use a setup.py. The current situation is that if you install something as an egg (which setuptools does by default anyways) then setuptools will put it before the stdlib and it’ll take precedence. This is a nice situation because it means that if you do run into a problem then it’s easier to debug because ``python -c import module; print(module.__file__)`` will always return the same answer in the “broken” environment. The alternative is often either a different name (which confuses people as to the relation) or monkey patching which means that module.__file__ might either be wrong if they just monkey patched the file and it always means that the behavior is going to change depending on what you’ve imported which is way more confusing then being able to override the stdlib. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Overriding stdlib http package
> On Jan 24, 2015, at 10:17 AM, Donald Stufft wrote: > > Or you have things like pdb++ which needs to replace the pdb import > because a lot of tools only have a flag like —pdb and do not provide > a way to switch it to a different import. The sys.path ordering means > that pdb++ has to do hacks in its setup.py[1] which means it won’t be > compatible with Wheel files or with a world where sdists don’t use > a setup.py. Sorry, forgot to link this https://bitbucket.org/antocuni/pdb/src/4669c3747a396e3766173feb40ebece32ab08aad/setup.py?at=default#cl-7 --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 468 (Ordered kwargs)
Wow, very cool. When I implemented the very first Python dict (cribbing from an algorithm in Knuth) I had no idea that 25 years later there would still be ways to improve upon it! I've got a feeling Knuth probably didn't expect this either... On Sat, Jan 24, 2015 at 2:51 AM, Maciej Fijalkowski wrote: > On Sat, Jan 24, 2015 at 12:50 PM, Maciej Fijalkowski > wrote: > > Hi > > > > I would like to point out that we implemented rhettingers idea in PyPy > > that makes all the dicts ordered by default and we don't have any > > adverse performance effects (in fact, there is quite significant > > memory saving coming from it). The measurments on CPython could be > > different, but in principle OrderedDict can be implemented as > > efficiently as normal dict. > > > > Writeup: > http://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html > > > > Previous discussion: > > https://mail.python.org/pipermail/python-dev/2012-December/123028.html > > > > Cheers, > > fijal > > also as a sidenote: PEP should maybe mention that PyPy is already > supporting it, a bit by chance > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can Python Be Built Without Distutils
On Fri Jan 23 2015 at 5:45:28 PM Gregory P. Smith wrote: > On Fri Jan 23 2015 at 11:20:02 AM M.-A. Lemburg wrote: > >> On 23.01.2015 19:48, Matthias Klose wrote: >> > On 01/23/2015 06:30 PM, Cyd Haselton wrote: >> >> Related to my earlier question regarding building Python on Android >> >> and an undefined reference to dlopen error...I have the following >> >> question: Is it possible to build and install Python without having >> >> to build and install...or use...distutils? >> >> >> >> Some background: >> >> I can build the python interpreter on my device, and I can build a >> >> bunch of modules. The problem appears when make reaches the part >> >> where setup.py is used to build and import modules...specifically when >> >> setup.py attempts to import distutils.core. >> > >> > you can do this using Setup.local. This works for me building additional >> > extensions as builtins. It might require some tweaking to build >> everything. >> >> You may want to have a look at the Setup files we're using >> in eGenix PyRun, which uses them to force static builds of the >> various built-in extensions. >> >> Look for these files: >> >> PyRun/Runtime/Setup.PyRun-2.7 >> PyRun/Runtime/Setup.PyRun-3.4 >> >> in the source archives: >> >> http://www.egenix.com/products/python/PyRun/ >> >> > Otoh, I would like to get rid off the setup.py altogether (/me ducks >> ...). >> >> Why ? It's great for finding stuff on your system and configuring >> everything without user intervention (well, most of the time :-)). >> > > Because our setup.py is a nightmare of arbitrary code run in a linear > fashion with ad-hoc checks for things that are unlike how any other project > on the planet determines what is available on your system. It may have > seemed "great" when it was created in 2001. It really shows its age now. > > It defeats build parallelism and dependency declaration. > It also prevents cross compilation. > > Building an interpreter with a limited standard library on your build host > so that you can run said interpreter to have it drive the remainder of your > build is way more self hosting that we rightfully need to be for CPython. > So are you suggesting to add the build rules to configure and the Makefile -- and Windows build file -- in order to drop setup.py? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 468 (Ordered kwargs)
On Sat, Jan 24, 2015 at 3:50 AM, Maciej Fijalkowski wrote: > I would like to point out that we implemented rhettingers idea in PyPy > that makes all the dicts ordered by default and we don't have any > adverse performance effects (in fact, there is quite significant > memory saving coming from it). The measurments on CPython could be > different, but in principle OrderedDict can be implemented as > efficiently as normal dict. > > Writeup: > http://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html > > Previous discussion: > https://mail.python.org/pipermail/python-dev/2012-December/123028.html Cool. I'll add a note to the PEP. -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 468 (Ordered kwargs)
Hi Guido. I *think* part of the reason why our implementation works is that machines are significantly different than at the times of Knuth. Avoiding cache misses is a very effective way to improve performance these days. Cheers, fijal On Sat, Jan 24, 2015 at 7:39 PM, Guido van Rossum wrote: > Wow, very cool. When I implemented the very first Python dict (cribbing from > an algorithm in Knuth) I had no idea that 25 years later there would still > be ways to improve upon it! I've got a feeling Knuth probably didn't expect > this either... > > On Sat, Jan 24, 2015 at 2:51 AM, Maciej Fijalkowski > wrote: >> >> On Sat, Jan 24, 2015 at 12:50 PM, Maciej Fijalkowski >> wrote: >> > Hi >> > >> > I would like to point out that we implemented rhettingers idea in PyPy >> > that makes all the dicts ordered by default and we don't have any >> > adverse performance effects (in fact, there is quite significant >> > memory saving coming from it). The measurments on CPython could be >> > different, but in principle OrderedDict can be implemented as >> > efficiently as normal dict. >> > >> > Writeup: >> > http://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html >> > >> > Previous discussion: >> > https://mail.python.org/pipermail/python-dev/2012-December/123028.html >> > >> > Cheers, >> > fijal >> >> also as a sidenote: PEP should maybe mention that PyPy is already >> supporting it, a bit by chance >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 468 (Ordered kwargs)
On Sat, Jan 24, 2015 at 11:11 AM, Maciej Fijalkowski wrote: > Hi Guido. > > I *think* part of the reason why our implementation works is that > machines are significantly different than at the times of Knuth. > Avoiding cache misses is a very effective way to improve performance > these days. > Right. We might look what Knuth has to say about data structures that are stored on disk and loaded into memory piecemeal. :-) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can Python Be Built Without Distutils
On 24.01.2015 21:23, Zachary Ware wrote: > On Saturday, January 24, 2015, Brett Cannon wrote: > >> On Fri Jan 23 2015 at 5:45:28 PM Gregory P. Smith > > wrote: >> >>> On Fri Jan 23 2015 at 11:20:02 AM M.-A. Lemburg >> > wrote: >>> On 23.01.2015 19:48, Matthias Klose wrote: > On 01/23/2015 06:30 PM, Cyd Haselton wrote: >> Related to my earlier question regarding building Python on Android >> and an undefined reference to dlopen error...I have the following >> question: Is it possible to build and install Python without having >> to build and install...or use...distutils? >> >> Some background: >> I can build the python interpreter on my device, and I can build a >> bunch of modules. The problem appears when make reaches the part >> where setup.py is used to build and import modules...specifically when >> setup.py attempts to import distutils.core. > > you can do this using Setup.local. This works for me building additional > extensions as builtins. It might require some tweaking to build everything. You may want to have a look at the Setup files we're using in eGenix PyRun, which uses them to force static builds of the various built-in extensions. Look for these files: PyRun/Runtime/Setup.PyRun-2.7 PyRun/Runtime/Setup.PyRun-3.4 in the source archives: http://www.egenix.com/products/python/PyRun/ > Otoh, I would like to get rid off the setup.py altogether (/me ducks ...). Why ? It's great for finding stuff on your system and configuring everything without user intervention (well, most of the time :-)). >>> >>> Because our setup.py is a nightmare of arbitrary code run in a linear >>> fashion with ad-hoc checks for things that are unlike how any other project >>> on the planet determines what is available on your system. It may have >>> seemed "great" when it was created in 2001. It really shows its age now. >>> >>> It defeats build parallelism and dependency declaration. >>> It also prevents cross compilation. >>> >>> Building an interpreter with a limited standard library on your build >>> host so that you can run said interpreter to have it drive the remainder of >>> your build is way more self hosting that we rightfully need to be for >>> CPython. >>> >> >> So are you suggesting to add the build rules to configure and the Makefile >> -- and Windows build file -- in order to drop setup.py? >> > > Windows already doesn't use setup.py. There are a lot more modules built-in > on Windows, and others have their own project files; distutils isn't used > at all. The Windows installers comes with all dependencies included. On other systems, this is not the case and so determining the various settings is done using Python, which is a lot easier to do than e.g. relying on autoconf finding everything and setting up the correct compile options for each built-in. All that said: if you can come up with a better system that's both easy to maintain and provides the same level of user friendliness, I'm sure this would be considered. BTW: Parallel execution, cross compilation can be added to setup.py. I don't think parallel execution is all that important, but cross compilation would certainly be an interesting feature to have (this currently has to be done using the Setup.in approach) and an option to build the modules statically linked to the interpreter would also be nice. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 24 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can Python Be Built Without Distutils
On Saturday, January 24, 2015, Brett Cannon wrote: > On Fri Jan 23 2015 at 5:45:28 PM Gregory P. Smith > wrote: > >> On Fri Jan 23 2015 at 11:20:02 AM M.-A. Lemburg > > wrote: >> >>> On 23.01.2015 19:48, Matthias Klose wrote: >>> > On 01/23/2015 06:30 PM, Cyd Haselton wrote: >>> >> Related to my earlier question regarding building Python on Android >>> >> and an undefined reference to dlopen error...I have the following >>> >> question: Is it possible to build and install Python without having >>> >> to build and install...or use...distutils? >>> >> >>> >> Some background: >>> >> I can build the python interpreter on my device, and I can build a >>> >> bunch of modules. The problem appears when make reaches the part >>> >> where setup.py is used to build and import modules...specifically when >>> >> setup.py attempts to import distutils.core. >>> > >>> > you can do this using Setup.local. This works for me building >>> additional >>> > extensions as builtins. It might require some tweaking to build >>> everything. >>> >>> You may want to have a look at the Setup files we're using >>> in eGenix PyRun, which uses them to force static builds of the >>> various built-in extensions. >>> >>> Look for these files: >>> >>> PyRun/Runtime/Setup.PyRun-2.7 >>> PyRun/Runtime/Setup.PyRun-3.4 >>> >>> in the source archives: >>> >>> http://www.egenix.com/products/python/PyRun/ >>> >>> > Otoh, I would like to get rid off the setup.py altogether (/me ducks >>> ...). >>> >>> Why ? It's great for finding stuff on your system and configuring >>> everything without user intervention (well, most of the time :-)). >>> >> >> Because our setup.py is a nightmare of arbitrary code run in a linear >> fashion with ad-hoc checks for things that are unlike how any other project >> on the planet determines what is available on your system. It may have >> seemed "great" when it was created in 2001. It really shows its age now. >> >> It defeats build parallelism and dependency declaration. >> It also prevents cross compilation. >> >> Building an interpreter with a limited standard library on your build >> host so that you can run said interpreter to have it drive the remainder of >> your build is way more self hosting that we rightfully need to be for >> CPython. >> > > So are you suggesting to add the build rules to configure and the Makefile > -- and Windows build file -- in order to drop setup.py? > Windows already doesn't use setup.py. There are a lot more modules built-in on Windows, and others have their own project files; distutils isn't used at all. -- Sent from Gmail Mobile ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can Python Be Built Without Distutils
On Sat, Jan 24, 2015 at 2:23 PM, Zachary Ware wrote: > On Saturday, January 24, 2015, Brett Cannon wrote: >> >> On Fri Jan 23 2015 at 5:45:28 PM Gregory P. Smith wrote: >>> >>> On Fri Jan 23 2015 at 11:20:02 AM M.-A. Lemburg wrote: On 23.01.2015 19:48, Matthias Klose wrote: > On 01/23/2015 06:30 PM, Cyd Haselton wrote: >> Related to my earlier question regarding building Python on Android >> and an undefined reference to dlopen error...I have the following >> question: Is it possible to build and install Python without having >> to build and install...or use...distutils? >> >> Some background: >> I can build the python interpreter on my device, and I can build a >> bunch of modules. The problem appears when make reaches the part >> where setup.py is used to build and import modules...specifically >> when >> setup.py attempts to import distutils.core. > > you can do this using Setup.local. This works for me building > additional > extensions as builtins. It might require some tweaking to build > everything. You may want to have a look at the Setup files we're using in eGenix PyRun, which uses them to force static builds of the various built-in extensions. Look for these files: PyRun/Runtime/Setup.PyRun-2.7 PyRun/Runtime/Setup.PyRun-3.4 in the source archives: http://www.egenix.com/products/python/PyRun/ > Otoh, I would like to get rid off the setup.py altogether (/me ducks > ...). Why ? It's great for finding stuff on your system and configuring everything without user intervention (well, most of the time :-)). >>> >>> >>> Because our setup.py is a nightmare of arbitrary code run in a linear >>> fashion with ad-hoc checks for things that are unlike how any other project >>> on the planet determines what is available on your system. It may have >>> seemed "great" when it was created in 2001. It really shows its age now. >>> >>> It defeats build parallelism and dependency declaration. >>> It also prevents cross compilation. >>> >>> Building an interpreter with a limited standard library on your build >>> host so that you can run said interpreter to have it drive the remainder of >>> your build is way more self hosting that we rightfully need to be for >>> CPython. >> >> >> So are you suggesting to add the build rules to configure and the Makefile >> -- and Windows build file -- in order to drop setup.py? > > > Windows already doesn't use setup.py. There are a lot more modules built-in > on Windows, and others have their own project files; distutils isn't used at > all. > > Is there a way to either a) remove setup.py from the Makefile install targets or b) remove it entirely? Regarding b) is it Setup.local that should be configured or Setup? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can Python Be Built Without Distutils
On Sat, 24 Jan 2015 21:53:06 +0100 "M.-A. Lemburg" wrote: > > BTW: Parallel execution, cross compilation can be added > to setup.py. I don't think parallel execution is all > that important, $ ./python setup.py build_ext --help [...] --parallel (-j) number of parallel build jobs Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Undefined dlopen When Building Module On Android
Android's dlopen() works slightly differently than the normal Unix dlopen() in
at least two different ways. I haven't seen your particular problem, but
that's probably because I'm cross-building CPython (building everything I need
on a Linux machine, and then copying the install directory to the Android
machine.)
(1) When building for Android you need to explicitly include the "-ldl" flag on
the command line for the link step. (In Linux the dlopen() routine is included
in libc, so "-ldl" is not necessary). I suspect that some part of your
distutils was not linked correctly. (Since I'm cross-building I've never run
distutils on the Android side, which is probably why I've not seen this.) Your
best bet would be to run under a debugger and figure out which line of C code
in which .so file is throwing the error. Then going back to the build scripts
and looking at how that .so file is getting linked. (For an example of someone
else having a similar problem see for example,
http://stackoverflow.com/questions/25846927/git-built-on-android-throws-undefined-reference-to-dlopen-error).
(2) The other possibility has to do with a quirk in Android's dlopen()
implementation. On most legacy Unix systems (including Linux) when you
dlopen() a library Z then all the already loaded libraries A, B, C, ... are
searched for any dependences of Z (even if Z was not explicitly linked against
any of A, B, C,...). On Android (perhaps for security reasons) that's not
true, if Z depends on A, then you need to have "-lA" when you link Z. An
example (and patch) for this problem is http://bugs.python.org/issue21668.
-Matt
> -Original Message-
> From: Cyd Haselton [mailto:chasel...@gmail.com]
> Sent: Friday, January 23, 2015 10:50 AM
> To: Brett Cannon
> Cc: Guido van Rossum; Python-Dev
> Subject: Re: [Python-Dev] Undefined dlopen When Building Module On Android
>
> I guess I'll keep waiting...given the zero responses I've gotten from the
> android side.
> :-(.
>
> In the meantime...in case anyone is interested... since I have the working
> binary, I
> executed it and went through each command in setup.py to see what throws the
> 'undefined reference to dlopen' error.
> Turns out that the distutils module is the culprit...specifically
> distutils.core.
>
>
> On Thu, Jan 22, 2015 at 8:33 AM, Brett Cannon wrote:
> > A mobile SIG is being formed, but it doesn't have a mailing list yet,
> > else that would be a good place to ask this question.
> >
> > On Wed Jan 21 2015 at 5:54:39 PM Guido van Rossum wrote:
> >>
> >> Maybe try a list focused on Android development? Few people in the
> >> Python core development community have any Android experience. But
> >> the issues and context you offer seem to be related to the Android world,
> >> not to
> Python.
> >> (dlopen is used by a lot of systems, not just Python.)
> >>
> >> On Wed, Jan 21, 2015 at 2:43 PM, Cyd Haselton wrote:
> >>>
> >>> On Mon, Jan 19, 2015 at 5:23 PM, Cyd Haselton
> >>> wrote:
> >>> > On Mon, Jan 19, 2015 at 8:51 AM, Cyd Haselton
> >>> >
> >>> > wrote:
> >>> >> Hello,
> >>> >> I'm struggling with a build issue on Android; I've posted to the
> >>> >> general python list with no result, so I'm re-posting here in
> >>> >> hopes that someone can help. If this is the wrong place feel
> >>> >> free to let me know.
> >>> >>
> >>> >> I'm attempting to build Python 2.7.8 on my Android device; I'm
> >>> >> using an environment that simulates a Linux filesystem within the
> >>> >> Android terminal using a port of fakechroot. Within that
> >>> >> environment I've ported and/or bootstrapped a number of Linux
> >>> >> utilities (curl, git, openssl, gcc)
> >>> >>
> >>> >> I run ./configure, then make, and the executable and library are
> >>> >> built. The problem occurs when build_ext is run; the newly built
> >>> >> python executable builds, then links _struct, and immediately
> >>> >> afterwards I get an 'undefined reference to dlopen' error.
> >>> >>
> >>> >> If I run ./python setup.py --verbose -library-dirs /path/to/lib
> >>> >> --libraries='c dl m' -f, the 'undefined reference to dlopen'
> >>> >> error is thrown again.
> >>> >>
> >>> >> If I run ./python setup.py --verbose -library-dirs /path/to/lib
> >>> >> --libraries='-lc -ldl -lm' -f the build continues past
> >>> >> _struct...even though ld throws the expected 'unable to find -l-lc'
> >>> >> and other
> errors.
> >>> >>
> >>> >> Let me know if you need me to provide additional information.
> >>> >> Any help would be greatly appreciated.
> >>> >>
> >>> >> Cyd
> >>> >
> >>> >
> >>> > Additionally I took a strace of the error producing command. The
> >>> > following is (hopefully) a relevant portion minus the various 'no
> >>> > such file' lines before the correct lib is found (which it always
> >>> > is)
> >>> >
> >>> > 16513
> >>> > open("/data/data/jackpal.androidterm/kbox2/bld/python/Python-
> 2.7.8/Lib/distutils/unixccompiler.py",
> >>> > O_RDONLY|O_LARGEFILE) = 3
Re: [Python-Dev] Undefined dlopen When Building Module On Android
Replies in body of message for clarity: On Sat, Jan 24, 2015 at 5:08 PM, Frank, Matthew I wrote: > Android's dlopen() works slightly differently than the normal Unix dlopen() > in at least two different ways. I haven't seen your particular problem, but > that's probably because I'm cross-building CPython (building everything I > need on a Linux machine, and then copying the install directory to the > Android machine.) Yeah...I have a Linux machine but it currently resides as a gunzipped tarball on an external drive...I lack the space to decompress and use it for building purposes. > (1) When building for Android you need to explicitly include the "-ldl" flag > on the command line for the link step. (In Linux the dlopen() routine is > included in libc, so "-ldl" is not necessary). I suspect that some part of > your distutils was not linked correctly. (Since I'm cross-building I've > never run distutils on the Android side, which is probably why I've not seen > this.) Your best bet would be to run under a debugger and figure out which > line of C code in which .so file is throwing the error. Then going back to > the build scripts and looking at how that .so file is getting linked. (For > an example of someone else having a similar problem see for example, > http://stackoverflow.com/questions/25846927/git-built-on-android-throws-undefined-reference-to-dlopen-error). Funny you should reference that post as I'm the one who started it; I ported git, which required curl and openssl ports, a while ago and...as I think I mentioned in that post...I had to go back and re-port openssl, making sure to throw in -ldl wherever possible. Python (which I'm building in the same environment in which I built openssl, git and curl) is a different beast altogether. Environment variables that make it to the Makefile don't make it into the modules built by setup.py, which is run by distutils, which I wasn't aware was built alongside Python until you mentioned it just now. Then there are the modules in Setup...which are built or not...seemingly independently of whether or not the line specifying the module is commented out or not; the hack workaround is to put every module that should be excluded in setup.py...even if it is commented out in Setup. Obviously I'm still trying to get a handle on the Python build process. > (2) The other possibility has to do with a quirk in Android's dlopen() > implementation. On most legacy Unix systems (including Linux) when you > dlopen() a library Z then all the already loaded libraries A, B, C, ... are > searched for any dependences of Z (even if Z was not explicitly linked > against any of A, B, C,...). On Android (perhaps for security reasons) > that's not true, if Z depends on A, then you need to have "-lA" when you link > Z. An example (and patch) for this problem is > http://bugs.python.org/issue21668. Thanks for the bit of info above...very useful. I've included -lc -ldl in the requisite places in Setup, and setup.py...although it's entirely possible I've not added it to places that need it given that I'm still struggling to understand the build process. Right now I'm at the point where running configure && make finishes successfully but throws warnings during the module build and import process. Running make install fails completely with the same 'undefined reference to dlopen' because, for some reason, make install requires the sharedmods target to be rebuilt again, which fails because setup.py fails when importing distutils.core. Why it doesn't do the same when running make is beyond me. Cyd ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Any grammar experts?
To finish PEP 448, I need to update the grammar for syntax such as
{**x for x in it}
and
{1:2, 3:4, **a}
It's been a long time since I've looked at grammars and I could really use
the advice of an expert. I'm considering replacing:
dictorsetmaker: ( (test ':' test (comp_for | (',' test ':' test)* [','])) |
(test (comp_for | (',' test)* [','])) )
with:
dictpopulator: test ':' test | '**' test
dictorsetmaker: ( (dictpopulator (comp_for | (',' dictpopulator)* [','])) |
(test (comp_for | (',' test)* [','])) )
Am I headed in the right direction? Of course I will need to edit
parsermodule.c and ast.c.
Best,
Neil
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Undefined dlopen When Building Module On Android
Closer. make && make install ran successfully; however running the newly installed binary results in the same error. On Sat, Jan 24, 2015 at 5:08 PM, Frank, Matthew I wrote: > Android's dlopen() works slightly differently than the normal Unix dlopen() > in at least two different ways. I haven't seen your particular problem, but > that's probably because I'm cross-building CPython (building everything I > need on a Linux machine, and then copying the install directory to the > Android machine.) > > (1) When building for Android you need to explicitly include the "-ldl" flag > on the command line for the link step. (In Linux the dlopen() routine is > included in libc, so "-ldl" is not necessary). I suspect that some part of > your distutils was not linked correctly. (Since I'm cross-building I've > never run distutils on the Android side, which is probably why I've not seen > this.) Your best bet would be to run under a debugger and figure out which > line of C code in which .so file is throwing the error. Then going back to > the build scripts and looking at how that .so file is getting linked. (For > an example of someone else having a similar problem see for example, > http://stackoverflow.com/questions/25846927/git-built-on-android-throws-undefined-reference-to-dlopen-error). > > (2) The other possibility has to do with a quirk in Android's dlopen() > implementation. On most legacy Unix systems (including Linux) when you > dlopen() a library Z then all the already loaded libraries A, B, C, ... are > searched for any dependences of Z (even if Z was not explicitly linked > against any of A, B, C,...). On Android (perhaps for security reasons) > that's not true, if Z depends on A, then you need to have "-lA" when you link > Z. An example (and patch) for this problem is > http://bugs.python.org/issue21668. > > -Matt > >> -Original Message- >> From: Cyd Haselton [mailto:chasel...@gmail.com] >> Sent: Friday, January 23, 2015 10:50 AM >> To: Brett Cannon >> Cc: Guido van Rossum; Python-Dev >> Subject: Re: [Python-Dev] Undefined dlopen When Building Module On Android >> >> I guess I'll keep waiting...given the zero responses I've gotten from the >> android side. >> :-(. >> >> In the meantime...in case anyone is interested... since I have the working >> binary, I >> executed it and went through each command in setup.py to see what throws the >> 'undefined reference to dlopen' error. >> Turns out that the distutils module is the culprit...specifically >> distutils.core. >> >> >> On Thu, Jan 22, 2015 at 8:33 AM, Brett Cannon wrote: >> > A mobile SIG is being formed, but it doesn't have a mailing list yet, >> > else that would be a good place to ask this question. >> > >> > On Wed Jan 21 2015 at 5:54:39 PM Guido van Rossum wrote: >> >> >> >> Maybe try a list focused on Android development? Few people in the >> >> Python core development community have any Android experience. But >> >> the issues and context you offer seem to be related to the Android world, >> >> not to >> Python. >> >> (dlopen is used by a lot of systems, not just Python.) >> >> >> >> On Wed, Jan 21, 2015 at 2:43 PM, Cyd Haselton wrote: >> >>> >> >>> On Mon, Jan 19, 2015 at 5:23 PM, Cyd Haselton >> >>> wrote: >> >>> > On Mon, Jan 19, 2015 at 8:51 AM, Cyd Haselton >> >>> > >> >>> > wrote: >> >>> >> Hello, >> >>> >> I'm struggling with a build issue on Android; I've posted to the >> >>> >> general python list with no result, so I'm re-posting here in >> >>> >> hopes that someone can help. If this is the wrong place feel >> >>> >> free to let me know. >> >>> >> >> >>> >> I'm attempting to build Python 2.7.8 on my Android device; I'm >> >>> >> using an environment that simulates a Linux filesystem within the >> >>> >> Android terminal using a port of fakechroot. Within that >> >>> >> environment I've ported and/or bootstrapped a number of Linux >> >>> >> utilities (curl, git, openssl, gcc) >> >>> >> >> >>> >> I run ./configure, then make, and the executable and library are >> >>> >> built. The problem occurs when build_ext is run; the newly built >> >>> >> python executable builds, then links _struct, and immediately >> >>> >> afterwards I get an 'undefined reference to dlopen' error. >> >>> >> >> >>> >> If I run ./python setup.py --verbose -library-dirs /path/to/lib >> >>> >> --libraries='c dl m' -f, the 'undefined reference to dlopen' >> >>> >> error is thrown again. >> >>> >> >> >>> >> If I run ./python setup.py --verbose -library-dirs /path/to/lib >> >>> >> --libraries='-lc -ldl -lm' -f the build continues past >> >>> >> _struct...even though ld throws the expected 'unable to find -l-lc' >> >>> >> and other >> errors. >> >>> >> >> >>> >> Let me know if you need me to provide additional information. >> >>> >> Any help would be greatly appreciated. >> >>> >> >> >>> >> Cyd >> >>> > >> >>> > >> >>> > Additionally I took a strace of the error producing command. The >> >>> > following is (hopefully) a
Re: [Python-Dev] Any grammar experts?
Have you tried it yet?
I think you have to inline dictpopulator, because dictpopulator can start
with the same tokens as test, and the parser doesn't backtrack. So it
wouldn't know how to decide whether to take the dictpopulator branch or the
set branch. If you inline it, the parser will know, because it does
something clever within the rule.
As-is, I get a lot of errors from pgen about ambiguity. This one seems to
work (but you still have to adjust the code generator of course):
dictorsetmaker: ( ((test ':' test | '**' test) (comp_for | (',' (test ':'
test | '**' test))* [','])) |
(test (comp_for | (',' test)* [','])) )
Also I presume you want a similar treatment for the set branch (replace
both tests with (test | '*' test).
Good luck! There's plenty of code to crib from for the code generation.
--Guido
On Sat, Jan 24, 2015 at 6:10 PM, Neil Girdhar wrote:
> To finish PEP 448, I need to update the grammar for syntax such as
>
> {**x for x in it}
>
> and
>
> {1:2, 3:4, **a}
>
> It's been a long time since I've looked at grammars and I could really use
> the advice of an expert. I'm considering replacing:
>
> dictorsetmaker: ( (test ':' test (comp_for | (',' test ':' test)* [','])) |
> (test (comp_for | (',' test)* [','])) )
>
> with:
>
> dictpopulator: test ':' test | '**' test
> dictorsetmaker: ( (dictpopulator (comp_for | (',' dictpopulator)* [','])) |
>(test (comp_for | (',' test)* [','])) )
>
> Am I headed in the right direction? Of course I will need to edit
> parsermodule.c and ast.c.
>
> Best,
>
> Neil
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Any grammar experts?
Thanks, I had tried it and got the ambiguities, but I wasn't sure if those
would disappear with editing some peripheral file.
Yes, you're right about the set branch.
Thank you,
Neil
On Sat, Jan 24, 2015 at 10:29 PM, Guido van Rossum wrote:
> Have you tried it yet?
>
> I think you have to inline dictpopulator, because dictpopulator can start
> with the same tokens as test, and the parser doesn't backtrack. So it
> wouldn't know how to decide whether to take the dictpopulator branch or the
> set branch. If you inline it, the parser will know, because it does
> something clever within the rule.
>
> As-is, I get a lot of errors from pgen about ambiguity. This one seems to
> work (but you still have to adjust the code generator of course):
>
> dictorsetmaker: ( ((test ':' test | '**' test) (comp_for | (',' (test ':'
> test | '**' test))* [','])) |
>(test (comp_for | (',' test)* [','])) )
>
> Also I presume you want a similar treatment for the set branch (replace
> both tests with (test | '*' test).
>
> Good luck! There's plenty of code to crib from for the code generation.
>
> --Guido
>
> On Sat, Jan 24, 2015 at 6:10 PM, Neil Girdhar
> wrote:
>
>> To finish PEP 448, I need to update the grammar for syntax such as
>>
>> {**x for x in it}
>>
>> and
>>
>> {1:2, 3:4, **a}
>>
>> It's been a long time since I've looked at grammars and I could really
>> use the advice of an expert. I'm considering replacing:
>>
>> dictorsetmaker: ( (test ':' test (comp_for | (',' test ':' test)* [',']))
>> |
>> (test (comp_for | (',' test)* [','])) )
>>
>> with:
>>
>> dictpopulator: test ':' test | '**' test
>> dictorsetmaker: ( (dictpopulator (comp_for | (',' dictpopulator)* [',']))
>> |
>>(test (comp_for | (',' test)* [','])) )
>>
>> Am I headed in the right direction? Of course I will need to edit
>> parsermodule.c and ast.c.
>>
>> Best,
>>
>> Neil
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why are generated files in the repository?
On 25 Jan 2015 01:09, "Benjamin Peterson" wrote: > > > > On Sat, Jan 24, 2015, at 03:00, Nick Coghlan wrote: > > On 20 January 2015 at 10:53, Benjamin Peterson > > wrote: > > > > > > > > > On Mon, Jan 19, 2015, at 19:40, Neil Girdhar wrote: > > >> I was also wondering why files like Python/graminit.c are in the > > >> respository? They generate spurious merge conflicts. > > > > > > Convenience mostly. > > > > It also gets us a round a couple of bootstrapping problems, where > > generating some of those files requires a working Python interpreter, > > which you may not have if you just cloned the source tree or unpacked > > the tarball. > > We could distribute the generated files in tarballs as part of the > release process. It's far more developer friendly to aim to have builds from a source check-out "just work" if we can. That's pretty much where we are today (getting external dependencies for the optional parts on *nix can still be a bit fiddly - it may be worth maintaining instructions for at least apt and yum in the developer guide that cover that) Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why are generated files in the repository?
But you can remove Python/graminit.c and "make clean && make" works, right? On Sat, Jan 24, 2015 at 11:00 PM, Nick Coghlan wrote: > > On 25 Jan 2015 01:09, "Benjamin Peterson" wrote: > > > > > > > > On Sat, Jan 24, 2015, at 03:00, Nick Coghlan wrote: > > > On 20 January 2015 at 10:53, Benjamin Peterson > > > wrote: > > > > > > > > > > > > On Mon, Jan 19, 2015, at 19:40, Neil Girdhar wrote: > > > >> I was also wondering why files like Python/graminit.c are in the > > > >> respository? They generate spurious merge conflicts. > > > > > > > > Convenience mostly. > > > > > > It also gets us a round a couple of bootstrapping problems, where > > > generating some of those files requires a working Python interpreter, > > > which you may not have if you just cloned the source tree or unpacked > > > the tarball. > > > > We could distribute the generated files in tarballs as part of the > > release process. > > It's far more developer friendly to aim to have builds from a source > check-out "just work" if we can. That's pretty much where we are today > (getting external dependencies for the optional parts on *nix can still be > a bit fiddly - it may be worth maintaining instructions for at least apt > and yum in the developer guide that cover that) > > Cheers, > Nick. > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can Python Be Built Without Distutils
Why doesn't our Makefile supply that flag with the make parallelism level in the sharedmods step? On Sat Jan 24 2015 at 2:25:45 PM Antoine Pitrou wrote: > On Sat, 24 Jan 2015 21:53:06 +0100 > "M.-A. Lemburg" wrote: > > > > BTW: Parallel execution, cross compilation can be added > > to setup.py. I don't think parallel execution is all > > that important, > > $ ./python setup.py build_ext --help > [...] > --parallel (-j) number of parallel build jobs > > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > greg%40krypto.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
