[Python-Dev] "Some" .pyc files not ending up in __pycache__ during installation
Hi, sorry for asking here instead of filing a bug, but given that 3.2 final is pretty close, I wanted to make sure this gets considered. A Cython user noticed that the installation (setup.py install or bdist) puts several .pyc files into the installed source directory, instead of moving them into __pycache__. The latter is still used, and most of the .pyc files end up there, but not all of them. Some even end up in both directories, once with a ".cpython-32.pyc" suffix in __pycache__ and simply as ".pyc" next to the sources. A specialty of the Cython installation is that a tiny part of Cython gets imported at the beginning, then setup() is called. Next, 2to3 is run over the sources, and during the extension building phase, the initially imported part is removed from sys.modules and Cython is imported completely from the converted sources and runs to compile parts of itself. Finally, the installation continues and copies/byte-compiles the 2to3 converted .py files. What I think is happening here, is that the normal import mechanism byte compiles the 2to3 converted sources into the __pycache__ directory (when invoked at extension building time), but then distutils' byte compilation seems to decide that it's better to keep the .pyc files where the sources are. And thus a mix of both ends up in the installation. Wouldn't it be better to let distutils *always* byte-compile the .py files into the appropriate __pycache__ directory, just like the import mechanism does? Or is there something else broken here? Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Some" .pyc files not ending up in __pycache__ during installation
> sorry for asking here instead of filing a bug, but given that 3.2 final > is pretty close, I wanted to make sure this gets considered. If you want it decided before the 3.2 release, it must be a release-critical bug report in the tracker. Posting it here does not make sure it gets considered. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Some" .pyc files not ending up in __pycache__ during installation
On Sat, Feb 19, 2011 at 10:37 PM, Stefan Behnel wrote: > What I think is happening here, is that the normal import mechanism byte > compiles the 2to3 converted sources into the __pycache__ directory (when > invoked at extension building time), but then distutils' byte compilation > seems to decide that it's better to keep the .pyc files where the sources > are. And thus a mix of both ends up in the installation. > > Wouldn't it be better to let distutils *always* byte-compile the .py files > into the appropriate __pycache__ directory, just like the import mechanism > does? Or is there something else broken here? A quick look [1] suggests that distutils.util.bytes_compile is indeed second-guessing py_compile.compile and forcing use of the legacy .pyc location rather than accepting the default location. (Why? I have no idea, it's distutils...) While this is definitely untidy, it doesn't strike me as a release blocker. More of a "fix it in 3.2.1", since the status quo will *work*, it just means the precompiled file will be ignored on first execution with newer Python versions. Cheers, Nick. [1] http://svn.python.org/view/python/branches/py3k/Lib/distutils/util.py?view=markup -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Some" .pyc files not ending up in __pycache__ during installation
On Sat, 19 Feb 2011 23:07:17 +1000 Nick Coghlan wrote: > > While this is definitely untidy, it doesn't strike me as a release > blocker. More of a "fix it in 3.2.1", since the status quo will > *work*, it just means the precompiled file will be ignored on first > execution with newer Python versions. Are you sure? If the package gets installed in a root-writable-only directory, later execution cannot create the right pyc files. This certainly looks like a critical issue (hopefully not release blocker). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] svn outage on Friday
On Fri, 18 Feb 2011 08:09:12 +0100 Dirkjan Ochtman wrote: > On Fri, Feb 18, 2011 at 00:17, "Martin v. Löwis" wrote: > > I think it's fair to say that the project currently rests, lacking > > a project lead. The most recent timeline is that conversion should > > be completed by PyCon, and, failing that, should start at PyCon. > > It's not exactly resting; I've been pushing around the > cvs2svn-converted tags to get them to behave sensibly, and I've been > having good discussions with Antoine and Georg about several things we > need to hash out in IRC. Sorry I haven't been doing more progress > reports here. > > But yes, it would be nice if we could actually switch by PyCon, but at > the very least there should be a fresh test repo and consensus on most > of the workflow issues by PyCon (for those interested who live in > Europe, I'm going to be at a hg sprint in Karlsruhe during the PyCon > weekend). Would be nice if at least a simple repo (e.g. peps) got fully migrated as soon as possible. Thanks Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Some" .pyc files not ending up in __pycache__ during installation
Am 19.02.2011 14:14, schrieb Antoine Pitrou: > On Sat, 19 Feb 2011 23:07:17 +1000 > Nick Coghlan wrote: >> >> While this is definitely untidy, it doesn't strike me as a release >> blocker. More of a "fix it in 3.2.1", since the status quo will >> *work*, it just means the precompiled file will be ignored on first >> execution with newer Python versions. > > Are you sure? If the package gets installed in a root-writable-only > directory, later execution cannot create the right pyc files. It will certainly work. It won't create pyc files, but it will still work - right? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Some" .pyc files not ending up in __pycache__ during installation
On Sat, Feb 19, 2011 at 11:14 PM, Antoine Pitrou wrote: > On Sat, 19 Feb 2011 23:07:17 +1000 > Nick Coghlan wrote: >> >> While this is definitely untidy, it doesn't strike me as a release >> blocker. More of a "fix it in 3.2.1", since the status quo will >> *work*, it just means the precompiled file will be ignored on first >> execution with newer Python versions. > > Are you sure? If the package gets installed in a root-writable-only > directory, later execution cannot create the right pyc files. Worst case, it will run from the source file. It may even use the legacy .pyc file in the case where it can't write to __pycache__ (I don't remember how that particular subtlety of PEP 3147 played out). Certainly not ideal from a performance point of view, but also not difficult to workaround once discovered. > This certainly looks like a critical issue (hopefully not release > blocker). As a performance problem that only arises in some situations when using distutils to do bulk compilation, and with running "compileall" as root available as a relatively straightforward workaround, I personally think it can wait until 3.2.1. So I'd agree with the critical-but-not-a-release-blocker assessment, but it's ultimately Georg's call. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Some" .pyc files not ending up in __pycache__ during installation
Le samedi 19 février 2011 à 14:27 +0100, "Martin v. Löwis" a écrit : > Am 19.02.2011 14:14, schrieb Antoine Pitrou: > > On Sat, 19 Feb 2011 23:07:17 +1000 > > Nick Coghlan wrote: > >> > >> While this is definitely untidy, it doesn't strike me as a release > >> blocker. More of a "fix it in 3.2.1", since the status quo will > >> *work*, it just means the precompiled file will be ignored on first > >> execution with newer Python versions. > > > > Are you sure? If the package gets installed in a root-writable-only > > directory, later execution cannot create the right pyc files. > > It will certainly work. It won't create pyc files, but it will still > work - right? Right, but that's a big performance regression if it doesn't use cached bytecode file. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] What's L2R status?
Ref: http://bugs.python.org/issue448679
Has this bug fixed already?
This bug seems not be fixed for Python 2.6 and Python 3.2rc3.
Python 3.2rc3 (r32rc3:88413, Feb 15 2011, 18:31:14)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> C=0
>>> def x():
... global C
... C += 1
... return C
...
>>> {x(): x(), x(): x()}
{2: 1, 4: 3}
--
INADA Naoki
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What's L2R status?
Sorry. There is a issue already.
http://bugs.python.org/issue11205
On Sun, Feb 20, 2011 at 1:19 AM, INADA Naoki wrote:
> Ref: http://bugs.python.org/issue448679
>
> Has this bug fixed already?
> This bug seems not be fixed for Python 2.6 and Python 3.2rc3.
>
> Python 3.2rc3 (r32rc3:88413, Feb 15 2011, 18:31:14)
> [GCC 4.4.5] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
C=0
def x():
> ... global C
> ... C += 1
> ... return C
> ...
{x(): x(), x(): x()}
> {2: 1, 4: 3}
>
>
> --
> INADA Naoki
>
--
INADA Naoki
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What's L2R status?
INADA> Sorry. There is a issue already. INADA> http://bugs.python.org/issue11205 >From Raymond's last comment on the above ticket: > How much to change and how far to backport may make for a good python-dev > discussion. I think the simple patch I posted for 2.7 is probably all that should be done for the 2.x series. Briefly, change the visit order to evaluate the key before the value, then use ROT_TWO to reorder the top of the stack for STORE_MAP. > I don't have any feelings about it one way or the other, but it would > great to see Py3.2.1 get fixed properly. I don't know if bumping the bytecode version is okay after 3.2 is released, or if that's something which could only happen with 3.3. Whatever is deemed proper by the release managers, the better change would be to bump the version number, remove the ROT_TWO from my basic patch and change the order of arguments expected by STORE_MAP. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Some" .pyc files not ending up in __pycache__ during installation
Am 19.02.2011 14:29, schrieb Nick Coghlan: > On Sat, Feb 19, 2011 at 11:14 PM, Antoine Pitrou wrote: >> On Sat, 19 Feb 2011 23:07:17 +1000 >> Nick Coghlan wrote: >>> >>> While this is definitely untidy, it doesn't strike me as a release >>> blocker. More of a "fix it in 3.2.1", since the status quo will >>> *work*, it just means the precompiled file will be ignored on first >>> execution with newer Python versions. >> >> Are you sure? If the package gets installed in a root-writable-only >> directory, later execution cannot create the right pyc files. > > Worst case, it will run from the source file. It may even use the > legacy .pyc file in the case where it can't write to __pycache__ (I > don't remember how that particular subtlety of PEP 3147 played out). > Certainly not ideal from a performance point of view, but also not > difficult to workaround once discovered. > >> This certainly looks like a critical issue (hopefully not release >> blocker). > > As a performance problem that only arises in some situations when > using distutils to do bulk compilation, and with running "compileall" > as root available as a relatively straightforward workaround, I > personally think it can wait until 3.2.1. > > So I'd agree with the critical-but-not-a-release-blocker assessment, > but it's ultimately Georg's call. Given that we are only hours from the final, I'm quite unwilling to call this a blocker, seeing that running from the .py file works well (and I'm not really of Antoine's opinion that that is such a big performance hit). BTW, I haven't seen an issue yet. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Some" .pyc files not ending up in __pycache__ during installation
On Sun, Feb 20, 2011 at 6:53 AM, Georg Brandl wrote: > Given that we are only hours from the final, I'm quite unwilling to > call this a blocker, seeing that running from the .py file works well > (and I'm not really of Antoine's opinion that that is such a big > performance hit). How significant the performance hit is depends on at least a few different factors: 1. How many files are affected 2. How many files are implicitly imported when the application starts 3. How big/complicated those files 4. How long the actual "do work" part of the application takes I expect something like "hg diff" on a small working copy would be severely impacted if the application files for hg itself weren't cached properly. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Bug in linking to gomp with python; causes crash in matplotlib.
Hello,
I've encountered a strange bug that appears to be either in gcc's gomp
implementation or in how python loads extension modules linked against
gomp. Here's the error:
Using gcc (multiple versions) on linux, I compile an empty c extension
module and pass -lgomp as a linker arg. If I import it, running a
simple script in matplotlib causes a segfault. Not passing -lgomp or
not loading the empty module makes the code works fine. More
specifically, if I compile:
#include "Python.h"
static struct PyMethodDef methods[] = {
{0, 0, 0, 0}
};
PyMODINIT_FUNC initempty(void) {
Py_InitModule4("empty", methods, 0, 0, PYTHON_API_VERSION);
}
using ``ext_modules = [Extension("empty", ["empty.c"], extra_link_args
= ["-lgomp"])]``, then
import empty
import matplotlib.pylab as plt
plt.figure()
plt.plot([0,1], [0,1], '-b')
plt.show()
causes the program to segfault (removing ``import empty`` makes it
fine). Looking at a traceback:
#0 0x00f78bc7 in __cxa_allocate_exception () from /usr/lib/libstdc++.so.6
#1 0x008f51f2 in py_to_agg_transformation_matrix (obj=0x8223f58,
errors=false) at src/agg_py_transforms.cpp:20
#2 0x008fdd73 in _path_module::update_path_extents (this=0x8e45f90,
args=...) at src/path.cpp:378
#3 0x009048bd in
Py::ExtensionModule<_path_module>::invoke_method_varargs (this=, method_def=0x8e9ae30, args=...) at
./CXX/Python2/ExtensionModule.hxx:184
#4 0x008f0d96 in method_varargs_call_handler
(_self_and_name_tuple=0x8e6eeac, _args=0x94e683c) at
CXX/Python2/cxx_extensions.cxx:1714
#5 0x080dc0d0 in PyEval_EvalFrameEx ()
#6 0x080dddf2 in PyEval_EvalCodeEx ()
While occurring in some of matplotlib's extension code (and I haven't
found another library that crashes it), the fact that the deciding
factor is whether I link against gomp indicates the it's probably
upstream somewhere.
I encountered this error a year ago and asked about it on the
matplotlib mailing list, but found a quick workaround then, and with
deadline pressure I forgot about it. However, it's come up again, and
then I was asked to bump it to python-dev, which is why I'm posting it
here.
I can reproduce it on the following systems. In all cases, matplotlib
is compiled from source on the development branch (r8969) and uses
QT4Agg as the backend, as is numpy, scipy, etc. If needed, I can
track down more versions.
gcc (Ubuntu/Linaro 4.4.4-14ubuntu5) 4.4.4, 64bit, Python 2.6.6, ubuntu 10.10
gcc (Ubuntu 4.4.3-4ubuntu5) 4.4.3, 64bit, Python 2.6.5, ubuntu 10.04
gcc (Ubuntu 4.4.1-4ubuntu9) 4.4.1, 32bit, Python 2.6.4, ubuntu 9.10
gcc 4.5.2 (source build), Python 2.6.5, ubuntu 10.04. On this build,
the given source example does not produce the result, and I haven't
been able to tweak it so it does. However, linking to a much larger
extension library that uses many different parts of openmp causes
exactly the same crash. If I recompile that library without openmp
support, then everything works fine; with openmp support it corrupts
something and matplotlib crashes in exactly the same way.
gcc 4.3.2, Python 2.6.2, ubuntu 9.04 (I don't have access to this
system any more, since it got upgraded, but it had the same problem a
year ago).
I'd be happy to provide any more information if needed. I attached
example code that reproduces it. Let me know if I should file a bug
report (and where to file it -- which is why I haven't yet).
Thanks,
--Hoyt
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ [email protected]
++
python-gomp-bug.tar.gz
Description: GNU Zip compressed data
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug in linking to gomp with python; causes crash in matplotlib.
On Sun, Feb 20, 2011 at 3:43 PM, Hoyt Koepke wrote: > I'd be happy to provide any more information if needed. I attached > example code that reproduces it. Let me know if I should file a bug > report (and where to file it -- which is why I haven't yet). An associated bug report would be appreciated (mailing list discussions are useful for raising awareness, but are more likely to be forgotten over time than bug reports): http://bugs.python.org Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Finally fix installer to add Python to %PATH% on Windows
I've got a feeling that policy is evil and can not be applied cleanly when change falls out of scope of Python core .c sources and .py files from standard library. Right now the proposal is to add Python to %PATH% to make Python more user friendly for newbies. I can't see what can become worse than it is now. People are discussing advanced scenarios with multiple Python installations, but I propose first to make Python a normal command line user-friendly system application for Windows, and then discuss more advanced usage scenarios to make it developer-friendly in subsequent versions. Time is passing by and nothing happens. -- anatoly t. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] w9xpopen.exe is still in 3.2
Python definitely needs a development Roadmap to avoid things like w9xpopen.exe slipping off radar from release to release. We don't support Windows 9x since Python 2.6. What this file does in 3.x distributions? http://bugs.python.org/issue2405 -- anatoly t. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
