Re: [openstack-dev] battling stale .pyc files
On Sep 15, 2014, at 4:34 AM, Lucas Alvares Gomes lucasago...@gmail.com wrote:
So, although I like the fix proposed and I would +1 that idea, I'm
also not very concerned if most of the people don't want that. Because
as you just said we can fix it locally easily. I didn't set it to my
.local but the way I do nowadays is to have a small bash function in
my .bashrc to delete the pyc files from the current directory:
function delpyc () {
find . -name *.pyc -exec rm -rf {} \;
}
So I just invoke it when needed :)
fyi there is a -delete option to find which is probably a little safer then
exec with a rm -rf. Also it is really convienient to do this as a git alias so
it happens automatically when switching branches:
In ~/.gitconfig:
[alias]
cc = !TOP=$(git rev-parse --show-toplevel) find $TOP -name '*.pyc'
-delete; git-checkout”
now you can git cc branch instead of git checkout branch.
Vish
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] battling stale .pyc files
Hi Mike, Thanks for bringing it up. I wanna say that I'm not an expert in CPython, but I personally like the fix because I have had some problems with stale .pyc in Ironic before, and they are pretty annoying. On Fri, Sep 12, 2014 at 4:18 PM, Mike Bayer mba...@redhat.com wrote: I’ve just found https://bugs.launchpad.net/nova/+bug/1368661, Unit tests sometimes fail because of stale pyc files”. The issue as stated in the report refers to the phenomenon of .pyc files that remain inappropriately, when switching branches or deleting files. Specifically, the kind of scenario that in my experience causes this looks like this. One version of the code has a setup like this: mylibrary/mypackage/somemodule/__init__.py Then, a different version we switch to changes it to this: mylibrary/mypackage/somemodule.py But somemodule/__init__.pyc will still be sitting around, and then things break - the Python interpreter skips the module (or perhaps the other way around. I just ran a test by hand and it seems like packages trump modules in Python 2.7). This is an issue for sure, however the fix that is proposed I find alarming, which is to use the PYTHONDONTWRITEBYTECODE=1 flag written directly into the tox.ini file to disable *all* .pyc file writing, for all environments unconditionally, both human and automated. I think that approach is a mistake. .pyc files have a definite effect on the behavior of the interpreter. They can, for example, be the factor that causes a dictionary to order its elements in one way versus another; I’ve had many relying-on-dictionary-ordering issues (which make no mistake, are bugs) smoked out by the fact that a .pyc file would reveal the issue..pyc files also naturally have a profound effect on performance. I’d hate for the Openstack community to just forget that .pyc files ever existed, our tox.ini’s safely protecting us from them, and then we start seeing profiling results getting published that forgot to run the Python interpreter in it’s normal state of operation. If we put this flag into every tox.ini, it means the totality of openstack testing will not only run more slowly, it also means our code will never be run within the Python runtime environment that will actually be used when code is shipped. The Python interpreter is incredibly stable and predictable and a small change like this is hardly something that we’d usually notice…until something worth noticing actually goes wrong, and automated testing is where that should be found, not after shipment. So this ordering thing, I don't think that it's caused by the PYTHONDONTWRITEBYTECODE, I googled that but couldn't find anything relating this option to the way python hash things (please point me to a document/code if I'm wrong). Are you sure you're not confusing it with the PYTHONHASHSEED option? So PYTHONHASHSEED yes does affect the ordering of the dict keys[1][2]. And I think that you'll find it more alarming because in the tox.ini we are already disabling that random hash seed[3] (but note that there's a comment there, disabling it seems to be a temporary thing) About the performance, this also doesn't seem to be true. I don't think .pyc affects the performance we run things at all, pyc are not meant to be an optimization in python. It DOES affect the startup of the application tho, because it will have to regenerate the bytecode all the time, see [4]: A program doesn't run any faster when it is read from a ‘.pyc’ or ‘.pyo’ file than when it is read from a ‘.py’ file; the only thing that's faster about ‘.pyc’ or ‘.pyo’ files is the speed with which they are loaded. [1] https://docs.python.org/2/using/cmdline.html#envvar-PYTHONHASHSEED [2] https://docs.python.org/2/using/cmdline.html#cmdoption-R [3] https://github.com/openstack/nova/blob/master/tox.ini#L12 [4] http://www.network-theory.co.uk/docs/pytut/CompiledPythonfiles.html The issue of the occasional unmatched .pyc file whose name happens to still be imported by the application is not that frequent, and can be solved by just making sure unmatched .pyc files are deleted ahead of time.I’d favor a utility such as in oslo.utils which performs this simple step of finding all unmatched .pyc files and deleting (taking care to be aware of __pycache__ / pep3147), and can be invoked from tox.ini as a startup command. But guess what - suppose you totally disagree and you really want to not have any .pyc files in your dev environment. Simple! Put PYTHONDONTWRITEBYTECODE=1 into *your* environment - it doesn’t need to be in tox.ini, just stick it in your .profile. Let’s put it up on the wikis, let’s put it into the dev guides, let’s go nuts. Banish .pyc files from your machine all you like. But let’s *not* do this on our automated test environments, and not force it to happen in *my* environment. So, although I like the fix proposed and I would +1 that idea, I'm
[openstack-dev] battling stale .pyc files
I’ve just found https://bugs.launchpad.net/nova/+bug/1368661, Unit tests sometimes fail because of stale pyc files”. The issue as stated in the report refers to the phenomenon of .pyc files that remain inappropriately, when switching branches or deleting files. Specifically, the kind of scenario that in my experience causes this looks like this. One version of the code has a setup like this: mylibrary/mypackage/somemodule/__init__.py Then, a different version we switch to changes it to this: mylibrary/mypackage/somemodule.py But somemodule/__init__.pyc will still be sitting around, and then things break - the Python interpreter skips the module (or perhaps the other way around. I just ran a test by hand and it seems like packages trump modules in Python 2.7). This is an issue for sure, however the fix that is proposed I find alarming, which is to use the PYTHONDONTWRITEBYTECODE=1 flag written directly into the tox.ini file to disable *all* .pyc file writing, for all environments unconditionally, both human and automated. I think that approach is a mistake. .pyc files have a definite effect on the behavior of the interpreter. They can, for example, be the factor that causes a dictionary to order its elements in one way versus another; I’ve had many relying-on-dictionary-ordering issues (which make no mistake, are bugs) smoked out by the fact that a .pyc file would reveal the issue..pyc files also naturally have a profound effect on performance. I’d hate for the Openstack community to just forget that .pyc files ever existed, our tox.ini’s safely protecting us from them, and then we start seeing profiling results getting published that forgot to run the Python interpreter in it’s normal state of operation. If we put this flag into every tox.ini, it means the totality of openstack testing will not only run more slowly, it also means our code will never be run within the Python runtime environment that will actually be used when code is shipped. The Python interpreter is incredibly stable and predictable and a small change like this is hardly something that we’d usually notice…until something worth noticing actually goes wrong, and automated testing is where that should be found, not after shipment. The issue of the occasional unmatched .pyc file whose name happens to still be imported by the application is not that frequent, and can be solved by just making sure unmatched .pyc files are deleted ahead of time.I’d favor a utility such as in oslo.utils which performs this simple step of finding all unmatched .pyc files and deleting (taking care to be aware of __pycache__ / pep3147), and can be invoked from tox.ini as a startup command. But guess what - suppose you totally disagree and you really want to not have any .pyc files in your dev environment. Simple! Put PYTHONDONTWRITEBYTECODE=1 into *your* environment - it doesn’t need to be in tox.ini, just stick it in your .profile. Let’s put it up on the wikis, let’s put it into the dev guides, let’s go nuts. Banish .pyc files from your machine all you like. But let’s *not* do this on our automated test environments, and not force it to happen in *my* environment. I also want to note that the issue of stale .pyc files should only apply to within the library subject to testing as it lives in its source directory. This has nothing to do with the packages that are installed under .tox as those are full packages, unless there’s some use case I’m not aware of (possible), we don’t checkout code into .tox nor do we manipulate files there as a matter of course. Just my 2.5c on this issue as to the approach I think is best. Leave the Python interpreter’s behavior as much as “normal” as possible in our default test environment. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] battling stale .pyc files
On Fri, Sep 12 2014, Mike Bayer wrote: Just my 2.5c on this issue as to the approach I think is best. Leave the Python interpreter’s behavior as much as “normal” as possible in our default test environment. I definitely agree with all of that. :) -- Julien Danjou // Free Software hacker // http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
