Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 12/09/14 17:11, Doug Hellmann wrote: I also use git-hooks with a post-checkout script to remove pyc files any time I change between branches, which is especially helpful if the different branches have code being moved around: git-hooks: https://github.com/icefox/git-hooks The script: $ cat ~/.git_hooks/post-checkout/remove_pyc #!/bin/sh echo Removing pyc files from `pwd` find . -name '*.pyc' | xargs rm -f exit 0 Good thing that python modules can't have spaces in their names! But for the future, find has a -delete parameter that won't break horribly on strange filenames. find . -name '*.pyc' -delete -- Radomir Dopieralski ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 09/15/2014 09:29 AM, Radomir Dopieralski wrote: On 12/09/14 17:11, Doug Hellmann wrote: I also use git-hooks with a post-checkout script to remove pyc files any time I change between branches, which is especially helpful if the different branches have code being moved around: git-hooks: https://github.com/icefox/git-hooks The script: $ cat ~/.git_hooks/post-checkout/remove_pyc #!/bin/sh echo Removing pyc files from `pwd` find . -name '*.pyc' | xargs rm -f exit 0 Good thing that python modules can't have spaces in their names! But for the future, find has a -delete parameter that won't break horribly on strange filenames. find . -name '*.pyc' -delete GNU find has that as an extension, but POSIX does not guarantee it, and BSD find lacks it. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Mon, Sep 15, 2014 at 1:25 PM, Eric Blake ebl...@redhat.com wrote: On 09/15/2014 09:29 AM, Radomir Dopieralski wrote: On 12/09/14 17:11, Doug Hellmann wrote: I also use git-hooks with a post-checkout script to remove pyc files any time I change between branches, which is especially helpful if the different branches have code being moved around: git-hooks: https://github.com/icefox/git-hooks The script: $ cat ~/.git_hooks/post-checkout/remove_pyc #!/bin/sh echo Removing pyc files from `pwd` find . -name '*.pyc' | xargs rm -f exit 0 Good thing that python modules can't have spaces in their names! But for the future, find has a -delete parameter that won't break horribly on strange filenames. find . -name '*.pyc' -delete GNU find has that as an extension, but POSIX does not guarantee it, and BSD find lacks it. The workaround is -print0: find . -name '*.pyc' -print0 | xargs -0 rm -f - Brant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 09/15/2014 03:02 PM, Brant Knudson wrote: Good thing that python modules can't have spaces in their names! But for the future, find has a -delete parameter that won't break horribly on strange filenames. find . -name '*.pyc' -delete GNU find has that as an extension, but POSIX does not guarantee it, and BSD find lacks it. The workaround is -print0: find . -name '*.pyc' -print0 | xargs -0 rm -f Alas, both find -print0 and xargs -0 are also a GNU extensions not required by POSIX. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 09/15/2014 03:15 PM, Eric Blake wrote:
On 09/15/2014 03:02 PM, Brant Knudson wrote:
Good thing that python modules can't have spaces in their names! But for
the future, find has a -delete parameter that won't break horribly on
strange filenames.
find . -name '*.pyc' -delete
GNU find has that as an extension, but POSIX does not guarantee it, and
BSD find lacks it.
The workaround is -print0: find . -name '*.pyc' -print0 | xargs -0 rm -f
Alas, both find -print0 and xargs -0 are also a GNU extensions not
required by POSIX.
find . -name '*.pyc' -exec rm -f \{} +
is POSIX.
--
Eric Blake eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org
signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 09/12/2014 07:14 PM, Doug Hellmann wrote: On Sep 12, 2014, at 12:03 PM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:52 AM, Doug Hellmann wrote: On Sep 12, 2014, at 11:21 AM, Mike Bayer mba...@redhat.com wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. my VPN was down and I didn’t get this thread just now, but I am strongly -1 on this as added to tox.ini, my response is http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html. Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* environment. Don’t force it on our automated tests or on my environment. .pyc files make a difference in behavior, and if we banish them from all testing, then our code is never tested within the environment that it will normally be run in after shipment. I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc files only, if a change to tox.ini must be made. I have to agree with Mike here. Cleaning up our dev environments using a little automation is better than disabling a feature of the interpreter that may have unforeseen consequences in behavior or performance. The more we introduce unusual settings like this into our environments and tools, the more edge cases and weirdness we’re going to find in those tools that keep us from doing the work we really want to be doing. We could use a git hook (see my earlier message in this thread) or we could add a command to tox to remove them before starting the tests. Neither of those solutions would affect the runtime behavior in a way that makes our dev environments fundamentally different from a devstack or production deployment. You believe that unit tests are going to change in the way they run so dramatically with this change that it invalidates their use? Do we have examples of what changes if you do and don't have pyc files there? Remember, we're not changing integration testing with this. This is solely unit testing. The reason I don't like just fix it in your local env is you are then exporting the complexity to developers. For something that they should really not have to get bitten by... a lot. Adding a command to tox to remove the files would be less intrusive than disabling their creation. We have had bad experiences mixing features to produce unusual dev environments that are different from the way the software really runs. All of the issues we had with namespace packages were caused by a bug in that implementation exposed because we were doing something unusual in devstack, for example. Adding some variation of “find $(python setup.py --name) --name ‘*.pyc’ | xargs rm -f” to tox.ini before running testr solves the problem you have identified without introducing any side-effects. This sounds saner to me so, I'd definitely go for this one. (FWIW, I'm use to clean up `pyc` files by myself) Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. This prevents pyc files from being writen in your git tree (win!). It doesn't seem to impact what pip installs... and if anyone knows how to prevent those pyc files from getting created, that would be great. But it's something which hopefully causes less perceived developer fragility of the system. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
+1!!! This is awesome. I *always* ran into this was about to get find . -name *.pyc -delete tattooed on the inside of my forearm. Now I don't have to. Thanks!!! --Brad Brad Topol, Ph.D. IBM Distinguished Engineer OpenStack (919) 543-0646 Internet: bto...@us.ibm.com Assistant: Kendra Witherspoon (919) 254-0680 From: Sean Dague s...@dague.net To: openstack-dev@lists.openstack.org openstack-dev@lists.openstack.org, Date: 09/12/2014 07:40 AM Subject:[openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. This prevents pyc files from being writen in your git tree (win!). It doesn't seem to impact what pip installs... and if anyone knows how to prevent those pyc files from getting created, that would be great. But it's something which hopefully causes less perceived developer fragility of the system. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. This prevents pyc files from being writen in your git tree (win!). It doesn't seem to impact what pip installs... and if anyone knows how to prevent those pyc files from getting created, that would be great. But it's something which hopefully causes less perceived developer fragility of the system. -Sean I also use git-hooks with a post-checkout script to remove pyc files any time I change between branches, which is especially helpful if the different branches have code being moved around: git-hooks: https://github.com/icefox/git-hooks The script: $ cat ~/.git_hooks/post-checkout/remove_pyc #!/bin/sh echo Removing pyc files from `pwd` find . -name '*.pyc' | xargs rm -f exit 0 -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. my VPN was down and I didn’t get this thread just now, but I am strongly -1 on this as added to tox.ini, my response is http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html. Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* environment. Don’t force it on our automated tests or on my environment. .pyc files make a difference in behavior, and if we banish them from all testing, then our code is never tested within the environment that it will normally be run in after shipment. I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc files only, if a change to tox.ini must be made. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 09/12/2014 11:21 AM, Mike Bayer wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. my VPN was down and I didn’t get this thread just now, but I am strongly -1 on this as added to tox.ini, my response is http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html. Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* environment. Don’t force it on our automated tests or on my environment. .pyc files make a difference in behavior, and if we banish them from all testing, then our code is never tested within the environment that it will normally be run in after shipment. I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc files only, if a change to tox.ini must be made. Your example in the other thread includes the random seed behavior, which is already addressed in new tox. So I don't see that as an issue. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Sep 12, 2014, at 11:24 AM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:21 AM, Mike Bayer wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. my VPN was down and I didn’t get this thread just now, but I am strongly -1 on this as added to tox.ini, my response is http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html. Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* environment. Don’t force it on our automated tests or on my environment. .pyc files make a difference in behavior, and if we banish them from all testing, then our code is never tested within the environment that it will normally be run in after shipment. I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc files only, if a change to tox.ini must be made. Your example in the other thread includes the random seed behavior, which is already addressed in new tox. So I don't see that as an issue. Will these patches all be accompanied by corresponding PYTHONHASHSEED settings? Also why don’t you want to place PYTHONDONTWRITEBYTECODE into your own environment?I don’t want this flag on my machine. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Sep 12, 2014, at 11:33 AM, Mike Bayer mba...@redhat.com wrote: On Sep 12, 2014, at 11:24 AM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:21 AM, Mike Bayer wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. my VPN was down and I didn’t get this thread just now, but I am strongly -1 on this as added to tox.ini, my response is http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html. Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* environment. Don’t force it on our automated tests or on my environment. .pyc files make a difference in behavior, and if we banish them from all testing, then our code is never tested within the environment that it will normally be run in after shipment. I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc files only, if a change to tox.ini must be made. Your example in the other thread includes the random seed behavior, which is already addressed in new tox. So I don't see that as an issue. Will these patches all be accompanied by corresponding PYTHONHASHSEED settings? Also why don’t you want to place PYTHONDONTWRITEBYTECODE into your own environment?I don’t want this flag on my machine. not to mention PYTHONHASHSEED only works on Python 3. What is the issue in tox you’re referring to ? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Sep 12, 2014, at 11:21 AM, Mike Bayer mba...@redhat.com wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. my VPN was down and I didn’t get this thread just now, but I am strongly -1 on this as added to tox.ini, my response is http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html. Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* environment. Don’t force it on our automated tests or on my environment. .pyc files make a difference in behavior, and if we banish them from all testing, then our code is never tested within the environment that it will normally be run in after shipment. I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc files only, if a change to tox.ini must be made. I have to agree with Mike here. Cleaning up our dev environments using a little automation is better than disabling a feature of the interpreter that may have unforeseen consequences in behavior or performance. The more we introduce unusual settings like this into our environments and tools, the more edge cases and weirdness we’re going to find in those tools that keep us from doing the work we really want to be doing. We could use a git hook (see my earlier message in this thread) or we could add a command to tox to remove them before starting the tests. Neither of those solutions would affect the runtime behavior in a way that makes our dev environments fundamentally different from a devstack or production deployment. Doug ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Fri, Sep 12 2014, Sean Dague wrote: Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. This prevents pyc files from being writen in your git tree (win!). It doesn't seem to impact what pip installs... and if anyone knows how to prevent those pyc files from getting created, that would be great. But it's something which hopefully causes less perceived developer fragility of the system. I understand it's generating .pyc could be something, but I don't really like that patch. I guess the problem is more likely that testrepository load the tests From the source directory whereas maybe we could make it load them from what's installed into the venv? -- Julien Danjou /* Free Software hacker http://julien.danjou.info */ signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 09/12/2014 11:33 AM, Mike Bayer wrote: On Sep 12, 2014, at 11:24 AM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:21 AM, Mike Bayer wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. my VPN was down and I didn’t get this thread just now, but I am strongly -1 on this as added to tox.ini, my response is http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html. Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* environment. Don’t force it on our automated tests or on my environment. .pyc files make a difference in behavior, and if we banish them from all testing, then our code is never tested within the environment that it will normally be run in after shipment. I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc files only, if a change to tox.ini must be made. Your example in the other thread includes the random seed behavior, which is already addressed in new tox. So I don't see that as an issue. Will these patches all be accompanied by corresponding PYTHONHASHSEED settings? Also why don’t you want to place PYTHONDONTWRITEBYTECODE into your own environment?I don’t want this flag on my machine. This was the set of tox changes that went in in August. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 09/12/2014 11:52 AM, Doug Hellmann wrote: On Sep 12, 2014, at 11:21 AM, Mike Bayer mba...@redhat.com wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. my VPN was down and I didn’t get this thread just now, but I am strongly -1 on this as added to tox.ini, my response is http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html. Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* environment. Don’t force it on our automated tests or on my environment. .pyc files make a difference in behavior, and if we banish them from all testing, then our code is never tested within the environment that it will normally be run in after shipment. I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc files only, if a change to tox.ini must be made. I have to agree with Mike here. Cleaning up our dev environments using a little automation is better than disabling a feature of the interpreter that may have unforeseen consequences in behavior or performance. The more we introduce unusual settings like this into our environments and tools, the more edge cases and weirdness we’re going to find in those tools that keep us from doing the work we really want to be doing. We could use a git hook (see my earlier message in this thread) or we could add a command to tox to remove them before starting the tests. Neither of those solutions would affect the runtime behavior in a way that makes our dev environments fundamentally different from a devstack or production deployment. You believe that unit tests are going to change in the way they run so dramatically with this change that it invalidates their use? Do we have examples of what changes if you do and don't have pyc files there? Remember, we're not changing integration testing with this. This is solely unit testing. The reason I don't like just fix it in your local env is you are then exporting the complexity to developers. For something that they should really not have to get bitten by... a lot. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Sep 12, 2014, at 12:03 PM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:33 AM, Mike Bayer wrote: On Sep 12, 2014, at 11:24 AM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:21 AM, Mike Bayer wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. my VPN was down and I didn’t get this thread just now, but I am strongly -1 on this as added to tox.ini, my response is http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html. Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* environment. Don’t force it on our automated tests or on my environment. .pyc files make a difference in behavior, and if we banish them from all testing, then our code is never tested within the environment that it will normally be run in after shipment. I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc files only, if a change to tox.ini must be made. Your example in the other thread includes the random seed behavior, which is already addressed in new tox. So I don't see that as an issue. Will these patches all be accompanied by corresponding PYTHONHASHSEED settings? Also why don’t you want to place PYTHONDONTWRITEBYTECODE into your own environment?I don’t want this flag on my machine. This was the set of tox changes that went in in August. corresponding to PYTHONHASHSEED, right? That whole thing is Python 3 only. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 09/12/2014 12:07 PM, Mike Bayer wrote: On Sep 12, 2014, at 12:03 PM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:33 AM, Mike Bayer wrote: On Sep 12, 2014, at 11:24 AM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:21 AM, Mike Bayer wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be found, and yet it fails. And then you realize that part of oslo doesn't exist any more except there are still pyc files laying around. Gah! I've proposed the following to Nova and Python novaclient - https://review.openstack.org/#/c/121044/ Which sets PYTHONDONTWRITEBYTECODE=true in the unit tests. my VPN was down and I didn’t get this thread just now, but I am strongly -1 on this as added to tox.ini, my response is http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html. Short answer: if you want this feature, put PYTHONDONTWRITEBYTECODE into *your* environment. Don’t force it on our automated tests or on my environment. .pyc files make a difference in behavior, and if we banish them from all testing, then our code is never tested within the environment that it will normally be run in after shipment. I’d far prefer a simple script added to tox.ini which deletes orphaned .pyc files only, if a change to tox.ini must be made. Your example in the other thread includes the random seed behavior, which is already addressed in new tox. So I don't see that as an issue. Will these patches all be accompanied by corresponding PYTHONHASHSEED settings? Also why don’t you want to place PYTHONDONTWRITEBYTECODE into your own environment?I don’t want this flag on my machine. This was the set of tox changes that went in in August. corresponding to PYTHONHASHSEED, right? That whole thing is Python 3 only. It very much is not only python 3. We have to pin in on a bunch of our python 2 tests now until we clean them up. There was a giant cross project effort on this all through July August to bring this in. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 2014-09-12 11:36:20 -0400 (-0400), Mike Bayer wrote: [...] not to mention PYTHONHASHSEED only works on Python 3. What is the issue in tox you’re referring to ? Huh? The overrides we added to numerous projects' tox.ini files to stem the breakage in Python 2.x unit tests from hash seed randomization in newer tox releases would seem to contradict your assertion. Also documentation... https://docs.python.org/2.7/using/cmdline.html#envvar-PYTHONHASHSEED (New in version 2.6.8.) -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 2014-09-12 12:07:41 -0400 (-0400), Mike Bayer wrote: [...] corresponding to PYTHONHASHSEED, right? That whole thing is Python 3 only. See other reply, but I really don't understand where you got that idea. Yes Python 2.x does not randomize the hash seed by default like Py3K (you have to pass -R to get that behavior) but you can still totally override the hash seed from the environment in 2.x (and more recent versions of tox happily do this for you and print out the hash seed which was chosen for a given test run). -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On 2014-09-12 17:16:11 +0100 (+0100), Daniel P. Berrange wrote: [...] Agreed, the problem with stale .pyc files is that it never occurs to developers that .pyc files are causing the problem until after you've wasted (potentially hours of) time debugging the problem. Avoiding this pain for all developers out of the box is a clear win overall and makes openstack development less painful. I've been bitten by similar issues often enough that I regularly git clean -dfx my checkouts or at least pass -r to tox so that it will recreate its virtualenvs from scratch. Yes it does add some extra time to the next test run, but you can iterate fairly tightly after that as long as you're not actively moving stuff around while you troubleshoot (and coupled with a git hook like Doug described for cleaning on topic branch changes would be a huge boon as well). -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Fri, 12 Sep 2014, Julien Danjou wrote: I guess the problem is more likely that testrepository load the tests From the source directory whereas maybe we could make it load them from what's installed into the venv? This rather ruins TDD doesn't it? -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Fri, Sep 12, 2014 at 04:23:09PM +, Jeremy Stanley wrote: On 2014-09-12 17:16:11 +0100 (+0100), Daniel P. Berrange wrote: [...] Agreed, the problem with stale .pyc files is that it never occurs to developers that .pyc files are causing the problem until after you've wasted (potentially hours of) time debugging the problem. Avoiding this pain for all developers out of the box is a clear win overall and makes openstack development less painful. I've been bitten by similar issues often enough that I regularly git clean -dfx my checkouts or at least pass -r to tox so that it will recreate its virtualenvs from scratch. Yes it does add some extra time to the next test run, but you can iterate fairly tightly after that as long as you're not actively moving stuff around while you troubleshoot (and coupled with a git hook like Doug described for cleaning on topic branch changes would be a huge boon as well). I'm not debating whether there are ways to clean up your env to avoid the problem /after/ it occurs. The point is to stop the problem occuring in the first place to avoid placing this uneccessary clean up burden on devs. Intentionally leaving things setup so that contributors hit bugs like stale .pyc files is just user hostile. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Sep 12, 2014, at 12:13 PM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-09-12 11:36:20 -0400 (-0400), Mike Bayer wrote: [...] not to mention PYTHONHASHSEED only works on Python 3. What is the issue in tox you’re referring to ? Huh? The overrides we added to numerous projects' tox.ini files to stem the breakage in Python 2.x unit tests from hash seed randomization in newer tox releases would seem to contradict your assertion. Also documentation... https://docs.python.org/2.7/using/cmdline.html#envvar-PYTHONHASHSEED (New in version 2.6.8.) Python 3’s documentation says “new in version 3.2.3”, so, confusing that they backported it to 2.6 at the same time but google searches tend to point you right here: https://docs.python.org/3.3/using/cmdline.html#envvar-PYTHONHASHSEED ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini
On Sep 12, 2014, at 12:29 PM, Daniel P. Berrange berra...@redhat.com wrote: On Fri, Sep 12, 2014 at 04:23:09PM +, Jeremy Stanley wrote: On 2014-09-12 17:16:11 +0100 (+0100), Daniel P. Berrange wrote: [...] Agreed, the problem with stale .pyc files is that it never occurs to developers that .pyc files are causing the problem until after you've wasted (potentially hours of) time debugging the problem. Avoiding this pain for all developers out of the box is a clear win overall and makes openstack development less painful. I've been bitten by similar issues often enough that I regularly git clean -dfx my checkouts or at least pass -r to tox so that it will recreate its virtualenvs from scratch. Yes it does add some extra time to the next test run, but you can iterate fairly tightly after that as long as you're not actively moving stuff around while you troubleshoot (and coupled with a git hook like Doug described for cleaning on topic branch changes would be a huge boon as well). I'm not debating whether there are ways to clean up your env to avoid the problem /after/ it occurs. The point is to stop the problem occuring in the first place to avoid placing this uneccessary clean up burden on devs. Intentionally leaving things setup so that contributors hit bugs like stale .pyc files is just user hostile. if we’re going to start diluting the test environment to suit developer environments, then the CI builds should use a different tox target that does *not* specify this environment variable. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
