Re: leftover pyc files
On Thu, 03 Nov 2011 17:54:52 +, Andrea Crotti wrote: All these ideas (shell and git hooks) are nice, but unfortunately - it's svn not git - it's windows not *nix - we have to remove only the ones without the corresponding *py... Does it matter? The other .pyc files will be recreated when they are next needed. Just nuke all the .pyc files and be done with it. Or if you can't bear having to wait for Python to compile them as needed, you can force compilation by running your test suite (you do have a test suite, don't you?) or by using the compileall.py script. python -m compileall some_directory should do the trick. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
I like to install a Bash shell of some kind on windows boxes I work on, specifically so I can use shell commands like this, just like on any other operating system. Cywin works just fine for this. svn also has hooks, but sadly not a checkout hook: http://svnbook.red-bean.com/en/1.1/ch05s02.html I guess you could write your own two-line wrapper script which does the checkout and then deletes the pyc files. I don't understand why deleting only some pyc files is important. Can't you just delete them all and let Python re generate the ones you need? I once saw someone create the longest python file they could to see how long generating pyc files takes, and it is very very quick indeed. A human could not notice the delay even for the largest of projects. Finally, someone asked about skipping .svn dirs: find has a '-prune' option, you can read about it on the manpage. -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
Apologies for all my messasges appearing twice. I'm using google groups web ui and have no idea why it's doing that. I'll stop using it. -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
On 11/04/2011 09:27 AM, Jonathan Hartley wrote: I like to install a Bash shell of some kind on windows boxes I work on, specifically so I can use shell commands like this, just like on any other operating system. Cywin works just fine for this. svn also has hooks, but sadly not a checkout hook: http://svnbook.red-bean.com/en/1.1/ch05s02.html I guess you could write your own two-line wrapper script which does the checkout and then deletes the pyc files. I don't understand why deleting only some pyc files is important. Can't you just delete them all and let Python re generate the ones you need? I once saw someone create the longest python file they could to see how long generating pyc files takes, and it is very very quick indeed. A human could not notice the delay even for the largest of projects. Finally, someone asked about skipping .svn dirs: find has a '-prune' option, you can read about it on the manpa Uhm yes it makes sense also to just remove all of them, I don't know why it was done like this but probably for performance reasons. I will try both ways, but I would definitively avoid the shell scripting, because it's mainly windows but it has to be multi-platform.. -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
On Fri, Nov 4, 2011 at 9:34 PM, Andrea Crotti andrea.crott...@gmail.com wrote: Uhm yes it makes sense also to just remove all of them, I don't know why it was done like this but probably for performance reasons. I will try both ways, but I would definitively avoid the shell scripting, because it's mainly windows but it has to be multi-platform.. If you're removing them all, you don't need to use a powerful shell. Much much easier! Just recursively del *.pyc and you're done. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
On 11/04/2011 10:39 AM, Chris Angelico wrote: If you're removing them all, you don't need to use a powerful shell. Much much easier! Just recursively del *.pyc and you're done. ChrisA Discussing with the guy that did it I think it's actually a good idea instead, because removing them *all* means. - remove a few thousand files every time where maybe 0 would be removed otherwise - recompile the same thousand of files It might not be so relevant but for 10 lines of code more I guess it's fine.. -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
Uhn, thanks for the easy way Just delete all *.pyc recursively. spend another 5-20 minutes to recompile all to get everything sync.. That is trivial! -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
For those not aware, the compiled file caching and import system was changed for 3.2. Given test.py, the compiled file is no longer test.pyc, in the same directory, but (for cpython32) __pycache__/test.cpython-32.pyc. Given the statement 'import test', the __pycache__ directory is only searched (for the name given above) after finding test.py. So if 'test.py' is deleted or renamed to 'mytest.py', an unchanged 'import test' will *fail* instead of importing the obsolete .pyc file. So there is no longer a functional reason to delete such obsolete files. However, the OP is stuck with 2.5. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
On Fri, 04 Nov 2011 16:01:14 -0400, Terry Reedy wrote: For those not aware, the compiled file caching and import system was changed for 3.2. Given test.py, the compiled file is no longer test.pyc, in the same directory, but (for cpython32) __pycache__/test.cpython-32.pyc. Given the statement 'import test', the __pycache__ directory is only searched (for the name given above) after finding test.py. So if 'test.py' is deleted or renamed to 'mytest.py', an unchanged 'import test' will *fail* instead of importing the obsolete .pyc file. So there is no longer a functional reason to delete such obsolete files. However, the OP is stuck with 2.5. Oh, I don't know, removing obsolete and useless crud from your file system seems like a good enough functional reason to me :) However, the behaviour of Python 3.2 is a little more complicated than the above, since it must still support .pyc only software. If you have a .pyc file in the PYTHONPATH, but *outside* of a __pycache__ directory, and it is compiled for the correct version of Python, it will still be imported as usual. I'm not sure what happens if you have two .pyc files, one inside the cache and one outside. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
A task like this is more suited to bash than Python:
find . -name '*.pyc' -exec rm '{}' ';'
--
http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
This can be added to git as a post-checkout hook:
In your project's .git/hooks/post-checkout:
#!/usr/bin/env bash
cd ./$(git rev-parse --show-cdup)
find . -name '*.pyc' -exec rm '{}' ';'
--
http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
This can be added to your project's .git/hooks/post-checkout:
#!/usr/bin/env bash
cd ./$(git rev-parse --show-cdup)
find . -name '*.pyc' -exec rm '{}' ';'
--
http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
Jonathan Hartley wrote:
A task like this is more suited to bash than Python:
find . -name '*.pyc' -exec rm '{}' ';'
You forgot to exclude the .svn directory from the search and didn't limit
the deletion to pyc-files whose corresponding py-file doesn't exist.
How would your line look with these modifications?
--
http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
All these ideas (shell and git hooks) are nice, but unfortunately - it's svn not git - it's windows not *nix - we have to remove only the ones without the corresponding *py... -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
On Fri, Nov 4, 2011 at 4:54 AM, Andrea Crotti andrea.crott...@gmail.com wrote: All these ideas (shell and git hooks) are nice, but unfortunately - it's svn not git - it's windows not *nix - we have to remove only the ones without the corresponding *py... Windows? Well, Windows shell scripting isn't quite as rich as bash/csh/etc, but it's possible. I'll leave recursion as an exercise for the reader, but this (untested) should do it for one directory: for %d in (*.pyc) do ( set fn=%d if not exist !fn:~0,-1! del %d ) This needs to be run with 'cmd /v' to enable delayed variable evaluation. Of course, it'd be really easy to do if Windows Batch had anything like decent string manipulation. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
On Thu, Nov 3, 2011 at 1:54 PM, Andrea Crotti andrea.crott...@gmail.com wrote: All these ideas (shell and git hooks) are nice, but unfortunately - it's svn not git - it's windows not *nix - we have to remove only the ones without the corresponding *py... -- http://mail.python.org/mailman/listinfo/python-list Barely tested. Change the print functions to removes. Pass the top directory as the argument. import os, sys for dirpath, dirnames, filenames in os.walk(sys.argv[1]): if dirpath.endswith(__pycache__): thispath = dirpath[:-11] for f in filenames: if f.endswith(.pyc): if not os.path.exists(os.path.join(thispath,f[:-1])): print(delete + os.path.join(thispath,f)) else: for f in filenames: if f.endswith(.pyc): if not os.path.exists(os.path.join(dirpath,f[:-1])): print(delete + os.path.join(dirpath,f)) #os.remove(os.path.join(dirpath,f)) -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
Andrea Crotti wrote: In our current setup, the script in charge to run our applications also scan all the directories and if it finds any pyc file without the corresponding module, it removes it. This was done because when you rename something, the old pyc remains there and can cause problems. Is it the only way possible to scan through and remove all of them or is there a smarter way? Switch to Python3.2 ;) http://docs.python.org/dev/py3k/whatsnew/3.2.html#pep-3147-pyc-repository-directories http://www.python.org/dev/peps/pep-3147/#case-4-legacy-pyc-files-and-source-less-imports -- http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
On 11/02/2011 03:03 PM, Peter Otten wrote:
Switch to Python3.2 ;)
Yes I saw that and it would be great, unfortunately we're stuck with
Python 2.5 :O
for some more time.
Anyway now the code that does it is a recursive thing ilke
def _clean_orphaned_pycs(self, directory, simulate=False):
Remove all the pyc files without a correspondent module
files = listdir(directory)
# going down the tree recursively (like os.walk)
for x in files:
if x.endswith('.pyc'):
py_file = x.split('.pyc')[0] + '.py'
if (not simulate) and (py_file not in files):
deletable_pyc = path.join(directory, x)
print 'DELETING pyc %s as it has no py %s' % \
(deletable_pyc, py_file)
remove(deletable_pyc)
#TODO: move out these special cases
if path.isdir(path.join(directory, x)) \
and x != '.svn' \
and not x.endswith('.egg-info'):
self._clean_orphaned_pycs(path.join(directory, x))
Can be done much better probably with os.walk or os.path.walk,
any suggestions?
--
http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
If you can at least upgrade to Python 2.6, another option, if you
don't mind losing the byte code caching all together (it may worsen
load times, however probably not significantly), you can set
PYTHONDONTWRITEBYTECODE to prevent the writing of .pyc files.
Alternatively, again in 2.6+, you can also specify the -B command-line
argument. See
http://docs.python.org/using/cmdline.html#envvar-PYTHONDONTWRITEBYTECODE
and http://docs.python.org/using/cmdline.html#cmdoption-unittest-discover-B
Chris
On Wed, Nov 2, 2011 at 8:45 AM, Andrea Crotti andrea.crott...@gmail.com wrote:
On 11/02/2011 03:03 PM, Peter Otten wrote:
Switch to Python3.2 ;)
Yes I saw that and it would be great, unfortunately we're stuck with Python
2.5 :O
for some more time.
Anyway now the code that does it is a recursive thing ilke
def _clean_orphaned_pycs(self, directory, simulate=False):
Remove all the pyc files without a correspondent module
files = listdir(directory)
# going down the tree recursively (like os.walk)
for x in files:
if x.endswith('.pyc'):
py_file = x.split('.pyc')[0] + '.py'
if (not simulate) and (py_file not in files):
deletable_pyc = path.join(directory, x)
print 'DELETING pyc %s as it has no py %s' % \
(deletable_pyc, py_file)
remove(deletable_pyc)
#TODO: move out these special cases
if path.isdir(path.join(directory, x)) \
and x != '.svn' \
and not x.endswith('.egg-info'):
self._clean_orphaned_pycs(path.join(directory, x))
Can be done much better probably with os.walk or os.path.walk,
any suggestions?
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: leftover pyc files
On 11/02/2011 04:06 PM, Chris Kaynor wrote: If you can at least upgrade to Python 2.6, another option, if you don't mind losing the byte code caching all together (it may worsen load times, however probably not significantly), you can set PYTHONDONTWRITEBYTECODE to prevent the writing of .pyc files. Alternatively, again in 2.6+, you can also specify the -B command-line argument. See http://docs.python.org/using/cmdline.html#envvar-PYTHONDONTWRITEBYTECODE and http://docs.python.org/using/cmdline.html#cmdoption-unittest-discover-B Chris Unfortunately not possible :( So well the hand-made removal is also fine, maybe if I'm able to run it as a service it should not impact on the run-time either, which would be nice... -- http://mail.python.org/mailman/listinfo/python-list
